Commit Graph
2257 Commits
Author SHA1 Message Date
Bhargav 427ee1baf1 fix: 4021 logs page additional error information (#4029)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Introduced `SanitizedHtmlRenderer` for improved and secure HTML
content rendering in search results.

- **Enhancements**
- Improved error handling in the application, including updated error
messages with "TraceID" for better tracking.
- Enhanced user experience by allowing users to view additional error
details during search failures.
- Simplified error output by removing detailed error information from
user interfaces.

- **Bug Fixes**
- Adjusted handling of notifications to more accurately reflect error
states, especially for rate limiting scenarios.

- **Chores**
- Modified dependency version for `rudder-sdk-js` to a specific version,
potentially impacting update capabilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-24 13:46:25 +05:30
Hengfei Yang 6ca73b465c feat: add multiple tokio runtime (#4035) 2024-07-24 15:25:46 +08:00
Taiming Liu 4aa09c8bc6 feat: clean nats locks (#4032) 2024-07-24 11:14:24 +08:00
Bhargav b0d6c761fa Fix: package version (#4027)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit


- **Chores**
- Downgraded the `rudder-sdk-js` dependency version to enhance
compatibility and stability.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-23 12:01:43 +05:30
Sai Nikhil fdd88944e2 fix: #4008 fixed string selection in sql mode (#4016)
in sql mode whenever user changes a stream name it should reflect in the
below drop down and respective schema.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit


- **New Features**
	- Enhanced stream handling within the logs search component.
- Improved management of selected streams, ensuring previous selections
do not persist.
	- Added asynchronous processing capabilities for stream data fetching.

- **Bug Fixes**
- Resolved issues with stream selection clarity by resetting selected
streams appropriately.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-22 14:15:43 +05:30
BhargavandNeha P 979f4ea6b6 feat: Npm package upgrade (#3984)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Updated error messaging in the Logs UI to provide clearer feedback to
users when search queries fail.

- **Chores**
- Improved element selection specificity in Playwright tests to ensure
interactions with the correct UI elements.
- Upgraded dependencies to the latest versions for improved performance,
security, and compatibility.
- Excluded unnecessary directories from TypeScript compilation to
enhance project specificity and efficiency.
- Refined time zone handling across components for better accuracy in
date and time displays.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Neha P <neha@openobserve.ai>
2024-07-22 13:54:34 +05:30
ktx-vaidehi bd6f4e8822 fix: remove count of value from values api (#4020)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Introduced a `no_count` property to enhance data fetching, allowing
users to specify whether to include item counts in dashboard queries.
- Improved API call flexibility by adding the `no_count` parameter,
optimizing data retrieval based on user needs.

- **Bug Fixes**
- Adjusted the logic for counting items in dashboards, streamlining
performance and reducing unnecessary data payloads.

- **Documentation**
- Updated component and API documentation to reflect the changes in data
handling and query parameters.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-22 12:50:19 +05:30
Ashish Kolhe 3d21d8d672 fix: cache historical log records (#4019)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Improved caching logic to enhance the accuracy and relevance of search
results.
- Enhanced conditions for evaluating cached results, reducing the chance
of returning outdated data.

- **Bug Fixes**
- Refined cache expiration criteria to ensure timely and relevant data
delivery.

- **Style**
- Updated variable names for improved readability and clarity in the
caching logic.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-22 11:52:42 +05:30
Ashish Kolhe 866a655aba fix: caching issue (#4017)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Adjusted the default behavior for cache checks to prioritize
descending order results.
- Enhanced the caching mechanism by broadening the time range considered
for cache results.

- **Bug Fixes**
- Improved logic in the caching process to ensure more accurate timing
for cached data.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-22 06:20:45 +05:30
Taiming Liu 6e374f1c4c fix: default web url and correct report server endpoint (#4015) 2024-07-20 16:18:01 -07:00
Neha P 22a3c11fdd test: assert records to be unique (#4014)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Introduced enhanced test functions for more precise SQL queries,
improving the validation of result cache ratios.
- Added a new test for executing multiple SQL queries simultaneously,
expanding test coverage.

- **Bug Fixes**
- Updated assertions in existing tests to align with new query
structures and expected values.

- **Chores**
- Both new test functions are currently marked as skipped, indicating
they are not ready for execution.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-19 17:59:30 +05:30
Hengfei Yang 82fc1dfb58 feat: value API without count (#4013) 2024-07-19 20:08:28 +08:00
Bhargav 8f1af1a7b0 fix: Logs page histogram api duplicate call (#3998)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit


- **Bug Fixes**
- Improved reliability of histogram data loading by refining conditional
logic.
- Ensured `loadLogsData()` only runs when necessary, enhancing
performance and preventing unnecessary calls.

- **New Features**
- Enhanced control flow to check for query result hits before proceeding
with certain operations.

- **Tests**
- Introduced a delay in the UI test to improve the reliability of
visibility checks for error messages after refresh actions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-19 15:42:34 +05:30
Hengfei Yang 19818ad990 perf: keep improve inverted index (#4010)
- [x] Reduce unused field for search on inverted index
- [x] Reduce order by for search on inverted index
- [x] Change `term like '%x%'` to `term like 'x%'`
2024-07-18 10:28:00 -07:00
Hengfei Yang 766d5a0244 fix: search partition with order by time asc (#4009) 2024-07-18 23:27:07 +08:00
Huaijin 463d646b55 fix: rate calculate for count (#4005) 2024-07-18 21:31:07 +08:00
Neha PandHengfei Yang d9af0e3126 test: e2e quick mode without timestamp (#4006)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Updated URL generation for copying HTTP traces content in
OpenTelemetry component to improve accuracy.

- **Tests**
- Added a new test case to verify the display of results without
timestamps in quick mode.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-07-18 17:26:54 +05:30
Hengfei Yang ac6e75cad8 fix: result cache panic when there is no _timestamp (#4004) 2024-07-18 19:36:20 +08:00
ktx-abhay fe3c4459c1 fix: visualization css issues (#4002)
- `Add To Dashboard` pop up css issue.

![image](https://github.com/user-attachments/assets/b0a9a618-f501-415e-a09f-125603edd3a7)
- Confirmation dialog issue for logs visualize toggle button.
2024-07-18 12:42:22 +05:30
Hengfei Yang 99970f7d0c feat: use cache for value API (#3997)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
	- Improved caching behavior and response processing in search requests.
- Enhanced error handling and response generation for better user
experience.

- **Tests**
- Disabled specific UI interaction in Playwright tests to prevent test
failures.

- **Chores**
	- Updated configuration defaults to improve clarity and efficiency.
	- Made internal constants publicly accessible for broader usage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-18 08:09:12 +05:30
Ashish Kolhe 201f76c0e7 fix: enable cache by default & scan size lable issue (#3999) 2024-07-17 22:16:58 +08:00
ktx-abhayandktx-vaidehi e02d476dde fix: visualization stream selection issue (#3993)
- #3980 
- #3843 
- Stream name change issue on chart type change for add panel.
- Validation on `add to dashboard` click.
- CSS issue while showing `errorList` in `addPanel` and `visualization`
page.
- Breakdown reset issue for pie and donut chart.
- Reset old rendered chart on toggle.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Disabled a section of code that previously updated a field stream
under certain conditions.

- **New Features**
- Added validation and error notifications when adding new panels to the
dashboard to enhance user feedback.

- **Style**
- Updated layout and styling attributes in the dashboard panel view for
improved visual consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
2024-07-17 12:12:40 +05:30
Taiming Liu f78199ab4f fix: ScanSize label (#3992) 2024-07-16 21:58:37 -07:00
胡同里有猫 6a5291f8ea feat: add span metrics for traces (#3787) 2024-07-17 12:36:53 +08:00
Taiming Liu 2a4c296b3f feat: Routing support for all ingestion APIs (#3970)
feat: Routing support for all ingestion APIs. close #3941
2024-07-16 18:31:23 -07:00
Ashish Kolhe 428abe864d chore: add result stats to status api (#3991) 2024-07-17 08:01:06 +08:00
Ashish Kolhe 7eeb56ac69 fix: separate cache for results (#3990)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Introduced disk result caching for queries, along with metrics to
track usage.
  
- **Enhancements**
- Added support for different file types in disk caching, including data
and result files.
- Improved garbage collection to handle result files in addition to data
files.

- **Bug Fixes**
- Enhanced cache checking logic to correctly handle result timestamp
columns.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-16 18:12:41 +05:30
Hengfei Yang acf1c395b5 fix: optimize default setting for single node (#3989) 2024-07-16 20:02:09 +08:00
Hengfei Yang 672bc20aae feat: add CPU limit for work group (#3985) 2024-07-16 18:56:10 +08:00
baerwang 73be9a4e82 feat: support swagger enable open (#3986) 2024-07-16 18:02:40 +08:00
Ashish Kolhe 2ee9ba3b18 fix: result cache histogram issue (#3983)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Improved cache handling to simplify and streamline response merging,
ensuring more efficient and predictable search results.
- Adjusted the default sorting behavior to ascending order, optimizing
the default user experience for search results.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-16 12:51:38 +05:30
Huaijin 4831cf22bb fix: rate calculate (#3982) 2024-07-16 14:06:30 +08:00
Hengfei Yang 17287f7ba8 fix: consistent hashing with querier group (#3978)
- [x] Add two consistent hashing table for `Interactive` and
`Background` querier group
- [x] Only cache latest parquet files for `Interactive` queriers.
2024-07-15 10:25:15 -07:00
Ashish Kolhe 9284ad9811 fix: log result cache ratio to usage & delta scan size (#3974)
- Report delta scan size for UI 
- Report result cache ratio to usage

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Added `result_cache_ratio` field to enhance cache performance
tracking.
- **Enhancements**
- Updated search and logging functionalities to incorporate
`result_cache_ratio` for more detailed analytics.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-15 16:40:38 +05:30
Bharghava Varun Ayada 47bfe473d2 fix: Add PRIMARY KEY constraint to tables in postgres (#3972) 2024-07-15 17:30:30 +08:00
25bb19d846 feat: visualization for logs page (#3529)
- Visualize the logs data
- Save these visualizations to the dashboard

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Introduced a new `SelectDashboardDropdown` component for selecting and
adding dashboards from a dropdown list.
- Added a `SelectTabDropdown` component for selecting and managing tabs
within a dashboard.

- **Enhancements**
- Added conditional rendering for folder selection in the `AddDashboard`
component.
- Improved handling and validation for `activeFolderId` and
`showFolderSelection` props.
- Enhanced `PanelSchemaRenderer` to handle null or undefined values more
gracefully.
- Updated various components to use dependency injection for
`dashboardPanelDataPageKey`.

- **Bug Fixes**
- Removed unnecessary `click()` actions in multiple UI test scripts to
streamline test flows.
- Fixed various selector and conditional rendering issues in UI
components for better stability and performance.

- **Refactor**
- Refactored `QueryEditor` to include a nullish check, improving
stability.
- Restructured `DashboardQueryEditor` for better maintainability and
clarity.
- Adjusted import statements and removed unused dependencies across
multiple components.

- **Style**
- Adjusted styles and button functionalities in `SearchBar` for a more
intuitive user experience.

- **Tests**
- Updated and streamlined Playwright tests to remove redundant actions
and improve efficiency.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: Neha P <neha@openobserve.ai>
2024-07-15 10:57:45 +05:30
Hengfei Yang 10eecfa174 fix: inverted index query (#3969) 2024-07-13 23:45:19 +08:00
Hengfei Yang 491d95ebb6 fix: inverted index fast mode (#3967) 2024-07-13 20:40:04 +08:00
Hengfei Yang effdc28b3f ci: bump datafusion to 40 (#3968) 2024-07-13 20:16:18 +08:00
Hengfei YangandTaiming Liu d1c8335e40 fix: inverted index search with fast mode (#3964)
We have a `fast mode` that only return the top N parquet files in the
`inverted index` which contains the `term total count > query.limit`.

But after we remove the `time range` for index query sql, the first
terms which `total count > 250` maybe is not in the query time range,
then it will cause the real query found nothing.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Introduced a `fast mode` option to the file search functionality for
quicker initial page loads.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
2024-07-12 22:55:00 +05:30
Bhargav bac6869b92 feat: cron feature on alert page (#3923)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added the ability to set alerts using cron expressions, offering more
flexible scheduling options.
- Introduced a new "frequency_type" property to toggle between 'minutes'
and 'cron' settings for alerts.
	- New tooltip and input fields for configuring cron job activation.

- **UI/UX Improvements**
- Updated tooltips, titles, and input fields for easier configuration of
cron-based alerts.
- Conditional rendering of input fields based on selected frequency
type.

- **Localization**
- Added new translation key "crontitle" with value "Enable Cron" in the
English locale.

- **Enhancements**
- Improved time-related functionalities with new date and timezone
conversion utilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-12 17:55:04 +05:30
Subhra264 cb646ebc92 feat: Introduce otel grpc endpoint to export traces (#3963)
Fixes #3940 

## New ENVs:
- `ZO_TRACING_GRPC_ORGANIZATION` - The name of the organization used to
ingest traces
- `ZO_TRACING_GRPC_STREAM_NAME` - The stream name to use for trace
export.
- `OTEL_OTLP_GRPC_ENDPOINT` - The grpc endpoint to use for trace export.
E.g. - `grpc://localhost:5081`. If this is specified,
`OTEL_OTLP_HTTP_ENDPOINT` is ignored.

`ZO_TRACING_HEADER_VALUE` can be used for authorization token.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added configuration options for GRPC tracing, including URL, type, and
header settings.
- **Improvements**
- Enhanced tracer exporter logic for better configurability based on
tracing type.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-12 16:48:28 +05:30
Huaijin 10238a767d perf: use infer schema store in memtable (#3944) 2024-07-12 18:43:37 +08:00
Huaijin b6f0f9ef64 feat: add short vs long query in query manager (#3962) 2024-07-12 17:34:21 +08:00
BhargavandTaiming Liu 72aea59198 fix: sql function change with syntax guide (#3643)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Enhanced text snippet suggestions in the Query Editor with updated
labels and insert text.
- Introduced new functions for advanced search and matching operations
in the Query Editor.
  
- **Improvements**
- Improved search instructions for 'error' values in the Syntax Guide,
including clarifications on case sensitivity.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
2024-07-12 13:02:03 +05:30
Taiming Liu 9e81565d5f feat: role-group based node selection (#3960)
Categorizes nodes into role groups selected to serve different query
requests based on query search event type.

--- 
Each node can be configured into different role groups by using the new
env: `ZO_NODE_ROLE_GROUP`
- default: `` (empty: none)
- node for high-priority tasks: `interactive`
- node for low-priority tasks: `background`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Introduced `node_role_group` configuration to specify role groups for
nodes.
- Added `SearchEventType` to `SearchRequest` to enhance search
capabilities.
  - Implemented `RoleGroup` enum to categorize nodes based on priority.

- **Improvements**
- Updated node registration and status setting to include `role_group`.
  - Enhanced logging to include node's role group in cluster operations.
- Streamlined search logic by assigning `search_type` earlier in the
request handling process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-12 11:04:13 +05:30
Hengfei Yang 47cfcae30a fix: delete wal file when stream already deleted (#3961) 2024-07-12 12:26:47 +08:00
Subhra264 038f1e3a78 fix: Index stream rbac migration (#3959)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Improved index operations and migration handling for better
performance and reliability.
- Enhanced logging for detailed migration status and tuple information.

- **Bug Fixes**
- Added error handling to ensure proper setup of the OFGA model,
reducing potential failures during initialization.

- **Refactor**
- Streamlined the `init()` function to handle index stream migrations
and resource cleanup more efficiently.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-11 21:15:05 +05:30
Hengfei Yang f66ff1f3d2 refactor: separate search with cache (#3958) 2024-07-11 21:40:03 +08:00
Hengfei Yang de38ba776e fix: schema version filter for query (#3955) 2024-07-11 17:26:00 +08:00