this PR introduces o2 ai agent with this user able to ask o2 ai about
sql query , vrl query and logs
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Fixes#5131
Addresses #5408
### Revamp
- Created a new table to store organizations in the `organizations`
table. Earlier, we never stored organizations in any table, rather we
used to store the organizations in the meta table as part of the `user`
json.
- The users and organization membership relations are now moved to its
own separate tables - `users` and `org_users`.
### New changes
**API changes** -
- `/organizations` - `POST` api to create an org and save it in the meta
table. In non-enterprise version, currently only the root user can
create the org. For enterprise version with rbac enabled, anyone with
`CREATE` access for organizations can create the organization and
becomes an admin. Root user can always create an organization.
- `/{org_id}/rename` - `PUT` api to rename the existing org. Only root
user and the admins of the org can access this API.
The payload for following APIs are changed -
- `POST api/{org_id}/roles` - `{"name":"new_role"}` is changed to
`{"role":"new_role"}
- `POST api/{org_id}/users/{user}` -
`{"role":"viewer","custom_role":["role_name"]}` is the new payload where
`custom_role` is optional and is array of custom roles (only works in
enterprise with rbac enabled) which will be assigned to the user. The
value of `role` can be standard role or as well as any custom role (when
it is custom role, the standard role assigned to the user is decided
based on the enterprise env variable - `O2_DEX_DEFAULT_ROLE`).
- `POST api/{org_id}/users`- the payload -
```
{"email":"user@user.com","role":"viewer","custom_role":["create_org_permission"],"password":"12345678", "first_name": "", "last_name":""}
```
- `PUT api//{org_id}/users/{email_id}` the payload -
```
{"first_name":"Hello","last_name":"j","role":"viewer","isExternal":false,"change_password":false,"password":"","custom_role":["create_org_permission"]}
```
A new ENV variable is introduced - `ZO_CREATE_ORG_THROUGH_INGESTION`
(default `false`) which if true, an org will automatically created when
ingesting data to a new org. If false, user can not create org by
ingesting data to new org, they need to first call the `POST
/organizations` API.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
**New Features**
- Added a `cloud` feature for enhanced organizational management
capabilities.
- Introduced new methods for managing organizational users and
asynchronous message processing.
- Enhanced user invitation and membership features, including improved
role handling.
- Implemented configurable ingestion URL support across various
components.
- Added new functionality for user management and organization handling
in the super cluster architecture.
**Improvements**
- Refined user authentication and authorization processes, including
error handling.
- Updated database schema to support multi-organization architecture.
- Enhanced pagination settings and UI components for better user
experience.
- Improved handling of organization names and user roles in
notifications and alerts.
- Streamlined service account token management with visibility
enhancements.
- Enhanced the `AddUpdateOrganization` component with improved user
input handling and validation.
- Updated the `AppOrganizations` component to dynamically load the
appropriate organization management interface.
- Improved the `ServiceAccountsList` component with updated pagination
options and token handling logic.
**Bug Fixes**
- Resolved issues with user role assignments and organization management
inconsistencies.
- Corrected routing and navigation logic for login and logout processes.
- Addressed various error handling scenarios in user and organization
processes.
- Fixed issues with the visibility of service account options in the
`EditGroup` and `EditRole` components based on the cloud configuration.
**Breaking Changes**
- Significant restructuring of user and organization data models.
- Changes to authentication workflows and configuration handling.
- Removal of certain components related to user and organization
management.
- Updated the logic for determining the ingestion URL across various
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
Co-authored-by: harshiv-7 <harshiv@openobserve.ai>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: Zhao.wei <weizhao029@gmail.com>
Co-authored-by: ShyamOOAI <shyam@openobserve.ai>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
In some cases when limit was added to the sql query we would report more
results than that size was asked. This PR adds checks to never return
more results than the `size` parameter
These tests include
- Added a new alert
- Add a new folder
- Added a new alert in this folder
- Moved 1 alert in 1 folder
- Bulk move alerts from default folder to new created folder
- Deleted the alerts that were moved to new folder
- Deleted the folder
- Deleted the template and destination
This uses background jobs in order to download files asynchronously, and
a mpsc queue in order to send file names to the jobs threads. This
limits the number of threads we spawn to a fixed number configurable via
config var.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated GitHub Actions workflow to dynamically use branch names for
repository dispatch events
- Improved workflow flexibility by removing hardcoded repository branch
references
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Tests covered**
- Ingest 79k fields with 100 user defined schema
- Reset stream selection while reloading home page and navigating to
logs
- Selecting stream and checking schema and values
- Go to streams page and enable store original data
- Go to stresstest stream via stream explorer
- Check if results are visible
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added new Playwright test suite for pipeline management.
- Implemented comprehensive UI interaction tests.
- Included tests for pipeline creation, deletion, and log management.
- Enhanced test coverage with parallel test execution.
- Expanded integration tests with additional scenarios for pipeline
interactions.
- Introduced a new test file for improved CI integration.
- **New Features**
- Added functionality for managing remote destinations within the
pipeline.
- Introduced new methods for creating and deleting remote destinations.
- **Bug Fixes**
- Improved the logic for selecting the current date in the logs page to
ensure accurate date selection.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5463
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced auto-refresh interval configuration across multiple
components.
- Added minimum refresh interval support with configurable restrictions.
- Introduced new error handling and validation mechanisms for report
creation.
- Added new utility functions for validating frequency intervals.
- **Improvements**
- Introduced dynamic minimum refresh interval settings.
- Added user feedback mechanisms for refresh interval constraints.
- Improved error handling for alert creation and updates.
- Enhanced clarity and maintainability of code through formatting
adjustments.
- Updated frequency input logic to dynamically calculate minimum values.
- Added validation for refresh intervals during data refresh and URL
query restoration.
- **Configuration**
- Added new props for minimum refresh interval and related messages.
- Implemented dynamic minimum value calculations for frequency inputs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
impl #5656
## Todo
- [x] Router consistent hashing for PromQL
- [x] Remove the key from index if the key get nothing from cache
- [x] Some cache item is empty, it should not
- [x] Add cache control ENV: `ZO_METRICS_CACHE_ENABLED=true`
- [ ] Optimize cache gc logic
- [ ] New encoding of RangeValue for cache
- [x] New cache key generate
- [x] Load cache from disk
- [ ] Histogram query OOM without cache for 2 days
- [x] Fix response series order
- [x] Need to check stream is exists
- [x] Add trace_id to the response
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new suite of Playwright tests for validating enrichment
table functionality.
- Added new functions for enhanced navigation and interaction within the
pipeline UI.
- Added tests to assert the data in every derived stream with condition,
function and only source + derived stream
- Added tests to assert if user clicks on any menu option and cancel on
dialog box, user stays on pipeline page
- Added tests to assert if user clicks on dashboard menu option and
accept on dialog box, user navigates to dashboard page
- **Bug Fixes**
- Improved error handling in tests to ensure accurate feedback during
interactions.
- **Chores**
- Updated the GitHub Actions workflow to include additional integration
tests.
- Removed outdated test files related to function management, Kinesis
ingestion, and reporting functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Updated the locator for the management menu item to enhance navigation
reliability.
- Streamlined the control flow by removing unnecessary wait times in the
management navigation process.
- **New Features**
- Added methods for managing service accounts, including adding,
validating, and deleting service accounts.
- Introduced a new test suite for validating service account
functionality, covering error handling and successful operations during
creation.
- Added a new method for automatic login based on visibility of the
login option.
- **Improvements**
- Updated the login method to ensure it waits for the correct login page
URL before proceeding.
- Simplified the login process in UI tests by removing unnecessary
parameters and wait times.
- Enhanced test scripts with improved structure and clarity, including
comments for better understanding.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced comprehensive UI testing for the Home, Ingestion, Login,
Logs, Metrics, Traces, Pipelines, Rum, Dashboard, Alerts, Reports, Data,
IAM, and About pages.
- Added functionality to handle organization changes and validations
across multiple pages, including default organization selections.
- New test suite for validating organization change scenarios across
various pages.
- Enhanced login functionality for internal users.
- Introduced new classes and methods for the Metrics, Streams, Rum, and
Alerts pages to support UI testing.
- Added new classes for the Data and IAM pages, expanding testing
capabilities.
- Enhanced Management page with navigation and validation capabilities.
- New methods for validating page URLs and content across various pages.
- **Bug Fixes**
- Improved validation methods to ensure proper navigation and content
checks on various pages.
- **Documentation**
- Updated testing framework to include new test cases and improved
structure for organization change tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix#5218
Backend:
- [x] allow condition node to evaluate `null` and `""`, empty, for both
`=` and `!=` conditions
- [x] stop ingestion when resolved dynamic stream name is empty
**How to Use(In ConditionNode):**
To evaluate field value is not empty:
`app_name != ""`
To evaluate field value is not null:
`app_name != null`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced `LogsPage` with improved date and time selection, timezone
management, and clipboard operations.
- Introduced `ManagementPage` for UI testing, allowing logo updates and
management functionalities.
- Added new tests for time zone changes and streamlined test scenarios
on the logs page.
- Introduced a suite of tests for joining logs, validating various join
scenarios.
- **Bug Fixes**
- Updated confirm button interaction in `ReportsPage` to ensure
visibility before clicking.
- **Documentation**
- Improved clarity and purpose of test scenarios through restructuring
and simplification.
- **Chores**
- Added `logoManagement.spec.js` and `join.spec.js` to the Playwright UI
integration tests for expanded coverage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
udev
Neha P
ktx-akshay
dependabot[bot]
Yashodhan JoshiandYashodhan Joshi
Chaitanya Sistla
Prabhat SharmaandHengfei Yang
Omkar Kesarkhane