### **PR Type**
Bug fix
___
### **Description**
- Include role in user-org tuple creation
- Pass `org.role` to `get_user_org_tuple`
- Add default role in custom role mapping
___
### **Changes diagram**
```mermaid
flowchart LR
PT["process_token"] -- "pass role" --> GUT["get_user_org_tuple"]
MG["map_group_to_custom_role"] -- "pass default role" --> GUT
```
___
### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>jwt.rs</strong><dd><code>Add role parameter to tuple
function</code>
</dd></summary>
<hr>
src/handler/http/auth/jwt.rs
<li>Added role parameter to <code>get_user_org_tuple</code><br> <li>
Passed <code>org.role</code> in <code>process_token</code><br> <li>
Included default role in custom role mapping
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7560/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+15/-3</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
___
> <details> <summary> Need help?</summary><li>Type <code>/help how to
...</code> in the comments thread for any questions about PR-Agent
usage.</li><li>Check out the <a
href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a>
for more information.</li></details>
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Cache Intermediate record batches for aggregate sql queries
## TODO
- [ ] Add in memory cache file index
- [ ] Modify exiting delete stream results api to also delete
`aggregations` cache
- [ ] use bounded Chanel for `CACHE_QUEUE` and and an env for queue size
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Fixes permission check for organization related api. In ofga when we set
a role with org perms, it sets to `org:_all_{org}` but in the check , we
check for permission on `org:{org}` which causes users with role having
permission on org, not be able to rename org. This fixes it by updating
it to correct check.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Fixes#5131
Addresses #5408
### Revamp
- Created a new table to store organizations in the `organizations`
table. Earlier, we never stored organizations in any table, rather we
used to store the organizations in the meta table as part of the `user`
json.
- The users and organization membership relations are now moved to its
own separate tables - `users` and `org_users`.
### New changes
**API changes** -
- `/organizations` - `POST` api to create an org and save it in the meta
table. In non-enterprise version, currently only the root user can
create the org. For enterprise version with rbac enabled, anyone with
`CREATE` access for organizations can create the organization and
becomes an admin. Root user can always create an organization.
- `/{org_id}/rename` - `PUT` api to rename the existing org. Only root
user and the admins of the org can access this API.
The payload for following APIs are changed -
- `POST api/{org_id}/roles` - `{"name":"new_role"}` is changed to
`{"role":"new_role"}
- `POST api/{org_id}/users/{user}` -
`{"role":"viewer","custom_role":["role_name"]}` is the new payload where
`custom_role` is optional and is array of custom roles (only works in
enterprise with rbac enabled) which will be assigned to the user. The
value of `role` can be standard role or as well as any custom role (when
it is custom role, the standard role assigned to the user is decided
based on the enterprise env variable - `O2_DEX_DEFAULT_ROLE`).
- `POST api/{org_id}/users`- the payload -
```
{"email":"user@user.com","role":"viewer","custom_role":["create_org_permission"],"password":"12345678", "first_name": "", "last_name":""}
```
- `PUT api//{org_id}/users/{email_id}` the payload -
```
{"first_name":"Hello","last_name":"j","role":"viewer","isExternal":false,"change_password":false,"password":"","custom_role":["create_org_permission"]}
```
A new ENV variable is introduced - `ZO_CREATE_ORG_THROUGH_INGESTION`
(default `false`) which if true, an org will automatically created when
ingesting data to a new org. If false, user can not create org by
ingesting data to new org, they need to first call the `POST
/organizations` API.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
**New Features**
- Added a `cloud` feature for enhanced organizational management
capabilities.
- Introduced new methods for managing organizational users and
asynchronous message processing.
- Enhanced user invitation and membership features, including improved
role handling.
- Implemented configurable ingestion URL support across various
components.
- Added new functionality for user management and organization handling
in the super cluster architecture.
**Improvements**
- Refined user authentication and authorization processes, including
error handling.
- Updated database schema to support multi-organization architecture.
- Enhanced pagination settings and UI components for better user
experience.
- Improved handling of organization names and user roles in
notifications and alerts.
- Streamlined service account token management with visibility
enhancements.
- Enhanced the `AddUpdateOrganization` component with improved user
input handling and validation.
- Updated the `AppOrganizations` component to dynamically load the
appropriate organization management interface.
- Improved the `ServiceAccountsList` component with updated pagination
options and token handling logic.
**Bug Fixes**
- Resolved issues with user role assignments and organization management
inconsistencies.
- Corrected routing and navigation logic for login and logout processes.
- Addressed various error handling scenarios in user and organization
processes.
- Fixed issues with the visibility of service account options in the
`EditGroup` and `EditRole` components based on the cloud configuration.
**Breaking Changes**
- Significant restructuring of user and organization data models.
- Changes to authentication workflows and configuration handling.
- Removal of certain components related to user and organization
management.
- Updated the logic for determining the ingestion URL across various
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
Co-authored-by: harshiv-7 <harshiv@openobserve.ai>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: Zhao.wei <weizhao029@gmail.com>
Co-authored-by: ShyamOOAI <shyam@openobserve.ai>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Makes the following fixes to these routes:
- `ListAlerts`
- adds `enabled` to the response body
- adds `name_substring` to the request body
- `MoveAlerts`
- changes the method type to `PATCH` since this is a non-idempotent
update
- changing to `PATCH` coincidentally happens to fix a routing issue
where requests to `MoveAlerts` were being routed to the `UpdateAlerts`
handler
- `EnableAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
- `TriggerAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated project dependencies in test configuration
- Added `tink` library (version 1.10.0+)
- Added `protobuf` library
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Removed several RUM and search-related test functions in API testing
- Skipped a search history test case in UI testing
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
rum search updated pytest
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Enhanced RUM (Real User Monitoring) test suite with improved logging
and data validation.
- Added print statements to output response payloads for better test
visibility.
- Introduced additional assertions for geographical data and log message
checks in search tests.
- Commented out specific actions in UI tests to refine test execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive end-to-end (E2E) tests for the alert management
API.
- Implemented tests for retrieving alerts, creating new alerts,
enabling/disabling alerts, triggering alerts, and deleting alerts.
- Included validation checks and response status code assertions for all
API functionalities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Updated test cases for pipeline creation and action handling with
corrected boolean value casing.
- Added new test case to validate pipeline creation failure scenario.
- Enhanced UI testing for enrichment table functionality by adding
date-time button interaction and visibility checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive test cases for API function creation
- Included tests for invalid payloads, text-only inputs, and special
character scenarios
- Verified successful function creation with valid payloads
- **Chores**
- Updated `.gitignore` to exclude temporary directories, configuration
files, and build artifacts
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive end-to-end tests for service account management.
- Implemented tests for retrieving, creating, updating, refreshing, and
deleting service accounts.
- Included validation for handling invalid email formats during service
account creation.
- Ensured proper validation of API responses and status codes for
service account operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive test cases for dashboard and alerts folder
functionalities.
- Introduced tests for unauthorized access scenarios.
- Added error handling tests for invalid inputs and folder IDs.
- Improved test coverage for various edge cases, including responses for
empty and invalid folder IDs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
New Folders API test cases
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive API testing for dashboard and alert
functionalities.
- Implemented test cases for retrieving, creating, updating, and
deleting dashboard and alert folders.
- Ensured proper response status code validation for various API
operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved clarity in passcode-related tests by renaming variables and
updating assertion messages.
- Enhanced specificity in testing passcode responses and updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
added PyTest for reset passcode
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added a new end-to-end test for the passcode reset functionality,
ensuring successful operation and response validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close#5420
- [x] set tantivy index as default index
- [x] support cross stream type join like below
```
select t1.ip, t2.name from logs.t1 as t1 join enrich.t2 as t1 on t1.ip = t2.ip
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `TableReferenceExt` module to enhance SQL-related
functionalities.
- Added a new `StreamTypeProvider` struct for managing table providers.
- **Improvements**
- Updated configuration defaults for inverted index formats.
- Enhanced handling of stream names and types throughout various
functions for better type safety.
- Improved logging with structured stream information.
- Refined the processing of the `MatchAll` condition in SQL-like
expressions.
- **Bug Fixes**
- Expanded string matching for `StreamType` to accommodate additional
input formats.
- **Documentation**
- Retained help comments for context on configuration fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new test suite for end-to-end testing of fuzzy matching
SQL queries.
- Added new test cases for verifying API data display and error message
handling in the UI.
- **Bug Fixes**
- Improved reliability of existing tests by enhancing visibility checks
before UI interactions.
- **Tests**
- Added helper functions for dynamic timestamp generation and SQL query
posting in the testing framework.
- Updated existing tests to streamline processes and improve interaction
logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new suite of Playwright tests for validating enrichment
table functionality.
- Added new functions for enhanced navigation and interaction within the
pipeline UI.
- Added tests to assert the data in every derived stream with condition,
function and only source + derived stream
- Added tests to assert if user clicks on any menu option and cancel on
dialog box, user stays on pipeline page
- Added tests to assert if user clicks on dashboard menu option and
accept on dialog box, user navigates to dashboard page
- **Bug Fixes**
- Improved error handling in tests to ensure accurate feedback during
interactions.
- **Chores**
- Updated the GitHub Actions workflow to include additional integration
tests.
- Removed outdated test files related to function management, Kinesis
ingestion, and reporting functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Neha P
Ashish Kolhe
Loakesh Indiranandsai nikhil kethe
udevandAshish Kolhe
Hengfei Yang
Huaijin
Anand Kumar Dubey
Yashodhan JoshiandYashodhan Joshi
Patrick Sullivan
