### **PR Type**
Bug fix
___
### **Description**
- Include role in user-org tuple creation
- Pass `org.role` to `get_user_org_tuple`
- Add default role in custom role mapping
___
### **Changes diagram**
```mermaid
flowchart LR
PT["process_token"] -- "pass role" --> GUT["get_user_org_tuple"]
MG["map_group_to_custom_role"] -- "pass default role" --> GUT
```
___
### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>jwt.rs</strong><dd><code>Add role parameter to tuple
function</code>
</dd></summary>
<hr>
src/handler/http/auth/jwt.rs
<li>Added role parameter to <code>get_user_org_tuple</code><br> <li>
Passed <code>org.role</code> in <code>process_token</code><br> <li>
Included default role in custom role mapping
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7560/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+15/-3</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
___
> <details> <summary> Need help?</summary><li>Type <code>/help how to
...</code> in the comments thread for any questions about PR-Agent
usage.</li><li>Check out the <a
href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a>
for more information.</li></details>
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Cache Intermediate record batches for aggregate sql queries
## TODO
- [ ] Add in memory cache file index
- [ ] Modify exiting delete stream results api to also delete
`aggregations` cache
- [ ] use bounded Chanel for `CACHE_QUEUE` and and an env for queue size
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Fixes permission check for organization related api. In ofga when we set
a role with org perms, it sets to `org:_all_{org}` but in the check , we
check for permission on `org:{org}` which causes users with role having
permission on org, not be able to rename org. This fixes it by updating
it to correct check.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Fixes#5131
Addresses #5408
### Revamp
- Created a new table to store organizations in the `organizations`
table. Earlier, we never stored organizations in any table, rather we
used to store the organizations in the meta table as part of the `user`
json.
- The users and organization membership relations are now moved to its
own separate tables - `users` and `org_users`.
### New changes
**API changes** -
- `/organizations` - `POST` api to create an org and save it in the meta
table. In non-enterprise version, currently only the root user can
create the org. For enterprise version with rbac enabled, anyone with
`CREATE` access for organizations can create the organization and
becomes an admin. Root user can always create an organization.
- `/{org_id}/rename` - `PUT` api to rename the existing org. Only root
user and the admins of the org can access this API.
The payload for following APIs are changed -
- `POST api/{org_id}/roles` - `{"name":"new_role"}` is changed to
`{"role":"new_role"}
- `POST api/{org_id}/users/{user}` -
`{"role":"viewer","custom_role":["role_name"]}` is the new payload where
`custom_role` is optional and is array of custom roles (only works in
enterprise with rbac enabled) which will be assigned to the user. The
value of `role` can be standard role or as well as any custom role (when
it is custom role, the standard role assigned to the user is decided
based on the enterprise env variable - `O2_DEX_DEFAULT_ROLE`).
- `POST api/{org_id}/users`- the payload -
```
{"email":"user@user.com","role":"viewer","custom_role":["create_org_permission"],"password":"12345678", "first_name": "", "last_name":""}
```
- `PUT api//{org_id}/users/{email_id}` the payload -
```
{"first_name":"Hello","last_name":"j","role":"viewer","isExternal":false,"change_password":false,"password":"","custom_role":["create_org_permission"]}
```
A new ENV variable is introduced - `ZO_CREATE_ORG_THROUGH_INGESTION`
(default `false`) which if true, an org will automatically created when
ingesting data to a new org. If false, user can not create org by
ingesting data to new org, they need to first call the `POST
/organizations` API.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
**New Features**
- Added a `cloud` feature for enhanced organizational management
capabilities.
- Introduced new methods for managing organizational users and
asynchronous message processing.
- Enhanced user invitation and membership features, including improved
role handling.
- Implemented configurable ingestion URL support across various
components.
- Added new functionality for user management and organization handling
in the super cluster architecture.
**Improvements**
- Refined user authentication and authorization processes, including
error handling.
- Updated database schema to support multi-organization architecture.
- Enhanced pagination settings and UI components for better user
experience.
- Improved handling of organization names and user roles in
notifications and alerts.
- Streamlined service account token management with visibility
enhancements.
- Enhanced the `AddUpdateOrganization` component with improved user
input handling and validation.
- Updated the `AppOrganizations` component to dynamically load the
appropriate organization management interface.
- Improved the `ServiceAccountsList` component with updated pagination
options and token handling logic.
**Bug Fixes**
- Resolved issues with user role assignments and organization management
inconsistencies.
- Corrected routing and navigation logic for login and logout processes.
- Addressed various error handling scenarios in user and organization
processes.
- Fixed issues with the visibility of service account options in the
`EditGroup` and `EditRole` components based on the cloud configuration.
**Breaking Changes**
- Significant restructuring of user and organization data models.
- Changes to authentication workflows and configuration handling.
- Removal of certain components related to user and organization
management.
- Updated the logic for determining the ingestion URL across various
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
Co-authored-by: harshiv-7 <harshiv@openobserve.ai>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: Zhao.wei <weizhao029@gmail.com>
Co-authored-by: ShyamOOAI <shyam@openobserve.ai>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Makes the following fixes to these routes:
- `ListAlerts`
- adds `enabled` to the response body
- adds `name_substring` to the request body
- `MoveAlerts`
- changes the method type to `PATCH` since this is a non-idempotent
update
- changing to `PATCH` coincidentally happens to fix a routing issue
where requests to `MoveAlerts` were being routed to the `UpdateAlerts`
handler
- `EnableAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
- `TriggerAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated project dependencies in test configuration
- Added `tink` library (version 1.10.0+)
- Added `protobuf` library
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Removed several RUM and search-related test functions in API testing
- Skipped a search history test case in UI testing
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
rum search updated pytest
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Enhanced RUM (Real User Monitoring) test suite with improved logging
and data validation.
- Added print statements to output response payloads for better test
visibility.
- Introduced additional assertions for geographical data and log message
checks in search tests.
- Commented out specific actions in UI tests to refine test execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive end-to-end (E2E) tests for the alert management
API.
- Implemented tests for retrieving alerts, creating new alerts,
enabling/disabling alerts, triggering alerts, and deleting alerts.
- Included validation checks and response status code assertions for all
API functionalities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Updated test cases for pipeline creation and action handling with
corrected boolean value casing.
- Added new test case to validate pipeline creation failure scenario.
- Enhanced UI testing for enrichment table functionality by adding
date-time button interaction and visibility checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive test cases for API function creation
- Included tests for invalid payloads, text-only inputs, and special
character scenarios
- Verified successful function creation with valid payloads
- **Chores**
- Updated `.gitignore` to exclude temporary directories, configuration
files, and build artifacts
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive end-to-end tests for service account management.
- Implemented tests for retrieving, creating, updating, refreshing, and
deleting service accounts.
- Included validation for handling invalid email formats during service
account creation.
- Ensured proper validation of API responses and status codes for
service account operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive test cases for dashboard and alerts folder
functionalities.
- Introduced tests for unauthorized access scenarios.
- Added error handling tests for invalid inputs and folder IDs.
- Improved test coverage for various edge cases, including responses for
empty and invalid folder IDs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
New Folders API test cases
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive API testing for dashboard and alert
functionalities.
- Implemented test cases for retrieving, creating, updating, and
deleting dashboard and alert folders.
- Ensured proper response status code validation for various API
operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved clarity in passcode-related tests by renaming variables and
updating assertion messages.
- Enhanced specificity in testing passcode responses and updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
added PyTest for reset passcode
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added a new end-to-end test for the passcode reset functionality,
ensuring successful operation and response validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close#5420
- [x] set tantivy index as default index
- [x] support cross stream type join like below
```
select t1.ip, t2.name from logs.t1 as t1 join enrich.t2 as t1 on t1.ip = t2.ip
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `TableReferenceExt` module to enhance SQL-related
functionalities.
- Added a new `StreamTypeProvider` struct for managing table providers.
- **Improvements**
- Updated configuration defaults for inverted index formats.
- Enhanced handling of stream names and types throughout various
functions for better type safety.
- Improved logging with structured stream information.
- Refined the processing of the `MatchAll` condition in SQL-like
expressions.
- **Bug Fixes**
- Expanded string matching for `StreamType` to accommodate additional
input formats.
- **Documentation**
- Retained help comments for context on configuration fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new test suite for end-to-end testing of fuzzy matching
SQL queries.
- Added new test cases for verifying API data display and error message
handling in the UI.
- **Bug Fixes**
- Improved reliability of existing tests by enhancing visibility checks
before UI interactions.
- **Tests**
- Added helper functions for dynamic timestamp generation and SQL query
posting in the testing framework.
- Updated existing tests to streamline processes and improve interaction
logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new suite of Playwright tests for validating enrichment
table functionality.
- Added new functions for enhanced navigation and interaction within the
pipeline UI.
- Added tests to assert the data in every derived stream with condition,
function and only source + derived stream
- Added tests to assert if user clicks on any menu option and cancel on
dialog box, user stays on pipeline page
- Added tests to assert if user clicks on dashboard menu option and
accept on dialog box, user navigates to dashboard page
- **Bug Fixes**
- Improved error handling in tests to ensure accurate feedback during
interactions.
- **Chores**
- Updated the GitHub Actions workflow to include additional integration
tests.
- Removed outdated test files related to function management, Kinesis
ingestion, and reporting functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5378
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Enhanced error handling for invalid stream names during ingestion,
ensuring appropriate logging and response for empty or invalid entries.
- Updated assertions in tests to reflect the correct handling of invalid
bulk payloads by the API, ensuring errors are present for invalid
submissions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Introduced a new end-to-end test for pipeline creation, enabling, and
disabling.
- Utilized parameterization to test various scenarios with different
pipeline names, actions, and expected HTTP status codes.
- Removed the previous end-to-end tests related to user token
management, which included user token retrieval and email verification
processes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced end-to-end testing for bulk ingestion with comprehensive
coverage.
- Enhanced functionality for managing enrichment tables, including
adding, uploading, and deleting entries.
- **Bug Fixes**
- Improved error handling and user interaction checks in the enrichment
table upload process.
- **Tests**
- Added multiple new test cases for enrichment table functionality,
including upload and error handling scenarios.
- Updated existing test cases for consistency and improved logic.
- **Chores**
- Standardized import statements and improved overall test structure for
clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced dashboard management with new methods for retrieving
properties like ID, owner, title, and description.
- Added functionality for creating, updating, listing, and deleting
dashboards through a new service module.
- Introduced a new `dashboards` table in the database, with migrations
for creation and population.
- **Bug Fixes**
- Improved error handling for dashboard retrieval and management
operations, providing clearer responses for various conditions.
- **Documentation**
- Updated documentation to reflect new methods and error handling
improvements.
- **Refactor**
- Streamlined the dashboard handling logic in service and database
modules, reducing complexity and enhancing modularity.
- Enhanced error handling logic in dashboard management functions for
better clarity and control flow.
- Refined the dashboard retrieval process to improve clarity in control
flow and error handling.
- Updated the logic for dashboard operations to simplify version
handling and reduce redundancy.
- Improved the overall structure and organization of the dashboard
management code.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The previous mechanism for generating hashes for dashboard models
depended on hashing the JSON representation of the dashboard that was
stored in the `meta` table.
As part of the migration of dashboards out of the `meta` table and into
a dedicated `dashboards` table, this ☝️ presented some problems:
Specifically when we want to translate from the Dashboard ORM model to a
`core::meta::dashboards::Dashboard` model, we need to generate the hash
at the time of translation since `hash` is a field on
`core::meta::dashboards::Dashboard`. We can't simply generate a hash
from the JSON stored in the `dashboards.data` column because this JSON
no longer contains fields for `title`, `owner`, `description`, and
`role`. Those fields are now stored in DB columns, and changes to those
columns in the DB columns would *not* result in changes to the hash on
the JSON blob.
Therefore, rather than hashing the JSON blob, this PR implements hash
functions on the Rust types for the dashboards. Therefore changes to
fields in those Rust structs will result in changes to the hash values,
but changes to the way in which the data is stored in the DB will not
affect the hash values.
⚠️ Note that this PR is changing the hash function on the dashboard
models, so hashes generated for the same model before and after this
deployment will not be the same. My understanding is that this shouldn't
be an issue since hashes are just used as a way of checking on the UI if
the dashboard model has been updated to avoid overwriting the changes of
another user. I believe that this change to the dashboard function might
just have the effect (around the time of deployment) where it would
appear to the UI that the dashboard has been updated when, in fact, it
has not.
https://github.com/openobserve/o2-enterprise/actions/runs/11979723563
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new type alias `OrdF64` for enhanced precision in
floating-point operations.
- Added the `Hash` trait to multiple dashboard-related structs and
enums, enabling their use in hash-based collections.
- **Bug Fixes**
- Improved error handling and assertion clarity in API testing for
folder and dashboard operations.
- **Chores**
- Standardized hashing mechanism for dashboard objects in database
operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new JSON file for SQL join query test cases, enhancing
the variety and coverage of tests.
- Added multiple new test cases for various SQL join operations,
improving testing capabilities.
- **Bug Fixes**
- Improved error handling in data ingestion steps to ensure successful
test execution.
- **Refactor**
- Transitioned from hardcoded test data to a parameterized approach for
better maintainability and flexibility in testing.
- Updated method signature to streamline test parameters.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR implements the new design of **Pipeline** introduced by #4432
---
**2 types of pipelines differentiates by the source**
Type I: **Realtime**
- source is a stream (Logs, Metrics, & Traces)
- takes effect when the source stream is being ingested
Type II: **Scheduled** (aka DerivedSteram)
- source is a SQL query
- takes effect based on the given schedule
The new pipeline is represented as a graph consists of different types
of Nodes that are connected by edges.
**Pipeline Data Model**
```rs
pub struct Pipeline {
pub id: String,
pub version: i32,
pub enabled: bool,
pub org: String,
pub name: String,
pub description: String,
pub source: PipelineSource,
pub nodes: Vec<Node>,
pub edges: Vec<Edge>,
}
```
**4 types of node**
I. StreamNode: either used for source node of realtime pipeline or for
destination nodes
II. QueryNode: used for source node of scheduled pipeline
III. FunctionNode: used for executing vrl functions
IV. ConditionNode: used for checking routing conditions
Rules applied to validating a pipeline when it's created:
1. non-empty nodes list
2. non-empty edges list
3. 1st node in nodes list is either StreamNode or QueryNode
4. non-empty `conditions` in all ConditionNode nodes in nodes list
5. all leaf nodes are of type StreamNode
6. In the same branch, unchecked `after_flattened` FunctionNode can't
follow checked
`after_flattened` checked FunctionNode
Pipeline execution is implemented with the struct `ExecutablePipeline`,
a ready-execute pipeline object cached in memory based on the Pipeline
objects modified on the UI side.
`ExecutablePipeline` object
- builds the relationships among all the nodes based on the edges
- topologically sorts the nodes based on the level to determine which
node to process first
- registers all the vrl function node once
`ExecutablePipeline` object processes ingested records in batch.
Starting from the source node, each connecting edge has a channel
connecting the parent and child node, used for passing the records. This
approach enables batch processing in parallel.
---
**Deprecated Features**
1. previous implementation of pipeline
2. Function x Stream associations
- Functions can still be added/edited/removed as before
- Needs to create a pipeline in order to apply chosen functions to the
desired stream
The new release when this pr is merged will automatically migrate **old
pipelines** and **Function x Stream associations** to new pipeline
format and can be found in `Stream Pipelines` tab with `Migrated-`
prefix names.
**_Note: Auto generated pipelines are paused by default. Please verify
the pipelines before enabling them_**
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Introduced a new parameterized test suite for validating SQL queries
involving `cast` and `try_cast` operations.
- Enhanced testing framework to ensure API reliability when handling
various SQL casting scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Ashish KolheandShrinath Rao
Hengfei Yang
Loakesh IndiranandHengfei Yang
Yashodhan JoshiandYashodhan Joshi
Patrick Sullivan
Neha P
