### **User description**
Currently role names and groups names are getting lowercased when
creating them (as part of the org revamp changes). This PR uses standard
format_role_name function to follow a standard role and group name
format.
___
### **PR Type**
Enhancement
___
### **Description**
- Introduce reusable name-formatting function
- Enforce regex formatting for roles and groups
- Update HTTP handlers to apply formatting
- Standardize success responses for role/group
___
### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>jwt.rs</strong><dd><code>Extract and centralize name
formatting</code>
</dd></summary>
<hr>
src/handler/http/auth/jwt.rs
<li>Extract regex-based formatting into helper<br> <li> Change
<code>format_role_name</code> to accept <code>&str</code><br> <li> Add
public <code>format_role_name_only</code> function<br> <li> Delegate
formatting to new helper function
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7224/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+9/-3</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>fga.rs</strong><dd><code>Apply standardized formatting
in handlers</code>
</dd></summary>
<hr>
src/handler/http/request/authz/fga.rs
<li>Import <code>format_role_name_only</code> helper<br> <li> Replace
lowercase trim with regex formatting<br> <li> Update handler logic to
use new helper<br> <li> Standardize response messages and payloads
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7224/files#diff-e7c99f37f3abbb5e0507d2760bbf67df010d9218704b38dce84c5477acecd22e">+15/-17</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
___
> <details> <summary> Need help?</summary><li>Type <code>/help how to
...</code> in the comments thread for any questions about PR-Agent
usage.</li><li>Check out the <a
href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a>
for more information.</li></details>
Cache Intermediate record batches for aggregate sql queries
## TODO
- [ ] Add in memory cache file index
- [ ] Modify exiting delete stream results api to also delete
`aggregations` cache
- [ ] use bounded Chanel for `CACHE_QUEUE` and and an env for queue size
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Migrates reports out of the `meta` table into new `reports` and
`report_dashboards` tables.
- Creates a new `reports` table
- Creates a new `report_dashboards` junction/join table that establishes
a many-to-many relation between reports and dashboards.
- Transforms data from the `meta` table into the new tables
- Creates a new `"default"` report folder for any organization that has
a report.
- Creates a new `reports` record for each report in the `meta` table.
- Creates a new `report_dashboards` record to link each report to each
of the dashboards it references.
- Previously, in the `meta` table, reports referenced dashboards by
folder ID and dashboard ID that was stored in JSON. This meant that
moving a dashboard would cause the references to that dashboard in
different reports to break. Since the `report_dashboards` table links
reports and dashboards using foreign keys to their KSUIDs, moving a
dashboard can no longer invalidate this reference.
- When `dashboards` or `reports` records are deleted, those deletions
are cascaded to the `report_dashboards` table.
- Updates the existing reports service layer logic to use the new
`reports` table rather than the `meta` table.
- Adds logic so that when a dashboard is deleted, any report which
referenced it is deleted as well.
- ⚠️ This PR removes the reports cache. I'm not sure if this is actually
something that will be OK or if doing this would create problems.
- Rather than trying to update the existing reports cache logic to work
with the other changes introduced in this PR, I thought that it might be
easier to remove the cache and then add it back later in an updated form
which uses the KSUIDs as cache keys rather than the report names since
eventually we will want to make it possible to have reports with
duplicate names.
- If we do actually need the report cache, I can add it back to this PR
and make the necessary updates so that it continues functioning.
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
This PR adds support for running unit tests with all three - sqlite,
postgres and mysql. This way we cover their code as well.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
After sso login, we return the access_token and refresh_token as base64
encoded string through cookies. But when handling the `/dex_refresh`
endpoint, we don't decode the cookie, hence the dex_refresh does not
really work.
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
closes#6996
This adds support for HEC data format in ingestion via `/org_id/_hec`
api path. See the issue for more details.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
with this POC we will let FE handle all short url redirections. We keep
the old routes intact for backwards compatibility
---------
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
1.This PR adds the output format to the external destination either json
/ ndjson , bydefault it is json
---------
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
impl #6948
- [x] When we run query, we get node list and check the node CPU, drop
at most 20% querier nodes.
- [x] When download file, we first try to download from original node(
by consistent hashing), then download from object store.
- [ ] ? should also apply this logic for ingester node ?
- [x] Fixed metrics keep increase issue of DownloadManager
this PR introduces o2 ai agent with this user able to ask o2 ai about
sql query , vrl query and logs
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
The issue is when we get cluster list we didn't filter the alert querier
cluster then the stats duplicated. The fix in o2 enterprise. this pr
only update some logs.
Fixes#5131
Addresses #5408
### Revamp
- Created a new table to store organizations in the `organizations`
table. Earlier, we never stored organizations in any table, rather we
used to store the organizations in the meta table as part of the `user`
json.
- The users and organization membership relations are now moved to its
own separate tables - `users` and `org_users`.
### New changes
**API changes** -
- `/organizations` - `POST` api to create an org and save it in the meta
table. In non-enterprise version, currently only the root user can
create the org. For enterprise version with rbac enabled, anyone with
`CREATE` access for organizations can create the organization and
becomes an admin. Root user can always create an organization.
- `/{org_id}/rename` - `PUT` api to rename the existing org. Only root
user and the admins of the org can access this API.
The payload for following APIs are changed -
- `POST api/{org_id}/roles` - `{"name":"new_role"}` is changed to
`{"role":"new_role"}
- `POST api/{org_id}/users/{user}` -
`{"role":"viewer","custom_role":["role_name"]}` is the new payload where
`custom_role` is optional and is array of custom roles (only works in
enterprise with rbac enabled) which will be assigned to the user. The
value of `role` can be standard role or as well as any custom role (when
it is custom role, the standard role assigned to the user is decided
based on the enterprise env variable - `O2_DEX_DEFAULT_ROLE`).
- `POST api/{org_id}/users`- the payload -
```
{"email":"user@user.com","role":"viewer","custom_role":["create_org_permission"],"password":"12345678", "first_name": "", "last_name":""}
```
- `PUT api//{org_id}/users/{email_id}` the payload -
```
{"first_name":"Hello","last_name":"j","role":"viewer","isExternal":false,"change_password":false,"password":"","custom_role":["create_org_permission"]}
```
A new ENV variable is introduced - `ZO_CREATE_ORG_THROUGH_INGESTION`
(default `false`) which if true, an org will automatically created when
ingesting data to a new org. If false, user can not create org by
ingesting data to new org, they need to first call the `POST
/organizations` API.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
**New Features**
- Added a `cloud` feature for enhanced organizational management
capabilities.
- Introduced new methods for managing organizational users and
asynchronous message processing.
- Enhanced user invitation and membership features, including improved
role handling.
- Implemented configurable ingestion URL support across various
components.
- Added new functionality for user management and organization handling
in the super cluster architecture.
**Improvements**
- Refined user authentication and authorization processes, including
error handling.
- Updated database schema to support multi-organization architecture.
- Enhanced pagination settings and UI components for better user
experience.
- Improved handling of organization names and user roles in
notifications and alerts.
- Streamlined service account token management with visibility
enhancements.
- Enhanced the `AddUpdateOrganization` component with improved user
input handling and validation.
- Updated the `AppOrganizations` component to dynamically load the
appropriate organization management interface.
- Improved the `ServiceAccountsList` component with updated pagination
options and token handling logic.
**Bug Fixes**
- Resolved issues with user role assignments and organization management
inconsistencies.
- Corrected routing and navigation logic for login and logout processes.
- Addressed various error handling scenarios in user and organization
processes.
- Fixed issues with the visibility of service account options in the
`EditGroup` and `EditRole` components based on the cloud configuration.
**Breaking Changes**
- Significant restructuring of user and organization data models.
- Changes to authentication workflows and configuration handling.
- Removal of certain components related to user and organization
management.
- Updated the logic for determining the ingestion URL across various
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
Co-authored-by: harshiv-7 <harshiv@openobserve.ai>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: Zhao.wei <weizhao029@gmail.com>
Co-authored-by: ShyamOOAI <shyam@openobserve.ai>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
- [x] fix: alert can't query latest data for separate alert query
- [x] feat: add size limit for download from other nodes
- [x] fix: use real time id generator
This adds the error header to all cases of internal server error so they
show up properly in audit. This is done by one of two approaches -
1. preferably using the `map_error_to_http_response` function , so that
everything goes via the same function
2. directly adding the header where the above function cannot be used
because is is not an error per say like trying to ingest at
non-ingester.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Currently, the scheduler trace_ids are not searchable in the traces
page, because they are only used in the logs and not in the
instrumentation. This pr fixes that - so, the scheduler query trace id
(which can be obtained from the `triggers` stream, field name -
`scheduler_trace_id`, the trace_id after the `/` is the scheduler query
trace_id), can be used as `trace_id` to search the trace in the traces
page.
Also, the `grpc::search` and `grpc::search_multi` does not use the
trace_id from the grpc "request" for instrumentation. This pr fixes that
as well.
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
- [x] fix: add retry if download file get size mismatch
- [x] fix: delete from file_cache index if one file delete from disk
- [x] fix: remove trace_id for file_cache and download, it is noise
- [x] fix: download file other node, add file size mismatch check, and
skip the not found error
We should extent slice not replace slice for download with streaming.
Testing this feature need to enable ENV:
```
ZO_CACHE_LATEST_FILES_ENABLED = true
ZO_CACHE_LATEST_FILES_DOWNLOAD_FROM_NODE=true
```
- [x] fix: `tokio_tungstenite` max msg limit config
- [x] honor user cancel in ui (logs, dashboards) if
`should_client_retry` is `true`, and is in loop.
- [x] fix: `values` event to send `cancel` response if search query
cancelled
- [x] handle `values` event error in ui
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: udev <uddhav@openobserve.ai>
This pr uses the meta table to store the start_time and end_time of
enrichment table, instead of relying on the stream_stats table which
takes some time to update. When we cache enrichment table for vrls, we
use the above start_time and end_time instead of `BASE_TIMESTAMP`
(because when enrichment table is overridden, deletion of old data
happens in the background and is not realtime, so using `BASE_TIMESTAMP`
included old data)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Subhra264
Loakesh IndiranandHengfei Yang
Patrick SullivanandSubhradeep Chakraborty
Maheep Kumar
Yashodhan JoshiandYashodhan Joshi
Jaydeep Maddipati
Ashish Kolhe
udevandomkark06
Sai Nikhilandoasisk
ByteBaker
