Fixes#6928
Currently if the root user resets the ingestion passcode token, the
cache is not updated for root user, hence on refresh it still shows the
old passcode even though it is changed. As a result, the ingestion
passcode token shown on UI does not work.
impl #6948
- [x] When we run query, we get node list and check the node CPU, drop
at most 20% querier nodes.
- [x] When download file, we first try to download from original node(
by consistent hashing), then download from object store.
- [ ] ? should also apply this logic for ingester node ?
- [x] Fixed metrics keep increase issue of DownloadManager
Fixes permission check for organization related api. In ofga when we set
a role with org perms, it sets to `org:_all_{org}` but in the check , we
check for permission on `org:{org}` which causes users with role having
permission on org, not be able to rename org. This fixes it by updating
it to correct check.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
the values response hits always honor the size, but the total field
which indicates the count of responses is returning incorrect results.
This PR fixes it.
This PR also fixes a migration issue where the ID field were used as i64
ref: #5612
We see an overflow when we multiply the query duration to a large
storage size, to fix it we divide the storage size first with the data
retention time
this PR introduces o2 ai agent with this user able to ask o2 ai about
sql query , vrl query and logs
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
The issue is when we get cluster list we didn't filter the alert querier
cluster then the stats duplicated. The fix in o2 enterprise. this pr
only update some logs.
In dashboard panel edit mode, repeatedly adding and closing a drill-down
with an SQL query causes the UI to freeze due to a memory leak in the
Quasar dialog component.
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
demotes the log level to debug for when local cache insert fails. This
can happen when multiple queries on same stream for same range are
fired, and after getting file meta from pg, all of them try to insert in
local cache, and after one of them succeeds others will fail with sqlite
error of unique constraint violation. This will simply log it on debug
level instead of error, because even if insert fails in local cache, not
a big problem.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Fixes#5131
Addresses #5408
### Revamp
- Created a new table to store organizations in the `organizations`
table. Earlier, we never stored organizations in any table, rather we
used to store the organizations in the meta table as part of the `user`
json.
- The users and organization membership relations are now moved to its
own separate tables - `users` and `org_users`.
### New changes
**API changes** -
- `/organizations` - `POST` api to create an org and save it in the meta
table. In non-enterprise version, currently only the root user can
create the org. For enterprise version with rbac enabled, anyone with
`CREATE` access for organizations can create the organization and
becomes an admin. Root user can always create an organization.
- `/{org_id}/rename` - `PUT` api to rename the existing org. Only root
user and the admins of the org can access this API.
The payload for following APIs are changed -
- `POST api/{org_id}/roles` - `{"name":"new_role"}` is changed to
`{"role":"new_role"}
- `POST api/{org_id}/users/{user}` -
`{"role":"viewer","custom_role":["role_name"]}` is the new payload where
`custom_role` is optional and is array of custom roles (only works in
enterprise with rbac enabled) which will be assigned to the user. The
value of `role` can be standard role or as well as any custom role (when
it is custom role, the standard role assigned to the user is decided
based on the enterprise env variable - `O2_DEX_DEFAULT_ROLE`).
- `POST api/{org_id}/users`- the payload -
```
{"email":"user@user.com","role":"viewer","custom_role":["create_org_permission"],"password":"12345678", "first_name": "", "last_name":""}
```
- `PUT api//{org_id}/users/{email_id}` the payload -
```
{"first_name":"Hello","last_name":"j","role":"viewer","isExternal":false,"change_password":false,"password":"","custom_role":["create_org_permission"]}
```
A new ENV variable is introduced - `ZO_CREATE_ORG_THROUGH_INGESTION`
(default `false`) which if true, an org will automatically created when
ingesting data to a new org. If false, user can not create org by
ingesting data to new org, they need to first call the `POST
/organizations` API.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
**New Features**
- Added a `cloud` feature for enhanced organizational management
capabilities.
- Introduced new methods for managing organizational users and
asynchronous message processing.
- Enhanced user invitation and membership features, including improved
role handling.
- Implemented configurable ingestion URL support across various
components.
- Added new functionality for user management and organization handling
in the super cluster architecture.
**Improvements**
- Refined user authentication and authorization processes, including
error handling.
- Updated database schema to support multi-organization architecture.
- Enhanced pagination settings and UI components for better user
experience.
- Improved handling of organization names and user roles in
notifications and alerts.
- Streamlined service account token management with visibility
enhancements.
- Enhanced the `AddUpdateOrganization` component with improved user
input handling and validation.
- Updated the `AppOrganizations` component to dynamically load the
appropriate organization management interface.
- Improved the `ServiceAccountsList` component with updated pagination
options and token handling logic.
**Bug Fixes**
- Resolved issues with user role assignments and organization management
inconsistencies.
- Corrected routing and navigation logic for login and logout processes.
- Addressed various error handling scenarios in user and organization
processes.
- Fixed issues with the visibility of service account options in the
`EditGroup` and `EditRole` components based on the cloud configuration.
**Breaking Changes**
- Significant restructuring of user and organization data models.
- Changes to authentication workflows and configuration handling.
- Removal of certain components related to user and organization
management.
- Updated the logic for determining the ingestion URL across various
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
Co-authored-by: harshiv-7 <harshiv@openobserve.ai>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: Zhao.wei <weizhao029@gmail.com>
Co-authored-by: ShyamOOAI <shyam@openobserve.ai>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
There is one issue the `tantivy::TIMESTAMP_UPDATED_AT` never create
before call search, but maybe we already created a lot of index files.
So we need to init this time when the app start.
Issue was that we use a lock function to lock sqlite client when doing
write ops, because only one thread should do them at a time. However,
the client init also requires a lock. When running ent version, ent init
had a function which would lock as well as be the first one to access
the client, hence deadlock on the client lock.
This issue was not there when running for the first time, because in
migration function we init clients without the locks. However when
running second time, the migration returns early, and does not int
clients hence the ent function deadlocks, and subsequently locks
everything that need the db access.
The fix is simple, in migration function move the client init before
return, so that we always have an initialized client and do not get
deadlocked in ent or non ent version.
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
In some cases when limit was added to the sql query we would report more
results than that size was asked. This PR adds checks to never return
more results than the `size` parameter
Subhra264
Abhay Padamani
Hengfei Yang
Omkar Kesarkhane
Siddharth
Taiming Liu
Bhargav
Sai Nikhil
Ashish Kolhe
Shyam Panjiyar
Yashodhan JoshiandYashodhan Joshi
Huaijinandhengfei yang
udev
Patrick Sullivan
Loakesh Indiran
