#5253
Miscellaneous cleanup tasks for the new alerts v2 endpoints:
- fixes some of the new route paths
- removes utoipa schema attribute from a Unix timestamp field on the
HTTP model
- removes unnecessary call to DB in "delete" operation
#5253
Part 8 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces the new `List` endpoint which returns the parent
folder information for each folder in the response
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new endpoint for listing alerts, enhancing alert
management capabilities.
- Added pagination support for alert listing through
`ListAlertsQueryPageParams`.
- New functionality to retrieve alerts along with associated folders.
- **Bug Fixes**
- Improved error handling for alert management operations.
- **Refactor**
- Enhanced data structures for alert parameters and queries.
- Updated existing functions for better clarity and modularity in alert
management.
- **Documentation**
- Updated OpenAPI documentation to include the new alert listing
endpoint.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 7 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces the new `DeleteAlert`, `TriggerAlert`, and
`EnableAlert` endpoints which identify the alert by its KSUID
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new API endpoints for managing alerts: delete, enable, and
trigger alerts.
- Enhanced alert management capabilities with comprehensive CRUD
operations for alerts.
- **Bug Fixes**
- Improved error handling for alert operations, ensuring better
management of alert states.
- **Documentation**
- Updated API documentation to reflect new functionalities for alert
management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 6 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces the new `UpdateAlert` endpoint which identifies the
alert to update by its KSUID
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new HTTP endpoint for updating alerts.
- Added functionality to update existing alerts in the database.
- Enhanced OpenAPI documentation with the new alert update path.
- **Bug Fixes**
- Improved error handling for folder-related operations during alert
updates.
- **Refactor**
- Renamed several alert management functions for clarity and
consistency.
- Updated existing functions to streamline alert creation and updating
processes.
- **Documentation**
- Updated API documentation to reflect new alert management
capabilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 5 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces the new `CreateAlert` endpoint which takes a
`folder_id` parameter to specify the parent folder of the new alert
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new HTTP POST endpoint for creating alerts.
- Added functionality for creating alerts in the database.
- Enhanced API documentation to include the new alert creation endpoint.
- **Bug Fixes**
- Improved error handling for alert creation, including a specific error
for non-existent folders.
- **Chores**
- Updated import statements to include necessary types for new
functionalities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 4 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces the new Get endpoint for getting an alert by its
KSUID ID
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new endpoint for retrieving alerts by ID.
- Added functionality to fetch alerts by ID in the alert management
system.
- **Bug Fixes**
- Improved error handling for alert retrieval and management.
- **Documentation**
- Updated OpenAPI documentation to include the new alert retrieval
endpoint.
- **Refactor**
- Renamed existing functions for clarity and consistency in alert
management.
- **Chores**
- Added necessary imports to support new functionalities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 3 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR deprecates old Alerts endpoints
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new namespace for alert services, marking existing routes
as deprecated.
- **Bug Fixes**
- Updated alert-related functions to reflect changes in their underlying
implementations while maintaining functionality.
- **Documentation**
- Revised OpenAPI documentation to reflect the transition to deprecated
alert handlers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 2 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR just renames various methods
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new alert management structures and request/response bodies
for creating, updating, and listing alerts.
- Added unique identifier support for alerts.
- **Bug Fixes**
- Enhanced error handling for alert creation and updates.
- **Documentation**
- Updated OpenAPI documentation to include new alert-related request and
response types.
- **Refactor**
- Renamed functions for clarity in alert management, including retrieval
and deletion by name.
- **Tests**
- Updated integration tests to reflect changes in alert deletion
methods.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 1 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces HTTP layer models for the new Alerts endpoints
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new dependencies to enhance database interactions and
unique identifier generation.
- Added a unique identifier field to the Alert structure.
- Established a comprehensive framework for managing alerts, including
new request and response structures for HTTP handling.
- Expanded OpenAPI documentation to include new alert-related
components.
- **Bug Fixes**
- Enhanced error handling for alert operations, providing more specific
error reporting.
- **Refactor**
- Updated function signatures to improve flexibility and modularity in
database interactions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Introduces domain-specific errors for alerts in the service layer. These
errors are then mapped to HTTP responses in the HTTP layer.
This PR will be followed up by another PR to introduce new alerts
endpoints for accessing alerts by their new KSUIDs and for accessing
information about each alert's parent folder.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `AlertError` enum for improved error handling in
alert operations.
- Enhanced error handling for alert management functions, allowing for
more specific error responses.
- **Bug Fixes**
- Streamlined return types for alert management functions to enhance
clarity and reduce complexity in error handling.
- **Documentation**
- Updated method signatures to reflect changes in error handling and
return types across various alert management functions.
- **Chores**
- Refined error reporting mechanisms in multiple modules for consistency
in error handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
impl #4281
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced organizational metrics with total counts for functions and
dashboards.
- Added new utility functions for formatting numerical data.
- **Bug Fixes**
- Improved display logic for stream and document counts in the user
interface.
- **Documentation**
- Updated localization keys for clarity and added new summary labels.
- **Refactor**
- Restructured summary data generation for better clarity and
organization.
- **Style**
- Increased width of certain layout elements for improved visibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added labels for "Index Size", "Docs Count", "Function", "Pipeline",
"Pipelines & Functions", "Alerts", and "Dashboards" in the home section.
- Enhanced the home view to display additional metrics: Index Size,
Document Count, Dashboard Count, and Pipelines alongside existing data.
- Introduced new summaries for pipelines, alerts, and dashboards in
organizational data.
- **Bug Fixes**
- Improved layout and styling for better data presentation in the home
view.
- Updated validation logic for the home page to reflect new label
changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
related to #5388
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced a gRPC search method for enhanced search capabilities.
- Added support for multi-query requests with the new
`MultiStreamRequest` struct.
- **Improvements**
- Enhanced error handling and tracing across various search functions.
- Simplified request handling by removing unnecessary conversion
methods.
- **Breaking Changes**
- Removed the previous `SearchRequest` and `SearchResponse` messages
from the protocol, impacting existing integrations.
- **Refactor**
- Restructured search request processing to streamline operations and
improve maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Overview
This adds new CRUDL folder endpoints that include `folder_type` as URL
parameter. These new endpoints can be used to
create/read/update/delete/list folders of the type specified in the
`folder_type` URL parameter.
The older CRUDL folder endpoints only worked on dashboard folders. They
are being deprecated in this PR. Once the UI stops calling these
endpoints they can be removed entirely.
## URL path design
he URLs for the new endpoints generally follow the pattern
`{org_id}/folders/{folder_type}/{folder_id}`. The position of the
`{folder_type}` parameter in these URLs is specifically intended to
represent the place of "folder types" within the hierarchy of
resources/concepts in the URL. Specifically the hierarchy is as
follows...
- `{org_id}` - represents an org which owns many folders
- `folders` - represents all folders
- `{folder_type}` - represents a class or subset of folders
- `{folder_id}` - identifies a single folder; this ID is only uniquely
identifying in the context of all concepts higher in the hierarchy,
including `{folder_type}`
## Alternatives
An alternative approach to this PR would be to update the existing
endpoints so that either
- `{folder_type}` is an optional final URL parameter (after
`{folder_id}`) that defaults to `dashboards`
- `{folder_type}` is an optional query parameter that defaults to
`dashboards`
Although pursuing one of these approaches would have reduced the need to
add new routes and deprecate old ones, I think the design of the URL
path would suffer and be less clear with either of these approaches
since in either of these approaches the path would not actually be
formatted to represent the resource hierarchy described above.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `FolderType` enum for folder data categorization.
- Enhanced folder management API with new functions that include a
`folder_type` parameter.
- **Bug Fixes**
- Deprecated old folder-related functions while maintaining backward
compatibility.
- **Documentation**
- Updated API documentation to reflect changes in folder management
functions, including new deprecated endpoints and the `FolderType`
schema.
- **Chores**
- Added deprecation warnings for the `folders` module to inform users
about outdated methods.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
The overall goal of this PR is to migrate alerts data out of `meta`
table into `alerts` table. To accomplish this three main changes are
made. Unfortunately, all three of these changes need to be synchronized
and submitted in the same PR.
1. A new migration script is added to
- Parse alerts JSON currently stored in the `meta` table into Rust
models for alerts. These Rust models are isolated to the migration
script since they represent the schema of alerts as they are stored in
the `meta` table before this migration executes.
- Convert those Rust models into Sea ORM entities for alerts.
- Insert all the Sea ORM entities for alerts into the new `alerts`
table.
2. Create get/put/delete/list methods for accessing the data in the
`alerts` table
- These functions use SeaORM to query and update the database.
- These functions must also convert between SeaORM alert entities and
the application's current data model for alerts (as defined in
`config::meta::alerts`.
- It's entirely expected that the models inside `config::meta::alerts`
will change over time. When this happens in the future, developers will
need to update the transformation functions between SeaORM alert
entities and the models in `config::meta::alerts`. Luckily, the compiler
should help to direct developers to where these transformation functions
need to be updated.
3. Update the old logic in `/src/service/db/alerts.rs` that previously
read and wrote alerts to the `meta` table so that it now reads and
writes alerts to the `alerts` table.
- The methods for accessing the `meta` table included logic for
triggering "cache update events" to the cluster coordinator and for
sending events to the super cluster queue. Since the new logic accesses
the `alerts` table rather than the `meta` table, this logic for emitting
events had to be reproduced.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `ListAlertsParams` struct for flexible querying of
alerts.
- Enhanced `TriggerCondition` struct with new timing parameters.
- Added new error variants for improved error handling in alert
management.
- Created a comprehensive alert management system with CRUD operations.
- Added integration tests for various functionalities related to alerts
and dashboards.
- Introduced a new `Alerts` variant in the `FolderType` enum for better
folder management.
- **Bug Fixes**
- Improved error handling for folder deletion when alerts are present.
- **Documentation**
- Updated documentation for several functions to clarify their scope and
functionality.
- **Chores**
- Added new dependencies to support additional functionality.
- Introduced migration scripts for database schema changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR introduces a feature to exempt data form the normal data
retention cycle and allows it to be stored for a longer period of time.
This time period is configurable via a global environment variable.
- [x] Add new feature: extend data retention
- [x] Fix issue #5418
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced a new field for extended data retention days in the
compacting process.
- Added a flexible update settings structure for managing stream
settings.
- Implemented a new time range management feature for deletion tasks.
- Enhanced configuration response to include retention days.
- **Bug Fixes**
- Improved error handling for stream settings related to retention days.
- **Tests**
- Expanded test suite with new tests for time range functionality.
- **Documentation**
- Updated OpenAPI documentation to reflect changes in stream settings
structures.
- Removed outdated components from OpenAPI definition.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
There are 3 types of partial responses:
1. VRL error
2. Super cluster error
3. Range error (max_query_limit)
Cache partial results only if there is a range error.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced search functionality with improved error handling for query
ranges.
- Introduced caching logic for partial results based on specific error
conditions.
- **Bug Fixes**
- Streamlined response generation and error reporting in search-related
functions, ensuring clarity and efficiency.
- **Documentation**
- Updated method signatures to reflect the latest changes in
functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
In this [other PR
](https://github.com/openobserve/openobserve/pull/5390) I need to be
able to add logic to create a default "alerts" folder.
In order to do that, we first need to enable callers to pass a
`folder_type` parameter to different internal methods for
getting/creating/updating/deleting/listing folders. Previously those
methods assumed that the folder type was always `Dashboards` since thus
far that was the only folder type that existed.
This fixes two bugs -
1. The _values api used `Count(*)` function in query to get count.
However in case of distinct streams we already aggregate the count in
`count` field. Thus we need to use `sum(count)` instead.
2. In https://github.com/openobserve/openobserve/pull/5220 , the logic
which adds/removes the distinct values when dashboard is added/deleted
was removed accidentally. This Also re-adds that logic back.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced bulk deletion of dashboards with associated distinct
values.
- Added new methods for managing distinct fields in dashboards.
- Enhanced search functionality with improved handling of distinct
streams.
- **Bug Fixes**
- Improved error handling for dashboard operations and search requests,
including new error messages.
- **Documentation**
- Updated documentation for methods to clarify functionality and usage.
- **Refactor**
- Streamlined query variable extraction logic for dashboards.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new endpoint for listing user roles, excluding certain
roles.
- Added a new endpoint for token exchange requests for service accounts.
- **Enhancements**
- Improved role management logic and error handling in user management
functions.
- Enhanced error handling for authentication and token validation
processes.
- Updated the handling of claims and user roles based on login flow.
- Enhanced the logic for retrieving login URLs and managing
authentication states.
- **Bug Fixes**
- Prevented unauthorized updates and retrievals for external service
accounts.
- **Documentation**
- Updated method signatures to reflect new parameters and functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced precision for storage size metrics across various components
by updating data types to floating-point numbers.
- **Bug Fixes**
- Adjusted calculations for size metrics to ensure accurate
floating-point operations.
- **Chores**
- Updated test cases to reflect new floating-point type handling for
storage metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR introduces the service accounts with tokens
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced a new module for managing service accounts, including
functionalities for listing, creating, updating, and deleting service
accounts.
- Added a new tab for "Service Accounts" in the Identity Access
Management interface.
- Enhanced user management interfaces to support service account
operations, including the ability to add and edit service accounts.
- Implemented a new component for managing service accounts within
groups and roles.
- Added new API endpoints for service account management, including
token operations.
- Introduced a comprehensive Vue component for managing service accounts
with enhanced features, including filtering, searching, and token
management.
- Enhanced permission checks and error handling for service accounts
across various functions.
- Added support for external user permissions in service account
operations.
- Enhanced error handling for various components to prevent unnecessary
notifications on permission-related issues.
- Added localization support for service account functionalities,
providing necessary labels for user interface elements.
- **Bug Fixes**
- Improved error handling and validation across user management and
service account functionalities.
- **Documentation**
- Updated localization files to support new service account features and
labels.
- **Style**
- Adjusted styles in various components to enhance usability and
consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Previously the `GetDashboard` and `DeleteDashboard` endpoints required
the client to pass in the `folder_id` query parameter. The query
parameter was actually superfluous because the `org_id` and
`dashboard_id` URL parameters provided to each endpoint already uniquely
identified a dashboard. Therefore, this PR removes `folder_id` as a
query parameter from those endpoints.
This PR introduces the optional `page_size` query parameter to the
`ListDashboards` endpoint. When set this will limit the size of the
result set to the specified value. The client/frontend will be able to
set this query parameter in calls to `ListDashboards` to limit the
number of results returned - specifically in requests to search
dashboards across folders.
This PR only implements limiting the result set size and does not
actually expose pagination as functionality of the `ListDashboards`
endpoint. My hope however is that this implementation will put us in a
good place to easily expose pagination on this endpoint if that's
something we want to do in the future - though it would by no means be
necessary.
There are a couple of idiosyncrasies here.
1. The `ListDashboards` endpoint does not read or use any `page_idx` or
`page_number` query parameter, as one might expect from a paginated
endpoint. Therefore if the client sets the `page_size` parameter to some
number N then the `ListDashboards` endpoint will always return the top N
results. (Considered in a different way, the endpoint always acts as if
the requested page index is `0`.)
- In the future we might want to add an additional `page_idx` query
parameter as well to enable "true" pagination (rather than simply
limiting the result set to the first N results). However there are no
immediate plans for this.
2. The `page_size` query parameter is only read/used by the endpoint
when the `title` parameter is **also** set. This means that when the
`title` parameter is **not** set the endpoint will ignore the
`page_size` parameter and will always return all dashboards that match
the other query parameter, `folder`.
- In the future we could choose to use this parameter any time it is
provided, not just when `title` is also provided. To do that there would
need to first be an update to the UI to address the issue described
below.
- For the time being this is necessary in order to not break the UI.
This is because currently the UI always sends the query parameter
`page_size=1000` to the endpoint, even though before now that query
parameter was never recognized by the endpoint. Additionally, the UI
assumes that the calls it makes to `ListDashboards` in order to show the
dashboards in a selected folder always return **all** dashboards, and in
such requests the frontend sets the `folder` query parameter but not the
`title` query parameter. Therefore, if we actually to use the
`page_size` parameter in **those** requests, then in its current
implementation the UI would fail to show all dashboards when a folder
with over 1000 dashboards is selected.
## Summary
This PR only refactors code and should not result in any behavioral
changes.
This PR follows #5220 and creates separate "HTTP Layer" models for each
our dashboard HTTP API endpoints.
By separating the "HTTP Layer" models from the "Service Layer" dashboard
models we will now (in future PRs) be able to...
- add folder name and folder ID fields to each item returned by the
`ListDashboards` endpoint - Step 3 of #5198
- remove dashboard details from the each item returned by the
`ListDashboards` endpoint - Step 1 of #5196
Unfortunately, this PR does not yet accomplish fully decoupling the
"HTTP Layer" dashboard model from the "DB Layer" dashboard model since
both layers still use the same same JSON de/serialization logic defined
on the structs inside `config/meta/dashboards`. So changes to that model
and the corresponding JSON de/serialization logic could still introduce
silent breaking changes into both the interface with the DB and the HTTP
API. This PR moves us closer to decoupling those models, though future
work will be necessary to fully decouple them.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new data structures for handling dashboard-related HTTP
requests and responses, enhancing clarity and type safety.
- Added support for creating, updating, and listing dashboards with
structured request and response bodies.
- **Documentation**
- Updated OpenAPI documentation to reflect new schemas for dashboard
operations, improving API clarity and specificity.
- **Refactor**
- Removed outdated dashboard structures and streamlined request handling
logic for improved maintainability.
- Updated integration tests to align with the new dashboard data
structures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
This PR only refactors code and should not result in any behavioral
changes.
This PR follows #5206 and attempts to refactor dashboards in a similar
way to how that PR refactors folders. This PR aims to divide our
dashboard code into distinct "HTTP Layer" and "Service Layer"s - where
the "HTTP Layer" contains only logic and models related to
serializing/deserializing for the REST HTTP API, and the "Service Layer"
contains only domain models/logic and is agnostic to the REST HTTP API.
This PR only partially achieves the above goals. Specifically, it makes
following changes...
1. Move all HTTP responses out of the `service` module and and into the
`http/request` module.
2. Introduce a new `DashboardError` in the `service` module to model all
types of "domain logic" errors that occur when interacting with folders.
3. Remove the `service/db/folders` module since, after the migration of
folders from the `meta` table to the `folders` table, this module mostly
just acted as a wrapper around the `infra/table/folders` module.
This PR is still missing the following changes. These changes will be
saved for a subsequent PR in order to keep the scope of this PR limited:
- Introduce new `...Response` and `...Request` types in the
`http/request/models` module whose role is specifically to define
serialization/deserialization definitions for the REST HTTP API.
⚠️ Notably, at this point the "HTTP Layer", "Service Layer", and "DB
Layer" models for dashboards are still tightly coupled together since
all still use the shared JSON serialization/deserialization logic
defined on the shared `Dashboard` struct in `config::meta`. Decoupling
the models and de/serialization logic at each layer will take more
effort than folders since the `Dashboard` struct is large and
complicated. I plan to start on this decoupling in the follow-up PR,
though that PR likely won't complete the process.
## Overall design goals
The design goals here are the same as in #5206 . Please see that PR's
description for a detailed explanation.
## Immediate motivation and next steps
Motivations and next steps are generally the same as in #5206 . Please
see that PR's description for a detailed explanation.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced dashboard creation and update processes with improved error
handling and response management.
- Added support for parsing dashboard data across multiple versions.
- **Bug Fixes**
- Streamlined error handling for dashboard operations, ensuring more
consistent and informative responses.
- **Chores**
- Refactored dashboard management logic to remove unnecessary
complexity, improving maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
This PR only refactors code and should not result in any behavioral
changes.
This PR divides our folder code into distinct "HTTP Layer" and "Service
Layer"s - where the "HTTP Layer" contains only logic and models related
to serializing/deserializing for the REST HTTP API, and the "Service
Layer" contains only domain models/logic and is agnostic to the REST
HTTP API.
Specifically, the following changes are made to the code...
1. Move all HTTP responses out of the `service` module and and into the
`http/request` module.
2. Introduce a new `FolderError` in the `service` module to model all
types of "domain logic" errors that occur when interacting with folders.
3. Introduce new `...Response` and `...Request` types in the
`http/request/models` module whose role is specifically to define
serialization/deserialization definitions for the REST HTTP API.
4. Remove the `service/db/folders` module since, after the migration of
folders from the `meta` table to the `folders` table, this module mostly
just acted as a wrapper around the `infra/table/folders` module.
## Overall design goals
The design goal here is to create a clearer divide between the "HTTP
Layer" and "Service Layer" of the application and to decouple the models
in these layers so that models in one layer can change without
introducing breaking changes to models in the other layer. Changes to a
model in one layer only need to be accompanied with appropriate changes
to the translation functions between that model and others.
## Immediate motivation and next steps
I would like to replicate this pattern with Dashboards as well as Alerts
and Reports. Implementing this pattern in Folders seems like a
reasonable place to start since Folders are much less complicated that
Dashboards or Alerts, and Dashboards (and soon-to-be Alerts) are
dependent on Folders since each Dashboard belongs to a parent Folder.
With Dashboards, I have an immediate need to divide "HTTP Layer" and
"Service Layer" models because as parts of #5198 and #5196 I need to
change the data structure of dashboards that are returned by
`ListDashboards` so that they are different from the `Dashboard` data
structure defined in the "Service Layer" - specifically I need each
"dashboard" returned by `ListDashboards` to contain the parent folder's
name and ID, and I need each "dashboard" returned by `ListDashboards` to
exclude the details inside the `v1`,...,`v5` fields.
Additionally, when implementing the migration of dashboards from the
`meta` table to the `dashboards` table, I ran into a lot of avoidable
bugs because our "HTTP Layer", "Service Layer", and "DB Layer" models
for dashboards are all coupled together - reusing the same structure and
serialization/deserialization logic. Therefore, when I changed the model
in which we stored dashboards in the DB, this also accidentally
introduced a breaking change (with no compiler error) to the way
dashboards are serialized/deserialized in the HTTP API. Maintaining
separate models and translation functions for these 3 layers would have
prevented these issues. Therefore, when I start migrating alerts to
there own table, I would like to start by dividing our models for alerts
into "HTTP Layer", "Service Layer", and "DB Layer" models to avoid the
same problems I faced while migrating dashboards.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced new data models for folder management in the API, enhancing
request and response structures.
- Added an asynchronous function to check for folder existence by ID.
- **Improvements**
- Enhanced error handling with a new `FolderError` enum for better
clarity in folder management operations.
- Improved type safety by specifying request body types for folder
creation and updates.
- Refactored folder management API endpoints for clearer response
structures.
- **Bug Fixes**
- Updated folder management API endpoints to ensure more accurate
responses and error handling.
- **Documentation**
- Updated OpenAPI documentation to reflect new folder management schemas
and remove outdated components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
https://github.com/openobserve/openobserve/issues/5198
This PR adds a new optional `title` query parameter to the
`ListDashboards` endpoint.
The endpoint now has the following behavior when these parameters are
provided:
- `folder` is provided and `title` is provided: The endpoint returns all
dashboards in the folder whose titles contain the value of `title` as
case-insensitive substring. This is new non-breaking behavior.
- `folder` is provided and `title` is not provided: The endpoint returns
all dashboards in the folder. This is the same behavior as before.
- `folder` is not provided and `title` is provided: The endpoint returns
all dashboards (regardless of folder) whose titles contain the value of
`title` as a case-insensitive substring. This is new non-breaking
behavior and enables the feature of searching for dashboards across
folders.
- `folder` is not provided and `title` is not provided: The endpoint
returns all dashboards inside the "default" folder. More obvious
behavior would be to return all dashboards, however the endpoint returns
only dashboards inside the "default" folder in order to maintain
backwards comparability and to avoid making a breaking change.
The PR should not result an any behavior changes on the UI.
This is the first PR for #5068 .
We need to move `dashboards` into the `config` crate so that DB access
functions in the `infra` crate can return types defined in the
`dashboards` module.
When the `dashboards` module is in the `openobserve` crate we cannot use
them in the `infra` crate since that would introduce a circular
dependency.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced a new function to move dashboards, enhancing dashboard
management capabilities.
- Added a new `SendReport` trait for improved report handling and
modularity.
- Enhanced error handling across various dashboard and report management
functions.
- **Improvements**
- Updated import paths for better clarity and organization, particularly
for dashboard-related modules.
- Improved timestamp handling for alerts to ensure accurate tracking of
updates.
- Enhanced OpenAPI documentation to reflect updated schema references
for dashboards.
- Expanded development dependencies to support improved functionality.
- Enhanced the logic for handling report triggers based on frequency
types.
- **Bug Fixes**
- Refined error handling in dashboard creation and deletion functions to
provide clearer responses.
- Standardized error handling across report management functions for
consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
#5069
**Description**
- Introduces migrations to
1. Create a new `folder` table
2. Transform folder records from `meta` table into `folder` table
3. Delete folder records from `meta` table
- Updates folder and dashboard service logic to use the new `folder`
table rather than the `meta` table
**Prior steps**
Before this PR is merged, two others must be merged first
1. https://github.com/openobserve/openobserve/pull/5095
2. https://github.com/openobserve/openobserve/pull/5096
After these other PRs are merged this PR will be updated to merge into
`main` rather than into `ps/5069/move-folder-2`.
**Next steps**
Feature #5069 will be complete with this PR.
This will open us up to starting on feature #5068.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new folder management functionalities, including creating,
updating, listing, and deleting folders.
- Added support for error handling improvements in dashboard and folder
operations.
- Implemented a new README for SeaORM entities detailing usage and
generation instructions.
- **Bug Fixes**
- Enhanced error handling in various functions to ensure appropriate
responses based on success or failure.
- **Documentation**
- Added documentation for SeaORM entities and their generation process.
- **Refactor**
- Streamlined folder and dashboard management functions for better
clarity and maintainability.
- Updated API structure for folder-related endpoints to improve clarity
and response handling.
- Improved error handling strategy across multiple service functions.
- Enhanced database interaction methods for folders and dashboards,
promoting modularity and consistency.
- Modified initialization workflow to ensure database schema updates on
application start.
- Simplified error handling in folder management service functions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR creates a general `self_reporting` module to be used internally
for both Usage and Error reporting
- channel-based queues to receive ReportingData (Usage or Error) and
ingest buffed data in batch determined by either `batch_size` or
`timeout`
- General `Error` data structure that can be extended to work with
different kinds of error types throughout the system
- Single interface, `publish_error()` can be used once Error type is
defined
- Removed usage gRPC ingestion service previously used only for
ingesting Usage data. Instead, using the general internal gRPC ingestion
service
This is the first PR for issue #5069 .
This PR moves various `folders` modules out of their parent `dashboards`
modules.
**Why?**
This is a purely organizational change. Going forward the concept of
folders will be treated as independent of dashboards so that a folder
can be used for different types of entities (dashboards, reports, and
alerts). Therefore we want the module organization to reflect this
conceptual change.
**Prior steps**
Before this PR can be merged the open PR
https://github.com/openobserve/openobserve/pull/5095 should be merged.
After that other PR is merged this PR will be updated to merge into
`main` rather than into `ps/5069/move-folder`.
**Next steps**
PR to migrate folders from `meta` table to `folder` table and to update
folder API endpoints.
This is the first PR for issue #5069 .
This PR moves the `Folder` model out of the `openobserve` crate into the
`config` crate.
**Why?**
We would like the `infra` crates DB access functions to return the
`Folder` model. However the `infra` crate cannot reference the `Folder`
model when it is inside the `openobserve` crate since this would require
a circular dependency. After we move the `Folder` model into the
`config` crate (with various other data models), we will be able to
reference that model from within the `infra` crate.
**Next steps**
1. PR to move various folder modules out of parent dashboard modules
since we want to treat folders as independent of dashboards.
https://github.com/openobserve/openobserve/pull/5096
2. PR to migrate folders from `meta` table to `folder` table and to
update folder API endpoints.
Hengfei Yang
Patrick Sullivan
Huaijin
Taiming Liu
Bhargavandhengfei yang
udevandHengfei Yang
Loakesh Indiranandoasisk
Yashodhan JoshiandYashodhan Joshi
Ashish Kolhe
