This is the first PR for #5068 .
We need to move `dashboards` into the `config` crate so that DB access
functions in the `infra` crate can return types defined in the
`dashboards` module.
When the `dashboards` module is in the `openobserve` crate we cannot use
them in the `infra` crate since that would introduce a circular
dependency.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced a new function to move dashboards, enhancing dashboard
management capabilities.
- Added a new `SendReport` trait for improved report handling and
modularity.
- Enhanced error handling across various dashboard and report management
functions.
- **Improvements**
- Updated import paths for better clarity and organization, particularly
for dashboard-related modules.
- Improved timestamp handling for alerts to ensure accurate tracking of
updates.
- Enhanced OpenAPI documentation to reflect updated schema references
for dashboards.
- Expanded development dependencies to support improved functionality.
- Enhanced the logic for handling report triggers based on frequency
types.
- **Bug Fixes**
- Refined error handling in dashboard creation and deletion functions to
provide clearer responses.
- Standardized error handling across report management functions for
consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
#5069
**Description**
- Introduces migrations to
1. Create a new `folder` table
2. Transform folder records from `meta` table into `folder` table
3. Delete folder records from `meta` table
- Updates folder and dashboard service logic to use the new `folder`
table rather than the `meta` table
**Prior steps**
Before this PR is merged, two others must be merged first
1. https://github.com/openobserve/openobserve/pull/5095
2. https://github.com/openobserve/openobserve/pull/5096
After these other PRs are merged this PR will be updated to merge into
`main` rather than into `ps/5069/move-folder-2`.
**Next steps**
Feature #5069 will be complete with this PR.
This will open us up to starting on feature #5068.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new folder management functionalities, including creating,
updating, listing, and deleting folders.
- Added support for error handling improvements in dashboard and folder
operations.
- Implemented a new README for SeaORM entities detailing usage and
generation instructions.
- **Bug Fixes**
- Enhanced error handling in various functions to ensure appropriate
responses based on success or failure.
- **Documentation**
- Added documentation for SeaORM entities and their generation process.
- **Refactor**
- Streamlined folder and dashboard management functions for better
clarity and maintainability.
- Updated API structure for folder-related endpoints to improve clarity
and response handling.
- Improved error handling strategy across multiple service functions.
- Enhanced database interaction methods for folders and dashboards,
promoting modularity and consistency.
- Modified initialization workflow to ensure database schema updates on
application start.
- Simplified error handling in folder management service functions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR creates a general `self_reporting` module to be used internally
for both Usage and Error reporting
- channel-based queues to receive ReportingData (Usage or Error) and
ingest buffed data in batch determined by either `batch_size` or
`timeout`
- General `Error` data structure that can be extended to work with
different kinds of error types throughout the system
- Single interface, `publish_error()` can be used once Error type is
defined
- Removed usage gRPC ingestion service previously used only for
ingesting Usage data. Instead, using the general internal gRPC ingestion
service
This is the first PR for issue #5069 .
This PR moves various `folders` modules out of their parent `dashboards`
modules.
**Why?**
This is a purely organizational change. Going forward the concept of
folders will be treated as independent of dashboards so that a folder
can be used for different types of entities (dashboards, reports, and
alerts). Therefore we want the module organization to reflect this
conceptual change.
**Prior steps**
Before this PR can be merged the open PR
https://github.com/openobserve/openobserve/pull/5095 should be merged.
After that other PR is merged this PR will be updated to merge into
`main` rather than into `ps/5069/move-folder`.
**Next steps**
PR to migrate folders from `meta` table to `folder` table and to update
folder API endpoints.
This is the first PR for issue #5069 .
This PR moves the `Folder` model out of the `openobserve` crate into the
`config` crate.
**Why?**
We would like the `infra` crates DB access functions to return the
`Folder` model. However the `infra` crate cannot reference the `Folder`
model when it is inside the `openobserve` crate since this would require
a circular dependency. After we move the `Folder` model into the
`config` crate (with various other data models), we will be able to
reference that model from within the `infra` crate.
**Next steps**
1. PR to move various folder modules out of parent dashboard modules
since we want to treat folders as independent of dashboards.
https://github.com/openobserve/openobserve/pull/5096
2. PR to migrate folders from `meta` table to `folder` table and to
update folder API endpoints.
Adds a new endpoint for `/functions` to find a list of associate
pipelines. This list is also used to warn user to remove associated pipelines
before deleting a function.
---------
Co-authored-by: Sai Nikhil <nikhil@openobserve.ai>
Fixes#4982
- Adds migration for pipelines for ofga tuple database.
- Fixes some rbac permission logic for pipelines.
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
This PR implements the new design of **Pipeline** introduced by #4432
---
**2 types of pipelines differentiates by the source**
Type I: **Realtime**
- source is a stream (Logs, Metrics, & Traces)
- takes effect when the source stream is being ingested
Type II: **Scheduled** (aka DerivedSteram)
- source is a SQL query
- takes effect based on the given schedule
The new pipeline is represented as a graph consists of different types
of Nodes that are connected by edges.
**Pipeline Data Model**
```rs
pub struct Pipeline {
pub id: String,
pub version: i32,
pub enabled: bool,
pub org: String,
pub name: String,
pub description: String,
pub source: PipelineSource,
pub nodes: Vec<Node>,
pub edges: Vec<Edge>,
}
```
**4 types of node**
I. StreamNode: either used for source node of realtime pipeline or for
destination nodes
II. QueryNode: used for source node of scheduled pipeline
III. FunctionNode: used for executing vrl functions
IV. ConditionNode: used for checking routing conditions
Rules applied to validating a pipeline when it's created:
1. non-empty nodes list
2. non-empty edges list
3. 1st node in nodes list is either StreamNode or QueryNode
4. non-empty `conditions` in all ConditionNode nodes in nodes list
5. all leaf nodes are of type StreamNode
6. In the same branch, unchecked `after_flattened` FunctionNode can't
follow checked
`after_flattened` checked FunctionNode
Pipeline execution is implemented with the struct `ExecutablePipeline`,
a ready-execute pipeline object cached in memory based on the Pipeline
objects modified on the UI side.
`ExecutablePipeline` object
- builds the relationships among all the nodes based on the edges
- topologically sorts the nodes based on the level to determine which
node to process first
- registers all the vrl function node once
`ExecutablePipeline` object processes ingested records in batch.
Starting from the source node, each connecting edge has a channel
connecting the parent and child node, used for passing the records. This
approach enables batch processing in parallel.
---
**Deprecated Features**
1. previous implementation of pipeline
2. Function x Stream associations
- Functions can still be added/edited/removed as before
- Needs to create a pipeline in order to apply chosen functions to the
desired stream
The new release when this pr is merged will automatically migrate **old
pipelines** and **Function x Stream associations** to new pipeline
format and can be found in `Stream Pipelines` tab with `Migrated-`
prefix names.
**_Note: Auto generated pipelines are paused by default. Please verify
the pipelines before enabling them_**
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
This implements a background job for consuming metrics generated by
openobsere directly in itself. This is controlled by these env vars -
- `ZO_SELF_METRIC_CONSUMPTION_ENABLED` - enable/disable self metric
consumption, false by default
- `ZO_SELF_METRIC_CONSUMPTION_INTERVAL` - interval to run the job in
seconds, 60 seconds by default
- `ZO_SELF_METRIC_CONSUMPTION_WHITELIST` - comma separated list of
metric names to ingest, empty by default. The job does not run if this
is empty.
On ingesters, this directly calls the jons metric ingestion function to
ingest the metrics from the node. Other nodes do an grpc request to one
of the ingesters and send the metrics to it.
All metrics are ingested under the `_meta` org, as that seemed sensible
to have the metric of openobserve in itself.
The Json encode code was taken from
https://crates.io/crates/prometheus-json-encoder , which unfortunately
does not have any original repo anymore. It had to be changed in some
places to fit openobserve's needs, and it is originally licensed under
Apache 2.0 . I have kept the original notice header in the file as-is.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced self-consumption of metrics, allowing users to configure
which metrics to self-consume and at what intervals.
- Added a JSON encoder for Prometheus metrics to facilitate integration
with other systems.
- Enhanced metrics ingestion capabilities to support JSON data.
- New configuration options for self-consumption of metrics, including
enabling/disabling and specifying a whitelist.
- **Bug Fixes**
- Improved error handling for metrics ingestion requests.
- **Documentation**
- Updated public interfaces to reflect new modules and functions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Addresses #4854
Currently, when updating organization settings, if relevant fields (like
`scrape_interval`, `trace_id_field_name` etc.) are not found, their
default values are used and hence all the fields that are not in the
request payload, are set to default.
This PR only updates the fields that are present in the request payload,
if none of the relevant fields are present, it returns 400 bad request
error.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `OrganizationSettingPayload` structure for optional
organization settings, enhancing flexibility in configuration.
- Updated the settings creation process to handle optional fields and
improve validation.
- **Bug Fixes**
- Enhanced error handling for organization settings, ensuring proper
responses for invalid or missing data.
- **Documentation**
- Updated method signatures to reflect changes in the expected input
structure for organization settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Closes: #4749
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced logic for updating stream settings, including synchronization
with index streams for data retention.
- Improved error handling for stream cache operations, returning
immediate responses for disabled caches.
- **Bug Fixes**
- Added checks to ensure proper handling of cache deletion based on
cache status.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `RedirectResponse` struct for improved HTTP
redirection handling.
- Enhanced authentication validation with a new redirect mechanism for
`/short/` requests.
- Updated the `retrieve` function to improve logging and response
handling.
- **Bug Fixes**
- Improved error management for authentication failures.
- **Refactor**
- Restructured route configurations to integrate new authentication
middleware and enhance service management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Addresses #4619
- [x] Multi timerange alerts
- [x] Add `is_partial` field to the `triggers` usage stream.
### Note: Only available in SQL alerts
<img width="1248" alt="Screenshot 2024-10-07 at 12 39 36 PM"
src="https://github.com/user-attachments/assets/57c99b29-6d59-496d-bb91-f958b6f2e0bd">
- Historical timeranges can be selected through the `Multi Window
Selection` field.
- When multi-window timeranges are selected, the vrl of the alert is
applied only after fetching all the timerange data and is applied to the
combined multi-timerange records.
- If `ResultArray` is specified in the vrl function, the input to the
vrl function will be multi-dimensional i.e. if two timeranges are
selected, then the input to the vrl function will be a two-dimensional
array containing a total of three arrays. Each array represents the data
of each timerange.
- If `ResultArray` is not specified, the input of vrl function is an
one-dimensional array containing records of all the timeranges in a
sorted order (from most recent to oldest).
- After applying the vrl function, the output array (irrespective of
multi/single dimension array) is first converted into a one-dimensional
array and then only the `threshold` rule is applied to the output of vrl
function. If the `threshold` satisfies then the alert triggers otherwise
not.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced support for multiple time ranges in alert configurations.
- Added a new `search_multi` function for handling multiple search
queries in a single request.
- Enhanced alert creation with new properties and methods for managing
multiple time ranges.
- New `is_partial` field added to `TriggerData` for tracking partial
evaluations.
- **Bug Fixes**
- Improved error handling for partial alert evaluations.
- **Documentation**
- Updated comments and documentation for better clarity on new features
and changes.
- **Refactor**
- Streamlined the logic for SQL query handling and alert evaluation
processes.
- **Style**
- Minor formatting adjustments across various components for improved
readability.
- **Tests**
- Enhanced validation checks and filtering logic in alert management
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Sai Nikhil <nikhil@openobserve.ai>
#4698
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new "Seconds" option in the date-time picker for more
precise time selection.
- Updated time formats to include seconds in the start and end time
fields.
- Enhanced the component's ability to manage relative time selections
with the addition of seconds.
- Improved handling of duration strings to recognize seconds.
- Adjusted dashboard interactions to handle date-time values more
effectively during zoom events.
- **Bug Fixes**
- Improved validation and logic for handling new time formats and
relative periods.
- Enhanced clarity and efficiency in query duration handling for search
functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
- Add `function`, `event`, `_timestamp` and `unit` to
`SearchHistoryHitResponse`.
- Only include `Search` events for search history
- Revert the change made in #4694 i.e. reuse `SearchHIstoryHitResponse`
for search history.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced search history functionality with additional metadata fields
for more detailed responses.
- Improved error handling and logging for search history operations.
- Updated data processing logic in the Search History component for
accurate representation of timestamps and durations.
- **Bug Fixes**
- Streamlined data processing and presentation in the Search History
component, ensuring accurate display of search history data.
- **Documentation**
- Updated component and method signatures to reflect changes in data
handling and structure.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Sai Nikhil <nikhil@openobserve.ai>
1.New Fields --> executed at introduced --> this helps in user to know
when this search got executed.
2. when expands Function defination got introduced if user searched with
function then it will be displayed.
3. Pagination got introduced to show all the results by default per page
user can able to see 100 rows.
4.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced search history component with pagination and improved data
handling.
- Added new fields in usage tracking to capture function-specific data.
- New method for dynamic pagination control in the search history
component.
- **Improvements**
- Streamlined search result processing by removing unnecessary
filtering.
- Improved timestamp handling in usage reporting.
- Flexible handling of the size parameter in search history retrieval.
- Refined partitioning logic and filtering capabilities in search
functionality.
- Enhanced user interface responsiveness in the search history
component.
- **Bug Fixes**
- Adjusted sorting functionality and pagination logic in the search
history component.
These updates enhance the user experience and improve data accuracy
throughout the application.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Hengfei Yang
Huaijin
Patrick SullivanandTaiming Liu
udev
Yashodhan Joshi
Subhra264andTaiming Liu
Loakesh Indiranandhengfei yang
Prabhat Sharma
Ashish Kolhe
