<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive test cases for API function creation
- Included tests for invalid payloads, text-only inputs, and special
character scenarios
- Verified successful function creation with valid payloads
- **Chores**
- Updated `.gitignore` to exclude temporary directories, configuration
files, and build artifacts
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive end-to-end tests for service account management.
- Implemented tests for retrieving, creating, updating, refreshing, and
deleting service accounts.
- Included validation for handling invalid email formats during service
account creation.
- Ensured proper validation of API responses and status codes for
service account operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive test cases for dashboard and alerts folder
functionalities.
- Introduced tests for unauthorized access scenarios.
- Added error handling tests for invalid inputs and folder IDs.
- Improved test coverage for various edge cases, including responses for
empty and invalid folder IDs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
New Folders API test cases
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added comprehensive API testing for dashboard and alert
functionalities.
- Implemented test cases for retrieving, creating, updating, and
deleting dashboard and alert folders.
- Ensured proper response status code validation for various API
operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved clarity in passcode-related tests by renaming variables and
updating assertion messages.
- Enhanced specificity in testing passcode responses and updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
added PyTest for reset passcode
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added a new end-to-end test for the passcode reset functionality,
ensuring successful operation and response validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close#5420
- [x] set tantivy index as default index
- [x] support cross stream type join like below
```
select t1.ip, t2.name from logs.t1 as t1 join enrich.t2 as t1 on t1.ip = t2.ip
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `TableReferenceExt` module to enhance SQL-related
functionalities.
- Added a new `StreamTypeProvider` struct for managing table providers.
- **Improvements**
- Updated configuration defaults for inverted index formats.
- Enhanced handling of stream names and types throughout various
functions for better type safety.
- Improved logging with structured stream information.
- Refined the processing of the `MatchAll` condition in SQL-like
expressions.
- **Bug Fixes**
- Expanded string matching for `StreamType` to accommodate additional
input formats.
- **Documentation**
- Retained help comments for context on configuration fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new test suite for end-to-end testing of fuzzy matching
SQL queries.
- Added new test cases for verifying API data display and error message
handling in the UI.
- **Bug Fixes**
- Improved reliability of existing tests by enhancing visibility checks
before UI interactions.
- **Tests**
- Added helper functions for dynamic timestamp generation and SQL query
posting in the testing framework.
- Updated existing tests to streamline processes and improve interaction
logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new suite of Playwright tests for validating enrichment
table functionality.
- Added new functions for enhanced navigation and interaction within the
pipeline UI.
- Added tests to assert the data in every derived stream with condition,
function and only source + derived stream
- Added tests to assert if user clicks on any menu option and cancel on
dialog box, user stays on pipeline page
- Added tests to assert if user clicks on dashboard menu option and
accept on dialog box, user navigates to dashboard page
- **Bug Fixes**
- Improved error handling in tests to ensure accurate feedback during
interactions.
- **Chores**
- Updated the GitHub Actions workflow to include additional integration
tests.
- Removed outdated test files related to function management, Kinesis
ingestion, and reporting functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5378
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Enhanced error handling for invalid stream names during ingestion,
ensuring appropriate logging and response for empty or invalid entries.
- Updated assertions in tests to reflect the correct handling of invalid
bulk payloads by the API, ensuring errors are present for invalid
submissions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Introduced a new end-to-end test for pipeline creation, enabling, and
disabling.
- Utilized parameterization to test various scenarios with different
pipeline names, actions, and expected HTTP status codes.
- Removed the previous end-to-end tests related to user token
management, which included user token retrieval and email verification
processes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced end-to-end testing for bulk ingestion with comprehensive
coverage.
- Enhanced functionality for managing enrichment tables, including
adding, uploading, and deleting entries.
- **Bug Fixes**
- Improved error handling and user interaction checks in the enrichment
table upload process.
- **Tests**
- Added multiple new test cases for enrichment table functionality,
including upload and error handling scenarios.
- Updated existing test cases for consistency and improved logic.
- **Chores**
- Standardized import statements and improved overall test structure for
clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced dashboard management with new methods for retrieving
properties like ID, owner, title, and description.
- Added functionality for creating, updating, listing, and deleting
dashboards through a new service module.
- Introduced a new `dashboards` table in the database, with migrations
for creation and population.
- **Bug Fixes**
- Improved error handling for dashboard retrieval and management
operations, providing clearer responses for various conditions.
- **Documentation**
- Updated documentation to reflect new methods and error handling
improvements.
- **Refactor**
- Streamlined the dashboard handling logic in service and database
modules, reducing complexity and enhancing modularity.
- Enhanced error handling logic in dashboard management functions for
better clarity and control flow.
- Refined the dashboard retrieval process to improve clarity in control
flow and error handling.
- Updated the logic for dashboard operations to simplify version
handling and reduce redundancy.
- Improved the overall structure and organization of the dashboard
management code.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The previous mechanism for generating hashes for dashboard models
depended on hashing the JSON representation of the dashboard that was
stored in the `meta` table.
As part of the migration of dashboards out of the `meta` table and into
a dedicated `dashboards` table, this ☝️ presented some problems:
Specifically when we want to translate from the Dashboard ORM model to a
`core::meta::dashboards::Dashboard` model, we need to generate the hash
at the time of translation since `hash` is a field on
`core::meta::dashboards::Dashboard`. We can't simply generate a hash
from the JSON stored in the `dashboards.data` column because this JSON
no longer contains fields for `title`, `owner`, `description`, and
`role`. Those fields are now stored in DB columns, and changes to those
columns in the DB columns would *not* result in changes to the hash on
the JSON blob.
Therefore, rather than hashing the JSON blob, this PR implements hash
functions on the Rust types for the dashboards. Therefore changes to
fields in those Rust structs will result in changes to the hash values,
but changes to the way in which the data is stored in the DB will not
affect the hash values.
⚠️ Note that this PR is changing the hash function on the dashboard
models, so hashes generated for the same model before and after this
deployment will not be the same. My understanding is that this shouldn't
be an issue since hashes are just used as a way of checking on the UI if
the dashboard model has been updated to avoid overwriting the changes of
another user. I believe that this change to the dashboard function might
just have the effect (around the time of deployment) where it would
appear to the UI that the dashboard has been updated when, in fact, it
has not.
https://github.com/openobserve/o2-enterprise/actions/runs/11979723563
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new type alias `OrdF64` for enhanced precision in
floating-point operations.
- Added the `Hash` trait to multiple dashboard-related structs and
enums, enabling their use in hash-based collections.
- **Bug Fixes**
- Improved error handling and assertion clarity in API testing for
folder and dashboard operations.
- **Chores**
- Standardized hashing mechanism for dashboard objects in database
operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new JSON file for SQL join query test cases, enhancing
the variety and coverage of tests.
- Added multiple new test cases for various SQL join operations,
improving testing capabilities.
- **Bug Fixes**
- Improved error handling in data ingestion steps to ensure successful
test execution.
- **Refactor**
- Transitioned from hardcoded test data to a parameterized approach for
better maintainability and flexibility in testing.
- Updated method signature to streamline test parameters.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR implements the new design of **Pipeline** introduced by #4432
---
**2 types of pipelines differentiates by the source**
Type I: **Realtime**
- source is a stream (Logs, Metrics, & Traces)
- takes effect when the source stream is being ingested
Type II: **Scheduled** (aka DerivedSteram)
- source is a SQL query
- takes effect based on the given schedule
The new pipeline is represented as a graph consists of different types
of Nodes that are connected by edges.
**Pipeline Data Model**
```rs
pub struct Pipeline {
pub id: String,
pub version: i32,
pub enabled: bool,
pub org: String,
pub name: String,
pub description: String,
pub source: PipelineSource,
pub nodes: Vec<Node>,
pub edges: Vec<Edge>,
}
```
**4 types of node**
I. StreamNode: either used for source node of realtime pipeline or for
destination nodes
II. QueryNode: used for source node of scheduled pipeline
III. FunctionNode: used for executing vrl functions
IV. ConditionNode: used for checking routing conditions
Rules applied to validating a pipeline when it's created:
1. non-empty nodes list
2. non-empty edges list
3. 1st node in nodes list is either StreamNode or QueryNode
4. non-empty `conditions` in all ConditionNode nodes in nodes list
5. all leaf nodes are of type StreamNode
6. In the same branch, unchecked `after_flattened` FunctionNode can't
follow checked
`after_flattened` checked FunctionNode
Pipeline execution is implemented with the struct `ExecutablePipeline`,
a ready-execute pipeline object cached in memory based on the Pipeline
objects modified on the UI side.
`ExecutablePipeline` object
- builds the relationships among all the nodes based on the edges
- topologically sorts the nodes based on the level to determine which
node to process first
- registers all the vrl function node once
`ExecutablePipeline` object processes ingested records in batch.
Starting from the source node, each connecting edge has a channel
connecting the parent and child node, used for passing the records. This
approach enables batch processing in parallel.
---
**Deprecated Features**
1. previous implementation of pipeline
2. Function x Stream associations
- Functions can still be added/edited/removed as before
- Needs to create a pipeline in order to apply chosen functions to the
desired stream
The new release when this pr is merged will automatically migrate **old
pipelines** and **Function x Stream associations** to new pipeline
format and can be found in `Stream Pipelines` tab with `Migrated-`
prefix names.
**_Note: Auto generated pipelines are paused by default. Please verify
the pipelines before enabling them_**
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Introduced a new parameterized test suite for validating SQL queries
involving `cast` and `try_cast` operations.
- Enhanced testing framework to ensure API reliability when handling
various SQL casting scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Introduced a series of end-to-end tests for SQL queries with various
join types to ensure functionality against the API.
- Added tests for `LIKE`, inner join, `IN`, left join, right join, and
full join SQL queries.
- Implemented parameterized testing for improved coverage and
maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added end-to-end tests for creating alerts based on SQL and float
values.
- Introduced a page object model for unflattened logs to streamline UI
interactions.
- Implemented a function to toggle quick mode in the test suite.
- **Bug Fixes**
- Enhanced error handling and validation in alert tests.
- Improved reliability of tests by ensuring UI elements are visible
before interactions.
- **Refactor**
- Refactored Playwright tests to utilize the `UnflattenedPage` object
for better maintainability.
- Consolidated parameters in fetch requests for clarity.
- **Chores**
- Removed redundant tests from the alert test suite to improve clarity
and maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Added more queries
Added test for match_all single character #4439
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced SQL query testing with a new data source, improving test
accuracy.
- Introduced multiple new test functions for comprehensive SQL
functionality coverage, including counting and coalescing.
- **Bug Fixes**
- Updated existing tests to align with the new data source, ensuring
reliable query validation.
- **Refactor**
- Replaced and renamed tests to reflect changes in the data source and
improve clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced enhanced test functions for more precise SQL queries,
improving the validation of result cache ratios.
- Added a new test for executing multiple SQL queries simultaneously,
expanding test coverage.
- **Bug Fixes**
- Updated assertions in existing tests to align with new query
structures and expected values.
- **Chores**
- Both new test functions are currently marked as skipped, indicating
they are not ready for execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added end-to-end test functions to validate SQL query handling.
- Introduced new log entry for end-to-end test tracking.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Fixed some breaking api tests
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Updated end-to-end tests related to SQL queries by commenting out
specific test functions.
- Enhanced SQL queries in tests with `ORDER BY` clauses for more precise
results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Introduced an end-to-end test for SQL query caching to improve
reliability.
- Modified a UI test description and marked it as skipped for future
refinement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Added more search queries under pytest
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added new end-to-end test functions for various SQL queries, including
count aggregation, match, max, in, and distinct queries, to ensure API
reliability and accuracy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced search functionality with indexed matching options.
- **Bug Fixes**
- Updated SQL queries in tests to align with new matching functions.
- **Improvements**
- Standardized naming conventions for matching functions to improve
clarity and consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
match_all renamed to match_all_raw
match_all_indexed renamed to match_all
same pattern applied to *_ignore_case functions
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved clarity in function names for better understanding and
usability.
- **Tests**
- Updated SQL queries in test cases to reflect changes in function
names, ensuring accurate test results.
- **Chores**
- Adjusted logging statements to provide better insights during
debugging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- added match all valid cases for histogram and sql
- match_all_indexed cases for histogram and sql
- match_all_ignore case for histogram and sql
- match_all_indexed_ignorecase for histogram and sql
---------
Co-authored-by: Ankur Srivastava <best.ankur@gmail.com>
Created a fixture which will be available during scope=session. This
will ingest data present in tests/test-data/logs_data.json to
org_pytest_data and stream_pytest_data
API Tests positive and negative scenarios -
- Search without payload
- Search with valid sql query
- search with invalid sql query
- search with limit and sql query
- valid histogram query
- histogram query with limit
---------
Co-authored-by: Neha P <neha@openobserve.ai>
Co-authored-by: Neha P <144308844+neha00290@users.noreply.github.com>
- Simplify the API integration tests by removing steps not needed (Node,
k8s, building docker, etc).
- Testing the UI, linting, formatting, etc should be done in a separate
workflow file
- Simplify Rust cache workflow.
- By default it caches /target
- Removed Dockerfile and k8s deployment from CI.
- You can just run `OpenObserve` in the background
- Replace `ubicloud-standard-16` with `ubuntu-latest` which is *FREE*.
- Using 16 cores, 64GB of ram is an overkill for running tests.
- Don't run the CI if changes to `.md` files are detected.
Notes:
- Before caching building `OpenObserve` takes around 5-7 mins, after
cache it goes down to 1-2 mins.
Neha P
Hengfei Yang
Shyam Panjiyar
Huaijin
Loakesh Indiranandhengfei yang
Patrick Sullivan
Yashodhan Joshi
Ashish Kolhe
Juan Calderon-Perez