This PR implements the new design of **Pipeline** introduced by #4432
---
**2 types of pipelines differentiates by the source**
Type I: **Realtime**
- source is a stream (Logs, Metrics, & Traces)
- takes effect when the source stream is being ingested
Type II: **Scheduled** (aka DerivedSteram)
- source is a SQL query
- takes effect based on the given schedule
The new pipeline is represented as a graph consists of different types
of Nodes that are connected by edges.
**Pipeline Data Model**
```rs
pub struct Pipeline {
pub id: String,
pub version: i32,
pub enabled: bool,
pub org: String,
pub name: String,
pub description: String,
pub source: PipelineSource,
pub nodes: Vec<Node>,
pub edges: Vec<Edge>,
}
```
**4 types of node**
I. StreamNode: either used for source node of realtime pipeline or for
destination nodes
II. QueryNode: used for source node of scheduled pipeline
III. FunctionNode: used for executing vrl functions
IV. ConditionNode: used for checking routing conditions
Rules applied to validating a pipeline when it's created:
1. non-empty nodes list
2. non-empty edges list
3. 1st node in nodes list is either StreamNode or QueryNode
4. non-empty `conditions` in all ConditionNode nodes in nodes list
5. all leaf nodes are of type StreamNode
6. In the same branch, unchecked `after_flattened` FunctionNode can't
follow checked
`after_flattened` checked FunctionNode
Pipeline execution is implemented with the struct `ExecutablePipeline`,
a ready-execute pipeline object cached in memory based on the Pipeline
objects modified on the UI side.
`ExecutablePipeline` object
- builds the relationships among all the nodes based on the edges
- topologically sorts the nodes based on the level to determine which
node to process first
- registers all the vrl function node once
`ExecutablePipeline` object processes ingested records in batch.
Starting from the source node, each connecting edge has a channel
connecting the parent and child node, used for passing the records. This
approach enables batch processing in parallel.
---
**Deprecated Features**
1. previous implementation of pipeline
2. Function x Stream associations
- Functions can still be added/edited/removed as before
- Needs to create a pipeline in order to apply chosen functions to the
desired stream
The new release when this pr is merged will automatically migrate **old
pipelines** and **Function x Stream associations** to new pipeline
format and can be found in `Stream Pipelines` tab with `Migrated-`
prefix names.
**_Note: Auto generated pipelines are paused by default. Please verify
the pipelines before enabling them_**
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Introduced a new parameterized test suite for validating SQL queries
involving `cast` and `try_cast` operations.
- Enhanced testing framework to ensure API reliability when handling
various SQL casting scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Introduced a series of end-to-end tests for SQL queries with various
join types to ensure functionality against the API.
- Added tests for `LIKE`, inner join, `IN`, left join, right join, and
full join SQL queries.
- Implemented parameterized testing for improved coverage and
maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added end-to-end tests for creating alerts based on SQL and float
values.
- Introduced a page object model for unflattened logs to streamline UI
interactions.
- Implemented a function to toggle quick mode in the test suite.
- **Bug Fixes**
- Enhanced error handling and validation in alert tests.
- Improved reliability of tests by ensuring UI elements are visible
before interactions.
- **Refactor**
- Refactored Playwright tests to utilize the `UnflattenedPage` object
for better maintainability.
- Consolidated parameters in fetch requests for clarity.
- **Chores**
- Removed redundant tests from the alert test suite to improve clarity
and maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Added more queries
Added test for match_all single character #4439
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced SQL query testing with a new data source, improving test
accuracy.
- Introduced multiple new test functions for comprehensive SQL
functionality coverage, including counting and coalescing.
- **Bug Fixes**
- Updated existing tests to align with the new data source, ensuring
reliable query validation.
- **Refactor**
- Replaced and renamed tests to reflect changes in the data source and
improve clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced enhanced test functions for more precise SQL queries,
improving the validation of result cache ratios.
- Added a new test for executing multiple SQL queries simultaneously,
expanding test coverage.
- **Bug Fixes**
- Updated assertions in existing tests to align with new query
structures and expected values.
- **Chores**
- Both new test functions are currently marked as skipped, indicating
they are not ready for execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added end-to-end test functions to validate SQL query handling.
- Introduced new log entry for end-to-end test tracking.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Fixed some breaking api tests
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Updated end-to-end tests related to SQL queries by commenting out
specific test functions.
- Enhanced SQL queries in tests with `ORDER BY` clauses for more precise
results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Introduced an end-to-end test for SQL query caching to improve
reliability.
- Modified a UI test description and marked it as skipped for future
refinement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Added more search queries under pytest
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Added new end-to-end test functions for various SQL queries, including
count aggregation, match, max, in, and distinct queries, to ensure API
reliability and accuracy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced search functionality with indexed matching options.
- **Bug Fixes**
- Updated SQL queries in tests to align with new matching functions.
- **Improvements**
- Standardized naming conventions for matching functions to improve
clarity and consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
match_all renamed to match_all_raw
match_all_indexed renamed to match_all
same pattern applied to *_ignore_case functions
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved clarity in function names for better understanding and
usability.
- **Tests**
- Updated SQL queries in test cases to reflect changes in function
names, ensuring accurate test results.
- **Chores**
- Adjusted logging statements to provide better insights during
debugging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- added match all valid cases for histogram and sql
- match_all_indexed cases for histogram and sql
- match_all_ignore case for histogram and sql
- match_all_indexed_ignorecase for histogram and sql
---------
Co-authored-by: Ankur Srivastava <best.ankur@gmail.com>
Created a fixture which will be available during scope=session. This
will ingest data present in tests/test-data/logs_data.json to
org_pytest_data and stream_pytest_data
API Tests positive and negative scenarios -
- Search without payload
- Search with valid sql query
- search with invalid sql query
- search with limit and sql query
- valid histogram query
- histogram query with limit
---------
Co-authored-by: Neha P <neha@openobserve.ai>
Co-authored-by: Neha P <144308844+neha00290@users.noreply.github.com>
- Simplify the API integration tests by removing steps not needed (Node,
k8s, building docker, etc).
- Testing the UI, linting, formatting, etc should be done in a separate
workflow file
- Simplify Rust cache workflow.
- By default it caches /target
- Removed Dockerfile and k8s deployment from CI.
- You can just run `OpenObserve` in the background
- Replace `ubicloud-standard-16` with `ubuntu-latest` which is *FREE*.
- Using 16 cores, 64GB of ram is an overkill for running tests.
- Don't run the CI if changes to `.md` files are detected.
Notes:
- Before caching building `OpenObserve` takes around 5-7 mins, after
cache it goes down to 1-2 mins.
ShyamOOAI
Neha P
Hengfei Yang
Yashodhan Joshi
Ashish Kolhe
Juan Calderon-Perez