mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 00:09:35 +00:00
udevandAshish Kolhe
1290512fe3
### **User description** Add separate envs for pipeline wal Improve metrics ___ ### **PR Type** Enhancement ___ ### **Description** - Add batch buffering for pipeline WAL writes - Schedule periodic buffer flush in enterprise mode - Introduce env configs and metric labels - Instrument WAL reader with timing logs ___ ### Diagram Walkthrough ```mermaid flowchart LR A["process_node"] B["group by batch_key"] C["BATCH_BUFFERS"] D["get_pipeline_wal_writer"] E["remote WAL"] F["accumulate"] G["pipeline flush task"] A -- "records" --> B B -- "add to buffer" --> C C -- "should_flush? yes" --> D D -- "write_wal" --> E C -- "should_flush? no" --> F G -- "flush_all_buffers" --> C ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody></tr></tbody></table> </details> ___ --------- Co-authored-by: Ashish Kolhe <ashish.j.kolhe@gmail.com>
pytest-openobserve
This is a repo to test openobserve API endpoints.
Introduction:
The tests are running using pytest against alpha1 for now.
Idea is to parametrize it for the other endpoints like production as well.
Getting Started
- We use
ryeto manage the python version and environment - https://github.com/mitsuhiko/rye - Once installed, simply run the following steps:
rye syncrye run pytest
TODOS:
- Parametrize tests to run against different environments
- Run the tests as cron-job every few hours or daily etc. for different environments
- Send slack-notification ( use github-actions ) in case something fails on
alpha1orproductionendpoints.