## Summary
- Adds a matrix strategy to the query agent SQL regression suite jobs in
both `api-testing.yml` and `query-agent.yml`
- Each job now runs with `ZO_FEATURE_SINGLE_NODE_OPTIMIZE_ENABLED=false`
and `ZO_FEATURE_SINGLE_NODE_OPTIMIZE_ENABLED=true`
- Verifies both DataFusion physical optimizer paths produce identical
query results
- Matrix legs run in parallel with `fail-fast: false`
- All artifact/log names include the matrix label to avoid collisions
## Test plan
- [ ] PR CI shows two `api_query_agent` matrix legs
(single_node_opt_false and single_node_opt_true), both pass
- [ ] `/query-agent` PR comment triggers both modes and posts per-mode
results comments
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Fixes a test gap where `match_all` Tantivy queries could return wrong
results after a DataFusion upgrade without being caught by existing
assertions.
### Problem
After a DataFusion upgrade merge, the DataFusion access plan was no
longer used for `match_all` queries, causing Tantivy to return wrong
results. The existing tests only compared row counts against a DuckDB
oracle that emulates `match_all` with `LIKE '%term%'` — so wrong rows
that still matched the LIKE pattern passed undetected.
Additionally, `node/flush` does not generate Tantivy indexes
immediately. The default `ZO_MAX_FILE_RETENTION_TIME=600s` means
WAL→parquet→Tantivy conversion can lag, so FTS queries may fall back to
a non-Tantivy path during tests.
### Changes
1. **Content verification**
([conftest.py](tests/api-testing/tests/query_agent/conftest.py)): Every
hit from a `match_all` query that returns the `log` field is now checked
— each search term word must be present in `log`. A hard assertion fails
if Tantivy returns rows that don't contain the search terms.
2. **Tantivy readiness gate**
([test_2_parquet.py](tests/api-testing/tests/query_agent/test_2_parquet.py)):
After `node/flush`, the test now polls with a `match_all('warehouse')
LIMIT 1` query (up to 120s timeout) to confirm Tantivy indexes are built
before running FTS assertions. Previously only COUNT(*) was checked.
3. **Multi-word match_all emulation**
([compute_counts.py](tests/test-data/query-agent/compute_counts.py)):
`match_all('ACK batch')` now translates to `(log LIKE '%ACK%' AND log
LIKE '%batch%')` — per-token AND conditions matching Tantivy's
token-level search semantics, instead of the old phrase match `log LIKE
'%ACK batch%'`.
4. **Regenerated expected results** across all query JSON files.
## Test plan
- [ ] Run query agent test suite against an environment with Tantivy
enabled
- [ ] Verify `match_all` content assertions catch intentionally
corrupted results
- [ ] Confirm post-flush Tantivy readiness poll works within the timeout
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
## Summary
- Adds **239 new tests** covering the vortex columnar file format across
78 scenarios from the test plan
- **`tests/api-testing/tests/vortex/`** — 9 test modules (83 tests):
format conversion, schema evolution, data integrity, FTS, downsampling,
API format awareness, server config, and performance/stress
- **`tests/api-testing/tests/query_agent/test_3_vortex.py`** — 156
tests: the full 155-query SQL correctness suite + scenario 39
(self-join) run post-flush against vortex files
- **`api-testing.yml`** — adds `--ignore=tests/vortex` to the OSS
default-set run; vortex tests require the enterprise binary with
`ZO_FILE_FORMAT=vortex` and are gated to enterprise CI only
## Test plan coverage
| Module | Scenarios |
|--------|-----------|
| `test_format_conversion.py` | 1–13, 7, 8, 9, 10 |
| `test_schema_evolution.py` | 14–25 |
| `test_data_integrity.py` | 55–62 |
| `test_fts.py` | 51–54 |
| `test_downsampling.py` | 47–50 |
| `test_api_format.py` | 71–73 |
| `test_config.py` | 66–67 |
| `test_performance.py` | 74–78 |
| `test_3_vortex.py` | 26–46 (155 queries + JOIN) |
## Enterprise CI
Two dedicated jobs added in the companion `openobserve/o2-enterprise` PR
on the same branch:
- **`api_query_agent_vortex`** — runs `test_3_vortex.py` with
`ZO_FILE_FORMAT=vortex`
- **`api_vortex_format_tests`** — runs the full `tests/vortex/` suite
with 4-phase server lifecycle (parquet→vortex→parquet→vortex+compaction)
OSS CI is unchanged except for the ignore rule — no vortex tests run
against the OSS binary.
## Test checklist
- [ ] `tests/vortex/` collects cleanly: `83 tests collected`
- [ ] `tests/query_agent/test_3_vortex.py` collects cleanly: `156 tests
collected`
- [ ] Enterprise CI `api_query_agent_vortex` job passes
- [ ] Enterprise CI `api_vortex_format_tests` job passes
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- ENT CI runners are slower than local/OSS runners — the
`ingest_query_agent_data` fixture was hitting the 120s timeout before
1550 records became searchable, causing all 310 query_agent tests to
fail at setup
- Bumped `wait_until` timeout to 300s
## Test plan
- [ ] OSS `api_query_agent_tests` passes as before
- [ ] ENT `api_query_agent_tests` no longer hits WaitTimeout on ingest
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- [x] Bulk ingestion panic
- [x] Create function panic
- [x] Dashboards/Alerts/Incidents panic when list data
- [x] Match http response error code
- [x] Skip some endpoint that fuzz can't generate data
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Comprehensive rework of the `tests/api-testing/` suite. 30+ commits
implementing all 6 phases of the [revamp
plan](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md).
**Before → After:**
| | Baseline | Final | Δ |
|---|---|---|---|
| passed | 421 | **443** | **+22 ✓** |
| failed | 3 | **2** | -1 (only same pre-existing) |
| skipped | 53 | 39 | -14 (unskipped dead tests now run) |
| xfailed | 1 | 2 | +1 (new visible backend-bug signal) |
| runtime | 588s | **387s** | **-201s (34% faster)** |
**Zero tests that passed in baseline are now failing.** The 2 remaining
failures are the exact pre-existing ones
(`test_enrichment_table_join.py` backend panic + `test_workflow.py` SMTP
env-miss).
## What's in the diff
- **New `support/` framework** — `OpenObserveClient`, endpoint wrappers
(`alerts`, `dashboards`, `streams`, `search`, etc.), `wait_until()`,
payload factories, pytest fixtures, SSE parser. 14 framework selfcheck
tests.
- **16 of 16 OLD-tier test files rewritten** — using the new framework,
unique-suffix names, body validation, `wait_until` instead of
`time.sleep`, parametrize where appropriate.
- **29 files moved into 12 feature folders** mirroring the UI suite
layout
(`tests/{alerts,dashboards,ingest,orgs,pipelines,regression,rum,search,streams,users,workflow}/`).
- **−6,500 LOC net** (copy-paste + dead code removal; new framework adds
back ~1,500).
- **Ruff config** in pyproject.toml: T201 ban on `print()`, PT pytest
rules, B bugbear, UP pyupgrade.
## CI workflow changes
- New job `api_regression_tests`: `strategy.matrix` over the 4 env-flag
combinations (UTF8_VIEW × JOIN_MATCH_ONE) — now runs in parallel instead
of as 4 sequential server reboots in the main job.
- New job `api_fuzz_tests`: schemathesis OpenAPI-driven fuzz against
`/api-doc/openapi.json` (214 endpoints tested). Informational
(`continue-on-error: true`). Each run surfaces **7–10 distinct 5xx
panics** (e.g., empty `sql` body returning 500 instead of 400, traces
endpoints crashing on unescaped path params that get interpolated into
SQL). Filed to backend team for separate triage.
- Server `o2_*.log` files uploaded as artifacts on failure (was:
silently lost).
- `api_tests_summary` now requires BOTH integration + regression to
pass.
### CI infrastructure fixes
A short hardening pass landed after the initial CI run surfaced several
issues:
- **pytest-sugar count inflation** — dropped `--force-sugar` from both
pytest invocations. The flag double-registered sugar on top of the
auto-loaded entry point, inflating the CI test counter ~3x (display
showed 1364 when collection was 486). Counter is now honest.
- **schemathesis 4.x flag migration** — `--base-url` → `-u`,
`--hypothesis-max-examples` → `-n`, `--hypothesis-deadline` →
`--request-timeout`, `--data-generation-method` → `-m`, dropped
`--auth-type basic`. The fuzz job was failing immediately on flag-parse
before this.
- **Regression matrix binary path** — used `${{ github.workspace
}}/target/debug/openobserve` so the absolute path resolves correctly
under `working-directory: tests/api-testing/` (was: every matrix job
failed with "No such file or directory").
- **`permissions: contents: read`** added to all 4 workflow jobs —
closes CodeQL alert #369.
- **Removed redundant server boot** in `api_integration_tests` (leftover
from main's structure after the regression matrix split; was silently
failing to bind to :5080 and getting masked by the first server).
## Backend bugs surfaced (for backend team to track)
The rework uncovered and visibly flagged these as `xfail`/documented
test comments:
1. `DELETE /api/organizations/{id}` returns 404 unconditionally even for
just-created orgs — endpoint appears unimplemented.
2. `test_dashboard_missing_required_fields` triggers backend panic
(returns 500 instead of 400/422).
3. JS `getEnrichmentTableData()` restricted to `_meta` org AND not
exposed in the `/functions/test` endpoint runtime — `ReferenceError`
even with explicit `enrichment_tables: [...]`.
4. Bulk API silently sanitizes invalid `_index` values (`****` → `_`)
without flagging — user data lands in unexpected index.
5. `AVG` of a single value drifts by ~1e-14 (likely float reduction
order; documented tolerance in `test_advanced_sql.py`).
6. Org name validation rejects hyphens (only
`A-Z/a-z/0-9/spaces/underscores`) — not in spec.
Plus 7–10 additional 5xx panics surfaced per fuzz run (input validation
gaps — separate triage list in fuzz job logs).
## Test bugs uncovered + fixed
- `test_pipeline.py`: had two function definitions named
`test_pipeline_creation_and_action_with_schedule` — Python's last-wins
meant the first test's body never executed for years.
- `test_alertsV2.py`: `assert alert_disabled == "False"` was indented
inside an `else:` branch that never executed because the response is
always a dict; AND `enabled` is a bool, not the string `"False"`. Two
layers of broken.
- `test_bulk.py`: variable typo (`valid_bulk_payload` reused on the POST
call) made the invalid-index test silently duplicate the valid case.
- `test_organisations.py`: `test_create_organization` used pre-pytest-4
`yield`-style fixtures — silently ignored by pytest 8.
- `test_js_result_array.py`: 8 tests skipped with `"Temporarily disabled
- failing test"` — the actual cause was that JS functions are restricted
to `_meta` org. One-line fix unskipped all 8.
- `test_fuzzy.py`: 4 tests `@pytest.mark.skip`-ed because they used a
1-minute time window against older static test data → 0 hits → flake →
skip. Wider window unskips them.
- 547 LOC of zombie commented-out test blocks deleted (15 commented `#
def test_` definitions, including one orphan case where the `def` was
commented but the indented body wasn't).
## Folder layout
```
tests/api-testing/
├── support/ # NEW framework
│ ├── client.py # OpenObserveClient
│ ├── endpoints/ # 7 per-resource wrappers
│ ├── factories.py # unique_name, time_window, *_payload
│ ├── wait.py # wait_until + WaitTimeout
│ ├── sse.py # read_sse_response
│ └── fixtures.py # client, temp_user_email, temp_stream_name, ...
├── helpers/ # Non-test shared code (was scattered in tests/)
│ ├── sourcemap_helpers.py
│ └── workflow/ # workflow chain + pages/ live here now
├── fixtures/
│ └── rum/ # Standalone RUM ingest scripts (was in tests/)
└── tests/
├── alerts/ # 3 files
├── dashboards/ # 4 files
├── ingest/ # 4 files
├── orgs/ # 2 files
├── pipelines/ # 10 files
├── regression/ # 2 files (env-flag matrix targets)
├── rum/ # 3 files
├── search/ # 8 files
├── streams/ # 1 file
├── users/ # 1 file
├── workflow/ # 3 files
├── conftest.py
└── test_framework_selfcheck.py
```
## Test plan
- [x] `rye run pytest --collect-only` — 486 tests discovered
- [x] Full default-set run (`--ignore=tests/regression`) — 443 pass / 2
fail / 39 skip / 2 xfail in 387s
- [x] `rye run pytest -m framework` — 14/14 selfcheck tests pass
- [x] `rye run ruff check support/ tests/test_framework_selfcheck.py` —
clean
- [x] CI run on this PR — `api_integration_tests` + all 4
`api_regression_tests` matrix jobs green
- [x] CI run — `api_fuzz_tests` runs cleanly (informational; surfaces
7–10 unique 5xx panics per run)
## Note on test count
The `test_streaming.py` rewrite folded 7 "tests" (`test_ingest_data`,
`test_ingest_join`, `test_disable_streaming`,
`test_update_max_query_range`, `test_enable_streaming`,
`test_delete_stream`, `test_delete_stream_join`) into pytest fixtures.
Each only checked `status_code == 200` of a setup/teardown operation —
they were imperative steps masquerading as tests. The fixtures still
perform the same operations and assert their success; they're just no
longer counted as discrete pytest items. **Functional coverage is
identical.**
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add merge_group event trigger to all required CI workflows so they run
when PRs enter the merge queue. Fix github.event.pull_request references
in dispatch-event-enterprise.yml to handle merge_group context.
Add comprehensive API tests for logs-to-logs joins with asymmetric data
sizes:
- 29 test cases covering INNER/LEFT/RIGHT/FULL joins, aggregations,
filters
- Subquery tests with IN/NOT IN, substr, length, CONCAT functions
- Tests run with both ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true and false
- Test data: logs_small.json (10 records) and logs_large.json (50
records)
---------
Co-authored-by: Claude <noreply@anthropic.com>
Fixes#5131
Addresses #5408
### Revamp
- Created a new table to store organizations in the `organizations`
table. Earlier, we never stored organizations in any table, rather we
used to store the organizations in the meta table as part of the `user`
json.
- The users and organization membership relations are now moved to its
own separate tables - `users` and `org_users`.
### New changes
**API changes** -
- `/organizations` - `POST` api to create an org and save it in the meta
table. In non-enterprise version, currently only the root user can
create the org. For enterprise version with rbac enabled, anyone with
`CREATE` access for organizations can create the organization and
becomes an admin. Root user can always create an organization.
- `/{org_id}/rename` - `PUT` api to rename the existing org. Only root
user and the admins of the org can access this API.
The payload for following APIs are changed -
- `POST api/{org_id}/roles` - `{"name":"new_role"}` is changed to
`{"role":"new_role"}
- `POST api/{org_id}/users/{user}` -
`{"role":"viewer","custom_role":["role_name"]}` is the new payload where
`custom_role` is optional and is array of custom roles (only works in
enterprise with rbac enabled) which will be assigned to the user. The
value of `role` can be standard role or as well as any custom role (when
it is custom role, the standard role assigned to the user is decided
based on the enterprise env variable - `O2_DEX_DEFAULT_ROLE`).
- `POST api/{org_id}/users`- the payload -
```
{"email":"user@user.com","role":"viewer","custom_role":["create_org_permission"],"password":"12345678", "first_name": "", "last_name":""}
```
- `PUT api//{org_id}/users/{email_id}` the payload -
```
{"first_name":"Hello","last_name":"j","role":"viewer","isExternal":false,"change_password":false,"password":"","custom_role":["create_org_permission"]}
```
A new ENV variable is introduced - `ZO_CREATE_ORG_THROUGH_INGESTION`
(default `false`) which if true, an org will automatically created when
ingesting data to a new org. If false, user can not create org by
ingesting data to new org, they need to first call the `POST
/organizations` API.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
**New Features**
- Added a `cloud` feature for enhanced organizational management
capabilities.
- Introduced new methods for managing organizational users and
asynchronous message processing.
- Enhanced user invitation and membership features, including improved
role handling.
- Implemented configurable ingestion URL support across various
components.
- Added new functionality for user management and organization handling
in the super cluster architecture.
**Improvements**
- Refined user authentication and authorization processes, including
error handling.
- Updated database schema to support multi-organization architecture.
- Enhanced pagination settings and UI components for better user
experience.
- Improved handling of organization names and user roles in
notifications and alerts.
- Streamlined service account token management with visibility
enhancements.
- Enhanced the `AddUpdateOrganization` component with improved user
input handling and validation.
- Updated the `AppOrganizations` component to dynamically load the
appropriate organization management interface.
- Improved the `ServiceAccountsList` component with updated pagination
options and token handling logic.
**Bug Fixes**
- Resolved issues with user role assignments and organization management
inconsistencies.
- Corrected routing and navigation logic for login and logout processes.
- Addressed various error handling scenarios in user and organization
processes.
- Fixed issues with the visibility of service account options in the
`EditGroup` and `EditRole` components based on the cloud configuration.
**Breaking Changes**
- Significant restructuring of user and organization data models.
- Changes to authentication workflows and configuration handling.
- Removal of certain components related to user and organization
management.
- Updated the logic for determining the ingestion URL across various
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
Co-authored-by: harshiv-7 <harshiv@openobserve.ai>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: Zhao.wei <weizhao029@gmail.com>
Co-authored-by: ShyamOOAI <shyam@openobserve.ai>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
- Simplify the API integration tests by removing steps not needed (Node,
k8s, building docker, etc).
- Testing the UI, linting, formatting, etc should be done in a separate
workflow file
- Simplify Rust cache workflow.
- By default it caches /target
- Removed Dockerfile and k8s deployment from CI.
- You can just run `OpenObserve` in the background
- Replace `ubicloud-standard-16` with `ubuntu-latest` which is *FREE*.
- Using 16 cores, 64GB of ram is an overkill for running tests.
- Don't run the CI if changes to `.md` files are detected.
Notes:
- Before caching building `OpenObserve` takes around 5-7 mins, after
cache it goes down to 1-2 mins.
Neha PandClaude Opus 4.7
Hengfei YangandClaude Opus 4.8
Shrinath RaoandClaude Opus 4.7
Ashish Kolhe
udev
Yashodhan JoshiandYashodhan Joshi
Subhra264
Simha G
Prabhat SharmaandmmosarafO2
dependabot[bot]
sktrpathi
Ankur Srivastava
Juan Calderon-Perez
