starred/openobserve
1
0
Fork 0
mirror of https://github.com/openobserve/openobserve.git synced 2026-11-04 16:09:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,925 Commits 635 Branches 244 Tags
4e805038fee978c8e86ea29965f14c467f8f1dc3
Commit Graph
3 Commits
Author SHA1 Message Date
ktx-akshay 33c6e9e0e1 test(dashboards): cover CTE/subquery match_all queries through the panel search path (#13838) 
## What

First test coverage for the dashboard **panel** endpoints and the
**dashboard
search path**, running CTE/subquery `match_all` queries as real
dashboard panels.

Nothing in the repo previously touched either:
`grep -rn "/panels\|search_type=dashboards" tests/api-testing/` returned
zero hits.

## Why

Panels query `_search?type=logs&search_type=dashboards&dashboard_id=…` —
a
different route into the engine than the Logs page (`search_type=ui`).
The
`match_all` + FST projection rewrite fixed in #13808 runs inside it, so
a query
that passes via Logs can still regress on a dashboard.

The rewrite has to preserve the projection of the block the predicate
sits in.
When that block is the whole statement, the projection is obvious. When
it's a
CTE body feeding a JOIN, a window, or a four-branch UNION ALL, it has to
preserve
the *CTE's* output schema while the outer query reads different columns
from it —
which is where it went wrong.

## Coverage

**71 distinct queries across 29 combination families**, all with
`match_all`
inside a `WITH …` body or a derived table:

| Axis | Variants |
|---|---|
| Nesting depth | 1, 2, 3, and 5-level CTE chains |
| Nesting kind | CTEs and derived tables (and one query using both) |
| UNION ALL | 2, 3, and 4 separate `match_all` branches |
| JOIN | CTE⋈CTE, derived⋈derived, cross-stream |
| Window | `ROW_NUMBER`, `RANK`, `DENSE_RANK`, partitioned aggregates |
| Negation | `NOT match_all`, and mixed with positive `match_all` |
| Combined with | `COUNT(DISTINCT)`, `CASE`, `HAVING`, `histogram()` |

Each case creates a real panel, runs it via the dashboard search path,
then
asserts row count against the DuckDB oracle and that no expected column
was
dropped from the projection. A further 30 cases run the same SQL through
**both**
Logs and dashboard and diff the results — that's what catches a rewrite
firing on
only one path.

Queries are reused from the existing query-agent corpus rather than
duplicated,
so the two suites can't drift: query-agent verifies the values via the
Logs path,
this verifies the same SQL survives the dashboard path.

Cases are **discovered by classifying the SQL**, not from a hand-written
ID list,
so new combinations added to the corpus are picked up automatically.
Test ids
name the shape (`Q1022-cte1_union_cdist_4ma`), so a failure identifies
the
combination without opening the corpus.

Plus 14 tests for the panel endpoints themselves — AddPanel /
UpdatePanel /
DeletePanel, contract + CRUD lifecycle + 7 error paths derived from
`DashboardError` in `src/core/src/http.rs`.

## Changes

| File | |
|---|---|
| `tests/dashboards/test_dashboard_cte_panels.py` | new — 103 tests |
| `tests/dashboards/test_dashboard_panels.py` | new — 14 tests |
| `tests/dashboards/conftest.py` | new — shared fixtures |
| `support/panel_queries.py` | new — discovery + panel helpers |
| `support/endpoints/dashboards.py` | +78, additive |
| `support/endpoints/search.py` | +36, additive |
| `support/factories.py` | +73, additive |

The three shared-file edits are **187 added lines, 0 modified** —
nothing
existing can break.

## Results

168 tests in tests/api-testing/tests/dashboards/ (117 new + 51
pre-existing)
166 passed, 1 skipped, 1 xfailed     # full nested-query corpus present
74 passed, 2 skipped, 1 xfailed     # current corpus on main


Repeated 2× and 4× respectively, no flakes.

Discovery scales with the corpus: on main it finds 8 nested `match_all`
cases,
and 71 once the larger sentinel set is present. Cases that aren't in the
corpus
skip with an explanatory message instead of failing, so this is safe to
merge in
either order.
 2026-08-17 09:47:17 +00:00
Shrinath Rao a647e6aee9 test: add retry_on_db_lock helper to support/wait.py (#12434) 
## Summary

- Adds `retry_on_db_lock()` to `tests/api-testing/support/wait.py`
- Companion to the enterprise-side fix (openobserve/o2-enterprise#TBD)
that applies the helper across all RBAC entity base tests

## Root cause

The `ent_rbac` CI shard runs with `-n 2` (two pytest-xdist workers).
SQLite's single-writer model means parallel workers racing on write
operations produce transient `500 {"message": "database is locked"}`
responses (SQLite error codes 5 / 517). These surface as flaky assertion
failures across many PRs.

## Fix

`retry_on_db_lock(fn)` wraps an HTTP call and retries up to 3 times with
exponential backoff (0.5 → 1 → 2 s) only when the response is `500` with
`"database is locked"` in the body. Parallelism is preserved; the brief
retry window absorbs the time the other writer needs to commit and
release the lock.

## Test plan

- [ ] OSS CI passes with this branch
- [ ] Enterprise PR with the same branch name picks up the new utility
and applies it in all RBAC entity base tests

🤖 Generated with [Claude Code](https://claude.com/claude-code)
 2026-06-03 12:23:48 +00:00
Shrinath RaoandClaude Opus 4.7 61aa772d57 test: revamp api-testing suite (framework + 16 file rewrites + CI matrix) (#12372) 
## Summary

Comprehensive rework of the `tests/api-testing/` suite. 30+ commits
implementing all 6 phases of the [revamp
plan](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md).

**Before → After:**

| | Baseline | Final | Δ |
|---|---|---|---|
| passed | 421 | **443** | **+22 ✓** |
| failed | 3 | **2** | -1 (only same pre-existing) |
| skipped | 53 | 39 | -14 (unskipped dead tests now run) |
| xfailed | 1 | 2 | +1 (new visible backend-bug signal) |
| runtime | 588s | **387s** | **-201s (34% faster)** |

**Zero tests that passed in baseline are now failing.** The 2 remaining
failures are the exact pre-existing ones
(`test_enrichment_table_join.py` backend panic + `test_workflow.py` SMTP
env-miss).

## What's in the diff

- **New `support/` framework** — `OpenObserveClient`, endpoint wrappers
(`alerts`, `dashboards`, `streams`, `search`, etc.), `wait_until()`,
payload factories, pytest fixtures, SSE parser. 14 framework selfcheck
tests.
- **16 of 16 OLD-tier test files rewritten** — using the new framework,
unique-suffix names, body validation, `wait_until` instead of
`time.sleep`, parametrize where appropriate.
- **29 files moved into 12 feature folders** mirroring the UI suite
layout
(`tests/{alerts,dashboards,ingest,orgs,pipelines,regression,rum,search,streams,users,workflow}/`).
- **−6,500 LOC net** (copy-paste + dead code removal; new framework adds
back ~1,500).
- **Ruff config** in pyproject.toml: T201 ban on `print()`, PT pytest
rules, B bugbear, UP pyupgrade.

## CI workflow changes

- New job `api_regression_tests`: `strategy.matrix` over the 4 env-flag
combinations (UTF8_VIEW × JOIN_MATCH_ONE) — now runs in parallel instead
of as 4 sequential server reboots in the main job.
- New job `api_fuzz_tests`: schemathesis OpenAPI-driven fuzz against
`/api-doc/openapi.json` (214 endpoints tested). Informational
(`continue-on-error: true`). Each run surfaces **7–10 distinct 5xx
panics** (e.g., empty `sql` body returning 500 instead of 400, traces
endpoints crashing on unescaped path params that get interpolated into
SQL). Filed to backend team for separate triage.
- Server `o2_*.log` files uploaded as artifacts on failure (was:
silently lost).
- `api_tests_summary` now requires BOTH integration + regression to
pass.

### CI infrastructure fixes

A short hardening pass landed after the initial CI run surfaced several
issues:

- **pytest-sugar count inflation** — dropped `--force-sugar` from both
pytest invocations. The flag double-registered sugar on top of the
auto-loaded entry point, inflating the CI test counter ~3x (display
showed 1364 when collection was 486). Counter is now honest.
- **schemathesis 4.x flag migration** — `--base-url` → `-u`,
`--hypothesis-max-examples` → `-n`, `--hypothesis-deadline` →
`--request-timeout`, `--data-generation-method` → `-m`, dropped
`--auth-type basic`. The fuzz job was failing immediately on flag-parse
before this.
- **Regression matrix binary path** — used `${{ github.workspace
}}/target/debug/openobserve` so the absolute path resolves correctly
under `working-directory: tests/api-testing/` (was: every matrix job
failed with "No such file or directory").
- **`permissions: contents: read`** added to all 4 workflow jobs —
closes CodeQL alert #369.
- **Removed redundant server boot** in `api_integration_tests` (leftover
from main's structure after the regression matrix split; was silently
failing to bind to :5080 and getting masked by the first server).

## Backend bugs surfaced (for backend team to track)

The rework uncovered and visibly flagged these as `xfail`/documented
test comments:

1. `DELETE /api/organizations/{id}` returns 404 unconditionally even for
just-created orgs — endpoint appears unimplemented.
2. `test_dashboard_missing_required_fields` triggers backend panic
(returns 500 instead of 400/422).
3. JS `getEnrichmentTableData()` restricted to `_meta` org AND not
exposed in the `/functions/test` endpoint runtime — `ReferenceError`
even with explicit `enrichment_tables: [...]`.
4. Bulk API silently sanitizes invalid `_index` values (`****` → `_`)
without flagging — user data lands in unexpected index.
5. `AVG` of a single value drifts by ~1e-14 (likely float reduction
order; documented tolerance in `test_advanced_sql.py`).
6. Org name validation rejects hyphens (only
`A-Z/a-z/0-9/spaces/underscores`) — not in spec.

Plus 7–10 additional 5xx panics surfaced per fuzz run (input validation
gaps — separate triage list in fuzz job logs).

## Test bugs uncovered + fixed

- `test_pipeline.py`: had two function definitions named
`test_pipeline_creation_and_action_with_schedule` — Python's last-wins
meant the first test's body never executed for years.
- `test_alertsV2.py`: `assert alert_disabled == "False"` was indented
inside an `else:` branch that never executed because the response is
always a dict; AND `enabled` is a bool, not the string `"False"`. Two
layers of broken.
- `test_bulk.py`: variable typo (`valid_bulk_payload` reused on the POST
call) made the invalid-index test silently duplicate the valid case.
- `test_organisations.py`: `test_create_organization` used pre-pytest-4
`yield`-style fixtures — silently ignored by pytest 8.
- `test_js_result_array.py`: 8 tests skipped with `"Temporarily disabled
- failing test"` — the actual cause was that JS functions are restricted
to `_meta` org. One-line fix unskipped all 8.
- `test_fuzzy.py`: 4 tests `@pytest.mark.skip`-ed because they used a
1-minute time window against older static test data → 0 hits → flake →
skip. Wider window unskips them.
- 547 LOC of zombie commented-out test blocks deleted (15 commented `#
def test_` definitions, including one orphan case where the `def` was
commented but the indented body wasn't).

## Folder layout

```
tests/api-testing/
├── support/                    # NEW framework
│   ├── client.py               # OpenObserveClient
│   ├── endpoints/              # 7 per-resource wrappers
│   ├── factories.py            # unique_name, time_window, *_payload
│   ├── wait.py                 # wait_until + WaitTimeout
│   ├── sse.py                  # read_sse_response
│   └── fixtures.py             # client, temp_user_email, temp_stream_name, ...
├── helpers/                    # Non-test shared code (was scattered in tests/)
│   ├── sourcemap_helpers.py
│   └── workflow/               # workflow chain + pages/ live here now
├── fixtures/
│   └── rum/                    # Standalone RUM ingest scripts (was in tests/)
└── tests/
    ├── alerts/                 # 3 files
    ├── dashboards/             # 4 files
    ├── ingest/                 # 4 files
    ├── orgs/                   # 2 files
    ├── pipelines/              # 10 files
    ├── regression/             # 2 files (env-flag matrix targets)
    ├── rum/                    # 3 files
    ├── search/                 # 8 files
    ├── streams/                # 1 file
    ├── users/                  # 1 file
    ├── workflow/               # 3 files
    ├── conftest.py
    └── test_framework_selfcheck.py
```

## Test plan

- [x] `rye run pytest --collect-only` — 486 tests discovered
- [x] Full default-set run (`--ignore=tests/regression`) — 443 pass / 2
fail / 39 skip / 2 xfail in 387s
- [x] `rye run pytest -m framework` — 14/14 selfcheck tests pass
- [x] `rye run ruff check support/ tests/test_framework_selfcheck.py` —
clean
- [x] CI run on this PR — `api_integration_tests` + all 4
`api_regression_tests` matrix jobs green
- [x] CI run — `api_fuzz_tests` runs cleanly (informational; surfaces
7–10 unique 5xx panics per run)

## Note on test count

The `test_streaming.py` rewrite folded 7 "tests" (`test_ingest_data`,
`test_ingest_join`, `test_disable_streaming`,
`test_update_max_query_range`, `test_enable_streaming`,
`test_delete_stream`, `test_delete_stream_join`) into pytest fixtures.
Each only checked `status_code == 200` of a setup/teardown operation —
they were imperative steps masquerading as tests. The fixtures still
perform the same operations and assert their success; they're just no
longer counted as discrete pytest items. **Functional coverage is
identical.**

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-01 07:19:48 +00:00