test: revamp api-testing suite (framework + 16 file rewrites + CI matrix) (#12372)

## Summary

Comprehensive rework of the `tests/api-testing/` suite. 30+ commits
implementing all 6 phases of the [revamp
plan](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md).

**Before → After:**

| | Baseline | Final | Δ |
|---|---|---|---|
| passed | 421 | **443** | **+22 ✓** |
| failed | 3 | **2** | -1 (only same pre-existing) |
| skipped | 53 | 39 | -14 (unskipped dead tests now run) |
| xfailed | 1 | 2 | +1 (new visible backend-bug signal) |
| runtime | 588s | **387s** | **-201s (34% faster)** |

**Zero tests that passed in baseline are now failing.** The 2 remaining
failures are the exact pre-existing ones
(`test_enrichment_table_join.py` backend panic + `test_workflow.py` SMTP
env-miss).

## What's in the diff

- **New `support/` framework** — `OpenObserveClient`, endpoint wrappers
(`alerts`, `dashboards`, `streams`, `search`, etc.), `wait_until()`,
payload factories, pytest fixtures, SSE parser. 14 framework selfcheck
tests.
- **16 of 16 OLD-tier test files rewritten** — using the new framework,
unique-suffix names, body validation, `wait_until` instead of
`time.sleep`, parametrize where appropriate.
- **29 files moved into 12 feature folders** mirroring the UI suite
layout
(`tests/{alerts,dashboards,ingest,orgs,pipelines,regression,rum,search,streams,users,workflow}/`).
- **−6,500 LOC net** (copy-paste + dead code removal; new framework adds
back ~1,500).
- **Ruff config** in pyproject.toml: T201 ban on `print()`, PT pytest
rules, B bugbear, UP pyupgrade.

## CI workflow changes

- New job `api_regression_tests`: `strategy.matrix` over the 4 env-flag
combinations (UTF8_VIEW × JOIN_MATCH_ONE) — now runs in parallel instead
of as 4 sequential server reboots in the main job.
- New job `api_fuzz_tests`: schemathesis OpenAPI-driven fuzz against
`/api-doc/openapi.json` (214 endpoints tested). Informational
(`continue-on-error: true`). Each run surfaces **7–10 distinct 5xx
panics** (e.g., empty `sql` body returning 500 instead of 400, traces
endpoints crashing on unescaped path params that get interpolated into
SQL). Filed to backend team for separate triage.
- Server `o2_*.log` files uploaded as artifacts on failure (was:
silently lost).
- `api_tests_summary` now requires BOTH integration + regression to
pass.

### CI infrastructure fixes

A short hardening pass landed after the initial CI run surfaced several
issues:

- **pytest-sugar count inflation** — dropped `--force-sugar` from both
pytest invocations. The flag double-registered sugar on top of the
auto-loaded entry point, inflating the CI test counter ~3x (display
showed 1364 when collection was 486). Counter is now honest.
- **schemathesis 4.x flag migration** — `--base-url` → `-u`,
`--hypothesis-max-examples` → `-n`, `--hypothesis-deadline` →
`--request-timeout`, `--data-generation-method` → `-m`, dropped
`--auth-type basic`. The fuzz job was failing immediately on flag-parse
before this.
- **Regression matrix binary path** — used `${{ github.workspace
}}/target/debug/openobserve` so the absolute path resolves correctly
under `working-directory: tests/api-testing/` (was: every matrix job
failed with "No such file or directory").
- **`permissions: contents: read`** added to all 4 workflow jobs —
closes CodeQL alert #369.
- **Removed redundant server boot** in `api_integration_tests` (leftover
from main's structure after the regression matrix split; was silently
failing to bind to :5080 and getting masked by the first server).

## Backend bugs surfaced (for backend team to track)

The rework uncovered and visibly flagged these as `xfail`/documented
test comments:

1. `DELETE /api/organizations/{id}` returns 404 unconditionally even for
just-created orgs — endpoint appears unimplemented.
2. `test_dashboard_missing_required_fields` triggers backend panic
(returns 500 instead of 400/422).
3. JS `getEnrichmentTableData()` restricted to `_meta` org AND not
exposed in the `/functions/test` endpoint runtime — `ReferenceError`
even with explicit `enrichment_tables: [...]`.
4. Bulk API silently sanitizes invalid `_index` values (`****` → `_`)
without flagging — user data lands in unexpected index.
5. `AVG` of a single value drifts by ~1e-14 (likely float reduction
order; documented tolerance in `test_advanced_sql.py`).
6. Org name validation rejects hyphens (only
`A-Z/a-z/0-9/spaces/underscores`) — not in spec.

Plus 7–10 additional 5xx panics surfaced per fuzz run (input validation
gaps — separate triage list in fuzz job logs).

## Test bugs uncovered + fixed

- `test_pipeline.py`: had two function definitions named
`test_pipeline_creation_and_action_with_schedule` — Python's last-wins
meant the first test's body never executed for years.
- `test_alertsV2.py`: `assert alert_disabled == "False"` was indented
inside an `else:` branch that never executed because the response is
always a dict; AND `enabled` is a bool, not the string `"False"`. Two
layers of broken.
- `test_bulk.py`: variable typo (`valid_bulk_payload` reused on the POST
call) made the invalid-index test silently duplicate the valid case.
- `test_organisations.py`: `test_create_organization` used pre-pytest-4
`yield`-style fixtures — silently ignored by pytest 8.
- `test_js_result_array.py`: 8 tests skipped with `"Temporarily disabled
- failing test"` — the actual cause was that JS functions are restricted
to `_meta` org. One-line fix unskipped all 8.
- `test_fuzzy.py`: 4 tests `@pytest.mark.skip`-ed because they used a
1-minute time window against older static test data → 0 hits → flake →
skip. Wider window unskips them.
- 547 LOC of zombie commented-out test blocks deleted (15 commented `#
def test_` definitions, including one orphan case where the `def` was
commented but the indented body wasn't).

## Folder layout

```
tests/api-testing/
├── support/                    # NEW framework
│   ├── client.py               # OpenObserveClient
│   ├── endpoints/              # 7 per-resource wrappers
│   ├── factories.py            # unique_name, time_window, *_payload
│   ├── wait.py                 # wait_until + WaitTimeout
│   ├── sse.py                  # read_sse_response
│   └── fixtures.py             # client, temp_user_email, temp_stream_name, ...
├── helpers/                    # Non-test shared code (was scattered in tests/)
│   ├── sourcemap_helpers.py
│   └── workflow/               # workflow chain + pages/ live here now
├── fixtures/
│   └── rum/                    # Standalone RUM ingest scripts (was in tests/)
└── tests/
    ├── alerts/                 # 3 files
    ├── dashboards/             # 4 files
    ├── ingest/                 # 4 files
    ├── orgs/                   # 2 files
    ├── pipelines/              # 10 files
    ├── regression/             # 2 files (env-flag matrix targets)
    ├── rum/                    # 3 files
    ├── search/                 # 8 files
    ├── streams/                # 1 file
    ├── users/                  # 1 file
    ├── workflow/               # 3 files
    ├── conftest.py
    └── test_framework_selfcheck.py
```

## Test plan

- [x] `rye run pytest --collect-only` — 486 tests discovered
- [x] Full default-set run (`--ignore=tests/regression`) — 443 pass / 2
fail / 39 skip / 2 xfail in 387s
- [x] `rye run pytest -m framework` — 14/14 selfcheck tests pass
- [x] `rye run ruff check support/ tests/test_framework_selfcheck.py` —
clean
- [x] CI run on this PR — `api_integration_tests` + all 4
`api_regression_tests` matrix jobs green
- [x] CI run — `api_fuzz_tests` runs cleanly (informational; surfaces
7–10 unique 5xx panics per run)

## Note on test count

The `test_streaming.py` rewrite folded 7 "tests" (`test_ingest_data`,
`test_ingest_join`, `test_disable_streaming`,
`test_update_max_query_range`, `test_enable_streaming`,
`test_delete_stream`, `test_delete_stream_join`) into pytest fixtures.
Each only checked `status_code == 200` of a setup/teardown operation —
they were imperative steps masquerading as tests. The fixtures still
perform the same operations and assert their success; they're just no
longer counted as discrete pytest items. **Functional coverage is
identical.**

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.github/workflows/api-testing.yml

@@ -57,6 +57,8 @@ jobs:
     needs: [check_changes]
     if: needs.check_changes.outputs.has_changes == 'true'
     runs-on: ubicloud-standard-4
+    permissions:
+      contents: read  # only need to checkout the repo
     steps:
       - name: Remove unused tools
         run: |
@@ -101,20 +103,112 @@ jobs:
           working-directory: "tests/api-testing/"
           version: "0.27.0"
 
-      - name: Start OpenObserve
-        run: target/debug/openobserve &
+      - name: Pin cpython
+        run: rye pin cpython@3.11.6
+        working-directory: tests/api-testing/
 
-      - name: Wait for start
+      - name: Rye sync
+        run: rye sync
+        working-directory: tests/api-testing/
+
+      - name: Run default-set tests
+        # Phase 6.b: regression matrix tests moved to their own
+        # api_regression_tests job below — they now run in parallel
+        # with this default-set job instead of sequentially within it.
+        # Cuts ~8 min off CI wall time at the cost of more runner minutes.
+        # NB: --force-sugar is intentionally omitted. pytest-sugar is already
+        # auto-loaded via the entry point; passing --force-sugar registers it
+        # a second time, which inflates the progress counter (CI displayed
+        # ~3x the real test count). Letting the plugin load via discovery
+        # alone keeps the counter honest.
         run: |
+          ${{ github.workspace }}/target/debug/openobserve > o2_default.log 2>&1 &
+          SERVER_PID_DEFAULT=$!
+          echo "OpenObserve PID for default tests: $SERVER_PID_DEFAULT"
           for i in $(seq 1 60); do
             if curl -sf http://localhost:5080/healthz > /dev/null 2>&1; then
-              echo "Server ready after ${i}s"
-              exit 0
+              echo "Server ready after ${i}s"; break
             fi
             sleep 1
           done
-          echo "Server failed to start in 60s"
-          exit 1
+          rye run pytest --ignore=tests/regression
+          kill $SERVER_PID_DEFAULT || true
+        working-directory: tests/api-testing/
+
+      - name: Upload OpenObserve server logs on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: openobserve-server-logs
+          path: tests/api-testing/o2_*.log
+          if-no-files-found: warn
+          retention-days: 14
+
+  # Phase 6.b: regression matrix split. The 4 env-flag-sensitive test
+  # configurations now run as 4 parallel jobs instead of 4 sequential
+  # restarts in one job. Each job builds OO once and runs its single
+  # configured pytest target.
+  api_regression_tests:
+    name: "api_regression / ${{ matrix.config.label }}"
+    needs: [check_changes]
+    if: needs.check_changes.outputs.has_changes == 'true'
+    runs-on: ubicloud-standard-4
+    permissions:
+      contents: read  # only need to checkout the repo
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - label: utf8_view_false
+            env: "ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true ZO_UTF8_VIEW_ENABLED=false"
+            target: tests/regression/test_subquery_join_match.py
+          - label: utf8_view_true
+            env: "ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true ZO_UTF8_VIEW_ENABLED=true"
+            target: tests/regression/test_subquery_join_match.py
+          - label: join_match_one_true
+            env: "ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true"
+            target: tests/regression/test_logs_to_logs_join.py
+          - label: join_match_one_false
+            env: "ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=false"
+            target: tests/regression/test_logs_to_logs_join.py
+    steps:
+      - name: Remove unused tools
+        run: |
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /opt/ghc
+          sudo rm -rf "/usr/local/share/boost"
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY"
+
+      - uses: actions/checkout@v5
+        with: { fetch-depth: 0 }
+
+      - name: Setup Rust Toolchain
+        uses: dtolnay/rust-toolchain@master
+        with:
+          toolchain: nightly-2025-11-11
+
+      - name: Setup Rust Cache
+        uses: Swatinem/rust-cache@v2
+        with:
+          cache-on-failure: true
+          shared-key: backend-debug-mimalloc
+          save-if: false  # main api_integration_tests job writes the cache
+
+      - name: Install Protoc
+        run: bash ./.github/protoc.sh
+
+      - name: Build OpenObserve debug binary
+        run: cargo build --features mimalloc
+
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.11.6" }
+
+      - name: Setup rye
+        uses: eifinger/setup-rye@v4
+        with:
+          enable-cache: true
+          working-directory: "tests/api-testing/"
+          version: "0.27.0"
 
       - name: Pin cpython
         run: rye pin cpython@3.11.6
@@ -124,89 +218,145 @@ jobs:
         run: rye sync
         working-directory: tests/api-testing/
 
-      - name: Run tests
+      - name: "Run ${{ matrix.config.label }} regression test"
+        # Server binary lives at the repo root (target/debug/openobserve);
+        # this step uses working-directory: tests/api-testing/, so we must
+        # invoke it via ${{ github.workspace }} or the shell will fail with
+        # "No such file or directory". --force-sugar omitted on purpose —
+        # pytest-sugar auto-loads via entry point, and passing the flag
+        # registers it twice and inflates the progress counter ~3x.
         run: |
-          wait_for_server() {
-            for i in $(seq 1 60); do
-              if curl -sf http://localhost:5080/healthz > /dev/null 2>&1; then
-                echo "Server ready after ${i}s"
-                return 0
-              fi
-              sleep 1
-            done
-            echo "Server failed to start in 60s"
-            return 1
-          }
-
-          echo "=== Running regression tests with ZO_UTF8_VIEW_ENABLED=false ==="
-          ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true ZO_UTF8_VIEW_ENABLED=false target/debug/openobserve > o2_utf8_false.log 2>&1 &
-          SERVER_PID_FALSE=$!
-          echo "OpenObserve PID for UTF8_VIEW=false: $SERVER_PID_FALSE"
-          wait_for_server
-
-          # Run regression tests with UTF8_VIEW=false
-          rye run pytest tests/test_subquery_join_match.py --force-sugar -v
-
-          # Kill the server
-          kill $SERVER_PID_FALSE || true
-          sleep 5
-
-          echo "=== Running regression tests with ZO_UTF8_VIEW_ENABLED=true ==="
-          ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true ZO_UTF8_VIEW_ENABLED=true target/debug/openobserve > o2_utf8_true.log 2>&1 &
-          SERVER_PID_TRUE=$!
-          echo "OpenObserve PID for UTF8_VIEW=true: $SERVER_PID_TRUE"
-          wait_for_server
-
-          # Run regression tests with UTF8_VIEW=true
-          rye run pytest tests/test_subquery_join_match.py --force-sugar -v
-
-          # Kill the server
-          kill $SERVER_PID_TRUE || true
-          sleep 5
-
-          echo "=== Running logs-to-logs join tests with ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true ==="
-          ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true target/debug/openobserve > o2_join_match_one_true.log 2>&1 &
-          SERVER_PID_JOIN_TRUE=$!
-          echo "OpenObserve PID for JOIN_MATCH_ONE=true: $SERVER_PID_JOIN_TRUE"
-          wait_for_server
-
-          # Run logs-to-logs join tests with JOIN_MATCH_ONE=true
-          rye run pytest tests/test_logs_to_logs_join.py --force-sugar -v
-
-          # Kill the server
-          kill $SERVER_PID_JOIN_TRUE || true
-          sleep 5
-
-          echo "=== Running logs-to-logs join tests with ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=false ==="
-          ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=false target/debug/openobserve > o2_join_match_one_false.log 2>&1 &
-          SERVER_PID_JOIN_FALSE=$!
-          echo "OpenObserve PID for JOIN_MATCH_ONE=false: $SERVER_PID_JOIN_FALSE"
-          wait_for_server
-
-          # Run logs-to-logs join tests with JOIN_MATCH_ONE=false
-          rye run pytest tests/test_logs_to_logs_join.py --force-sugar -v
-
-          # Kill the server
-          kill $SERVER_PID_JOIN_FALSE || true
-          sleep 5
-
-          echo "=== Running all other API tests with default settings ==="
-          target/debug/openobserve > o2_default.log 2>&1 &
-          SERVER_PID_DEFAULT=$!
-          echo "OpenObserve PID for other tests: $SERVER_PID_DEFAULT"
-          wait_for_server
-
-          # Run all other tests (excluding regression and feature flag tests)
-          rye run pytest --ignore=tests/test_subquery_join_match.py --ignore=tests/test_logs_to_logs_join.py --force-sugar
-
-          # Kill the server
-          kill $SERVER_PID_DEFAULT || true
+          ${{ matrix.config.env }} ${{ github.workspace }}/target/debug/openobserve > "o2_${{ matrix.config.label }}.log" 2>&1 &
+          SERVER_PID=$!
+          for i in $(seq 1 60); do
+            if curl -sf http://localhost:5080/healthz > /dev/null 2>&1; then
+              echo "Server ready after ${i}s"; break
+            fi
+            sleep 1
+          done
+          rye run pytest ${{ matrix.config.target }} -v
+          kill $SERVER_PID || true
         working-directory: tests/api-testing/
 
+      - name: Upload server log on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: "openobserve-regression-${{ matrix.config.label }}-log"
+          path: "tests/api-testing/o2_${{ matrix.config.label }}.log"
+          if-no-files-found: warn
+          retention-days: 14
+
+  # Phase 5: schemathesis fuzz job — informational (continue-on-error),
+  # surfaces 5xx panics across the 150 OpenAPI endpoints. Decoupled from
+  # api_integration_tests so its findings don't block merges yet; teams
+  # can promote it to blocking after the noise floor is understood.
+  api_fuzz_tests:
+    name: api_fuzz_tests
+    needs: [check_changes]
+    if: needs.check_changes.outputs.has_changes == 'true'
+    runs-on: ubicloud-standard-4
+    continue-on-error: true
+    permissions:
+      contents: read  # only need to checkout the repo
+    steps:
+      - name: Remove unused tools
+        run: |
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /opt/ghc
+          sudo rm -rf "/usr/local/share/boost"
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY"
+
+      - name: Checkout git repo
+        uses: actions/checkout@v5
+        with:
+          fetch-depth: 0
+
+      - name: Setup Rust Toolchain
+        uses: dtolnay/rust-toolchain@master
+        with:
+          toolchain: nightly-2025-11-11
+
+      - name: Setup Rust Cache
+        uses: Swatinem/rust-cache@v2
+        with:
+          cache-on-failure: true
+          shared-key: backend-debug-mimalloc
+          save-if: false  # don't write cache from this job (the main job does)
+
+      - name: Install Protoc
+        run: bash ./.github/protoc.sh
+
+      - name: Build OpenObserve debug binary
+        run: cargo build --features mimalloc
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11.6"
+
+      - name: Setup rye
+        uses: eifinger/setup-rye@v4
+        with:
+          enable-cache: true
+          working-directory: "tests/api-testing/"
+          version: "0.27.0"
+
+      - name: Pin cpython
+        run: rye pin cpython@3.11.6
+        working-directory: tests/api-testing/
+
+      - name: Rye sync (schemathesis dep installed)
+        run: rye sync
+        working-directory: tests/api-testing/
+
+      - name: Start OpenObserve
+        run: target/debug/openobserve > o2_fuzz.log 2>&1 &
+
+      - name: Wait for /healthz
+        run: |
+          for i in $(seq 1 60); do
+            if curl -sf http://localhost:5080/healthz > /dev/null 2>&1; then exit 0; fi
+            sleep 1
+          done
+          exit 1
+
+      - name: Run schemathesis fuzz tests
+        # Schemathesis 4.x CLI flag renames vs the 3.x originals:
+        #   --base-url               -> --url / -u
+        #   --auth-type basic        -> removed (USER:PASS implies basic)
+        #   --hypothesis-max-examples -> --max-examples / -n
+        #   --hypothesis-deadline    -> no equivalent; use --request-timeout
+        #                               (seconds, not ms) to bound per-request
+        #   --data-generation-method -> --mode / -m
+        # `--checks not_a_server_error` keeps focus on 5xx panics (actual
+        # backend bugs, not spec-conformance nits). `-n 20` bounds runtime;
+        # `--mode positive` keeps inputs realistic (negative inputs are a
+        # separate exercise once the noise floor here is understood).
+        run: |
+          rye run schemathesis run \
+            -u http://localhost:5080 \
+            --auth "$ZO_ROOT_USER_EMAIL:$ZO_ROOT_USER_PASSWORD" \
+            --checks not_a_server_error \
+            -n 20 \
+            --request-timeout 5 \
+            -m positive \
+            http://localhost:5080/api-doc/openapi.json
+        working-directory: tests/api-testing/
+
+      - name: Upload OpenObserve server log on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: openobserve-fuzz-log
+          path: tests/api-testing/o2_fuzz.log
+          if-no-files-found: warn
+          retention-days: 14
+
   api_tests_summary:
     runs-on: ubuntu-latest
     permissions: {}
-    needs: [check_changes, api_integration_tests]
+    needs: [check_changes, api_integration_tests, api_regression_tests]
     if: always()
     steps:
       - name: Check test results
@@ -215,10 +365,13 @@ jobs:
             echo "check_changes job did not succeed: ${{ needs.check_changes.result }}"
             exit 1
           fi
-          if [ "${{ needs.api_integration_tests.result }}" == "success" ] || [ "${{ needs.api_integration_tests.result }}" == "skipped" ]; then
-            echo "API tests passed or skipped"
+          # Both api_integration_tests AND api_regression_tests must pass
+          # (or be skipped because there were no changes).
+          ok() { [ "$1" == "success" ] || [ "$1" == "skipped" ]; }
+          if ok "${{ needs.api_integration_tests.result }}" && ok "${{ needs.api_regression_tests.result }}"; then
+            echo "All API test jobs passed or skipped"
             exit 0
           else
-            echo "API tests failed"
+            echo "API tests failed: integration=${{ needs.api_integration_tests.result }}, regression=${{ needs.api_regression_tests.result }}"
             exit 1
           fi