Files
openobserve/tests/api-testing/fixtures/rum/ingest_rum_errors.py
T
Shrinath RaoandClaude Opus 4.7 61aa772d57 test: revamp api-testing suite (framework + 16 file rewrites + CI matrix) (#12372)
## Summary

Comprehensive rework of the `tests/api-testing/` suite. 30+ commits
implementing all 6 phases of the [revamp
plan](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md).

**Before → After:**

| | Baseline | Final | Δ |
|---|---|---|---|
| passed | 421 | **443** | **+22 ✓** |
| failed | 3 | **2** | -1 (only same pre-existing) |
| skipped | 53 | 39 | -14 (unskipped dead tests now run) |
| xfailed | 1 | 2 | +1 (new visible backend-bug signal) |
| runtime | 588s | **387s** | **-201s (34% faster)** |

**Zero tests that passed in baseline are now failing.** The 2 remaining
failures are the exact pre-existing ones
(`test_enrichment_table_join.py` backend panic + `test_workflow.py` SMTP
env-miss).

## What's in the diff

- **New `support/` framework** — `OpenObserveClient`, endpoint wrappers
(`alerts`, `dashboards`, `streams`, `search`, etc.), `wait_until()`,
payload factories, pytest fixtures, SSE parser. 14 framework selfcheck
tests.
- **16 of 16 OLD-tier test files rewritten** — using the new framework,
unique-suffix names, body validation, `wait_until` instead of
`time.sleep`, parametrize where appropriate.
- **29 files moved into 12 feature folders** mirroring the UI suite
layout
(`tests/{alerts,dashboards,ingest,orgs,pipelines,regression,rum,search,streams,users,workflow}/`).
- **−6,500 LOC net** (copy-paste + dead code removal; new framework adds
back ~1,500).
- **Ruff config** in pyproject.toml: T201 ban on `print()`, PT pytest
rules, B bugbear, UP pyupgrade.

## CI workflow changes

- New job `api_regression_tests`: `strategy.matrix` over the 4 env-flag
combinations (UTF8_VIEW × JOIN_MATCH_ONE) — now runs in parallel instead
of as 4 sequential server reboots in the main job.
- New job `api_fuzz_tests`: schemathesis OpenAPI-driven fuzz against
`/api-doc/openapi.json` (214 endpoints tested). Informational
(`continue-on-error: true`). Each run surfaces **7–10 distinct 5xx
panics** (e.g., empty `sql` body returning 500 instead of 400, traces
endpoints crashing on unescaped path params that get interpolated into
SQL). Filed to backend team for separate triage.
- Server `o2_*.log` files uploaded as artifacts on failure (was:
silently lost).
- `api_tests_summary` now requires BOTH integration + regression to
pass.

### CI infrastructure fixes

A short hardening pass landed after the initial CI run surfaced several
issues:

- **pytest-sugar count inflation** — dropped `--force-sugar` from both
pytest invocations. The flag double-registered sugar on top of the
auto-loaded entry point, inflating the CI test counter ~3x (display
showed 1364 when collection was 486). Counter is now honest.
- **schemathesis 4.x flag migration** — `--base-url` → `-u`,
`--hypothesis-max-examples` → `-n`, `--hypothesis-deadline` →
`--request-timeout`, `--data-generation-method` → `-m`, dropped
`--auth-type basic`. The fuzz job was failing immediately on flag-parse
before this.
- **Regression matrix binary path** — used `${{ github.workspace
}}/target/debug/openobserve` so the absolute path resolves correctly
under `working-directory: tests/api-testing/` (was: every matrix job
failed with "No such file or directory").
- **`permissions: contents: read`** added to all 4 workflow jobs —
closes CodeQL alert #369.
- **Removed redundant server boot** in `api_integration_tests` (leftover
from main's structure after the regression matrix split; was silently
failing to bind to :5080 and getting masked by the first server).

## Backend bugs surfaced (for backend team to track)

The rework uncovered and visibly flagged these as `xfail`/documented
test comments:

1. `DELETE /api/organizations/{id}` returns 404 unconditionally even for
just-created orgs — endpoint appears unimplemented.
2. `test_dashboard_missing_required_fields` triggers backend panic
(returns 500 instead of 400/422).
3. JS `getEnrichmentTableData()` restricted to `_meta` org AND not
exposed in the `/functions/test` endpoint runtime — `ReferenceError`
even with explicit `enrichment_tables: [...]`.
4. Bulk API silently sanitizes invalid `_index` values (`****` → `_`)
without flagging — user data lands in unexpected index.
5. `AVG` of a single value drifts by ~1e-14 (likely float reduction
order; documented tolerance in `test_advanced_sql.py`).
6. Org name validation rejects hyphens (only
`A-Z/a-z/0-9/spaces/underscores`) — not in spec.

Plus 7–10 additional 5xx panics surfaced per fuzz run (input validation
gaps — separate triage list in fuzz job logs).

## Test bugs uncovered + fixed

- `test_pipeline.py`: had two function definitions named
`test_pipeline_creation_and_action_with_schedule` — Python's last-wins
meant the first test's body never executed for years.
- `test_alertsV2.py`: `assert alert_disabled == "False"` was indented
inside an `else:` branch that never executed because the response is
always a dict; AND `enabled` is a bool, not the string `"False"`. Two
layers of broken.
- `test_bulk.py`: variable typo (`valid_bulk_payload` reused on the POST
call) made the invalid-index test silently duplicate the valid case.
- `test_organisations.py`: `test_create_organization` used pre-pytest-4
`yield`-style fixtures — silently ignored by pytest 8.
- `test_js_result_array.py`: 8 tests skipped with `"Temporarily disabled
- failing test"` — the actual cause was that JS functions are restricted
to `_meta` org. One-line fix unskipped all 8.
- `test_fuzzy.py`: 4 tests `@pytest.mark.skip`-ed because they used a
1-minute time window against older static test data → 0 hits → flake →
skip. Wider window unskips them.
- 547 LOC of zombie commented-out test blocks deleted (15 commented `#
def test_` definitions, including one orphan case where the `def` was
commented but the indented body wasn't).

## Folder layout

```
tests/api-testing/
├── support/                    # NEW framework
│   ├── client.py               # OpenObserveClient
│   ├── endpoints/              # 7 per-resource wrappers
│   ├── factories.py            # unique_name, time_window, *_payload
│   ├── wait.py                 # wait_until + WaitTimeout
│   ├── sse.py                  # read_sse_response
│   └── fixtures.py             # client, temp_user_email, temp_stream_name, ...
├── helpers/                    # Non-test shared code (was scattered in tests/)
│   ├── sourcemap_helpers.py
│   └── workflow/               # workflow chain + pages/ live here now
├── fixtures/
│   └── rum/                    # Standalone RUM ingest scripts (was in tests/)
└── tests/
    ├── alerts/                 # 3 files
    ├── dashboards/             # 4 files
    ├── ingest/                 # 4 files
    ├── orgs/                   # 2 files
    ├── pipelines/              # 10 files
    ├── regression/             # 2 files (env-flag matrix targets)
    ├── rum/                    # 3 files
    ├── search/                 # 8 files
    ├── streams/                # 1 file
    ├── users/                  # 1 file
    ├── workflow/               # 3 files
    ├── conftest.py
    └── test_framework_selfcheck.py
```

## Test plan

- [x] `rye run pytest --collect-only` — 486 tests discovered
- [x] Full default-set run (`--ignore=tests/regression`) — 443 pass / 2
fail / 39 skip / 2 xfail in 387s
- [x] `rye run pytest -m framework` — 14/14 selfcheck tests pass
- [x] `rye run ruff check support/ tests/test_framework_selfcheck.py` —
clean
- [x] CI run on this PR — `api_integration_tests` + all 4
`api_regression_tests` matrix jobs green
- [x] CI run — `api_fuzz_tests` runs cleanly (informational; surfaces
7–10 unique 5xx panics per run)

## Note on test count

The `test_streaming.py` rewrite folded 7 "tests" (`test_ingest_data`,
`test_ingest_join`, `test_disable_streaming`,
`test_update_max_query_range`, `test_enable_streaming`,
`test_delete_stream`, `test_delete_stream_join`) into pytest fixtures.
Each only checked `status_code == 200` of a setup/teardown operation —
they were imperative steps masquerading as tests. The fixtures still
perform the same operations and assert their success; they're just no
longer counted as discrete pytest items. **Functional coverage is
identical.**

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-01 07:19:48 +00:00

198 lines
6.9 KiB
Python

"""
Ingest synthetic RUM error data for sourcemap UI testing.
This script injects RUM error events with minified stacktraces that can be
resolved using uploaded sourcemaps. This allows UI tests to run without
needing to build and serve the test app.
"""
import requests
import base64
import time
import json
import os
# Get config from environment
BASE_URL = os.getenv('ZO_BASE_URL', 'http://localhost:5080')
EMAIL = os.getenv('ZO_ROOT_USER_EMAIL', 'root@example.com')
PASSWORD = os.getenv('ZO_ROOT_USER_PASSWORD', 'Complexpass#123')
ORG_ID = os.getenv('ORGNAME', 'default')
def ingest_rum_errors():
"""Ingest RUM error events to OpenObserve."""
# Create session with Basic Auth for API calls
api_session = requests.Session()
basic_auth = base64.b64encode(f"{EMAIL}:{PASSWORD}".encode()).decode()
api_session.headers.update({"Authorization": f"Basic {basic_auth}"})
# Fetch RUM token for ingestion
print("Fetching RUM token...")
rum_token_response = api_session.get(f"{BASE_URL}/api/{ORG_ID}/rumtoken")
if rum_token_response.status_code != 200:
print(f"❌ Failed to fetch RUM token: {rum_token_response.status_code}")
rum_token = None
else:
rum_token_data = rum_token_response.json()
rum_token = rum_token_data.get('data', {}).get('rum_token') or rum_token_data.get('rum_token')
if rum_token:
print(f"✅ RUM token fetched successfully")
else:
print(f"⚠️ RUM token is empty, response: {rum_token_data}")
# Create RUM ingestion session with RUM token
session = requests.Session()
if rum_token:
# RUM token goes in Authorization header as Bearer token
session.headers.update({"Authorization": f"Bearer {rum_token}"})
# Current timestamp
now = int(time.time() * 1000)
# Sample RUM error events with minified stacktraces
# These match the format that the RUM SDK sends
rum_errors = [
{
"date": now - 60000, # 1 minute ago
"type": "error",
"error": {
"message": "TypeError: Cannot read property 'name' of undefined",
"type": "TypeError",
"stack": "TypeError: Cannot read property 'name' of undefined\n at Ht @ http://localhost:8089/assets/main.87e94092.js:1:2345\n at onClick @ http://localhost:8089/assets/main.87e94092.js:1:3456",
"source": "source",
"is_crash": False,
"resource": {
"url": "http://localhost:8089/"
}
},
"service": "o2-sourcemap-test-app",
"version": "1.0.0-api-test",
"session": {
"id": "test-session-001"
},
"view": {
"id": "test-view-001",
"referrer": "",
"url": "http://localhost:8089/"
},
"application": {
"id": "o2-sourcemap-test-app"
},
"context": {
"browser": {
"name": "Chrome",
"version": "120.0.0"
},
"os": {
"name": "Mac OS",
"version": "14.0"
}
}
},
{
"date": now - 120000, # 2 minutes ago
"type": "error",
"error": {
"message": "ReferenceError: undefinedVariable is not defined",
"type": "ReferenceError",
"stack": "ReferenceError: undefinedVariable is not defined\n at triggerReferenceError @ http://localhost:8089/assets/main.87e94092.js:1:4567\n at onClick @ http://localhost:8089/assets/main.87e94092.js:1:5678",
"source": "source",
"is_crash": False,
"resource": {
"url": "http://localhost:8089/"
}
},
"service": "o2-sourcemap-test-app",
"version": "1.0.0-api-test",
"session": {
"id": "test-session-002"
},
"view": {
"id": "test-view-002",
"referrer": "",
"url": "http://localhost:8089/"
},
"application": {
"id": "o2-sourcemap-test-app"
},
"context": {
"browser": {
"name": "Firefox",
"version": "119.0"
},
"os": {
"name": "Windows",
"version": "11"
}
}
},
{
"date": now - 180000, # 3 minutes ago
"type": "error",
"error": {
"message": "RangeError: Maximum call stack size exceeded",
"type": "RangeError",
"stack": "RangeError: Maximum call stack size exceeded\n at recursiveFunction @ http://localhost:8089/assets/main.87e94092.js:1:6789\n at recursiveFunction @ http://localhost:8089/assets/main.87e94092.js:1:6789",
"source": "source",
"is_crash": False,
"resource": {
"url": "http://localhost:8089/"
}
},
"service": "o2-sourcemap-test-app",
"version": "1.0.0-api-test",
"session": {
"id": "test-session-003"
},
"view": {
"id": "test-view-003",
"referrer": "",
"url": "http://localhost:8089/"
},
"application": {
"id": "o2-sourcemap-test-app"
},
"context": {
"browser": {
"name": "Safari",
"version": "17.0"
},
"os": {
"name": "Mac OS",
"version": "14.0"
}
}
}
]
print(f"\nIngesting {len(rum_errors)} RUM error events to {BASE_URL}")
# Use _rumdata stream (not _rumlog) because the UI queries _rumdata for errors
# The UI looks for records where type='error' in _rumdata stream
url = f"{BASE_URL}/api/{ORG_ID}/_rumdata/_json"
# Send as JSON array
response = api_session.post(
url,
json=rum_errors,
headers={'Content-Type': 'application/json'}
)
if response.status_code in [200, 201]:
print(f"✅ Successfully ingested {len(rum_errors)} RUM errors")
print(f" Status: {response.status_code}")
print(f" Stream: _rumdata (errors with type='error')")
print(f" Errors can be viewed in RUM Error Tracking page")
return True
else:
print(f"❌ Failed to ingest RUM errors")
print(f" Status: {response.status_code}")
print(f" Response: {response.text}")
return False
if __name__ == "__main__":
success = ingest_rum_errors()
exit(0 if success else 1)