openobserve/tests/api-testing/README.md

# OpenObserve API Tests

End-to-end pytest suite against the OpenObserve REST API. Runs in CI on every
PR (`.github/workflows/api-testing.yml`); also runnable locally.

## Quick start

```bash
# one-time setup
brew install rye           # or: curl -sSf https://rye.astral.sh/get | bash
rye sync                   # install Python + deps from requirements.lock
cargo build                # build the OO debug binary (~10 min first time)

# every run
ZO_ROOT_USER_EMAIL=root@example.com \
ZO_ROOT_USER_PASSWORD='Complexpass#123' \
ZO_BASE_URL=http://localhost:5080/ \
WS_ZO_BASE_URL=ws://localhost:5080/ \
target/debug/openobserve &
rye run pytest             # full suite
rye run pytest -m framework        # just the new-framework self-tests
rye run pytest tests/test_kvstore.py -v   # one file
```

## Layout

```
tests/api-testing/
├── pyproject.toml          # pytest + ruff config
├── conftest.py             # legacy session-wide fixtures (do not extend)
├── support/                # NEW FRAMEWORK — write new tests against this
│   ├── client.py           # OpenObserveClient: HTTP entry point
│   ├── endpoints/          # one wrapper class per resource group
│   │   ├── search.py       # client.search.sql(...), .hits(...), .count(...)
│   │   ├── streams.py      # client.streams.list(), .delete(...), .ingest_json(...)
│   │   ├── users.py
│   │   ├── dashboards.py
│   │   ├── alerts.py
│   │   ├── folders.py
│   │   └── kvstore.py
│   ├── factories.py        # payload builders (user_payload, dashboard_payload, …)
│   ├── wait.py             # wait_until(predicate, timeout, ...) — REPLACES time.sleep
│   └── fixtures.py         # client, temp_stream_name, temp_user_email, …
├── tests/                  # test files
│   ├── conftest.py
│   ├── test_framework_selfcheck.py   # 14 tests that prove support/ works
│   └── test_*.py           # 40+ legacy test files (in revamp progress)
├── helpers/                # non-test shared code
│   ├── sourcemap_helpers.py
│   └── workflow/           # workflow chain helpers (with embedded pages/)
├── fixtures/               # static test data
│   └── rum/                # RUM ingest scripts
└── README.md               # this file
```

## Writing a new test — the rules

**Use the new framework.** Don't copy-paste from old `test_*.py` files; they
are scheduled for rewrite (see [revamp plan](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md)).

```python
# tests/<area>/test_my_feature.py
from support.client import OpenObserveClient
from support.factories import user_payload
from support.fixtures import client, temp_user_email   # noqa: F401
from support.wait import wait_until


def test_create_user_returns_complete_record(client: OpenObserveClient, temp_user_email: str):
    """Create a user — verify the response has the email AND role we sent."""
    payload = user_payload(email=temp_user_email, role="admin")
    resp = client.users.create(payload)

    assert resp.status_code == 200, resp.text
    body = resp.json()
    # VALIDATE THE BODY, not just status_code
    assert body.get("data", {}).get("email") == temp_user_email
    assert body.get("data", {}).get("role") == "admin"
    # cleanup happens automatically via temp_user_email fixture
```

### Mandatory patterns

| ✅ Do | ❌ Don't |
|---|---|
| `from support.fixtures import client` | `session = create_session; url = base_url; org_id = "default"` (307 occurrences in legacy code) |
| `client.streams.list()` | `session.get(f"{url}api/{org_id}/streams")` |
| `body = resp.json(); assert body["foo"] == ...` | `assert resp.status_code == 200` and stop |
| `wait_until(lambda: client.search.count(sql) >= N, timeout=30)` | `time.sleep(5)` |
| `unique_email("user")` / `temp_user_email` fixture | hardcoded `pytests@gmail.com` |
| `logger.debug("got %s", resp.status_code)` | `print(resp.content)` |
| `start, end = time_window(minutes=15)` | hardcoded `start_time = 1700629279639000` |

### Anti-patterns that fail review

- `assert resp.status_code in [200, 400, 404, 500]` — pick one. If the API behaves wrongly, file a bug; don't paper over it.
- `pytest.skip(...)` when the test discovers the bug it's meant to catch — use `@pytest.mark.xfail(strict=True)`.
- Commented-out test functions — delete them.
- `pages.*` direct imports — that subtree is legacy. New tests use `support.client`.

### Lint

```bash
rye run ruff check support/ tests/test_framework_selfcheck.py  # strict mode
rye run ruff check --fix support/                              # autofix
```

Existing legacy tests have a per-file ruff-ignore (so `print` etc. don't fail
the whole suite); new tests under `support/` and any new well-formed test
file should pass strict ruff.

## Test markers

| Marker | Meaning |
|---|---|
| `framework` | Self-tests for `support/` framework. Run with `-m framework`. |
| `enterprise` | Needs an enterprise build (e.g. the on-call suite, whose routes an OSS server answers 404). Run with `-m enterprise`; deselect with `-m "not enterprise"`. |
| `slow` | Long-running test (e.g. the profiles 100k-stack ingest, ~2 min). Runs in the normal suite; deselect with `-m "not slow"`. |
| `serial` | Reads or changes server-wide state (root passcode, shared seed stream, org-wide counts). Jobs that run the default set with `-n` (OSS `api_integration_tests`, ENT `oss_in_ent`) run `-m "not serial" -n 4 --dist loadfile`, then `-m serial` alone; mark any new test of that kind. |
| `order(N)` | Run order pin (legacy — avoid in new tests). |

## Enterprise suites with their own setup

CI copies `o2-enterprise/tests/*` over `tests/` and runs these suites in their own shards, never in the full run. Locally, deselect them from `rye run pytest` with `--ignore=tests/fleet/ --ignore=tests/fleet_flag_off/ --ignore=tests/fleet_collector/ --ignore=tests/rbac/fleet/`.

| Suite | Needs | Run |
|---|---|---|
| `tests/fleet/` | An enterprise server with `O2_FLEET_ENABLED=true` (the default) | `rye run pytest tests/fleet/ -n 4 --dist loadfile` |
| `tests/fleet_flag_off/` | Two starts over the same `./data`: first with `O2_FLEET_ENABLED=true` to seed a real token, then with `O2_FLEET_ENABLED=false` (fails otherwise) | `FLEET_FLAG_OFF_SEED=/tmp/seed.json PYTHONPATH=.:tests rye run python -m fleet_flag_off.seed_token`, restart the server with the flag off, then `FLEET_FLAG_OFF_SEED=/tmp/seed.json rye run pytest tests/fleet_flag_off/` |
| `tests/fleet_collector/` | An enterprise server with `O2_FLEET_ENABLED=true`, a running Docker daemon, and the collector image pinned as `IMAGE_REF` in `tests/fleet_collector/collector_docker.py` (pulled on first use). Containers reach the server through `host.docker.internal`. | `rye run pytest tests/fleet_collector/` (no `-n`) |
| `tests/rbac/fleet/` | An enterprise server with `O2_OPENFGA_ENABLED=true`, `O2_OPENFGA_LIST_ONLY_PERMITTED=true` (the filtered-list checks need it) and a running OpenFGA server; CI runs it in the `ent_rbac` shard | `rye run pytest tests/rbac/fleet/` |

Fleet runs leave `fl_auto_*` orgs behind, because org delete is cloud-only. Most are empty, but the flag-off seed org keeps its fleet and live token, and the fleet suite's orgs keep a disabled ingestion token. Remove them by hand before reusing a local server.

## Troubleshooting

- **`KeyError: 'ZO_BASE_URL'` on collection** — `conftest.py` reads env vars at import. Export them before running pytest (even `--collect-only`).
- **On-call ack-token tests skip locally** — they read the paging mail to get the acknowledgement token, which exists only in the delivered message. Start Mailpit with `mailpit --smtp 127.0.0.1:1025 --listen 127.0.0.1:8025 --smtp-allowed-recipients '(?i)@test\.invalid$'` (the filter makes mail to the reachability tests' other domain a real refusal), point the server at it with `ZO_SMTP_ENABLED=true ZO_SMTP_HOST=127.0.0.1 ZO_SMTP_PORT=1025 ZO_SMTP_FROM_EMAIL=oncall@ci.openobserve.test`, and set `MAILPIT_BASE_URL` if it is not `http://127.0.0.1:8025`. With no mailbox they skip locally and fail when `CI` is set. The enterprise `ent_oncall` shard starts Mailpit v1.21.8 on 1025/8025 with that recipient filter and starts the server with the same SMTP settings.
- **`test_workflow.py` fails with SMTP error** — `ZO_SMTP_ENABLED=true` not set. CI does this; locally you'd need to wire an SMTP env.
- **Test_organisations.py warning about `yield`** — `test_create_organization` uses pre-pytest-4 yield-style and is silently ignored. Scheduled for rewrite in Phase 4.

## Revamp in progress

This suite is being incrementally reworked. See:
- [`../../tests/ui-testing/MD_Files/api tests revamp/API_TESTS_REVAMP.md`](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md) — the plan
- [`../../tests/ui-testing/MD_Files/api tests revamp/oss_rework_progress/`](../../tests/ui-testing/MD_Files/api%20tests%20revamp/oss_rework_progress/) — phase-by-phase progress