# Testing Guide for Windmill CLI

## Running Tests

```bash
# Run unit tests only (fast — no backend, no database, no cargo build)
bun run test:unit

# Run all tests (unit + integration — requires PostgreSQL + cargo)
DATABASE_URL=postgres://postgres:changeme@localhost:5432 bun run test

# Run specific test files
bun test test/sync_pull_push.test.ts
bun test test/workspace_conflicts_unit.test.ts
```

## Test Categories

### Unit tests (`*_unit.test.ts`)

Pure local tests — no backend, no database. Uses `bunfig.unit.toml` (no preload).

Examples: `git_unit`, `lint_command_unit`, `tar_creation_unit`, `workspace_conflicts_unit`

`mock.module()` mocks the module for the whole `bun test` process, not for the file
that installs it, and `mock.restore()` does not undo it. A file that mocks a module
must hand back its real exports in `afterAll` (see
`schedule_push_permissioned_as_unit`), or it silently rewires whichever file runs
next — and the run order is the directory's, so it differs between Linux and Windows.

### Integration tests

Require a running backend and PostgreSQL. The `setup.ts` preload builds the backend
binary and starts a shared backend instance.

Examples: `sync_pull_push`, `dev_server`, `standalone_commands`

## Module mocks

`mock.module` replaces a module for the **whole process**, and it does reach modules that
were already imported — a stub one file installs lands on a consumer an earlier file
loaded.

Handing the module back in `afterAll` is not a reliable undo. Files do run one at a time
(a root-level `afterAll` completes before the next file's body evaluates), so it looks
like it should be — but stubbing `bundle.ts` and restoring it that way still left
`raw_app_svelte_plugin_unit.test.ts` asserting against an empty bundle, green on Linux
and red on Windows, where the `readdir` file order differs. Treat a stub as permanent for
the run.

So the rule is about what you stub, not how you clean up: **stub only a module no other
in-process suite imports.** Check with `grep -rl "<exported fn>" test/` before reaching
for one. A suite that drives the CLI through a spawned process is out of reach of a
module mock and doesn't count.

The API client `gen/services.gen.ts` is the exception, and is stubbed only through
`mockServices` in `test/mock_services.ts`. Bun fixes a mocked module's export names at
the first `mock.module` call of the run, so a stub that replaced the module with just the
functions its suite needed left every other name undefined for the rest of the run, even
for suites that stubbed that name themselves. `test/mock_services.ts` therefore calls
`mock.module` exactly once, with every real export behind a dispatcher; `mockServices`
only swaps which stubs the dispatcher routes to, so it doesn't rely on the unreliable
`afterAll` re-mock described above.

`raw_app_push_policy_unit.test.ts` deliberately does not stub `bundle.ts`, which failed
the rule.

## AI Benchmark Caveats

The repo-level benchmark CLI lives under `ai_evals/`, but it currently depends on
mocked frontend flow execution in a few places. Treat `flow` benchmark passes as
artifact-shape signal, not full runtime correctness, when either of these apply:

- deterministic flow validation does not currently reject syntactically invalid
  `rawscript` module bodies
- frontend benchmark calls to `test_run_flow` and `test_run_step` return mocked
  completed jobs for `mock-job-id-*` workspaces instead of executing the flow

If a prompt change depends on flow wiring or script runtime behavior, verify it
with additional validation or a real run before trusting the benchmark result.

## Environment Variables

| Variable | Purpose | Default |
|----------|---------|---------|
| `DATABASE_URL` | PostgreSQL connection string (without database name) | `postgres://postgres:changeme@localhost:5432` |
| `TEST_BACKEND` | `cargo` or `docker` | `cargo` |
| `CI_MINIMAL_FEATURES` | `true` for CI mode (zip-only features) | unset |
| `EE_LICENSE_KEY` | Enterprise license for EE feature tests | unset |
| `TEST_FEATURES` | Additional cargo features (comma-separated) | unset |
| `TEST_CLI_RUNTIME` | `node` to test npm package | unset |
| `UNIT_ONLY` | `1` to skip backend setup in preload (used by `test:unit`) | unset |
| `VERBOSE` | `1` for backend process output | unset |

## Cleanup

Stale test databases (`windmill_test_*`) and orphaned backend processes from
previous crashed runs are automatically cleaned up when starting a new test run.

To manually check for leftovers:

```bash
# Check for stale test databases
psql postgres://postgres:changeme@localhost:5432/postgres -c \
  "SELECT datname FROM pg_database WHERE datname LIKE 'windmill_test_%';"

# Check for orphaned backend processes
ps aux | grep "target/debug/windmill" | grep -v grep
```