windmill/cli/TESTING.md

Testing Guide for Windmill CLI

Running Tests

# Run unit tests only (fast — no backend, no database, no cargo build)
bun run test:unit

# Run all tests (unit + integration — requires PostgreSQL + cargo)
DATABASE_URL=postgres://postgres:changeme@localhost:5432 bun run test

# Run specific test files
bun test test/sync_pull_push.test.ts
bun test test/workspace_conflicts_unit.test.ts

Test Categories

Unit tests (*_unit.test.ts)

Pure local tests — no backend, no database. Uses bunfig.unit.toml (no preload).

Examples: git_unit, lint_command_unit, tar_creation_unit, workspace_conflicts_unit

mock.module() mocks the module for the whole bun test process, not for the file that installs it, and mock.restore() does not undo it. A file that mocks a module must hand back its real exports in afterAll (see schedule_push_permissioned_as_unit), or it silently rewires whichever file runs next — and the run order is the directory's, so it differs between Linux and Windows.

Integration tests

Require a running backend and PostgreSQL. The setup.ts preload builds the backend binary and starts a shared backend instance.

Examples: sync_pull_push, dev_server, standalone_commands

Module mocks

mock.module replaces a module for the whole process, and it does reach modules that were already imported — a stub one file installs lands on a consumer an earlier file loaded.

Handing the module back in afterAll is not a reliable undo. Files do run one at a time (a root-level afterAll completes before the next file's body evaluates), so it looks like it should be — but stubbing bundle.ts and restoring it that way still left raw_app_svelte_plugin_unit.test.ts asserting against an empty bundle, green on Linux and red on Windows, where the readdir file order differs. Treat a stub as permanent for the run.

So the rule is about what you stub, not how you clean up: stub only a module no other in-process suite imports. Check with grep -rl "<exported fn>" test/ before reaching for one. A suite that drives the CLI through a spawned process is out of reach of a module mock and doesn't count.

The API client gen/services.gen.ts is the exception, and is stubbed only through mockServices in test/mock_services.ts. Bun fixes a mocked module's export names at the first mock.module call of the run, so a stub that replaced the module with just the functions its suite needed left every other name undefined for the rest of the run, even for suites that stubbed that name themselves. test/mock_services.ts therefore calls mock.module exactly once, with every real export behind a dispatcher; mockServices only swaps which stubs the dispatcher routes to, so it doesn't rely on the unreliable afterAll re-mock described above.

raw_app_push_policy_unit.test.ts deliberately does not stub bundle.ts, which failed the rule.

AI Benchmark Caveats

The repo-level benchmark CLI lives under ai_evals/, but it currently depends on mocked frontend flow execution in a few places. Treat flow benchmark passes as artifact-shape signal, not full runtime correctness, when either of these apply:

• deterministic flow validation does not currently reject syntactically invalid rawscript module bodies
• frontend benchmark calls to test_run_flow and test_run_step return mocked completed jobs for mock-job-id-* workspaces instead of executing the flow

If a prompt change depends on flow wiring or script runtime behavior, verify it with additional validation or a real run before trusting the benchmark result.

Environment Variables

Variable	Purpose	Default
DATABASE_URL	PostgreSQL connection string (without database name)	postgres://postgres:changeme@localhost:5432
TEST_BACKEND	cargo or docker	cargo
CI_MINIMAL_FEATURES	true for CI mode (zip-only features)	unset
EE_LICENSE_KEY	Enterprise license for EE feature tests	unset
TEST_FEATURES	Additional cargo features (comma-separated)	unset
TEST_CLI_RUNTIME	node to test npm package	unset
UNIT_ONLY	1 to skip backend setup in preload (used by test:unit)	unset
VERBOSE	1 for backend process output	unset

Cleanup

Stale test databases (windmill_test_*) and orphaned backend processes from previous crashed runs are automatically cleaned up when starting a new test run.

To manually check for leftovers:

# Check for stale test databases
psql postgres://postgres:changeme@localhost:5432/postgres -c \
  "SELECT datname FROM pg_database WHERE datname LIKE 'windmill_test_%';"

# Check for orphaned backend processes
ps aux | grep "target/debug/windmill" | grep -v grep