windmill/cli/TESTING.md

Testing Guide for Windmill CLI

Running Tests

# Run unit tests only (fast — no backend, no database, no cargo build)
bun run test:unit

# Run all tests (unit + integration — requires PostgreSQL + cargo)
DATABASE_URL=postgres://postgres:changeme@localhost:5432 bun run test

# Run specific test files
bun test test/sync_pull_push.test.ts
bun test test/workspace_conflicts_unit.test.ts

Test Categories

Unit tests (*_unit.test.ts)

Pure local tests — no backend, no database. Uses bunfig.unit.toml (no preload).

Examples: git_unit, lint_command_unit, tar_creation_unit, workspace_conflicts_unit

mock.module() mocks the module for the whole bun test process, not for the file that installs it, and mock.restore() does not undo it. A file that mocks a module must hand back its real exports in afterAll (see schedule_push_permissioned_as_unit), or it silently rewires whichever file runs next — and the run order is the directory's, so it differs between Linux and Windows.

Integration tests

Require a running backend and PostgreSQL. The setup.ts preload builds the backend binary and starts a shared backend instance.

Examples: sync_pull_push, dev_server, standalone_commands

Module mocks

mock.module replaces a module for the whole process, and it does reach modules that were already imported — a stub one file installs lands on a consumer an earlier file loaded.

Handing the module back in afterAll is not a reliable undo. Files do run one at a time (a root-level afterAll completes before the next file's body evaluates), so it looks like it should be — but stubbing bundle.ts and restoring it that way still left raw_app_svelte_plugin_unit.test.ts asserting against an empty bundle, green on Linux and red on Windows, where the readdir file order differs. Treat a stub as permanent for the run.

So the rule is about what you stub, not how you clean up: stub only a module no other in-process suite imports. Check with grep -rl "<exported fn>" test/ before reaching for one. A suite that drives the CLI through a spawned process is out of reach of a module mock and doesn't count.

raw_app_push_policy_unit.test.ts is the worked example: it stubs gen/services.gen.ts, which passes the rule because nothing else in test/ imports the three API functions it replaces, and deliberately does not stub bundle.ts, which failed it.

AI Benchmark Caveats

The repo-level benchmark CLI lives under ai_evals/, but it currently depends on mocked frontend flow execution in a few places. Treat flow benchmark passes as artifact-shape signal, not full runtime correctness, when either of these apply:

• deterministic flow validation does not currently reject syntactically invalid rawscript module bodies
• frontend benchmark calls to test_run_flow and test_run_step return mocked completed jobs for mock-job-id-* workspaces instead of executing the flow

If a prompt change depends on flow wiring or script runtime behavior, verify it with additional validation or a real run before trusting the benchmark result.

Environment Variables

Variable	Purpose	Default
DATABASE_URL	PostgreSQL connection string (without database name)	postgres://postgres:changeme@localhost:5432
TEST_BACKEND	cargo or docker	cargo
CI_MINIMAL_FEATURES	true for CI mode (zip-only features)	unset
EE_LICENSE_KEY	Enterprise license for EE feature tests	unset
TEST_FEATURES	Additional cargo features (comma-separated)	unset
TEST_CLI_RUNTIME	node to test npm package	unset
UNIT_ONLY	1 to skip backend setup in preload (used by test:unit)	unset
VERBOSE	1 for backend process output	unset

Cleanup

Stale test databases (windmill_test_*) and orphaned backend processes from previous crashed runs are automatically cleaned up when starting a new test run.

To manually check for leftovers:

# Check for stale test databases
psql postgres://postgres:changeme@localhost:5432/postgres -c \
  "SELECT datname FROM pg_database WHERE datname LIKE 'windmill_test_%';"

# Check for orphaned backend processes
ps aux | grep "target/debug/windmill" | grep -v grep