Files
windmill/cli/test/test_backend.ts
T
Ruben Fiszel a22d179903 add GitHub Actions CI and raw app sync tests (#7677)
* feat(cli): add GitHub Actions CI and raw app sync tests

- Add CLI tests GitHub Action that runs on Linux and Windows
- Add build check job that runs on CLI and openapi.yaml changes
- Uses Rust cargo backend instead of Docker for better CI compatibility
- Add cargo_backend.ts and test_backend.ts for test infrastructure
- Fix Windows path separator bug in raw_apps.ts (use "/" for relative paths)
- Fix PostgreSQL URL parsing in cargo_backend.ts
- Update tests to use gitBranches format instead of deprecated overrides
- Add raw_app_sync.test.ts for raw app sync workflow testing (ignored for now - needs EE)
- Skip tests that require EE features (git sync settings, raw apps)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(cli): Fix Windows path compatibility issues in tests

- Use fromFileUrl() in cargo_backend.ts for proper Windows path handling
- Normalize path separators to forward slashes in resource_folders.ts
- Fix readDirRecursive to return normalized paths in test helper
- Use forward slashes consistently in buildMetadataPath and detection functions

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(cli): Use SEP in test assertions instead of modifying logic

- Revert resource_folders.ts to use SEP as intended
- Update test assertions to use SEP for platform-specific paths
- Keep readDirRecursive normalization for consistent test comparisons

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(cli): Use SEP for all path separators in test assertions

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(cli): Use resolve() for proper cross-platform path handling in cargo_backend

String concatenation with path separators creates malformed paths on Windows.
Use path.resolve() instead for proper cross-platform path resolution.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(backend): Add cfg attributes for Windows compatibility

- Add #[cfg(unix)] to anyhow::anyhow import (only used in unix cfg block)
- Add #[cfg(not(windows))] to parse_file function (uses cat, only for cgroups)
- Remove unused std::io import, use std::io::Result directly

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: Windows compilation + convert integration tests to withTestBackend

- Fix unused import SYSTEM_ROOT in csharp_executor.rs on Windows by
  requiring both windows and csharp feature
- Fix unused variable id in handle_child.rs on Windows by adding
  #[allow(unused_variables)] since id is only used in cfg(unix) code
- Convert all RUN_INTEGRATION_TESTS dependent tests in sync_pull_push.test.ts
  to use withTestBackend pattern for automatic backend setup

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: configurable test features with CI_MINIMAL_FEATURES env var

- Default: full features (zip, private, enterprise) for local development
- CI mode: minimal features (zip only) when CI_MINIMAL_FEATURES=true
- Add shouldSkipOnCI() helper for tests requiring EE features
- Update EE-dependent tests to use shouldSkipOnCI()
- Add test instructions to cli/README.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: enable raw app tests (not EE-dependent)

Raw apps work with minimal features. 2 tests pass, 2 have test logic
bugs to investigate separately:
- "delete file and push" - file deletion not syncing correctly
- "dry-run push shows expected changes" - JSON output parsing issue

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: gate cgroups module to Linux only

cgroups are Linux-specific, the module was causing dead_code warnings
on Windows compilation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(ci): add CI_MINIMAL_FEATURES env var to CLI tests workflow

Set CI_MINIMAL_FEATURES=true in both Linux and Windows test jobs so
the backend compiles with minimal features (zip only) and EE-dependent
tests self-skip via shouldSkipOnCI().

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(cli): raw app tests and backend startup timing

- Add 5s delay after backend ready for migrations to complete
- Fix dry-run JSON output parsing (handle pretty-printed JSON)
- Temporarily ignore "delete file" test (needs isSuperset fix)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(cli): raw app file deletion sync

- Add deepEqual check for files in raw_apps.ts isSuperset comparison
- Handle raw_app file deletions in sync.ts by re-pushing the entire app
- Fix test to remove CSS import before deleting the file

When deleting a file from a raw app, the sync now properly updates
the backend with the new file list (excluding the deleted file).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(cli): Windows path separators in tests

Normalize paths for cross-platform comparison by converting
backslashes to forward slashes before path assertions.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(cli): normalize featurePaths in multi_instance_workspace test

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test(cli): add mixed case paths sync tests for Windows compatibility

Add comprehensive tests for sync pull/push with capitalized folder paths
to catch Windows case-insensitivity issues:
- Scripts in f/MyFolder/MyScript
- Flows in f/MyFlows/DataProcessor
- Apps in f/MyApps/Dashboard
- Variables in f/MyVars/ApiKey
- Deeply nested paths with mixed case
- Multiple resources in same capitalized folder
- CamelCase folder names with numbers

Each test verifies the full pull -> modify -> push -> verify cycle.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test(cli): add idempotency check to mixed case paths tests

After each push, pull again with --dry-run --json-output and verify
that no changes are detected. This ensures the sync is stable and
catches issues where pull/push cycles cause spurious diffs.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-25 20:38:33 +00:00

485 lines
14 KiB
TypeScript

/**
* Unified Test Backend Interface
*
* Provides a common interface for both Docker-based and Cargo-based backends.
* Use environment variable TEST_BACKEND to switch:
* - TEST_BACKEND=cargo (default) - Uses pre-built binary + local postgres
* - TEST_BACKEND=docker - Uses docker-compose (legacy)
*
* Prerequisites for cargo backend:
* - PostgreSQL running locally (default: postgres://postgres:changeme@localhost:5432)
* - Backend built: cd backend && cargo build --release (or debug)
*
* Usage:
* import { withTestBackend, cleanupTestBackend } from "./test_backend.ts";
*
* Deno.test("my test", async () => {
* await withTestBackend(async (backend, tempDir) => {
* const result = await backend.runCLICommand(["sync", "pull"], tempDir);
* // ...
* });
* });
*/
import { CargoBackend, CargoBackendConfig } from "./cargo_backend.ts";
import { ContainerizedBackend, ContainerConfig } from "./containerized_backend.ts";
/**
* Common interface for test backends
*/
export interface TestBackend {
readonly baseUrl: string;
readonly workspace: string;
readonly testConfigDir: string;
readonly token?: string;
start(): Promise<void>;
stop(): Promise<void>;
reset(): Promise<void>;
createCLICommand(args: string[], workingDir: string, workspaceName?: string): Deno.Command;
runCLICommand(args: string[], workingDir: string, workspaceName?: string): Promise<{
stdout: string;
stderr: string;
code: number;
}>;
// Optional methods that may not be on all backends
apiRequest?(path: string, options?: RequestInit): Promise<Response>;
seedTestData?(): Promise<void>;
getWorkspaceSettings?(): Promise<any>;
updateGitSyncConfig?(config: any): Promise<void>;
createAdditionalGitRepo?(repoPath: string, description: string): Promise<void>;
listAllScripts?(): Promise<any[]>;
listAllApps?(): Promise<any[]>;
listAllResources?(): Promise<any[]>;
listAllVariables?(): Promise<any[]>;
}
/**
* Adapter to make CargoBackend implement TestBackend
*/
class CargoBackendAdapter implements TestBackend {
private backend: CargoBackend;
constructor(config?: Partial<CargoBackendConfig>) {
this.backend = new CargoBackend(config);
}
get baseUrl(): string {
return this.backend.baseUrl;
}
get workspace(): string {
return this.backend.workspace;
}
get testConfigDir(): string {
return this.backend.testConfigDir;
}
get token(): string {
return this.backend.authToken;
}
async start(): Promise<void> {
await this.backend.start();
}
async stop(): Promise<void> {
await this.backend.stop();
}
async reset(): Promise<void> {
await this.backend.reset();
}
createCLICommand(args: string[], workingDir: string, workspaceName?: string): Deno.Command {
return this.backend.createCLICommand(args, workingDir, workspaceName);
}
async runCLICommand(args: string[], workingDir: string, workspaceName?: string) {
return this.backend.runCLICommand(args, workingDir, workspaceName);
}
async apiRequest(path: string, options?: RequestInit): Promise<Response> {
return this.backend.apiRequest(path, options);
}
async seedTestData(): Promise<void> {
// Create test folder first
await this.createTestFolder("test");
// Create test resources and variables
await this.createTestResource("f/test/my_resource", "Test resource description");
await this.createTestVariable("f/test/my_variable", "Test variable value");
// Create test group
await this.createTestGroup("test_group");
// Create a test app
await this.createTestApp("f/test/test_dashboard");
}
private async createTestApp(path: string): Promise<void> {
const response = await this.backend.apiRequest(`/api/w/${this.workspace}/apps/create`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
path,
value: {
type: "app",
grid: [],
hiddenInlineScripts: [],
css: {},
norefreshbar: false,
},
summary: "Test app",
policy: {
on_behalf_of: null,
on_behalf_of_email: null,
triggerables: {},
execution_mode: "viewer",
},
}),
});
if (!response.ok) {
const error = await response.text();
if (!error.includes("already exists")) {
console.warn(`Warning: Failed to create app ${path}: ${error}`);
}
} else {
await response.text();
}
}
private async createTestFolder(name: string): Promise<void> {
const response = await this.backend.apiRequest(`/api/w/${this.workspace}/folders/create`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ name }),
});
if (!response.ok) {
const error = await response.text();
if (!error.includes("already exists")) {
console.warn(`Warning: Failed to create folder ${name}: ${error}`);
}
} else {
await response.text();
}
}
private async createTestGroup(name: string): Promise<void> {
const response = await this.backend.apiRequest(`/api/w/${this.workspace}/groups/create`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ name, summary: `Test group ${name}` }),
});
if (!response.ok) {
const error = await response.text();
if (!error.includes("already exists")) {
console.warn(`Warning: Failed to create group ${name}: ${error}`);
}
} else {
await response.text();
}
}
private async createTestResource(path: string, description: string): Promise<void> {
// First ensure the folder exists
const folderPath = path.split("/").slice(0, 2).join("/"); // e.g., "f/test"
const folderName = folderPath.replace("f/", "");
try {
const folderResponse = await this.backend.apiRequest(`/api/w/${this.workspace}/folders/create`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ name: folderName }),
});
await folderResponse.text(); // Consume response body
} catch {
// Folder may already exist
}
const response = await this.backend.apiRequest(
`/api/w/${this.workspace}/resources/create`,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
path,
description,
resource_type: "any",
value: { test: "value" },
}),
}
);
if (!response.ok) {
const error = await response.text();
if (!error.includes("already exists")) {
console.warn(`Warning: Failed to create resource ${path}: ${error}`);
}
} else {
await response.text();
}
}
private async createTestVariable(path: string, value: string): Promise<void> {
const response = await this.backend.apiRequest(
`/api/w/${this.workspace}/variables/create`,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
path,
value,
is_secret: false,
description: "Test variable",
}),
}
);
if (!response.ok) {
const error = await response.text();
if (!error.includes("already exists")) {
console.warn(`Warning: Failed to create variable ${path}: ${error}`);
}
} else {
await response.text();
}
}
async getWorkspaceSettings(): Promise<any> {
const response = await this.backend.apiRequest(`/api/w/${this.workspace}/workspaces/get_settings`);
if (!response.ok) {
throw new Error(`Failed to get workspace settings: ${response.status}`);
}
return response.json();
}
async updateGitSyncConfig(config: any): Promise<void> {
const response = await this.backend.apiRequest(
`/api/w/${this.workspace}/workspaces/edit_git_sync_config`,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(config),
}
);
if (!response.ok) {
throw new Error(`Failed to update git sync config: ${response.status}`);
}
await response.text();
}
async createAdditionalGitRepo(repoPath: string, description: string): Promise<void> {
const gitRepo = {
path: repoPath,
description,
resource_type: "git_repository",
value: {
url: "https://github.com/windmill-labs/windmill-test.git",
branch: "main",
token: "",
},
};
const response = await this.backend.apiRequest(
`/api/w/${this.workspace}/resources/create`,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(gitRepo),
}
);
if (!response.ok) {
const error = await response.text();
if (!error.includes("already exists")) {
console.warn(`Failed to create git repo ${repoPath}: ${error}`);
}
} else {
await response.text();
}
}
async listAllScripts(): Promise<any[]> {
const response = await this.backend.apiRequest(`/api/w/${this.workspace}/scripts/list`);
if (!response.ok) return [];
return response.json();
}
async listAllApps(): Promise<any[]> {
const response = await this.backend.apiRequest(`/api/w/${this.workspace}/apps/list`);
if (!response.ok) return [];
return response.json();
}
async listAllResources(): Promise<any[]> {
const response = await this.backend.apiRequest(`/api/w/${this.workspace}/resources/list`);
if (!response.ok) return [];
return response.json();
}
async listAllVariables(): Promise<any[]> {
const response = await this.backend.apiRequest(`/api/w/${this.workspace}/variables/list`);
if (!response.ok) return [];
return response.json();
}
}
/**
* Adapter to make ContainerizedBackend implement TestBackend
*/
class ContainerizedBackendAdapter implements TestBackend {
private backend: ContainerizedBackend;
constructor(config?: Partial<ContainerConfig>) {
this.backend = new ContainerizedBackend(config);
}
get baseUrl(): string {
return this.backend.baseUrl;
}
get workspace(): string {
return this.backend.workspace;
}
get testConfigDir(): string {
return this.backend.testConfigDir;
}
get token(): string {
return this.backend.token;
}
async start(): Promise<void> {
await this.backend.start();
}
async stop(): Promise<void> {
await this.backend.stop();
}
async reset(): Promise<void> {
await this.backend.reset();
}
createCLICommand(args: string[], workingDir: string, workspaceName?: string): Deno.Command {
return this.backend.createCLICommand(args, workingDir, workspaceName);
}
async runCLICommand(args: string[], workingDir: string, workspaceName?: string) {
return this.backend.runCLICommand(args, workingDir, workspaceName);
}
async seedTestData(): Promise<void> {
await this.backend.seedTestData();
}
async getWorkspaceSettings(): Promise<any> {
return this.backend.getWorkspaceSettings();
}
async updateGitSyncConfig(config: any): Promise<void> {
await this.backend.updateGitSyncConfig(config);
}
async createAdditionalGitRepo(repoPath: string, description: string): Promise<void> {
await this.backend.createAdditionalGitRepo(repoPath, description);
}
async listAllScripts(): Promise<any[]> {
return this.backend.listAllScripts();
}
async listAllApps(): Promise<any[]> {
return this.backend.listAllApps();
}
async listAllResources(): Promise<any[]> {
return this.backend.listAllResources();
}
async listAllVariables(): Promise<any[]> {
return this.backend.listAllVariables();
}
}
// Global backend instance
let globalBackend: TestBackend | null = null;
/**
* Get the backend type from environment
*/
function getBackendType(): "cargo" | "docker" {
const envType = Deno.env.get("TEST_BACKEND")?.toLowerCase();
if (envType === "docker") {
return "docker";
}
return "cargo"; // Default to cargo
}
/**
* Create a new backend instance based on configuration
*/
export function createTestBackend(type?: "cargo" | "docker"): TestBackend {
const backendType = type || getBackendType();
if (backendType === "docker") {
console.log("📦 Using Docker-based test backend");
return new ContainerizedBackendAdapter();
} else {
console.log("🦀 Using Cargo-based test backend");
return new CargoBackendAdapter({
verbose: Deno.env.get("VERBOSE") === "1",
});
}
}
/**
* Get or create global backend instance
*/
export async function getTestBackend(): Promise<TestBackend> {
if (!globalBackend) {
globalBackend = createTestBackend();
await globalBackend.start();
}
return globalBackend;
}
/**
* Convenience function for tests - runs test with backend
*/
export async function withTestBackend<T>(
testFn: (backend: TestBackend, tempDir: string) => Promise<T>
): Promise<T> {
const backend = await getTestBackend();
const tempDir = await Deno.makeTempDir({ prefix: "windmill_cli_test_" });
try {
await backend.reset();
if (backend.seedTestData) {
await backend.seedTestData();
}
return await testFn(backend, tempDir);
} finally {
await Deno.remove(tempDir, { recursive: true });
}
}
/**
* Cleanup function for test suites
*/
export async function cleanupTestBackend(): Promise<void> {
if (globalBackend) {
await globalBackend.stop();
globalBackend = null;
}
}
// Re-export for convenience
export type { CargoBackendConfig } from "./cargo_backend.ts";
export type { ContainerConfig } from "./containerized_backend.ts";