mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-19 16:02:14 +00:00
sql-s3-streaming-issue
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ruben Fiszel
|
c258928ab6 |
fix(cli): reconcile case-only path drift during sync on case-insensitive filesystems (WIN-2020) (#9485)
* fix(cli): reconcile case-only path drift during sync on case-insensitive filesystems Windmill paths are case-sensitive, but Windows (and the default macOS setup) use case-insensitive filesystems. The real-world failure behind WIN-2020 is not a user authoring both f/Caps and f/caps — it is a single capitalized folder whose on-disk casing silently drifts (Windows stores and reports whatever case the directory was first created with, regardless of the server's path). The diff then sees the drifted local path as a brand-new item and emits a destructive "delete f/Caps + add f/caps" pair, so a capitalized folder appears to vanish and a lowercase clone shows up out of nowhere — and a push can clobber the real server item. Fix: on a case-insensitive filesystem, reconcile case-only drift before diffing. The server's path casing is authoritative, so compareDynFSElement now rewrites local keys that differ from a remote key only by case to the server's casing (canonicalizeCaseInsensitiveKeys), making the diff treat them as the same item. Case-insensitivity is auto-detected by probing the sync directory, with a WMILL_CASE_INSENSITIVE_FS=true/false override to force Windows behaviour (or emulate it for tests / cross-platform repos) on any host. Reconciled paths are summarized in a single info line. Genuinely unrepresentable collisions — two DISTINCT server paths that differ only by case — cannot be canonicalized to one target; those are detected and warned about on every platform so a case-sensitive-Linux author learns their tree won't round-trip for a Windows/macOS teammate. Tests: - Pure unit tests for findCaseInsensitiveCollisions, canonicalizeCaseInsensitiveKeys and summarizeCaseRewrites (platform independent). - An end-to-end drift test that runs on BOTH CI jobs: on the Windows runner it exercises the real case-insensitive NTFS + auto-probe; on Linux it reproduces the drift via rename, asserts the destructive phantom appears without the fix, and asserts a clean no-op push with the fix forced on. Fixes WIN-2020 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): canonicalize local-only descendants of drifted folders; dedupe nested case collisions Address two review findings on the WIN-2020 case-insensitive sync fix: P1 (correctness): canonicalizeCaseInsensitiveKeys previously only rewrote local keys with an exact full-path remote match. A brand-new local file under a drifted folder (e.g. adding f/caps/New.ts when the server has f/Caps but no f/caps/New.ts) had no exact match, so it kept its lowercase casing and push uploaded it as-is — recreating f/caps beside f/Caps and reintroducing the very collision the fix prevents. Canonicalization is now segment-by-segment against a trie of remote paths, so local-only descendants inherit the longest unambiguous server folder casing. A segment is only adopted when the server casing is unambiguous; at the first ambiguous/unknown segment the remainder keeps local casing. The original key's separator style is preserved so rewritten keys still round-trip. P2 (nit): findCaseInsensitiveCollisions reported the folder group AND a nested per-file group when case-variant folders held same-named files, inflating the "Found N path(s)" count. It now reports only the shallowest clash (drops a group whose ancestor prefix is itself a collision). Tests: add unit coverage for the new-file-under-drifted-folder rewrite, the stop-at-first-unguided-segment behavior, and shallowest-only collision reporting; extend the e2e drift test to assert a new item added under the drifted folder is pushed under the server's folder casing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
1c56148714 |
fix(cli-tests): stabilize flow lock-gen race + Windows path (#9080)
* fix(cli-tests): stabilize flow lock-gen race + Windows path Three CLI test failures on the latest main, all flaky on CI: 1. `Mixed Case Paths: pull and push flow with capitalized folder` and `Integration: Mixed scripts and flows with nonDottedPaths are idempotent`: flow create/update queues an async FlowDependencies job that fills inline-script lockfiles and rewrites flow.value. The tests pulled/pushed before the worker finished, so dry-run idempotency saw phantom `*.inline_script.lock` adds and `flow.yaml` edits. Added a `waitForFlowDependencyJob` helper that polls `/flows/get` for the latest `dependency_job` and `/jobs_u/completed/get` until it lands, and called it after each API/CLI flow write in both tests. 2. `HEADERS env var is forwarded on every CLI fetch` (Windows-only, added in #9075): the new test built the CLI entrypoint via `new URL("..", import.meta.url).pathname`, which yields `/C:/...` on Windows and `Bun.spawn` rejected before reaching the proxy, leaving `rejectedRequests.length` at 0. Switched to `fileURLToPath` + `node:path.join` to match `cargo_backend.ts`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli-tests): use /flows/deployment_status to actually wait for dep job CI reviewers (Claude, Codex) flagged the prior `waitForFlowDependencyJob` as a no-op: it read `flow.dependency_job` from `/api/w/{ws}/flows/get`, but `Flow` / `FlowWithStarred` (backend/windmill-types/src/flows.rs:20-60) do not include that field. The helper exited on the first iteration without polling. Switch to `/api/w/{ws}/flows/deployment_status/p/{path}`, which returns `{ lock_error_logs, job_id }`. `job_id` is the FlowDependencies UUID written into `deployment_metadata` in the same tx as the dep-job push (backend/windmill-api-flows/src/flows.rs:660-672 and :1275-1292), so by the time the create/update API call returns, the response carries the latest dep-job UUID. Then poll `/jobs_u/completed/get/{job_id}` as before. Local runtime for `mixed_case_paths.test.ts` jumps from ~9s to ~32s, confirming the helper now actually waits instead of returning immediately. The 404 short-circuit in `sync_pull_push.test.ts` still works — `get_deployment_status` returns 404 when the flow is absent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
4a14e9436e |
prevent async lock gen race condition in mixed case path tests (#8202)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
4fedfdfd11 |
feat(cli): add consistent get/list/new subcommands for all item types (#8047)
* feat(cli): add consistent get/list/new subcommands for all item types Make the CLI consistent so every item type (script, flow, app, resource, resource-type, variable, schedule, folder, trigger) supports get/list/new subcommands, enabling the CLI to be used as a full API client in bash scripts with jq piping. - Add --json flag to all list commands for machine-readable output - Register explicit "list" subcommand alongside default action - Add "get <path> [--json]" subcommand to fetch single items from API - Rename "bootstrap" to "new" for script/flow, keep "bootstrap" as alias - Add "new" subcommand for resource, resource-type, variable, schedule, folder, and trigger to create local template YAML files - Update cli-commands skill documentation for wmill init - Add integration tests for all new commands Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * feat: install wmill CLI in Docker images and use it for bash variable/resource access - Install windmill-cli via bun in all Dockerfiles that include bun - DockerfileCli: switch from node:slim to oven/bun:slim - CLI: auto-configure from WM_WORKSPACE/WM_TOKEN/BASE_INTERNAL_URL env vars as last-resort fallback when no workspace is configured - Frontend: replace curl-based bash snippets with wmill variable/resource get - Add backend integration tests for wmill CLI in bash scripts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): install windmill-cli in backend test workflow Ensures wmill is available on PATH for bash integration tests that use `wmill variable get` and `wmill resource get`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(cli): replace @std/* Deno dependencies with Node.js equivalents Replace @std/log with a lightweight custom logger (core/log.ts), @std/path with node:path, and @std/yaml with the yaml npm package. Also fix process hang on exit, add --node option to install_dev.sh, and add missing hasRequiredPermissions to NpmProvider. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * all * all * refactor(cli): replace @ayonli/jsext and @std/encoding with lightweight alternatives Replace @ayonli/jsext (8.4MB) with tar-stream (32kB) for tar creation, replace @std/encoding with Node.js Buffer.toString("hex"), and fix @windmill-labs/shared-utils to use direct npm instead of JSR mirror. Also resolve merge conflicts in sync.ts and fix pre-existing type errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): use singleQuote YAML output and pass yamlOptions in gitsync pull The yaml library defaults to double quotes, but the codebase (and tests) expect single-quoted strings. Add singleQuote: true to yamlOptions and pass yamlOptions to gitsync-settings pull writeFile calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * all * fix(cli): address code review feedback - Install CLI from source in backend tests instead of npm - Fix script bootstrap catch block to re-throw "File already exists" - Add type-safe local variable after trigger kind validation - Use created_by instead of policy.on_behalf_of for app get output - Note --kind is recommended for faster trigger lookup in help text - Document node symlink purpose in Dockerfiles Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): use /usr/bin for wmill wrapper to ensure it's in PATH Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): install wmill to ~/.local/bin to avoid permission issues Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci(backend): switch to Blacksmith runner and add cargo caching - Switch from ubicloud-standard-16 to blacksmith-16vcpu-ubuntu-2404 for faster NVMe-backed builds - Add stickydisk for cargo target directory (persistent NVMe cache across runs) - Add cache for cargo registry and git dependencies - Upgrade DuckDB FFI cache from actions/cache@v3 to useblacksmith/cache@v1 - Enable CARGO_INCREMENTAL=1 to benefit from persistent target cache Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix ci --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
a2cefdf0a2 |
refactor(cli): migrate CLI from Deno to Bun/Node.js (#8041)
* fix: only enable EE features in test backend when license key is available Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: skip EE tests without license key and exclude test-skills from test discovery Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: unskip passing tests and add duplicate (remote, workspaceId) check in addWorkspace Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(cli): migrate from Deno APIs to Node.js/Bun-compatible APIs Replace Deno-specific APIs with Node.js equivalents across the entire CLI codebase to enable running on Node.js/Bun. Switch build system from dnt to bun, update imports from jsr:/npm: prefixed to bare specifiers, and add package.json/tsconfig.json for the Node.js ecosystem. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * test(cli): expand test coverage with new integration and unit tests Add standalone_commands.test.ts covering folder list, schedule list, resource-type list/push/update, script show/run/bootstrap, and user commands. Add unit tests for filePathExtensionFromContentType and removeExtensionToPath. Add git_unit, local_encryption_unit, resource_folders_unit, and settings_unit test files. Fix schedule cron expressions (6-field format), add includeSchedules flag, improve test setup with pre-build and auto-cleanup, and support TEST_CLI_RUNTIME=node. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): replace Deno.readFile with node:fs in WASM loaders and add schema parsing tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(cli): switch WASM parsers from local files to npm packages Use published windmill-parser-wasm-* npm packages instead of local wasm/ files. A loadParser() helper uses createRequire to resolve the .wasm binary from node_modules and passes it to init() via readFileSync, avoiding fetch() and Deno.readFile() patches. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(cli): add coverage for --locks-required lint feature Add 15 tests covering the lock-checking functionality merged from main: - checkMissingLocks: standalone scripts (python, bun, bash), inline lock file resolution (valid, empty, missing), flow inline rawscripts (with/without locks, nested forloopflow), app inline scripts, raw apps without backend folder - runLint --locks-required integration: reports issues when locks missing, skips checks when flag absent, passes when locks exist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci(cli): replace Deno with Bun in CI workflows - cli-tests.yml: remove Deno setup, use `bun test` instead of `deno test`, add `bun install` step for dependency installation - npm_on_release.yml: replace Deno setup with Bun setup for CLI publishing - build.sh: add `bun install` before building so CI has dependencies Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): pre-start backend in test preload and remove Deno test leftovers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): normalize path separators for Windows compatibility Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * more tests + windows * ci(cli): use Blacksmith runner for Windows tests Switch test-windows job from windows-latest to blacksmith-16vcpu-windows-2025 for faster CI execution. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): fix Windows path separator expectations in unit tests buildMetadataPath and extractResourceName normalize to forward slashes internally, so tests should not expect platform-specific separators in their output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): fix Windows CI test failures for dev_server and script_run Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): set BUN_PATH and NODE_BIN_PATH for backend worker on Windows Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci(cli): add SSH debug step on Windows test failure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): use native path separators for ignore check in dev mode on Windows Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
a22d179903 |
add GitHub Actions CI and raw app sync tests (#7677)
* feat(cli): add GitHub Actions CI and raw app sync tests - Add CLI tests GitHub Action that runs on Linux and Windows - Add build check job that runs on CLI and openapi.yaml changes - Uses Rust cargo backend instead of Docker for better CI compatibility - Add cargo_backend.ts and test_backend.ts for test infrastructure - Fix Windows path separator bug in raw_apps.ts (use "/" for relative paths) - Fix PostgreSQL URL parsing in cargo_backend.ts - Update tests to use gitBranches format instead of deprecated overrides - Add raw_app_sync.test.ts for raw app sync workflow testing (ignored for now - needs EE) - Skip tests that require EE features (git sync settings, raw apps) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): Fix Windows path compatibility issues in tests - Use fromFileUrl() in cargo_backend.ts for proper Windows path handling - Normalize path separators to forward slashes in resource_folders.ts - Fix readDirRecursive to return normalized paths in test helper - Use forward slashes consistently in buildMetadataPath and detection functions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): Use SEP in test assertions instead of modifying logic - Revert resource_folders.ts to use SEP as intended - Update test assertions to use SEP for platform-specific paths - Keep readDirRecursive normalization for consistent test comparisons Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): Use SEP for all path separators in test assertions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): Use resolve() for proper cross-platform path handling in cargo_backend String concatenation with path separators creates malformed paths on Windows. Use path.resolve() instead for proper cross-platform path resolution. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(backend): Add cfg attributes for Windows compatibility - Add #[cfg(unix)] to anyhow::anyhow import (only used in unix cfg block) - Add #[cfg(not(windows))] to parse_file function (uses cat, only for cgroups) - Remove unused std::io import, use std::io::Result directly Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: Windows compilation + convert integration tests to withTestBackend - Fix unused import SYSTEM_ROOT in csharp_executor.rs on Windows by requiring both windows and csharp feature - Fix unused variable id in handle_child.rs on Windows by adding #[allow(unused_variables)] since id is only used in cfg(unix) code - Convert all RUN_INTEGRATION_TESTS dependent tests in sync_pull_push.test.ts to use withTestBackend pattern for automatic backend setup Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: configurable test features with CI_MINIMAL_FEATURES env var - Default: full features (zip, private, enterprise) for local development - CI mode: minimal features (zip only) when CI_MINIMAL_FEATURES=true - Add shouldSkipOnCI() helper for tests requiring EE features - Update EE-dependent tests to use shouldSkipOnCI() - Add test instructions to cli/README.md Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: enable raw app tests (not EE-dependent) Raw apps work with minimal features. 2 tests pass, 2 have test logic bugs to investigate separately: - "delete file and push" - file deletion not syncing correctly - "dry-run push shows expected changes" - JSON output parsing issue Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: gate cgroups module to Linux only cgroups are Linux-specific, the module was causing dead_code warnings on Windows compilation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(ci): add CI_MINIMAL_FEATURES env var to CLI tests workflow Set CI_MINIMAL_FEATURES=true in both Linux and Windows test jobs so the backend compiles with minimal features (zip only) and EE-dependent tests self-skip via shouldSkipOnCI(). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): raw app tests and backend startup timing - Add 5s delay after backend ready for migrations to complete - Fix dry-run JSON output parsing (handle pretty-printed JSON) - Temporarily ignore "delete file" test (needs isSuperset fix) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): raw app file deletion sync - Add deepEqual check for files in raw_apps.ts isSuperset comparison - Handle raw_app file deletions in sync.ts by re-pushing the entire app - Fix test to remove CSS import before deleting the file When deleting a file from a raw app, the sync now properly updates the backend with the new file list (excluding the deleted file). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): Windows path separators in tests Normalize paths for cross-platform comparison by converting backslashes to forward slashes before path assertions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): normalize featurePaths in multi_instance_workspace test Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(cli): add mixed case paths sync tests for Windows compatibility Add comprehensive tests for sync pull/push with capitalized folder paths to catch Windows case-insensitivity issues: - Scripts in f/MyFolder/MyScript - Flows in f/MyFlows/DataProcessor - Apps in f/MyApps/Dashboard - Variables in f/MyVars/ApiKey - Deeply nested paths with mixed case - Multiple resources in same capitalized folder - CamelCase folder names with numbers Each test verifies the full pull -> modify -> push -> verify cycle. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(cli): add idempotency check to mixed case paths tests After each push, pull again with --dry-run --json-output and verify that no changes are detected. This ensures the sync is stable and catches issues where pull/push cycles cause spurious diffs. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |