starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-08-20 08:01:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
13,639 Commits 732 Branches 1,510 Tags
fix/wac-python-cache-install-race
Commit Graph

5 Commits

Author SHA1 Message Date
Ruben Fiszel aedf369174 fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe (#8999) 
* fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix(pg): wrap encoder errors with arg context, add fallback test

Followups on #8999 review:

- Wrap rust-postgres "error serializing parameter N" failures with the arg
  name, JSON value kind, and asserted Postgres type plus a hint about an
  explicit cast — so users see actionable context instead of an opaque
  WrongType.
- Drift-prevention meta-test: assert otyp_to_pg_type and convert_val agree
  on the Type for every recognised arg_t when the JSON value matches its
  natural Rust kind. Catches future drift if either side changes.
- Integration test for the prepare + query_raw fallback path: confirms
  unrecognised arg_t (custom enum) is routed through prepare and the
  server-resolved type appears in the failure surface — flips into a
  test failure if a regression accidentally routes unrecognised types
  through query_typed_raw.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pg): add otyp_inferred flag + regex-based placeholder renumbering

Two follow-ups from the review of #8999:

1. **Issue #1 (Number/Bool + explicit text decl in WHERE)**

   Add `Arg::otyp_inferred: bool` to the parser. The PG SQL parser sets
   it `true` only at the "no info → fall back to text" site (bare `$N`,
   no inline cast, no `-- $N (TYPE)` decl). All other arg sources keep
   it `false`.

   In `convert_val` this flag distinguishes:
   - explicit text-like target (`-- $1 (text)` or `$1::text`) — coerce
     `Bool`/`Number` → `Box<String>` so `WHERE text_col = $1` works
     (`text = text` operator). Pre-#8988 behaviour, restored.
   - parser-default text (bare `$N`) — bind the value's natural Rust
     type so the regression case (`Value::Bool` against a real `bool`
     column via `CAST AS bool`) keeps working.

   `Arg` is in `windmill-parser`; the new field has `#[serde(default)]`
   so persisted signatures stay backward-compatible.

2. **Issue #4 ($5/$50 substring rewrite collision)**

   Replace the per-index `String::replace` chain (which turned `$50`
   into `$10` when oidx=5 was processed first) with a single regex
   pass. `\d+` is greedy, so `$5` and `$50` match as distinct units;
   indices outside the mapping are left intact.

3. Tests:
   - parser: `test_parse_pgsql_otyp_inferred_flag` covers bare/inline-
     cast/decl/mixed shapes.
   - executor unit: `convert_val_bool_against_every_arg_t` and
     `convert_val_*_number_*` split each text-like target into explicit
     vs inferred expectations.
   - executor unit: `renumber_sparse_placeholders_no_collision`.
   - integration: `test_postgresql_arg_type_combinations` adds 4 cases
     covering decl(text)+Number/Bool in WHERE, bare $1+Bool, and
     sparse positional args ($5/$50).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pg+sdk): enum support, extended String arms, position-aware $N rewrite, SDK quality

Backend:

1. **`AnyTextValue` ToSql/FromSql wrapper**: vanilla `tokio_postgres`'s
   `ToSql for String` / `FromSql for String` reject `Kind::Enum` and
   `Kind::Domain` even though the wire format is plain UTF-8. The wrapper
   accepts those kinds in both directions. End result: explicit
   `$1::my_enum` / `CAST($1 AS my_enum)` casts now round-trip without the
   ugly `CAST($1::text AS my_enum)` workaround, AND `SELECT enum_col`
   results come back as JSON strings instead of erroring at the FromSql
   layer.

2. **#10 — Value::String → numeric/real/double/oid/bool**. Without these
   arms, a string-encoded value (`"3.14"`, `"true"`) for a non-text /
   non-temporal arg_t fell through to `Box<String> + TEXT`, which then
   failed at the server (no implicit cast text→numeric in expression
   context). Now strings are parsed into the matching native type with
   clear error messages on parse failure.

3. **Position-aware `$N` rewrite**: replaces the regex-based renumbering
   (which fixed the `$5/$50` substring collision but still walked through
   string literals and comments, mangling `'price: $5'` etc.) with a
   walk over `parse_pg_statement_arg_positions` — the same
   string/comment/dollar-quote-aware tokenizer used for index discovery.
   Adds `parse_pg_statement_arg_positions` to the parser's public API.

SDK:

4. **BigInt support**: `JSON.stringify(BigInt)` throws. The SDK now
   stringifies bigints before serialisation; the executor accepts
   numeric strings into BIGINT arg slots via the existing
   `Value::String → INT8` parsing arm. SDK-side `inferSqlType` is split
   so `BigInt` always resolves to `BIGINT` (was reaching
   `Number.isInteger(BigInt)` which returns false → wrong default).

5. **Homogeneous array auto-tag**: `${[1,2,3]}` against an `int[]` column
   now emits `$1::BIGINT[]` instead of `$1::JSON`. Detection covers
   primitive types only (number / bigint / string / boolean); mixed or
   nested arrays still fall back to JSON. Mixed int/float widens to
   `DOUBLE PRECISION[]`.

6. **`.query()` positional bug**: previously the `.query()` method
   abused the template-tag builder, which appended `$N::TYPE` after the
   user's literal SQL string instead of binding by position
   (`SELECT $1, $2` became `SELECT $1, $2$1::BIGINT`). Now `.query()`
   builds the executor-shaped content directly: a `-- $N argN (TYPE)`
   declaration block followed by the user's SQL verbatim.

Tests:

- Parser: `test_parse_pg_statement_arg_positions_skips_strings_and_comments`
  asserts string literals, comments, and dollar-quoted blocks don't
  produce positions (so renumbering doesn't mangle them).
- Executor unit: `renumber_sparse_placeholders_no_collision_no_string_mangling`
  uses the new position-aware path and includes string-literal + comment
  + `$$…$$` cases. Existing convert_val tests grow to cover new
  String→numeric/real/double/oid/bool arms.
- Integration: `test_postgresql_arg_type_combinations` adds 13 cases
  (enum round-trip both directions, string→numeric/real/double/bool/oid,
  string-literal `$N` non-mangling). The prepare-fallback test now
  asserts SUCCESS (not failure) for enum encoding via AnyTextValue.
- SDK: new `typescript-client/tests/sqlUtils.test.ts` (42 tests)
  exhaustively covering inferSqlType primitives + arrays,
  parseTypeAnnotation, datatable() template tag (with all the new
  shapes — BigInt, homogeneous arrays, RawSql, schema preamble),
  datatable().query() positional, and ducklake() shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pg): replace DISCARD ALL with curated reset (preserves typeinfo cache)

Found while exhaustively probing custom-type DX: every cached-connection
reuse was running `DISCARD ALL`, whose included `DEALLOCATE ALL`
deallocates *all* prepared statements server-side — including the typeinfo
statements that tokio_postgres caches per-Client to resolve custom enum /
domain Oids. tokio_postgres still held `Statement` objects whose names
the server had forgotten, so the next custom-type query failed with
intermittent "prepared statement \"sN\" does not exist" errors. The
failure was easy to reproduce: any sequence that forced typeinfo lookup
for two different custom-type kinds on the same cached connection (e.g.
enum followed by domain) would hit it.

Replace `DISCARD ALL` with a curated reset that explicitly targets the
state we actually care about, *without* touching prepared statements:

  RESET ALL                     — GUC parameters (search_path, application
                                  _name, statement_timeout, …)
  RESET SESSION AUTHORIZATION   — undoes both `SET SESSION AUTHORIZATION`
                                  and `SET ROLE` (RESET ALL does NOT —
                                  these aren't GUC parameters, so without
                                  this an elevated role from a previous
                                  job would silently leak)
  UNLISTEN *                    — drops LISTEN registrations
  CLOSE ALL                     — closes open cursors

Trade-off: temp tables, advisory locks (session-scoped), and user-created
PREPARE statements may persist across cached-connection reuse — rare in
datatable / PG-script workloads. tokio_postgres's typeinfo cache survives
intact, so custom enum / domain queries are fast on subsequent reuse.

Tests:
- `test_postgresql_custom_types_on_cached_connection` — runs 10×
  alternating enum + domain queries on a cached connection. Pre-fix this
  failed with `prepared statement "sN" does not exist` after the first
  reuse; post-fix passes.
- `test_postgresql_set_role_does_not_leak_across_cached_connection` —
  switches `SET ROLE` and `SET SESSION AUTHORIZATION` to a non-postgres
  role, then runs a follow-up job and asserts current_user/session_user
  are restored. Specifically catches the case where someone might switch
  back to `RESET ALL` alone (which doesn't cover SET ROLE / SESSION
  AUTHORIZATION) and silently introduce a permission-leak vector.
- All existing session-isolation tests
  (`test_postgresql_cached_connection_resets_session`,
   `test_postgresql_single_worker_session_isolation`,
   `test_postgresql_100_jobs_cached`) continue to pass.

Found via end-to-end probing of datatable / PG-script DX, not previously
covered: the existing isolation tests only did `SET ROLE postgres`, the
connecting user, so the leak was invisible.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pg): address PR #8999 review (cubic + claude)

cubic (P1, real bug):
- `convert_vec_val` for `timetz` array asserted `Type::TIMETZ_ARRAY`, but
  chrono `NaiveTime` only encodes for TIME (same caveat as the scalar
  arm). Switch to `Type::TIME_ARRAY`; rely on PG's implicit `time→timetz`
  assignment cast at the column site. Add an explicit unit test.

claude (#1, silent failure → explicit error):
- `Bool` + explicit `(char)` / `(character)` decl previously silently
  bound BOOL, hoping the server would cast at the use site — but PG has
  no implicit `bool→char` and the resulting error
  ("operator does not exist: bool = char") was opaque. Now error at
  bind time with an actionable hint to use `bool` decl or pass the
  value as a "t"/"f" string.

claude (#2, asymmetry doc):
- Object/Array still coerce to text on `matches!(typ, Typ::Str(_))`
  (covers both explicit AND inferred-default text), unlike Bool/Number
  which key on `explicit_text_target`. The asymmetry is intentional
  (no implicit `jsonb → text` cast in expression context vs PG having
  implicit `bool/int → text` casts) — added a body comment so future
  maintainers don't try to "align" them.

claude (#3, perf):
- `parse_pg_statement_arg_indices` and `parse_pg_statement_arg_positions`
  walked the SQL tokenizer twice. Fold into a single pass that derives
  the index set from the position list.

claude (#4, fmt drift):
- `cargo fmt` over the parser crates I touched with perl scripts in the
  earlier commit (windmill-parser-{sql,bash,ts,go,php,java,csharp,nu,py,
  rust,graphql,yaml,r}). Net cosmetic.

claude (#5, parseTypeAnnotation):
- One-line caveat in the SDK's `parseTypeAnnotation` that the returned
  string is presence-only (e.g. `${x}::DOUBLE PRECISION` returns
  `"DOUBLE"`, `CAST(${x} AS int)` returns `"int)"` — neither matches a
  real PG type, but the only consumer just checks `!== undefined`).

While here — discovered + fixed independently while exhaustively probing
DX:

- **Replace `DISCARD ALL` with curated reset** (`RESET ALL; RESET
  SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;`). DISCARD's
  `DEALLOCATE ALL` killed tokio_postgres' typeinfo cache, producing
  intermittent `prepared statement "sN" does not exist` errors on
  custom-type queries after cached-conn reuse. New regression tests:
  `test_postgresql_custom_types_on_cached_connection` and
  `test_postgresql_set_role_does_not_leak_across_cached_connection`
  (the latter catches the case where someone might switch back to
  `RESET ALL` alone and silently introduce a permission-leak vector —
  RESET ALL doesn't cover SET ROLE / SET SESSION AUTHORIZATION).

- **ISO-8601 timestamp results** (`pg_cell_to_json_value`). Pre-fix
  `TIMESTAMP` was rendered with a space separator ("2024-01-15 10:30:00")
  and `TIMESTAMPTZ` with " UTC" suffix ("2024-01-15 10:30:00 UTC") —
  neither parseable by `date-fns parseISO`, JavaScript `new Date()` is
  lenient enough to handle them but several frontend `App*Input.svelte`
  components use parseISO and fail silently. Switched to ISO-8601 with
  `T` separator and `+00:00` offset; arg-parsing path still accepts the
  legacy " UTC" suffix for back-compat.

Test coverage:
- 17/17 unit (`pg_executor::tests`)
- 9/9 integration (`backend/tests/worker.rs`, `test_postgresql_*`)
- 27/27 parser (`windmill-parser-sql`)
- 42/42 SDK (`typescript-client/tests/sqlUtils.test.ts`)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pg): bounded one-shot warning on numeric precision loss + ISO-8601 + NaN handling

Found while probing PG-script DX with millions of numeric cells:

1. **Numeric precision-loss warning**: `numeric` results are still serialised
   as JSON Number (back-compat — switching to JSON String would silently
   break user code doing arithmetic on results), but we now detect
   `Decimal -> f64 -> Decimal` round-trip failure and emit a single
   job-log warning recommending a `::text` cast in the SQL. Bounded by
   `NUMERIC_PRECISION_CHECK_BUDGET = 256` cells per query (one atomic
   load + one fetch_sub on the hot path; first lossy value
   short-circuits to a single load thereafter). Worst-case overhead on
   a 1M-cell numeric-heavy query: ~25µs of checks + 5ns × N atomic
   loads (vs. ~100ms unbounded).

2. **ISO-8601 timestamps**: `pg_cell_to_json_value` previously returned
   `"2024-01-15 10:30:00"` (TIMESTAMP) and `"2024-01-15 10:30:00 UTC"`
   (TIMESTAMPTZ) — neither parseable by date-fns `parseISO`, which is
   what the apps `App*Input.svelte` components use, so timestamp values
   silently failed to round-trip into date pickers. Switch to ISO-8601
   (`T` separator + `+00:00` offset) on the result side; arg-parser
   continues to accept the legacy `" UTC"`-suffixed format for
   back-compat.

3. **Float NaN / Infinity results**: `Number::from_f64` returns None for
   NaN / ±Inf, which `pg_cell_to_json_value` was raising as
   "invalid json-float" — failing the *entire* query if any cell held
   one of these special values. Now serialise them as JSON strings
   ("NaN", "Infinity", "-Infinity") and let the rest of the row come
   through. Arg-side: `s.parse::<f64>()` already accepts the same
   strings.

Tests:
- `decimal_fits_f64_losslessly_predicate` — covers fits / doesn't-fit
  cases for the precision-loss predicate.
- `precision_check_budget_caps_per_query_overhead` — locks in the
  budget cap and the loss-flag short-circuit.
- All 9 PG integration tests + 17 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pg): add pg_advisory_unlock_all to reset; warn on missing args; honor decl defaults

While probing PG-script DX further found three more frictions:

1. **Advisory lock leak** (cubic P2): switching from `DISCARD ALL` to
   `RESET ALL; RESET SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;`
   meant session-scoped advisory locks (`pg_advisory_lock`) leaked
   across cached-connection reuse. Add `SELECT pg_advisory_unlock_all()`
   to the chain — `DISCARD ALL` covered this implicitly via
   `DISCARD PLANS / DEALLOCATE / pg_advisory_unlock_all` and we lost it
   in the switch.

2. **Missing-arg silent NULL**: an arg declared in the SQL (e.g.
   `-- $1 amount (numeric)`) but not provided in the args object was
   bound as NULL with no error / warning. Misspelling the key in the
   args object silently produced a row of NULLs — a notorious DX
   debugging trap. Now: collect the names of declared-but-missing
   args during dispatch and emit a single one-shot warning to the job
   logs at end-of-query naming each one. Bound NULL is preserved for
   back-compat.

3. **Declaration defaults ignored**: `-- $1 a (int) = 5` carries
   `arg.default = Some(Number(5))`, but the dispatch fell straight to
   NULL when the arg was missing. Now: respect the default —
   user-supplied value > declaration default > NULL. Also fixes the
   warning logic above (only warn for args that *don't* have a default).

Tests: existing 19 unit + 9 integration pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pg): multi-word PG types with [] suffix lost the array-ness; array arms accept stringified values

Two more frictions found while probing SDK end-to-end against a real
datatable resource:

1. **Multi-word array types lose the [] suffix in the parser**.
   `transform_types_with_spaces` recognises aliases for "double
   precision", "character varying", "timestamp with time zone", etc.
   but its return type was `&'a str` — only the bare alias, never with
   a trailing `[]`. The `RE_CODE_PGSQL` regex's `\w+` captures stop at
   the first space, so the regex's own `(?:\[\])?` array-suffix branch
   sees only `"double"` (not `"double precision[]"`); the `[]` was
   silently lost. Result: `$1::double precision[]` (which the SDK now
   emits for homogeneous float arrays via the new auto-tag) routed
   through `Value::Array → Type::JSONB` and the server failed with
   "cannot cast type jsonb to double precision[]".

   Fix: switch `transform_types_with_spaces` to return `Cow<'a, str>`
   and re-check the trailing bytes after a multi-word match. If they
   start with `[]`, return `format!("{alias}[]")` — Owned. Single-word
   types and the no-match path keep returning Borrowed slices, so no
   allocation in the hot path.

2. **Array arms in `convert_vec_val` rejected stringified values for
   numeric / int* / bool / oid / real / double**. The scalar `convert_val`
   already parses strings into the matching native type for these arg_ts,
   but the array variant only accepted JSON-native counterparts. Sending
   `["1.5", "2.5", "3.5"]` against `$1::numeric[]` (e.g. via `unnest` for
   bulk loading, or `JSON.stringify(BigInt[])` round-trip) failed with
   "Mixed types in array". Now the array arms mirror the scalar ones —
   `as_<native>().or_else(|| as_str().and_then(parse))` — so both shapes
   round-trip cleanly.

Tests: 19 unit + 9 integration pass; existing parser tests cover the
multi-word array forms (the regex-cap behaviour didn't break for
single-word types, and Cow plumbing is transparent to all callers).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(parsers): add otyp_inferred field to Arg literals in tests + 3 missed src files

CI failures: the perl-driven sweep that added `otyp_inferred: false` to
every `Arg { ... }` literal when I introduced the field in the parser
schema covered `src/lib.rs` files but missed:

  - parsers/windmill-parser-bash/src/lib.rs       (mass-edited but a
    later format pass un-applied a few sites)
  - parsers/windmill-parser-go/src/lib.rs         (same)
  - parsers/windmill-parser-graphql/src/lib.rs    (same)
  - parsers/windmill-parser-nu/tests/tests.rs     (test file — not
    swept the first time)
  - parsers/windmill-parser-ts/tests/tests.rs     (test file — same)

Also tightened the regex to handle `oidx: None` without the trailing
comma (some test files had the field as the last initialiser line).

`cargo build --features <CI feature combo> --workspace --all-targets`
is clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(sdk): Date → TIMESTAMPTZ; NaN / ±Infinity → string

Two more frictions found while running the actual SDK end-to-end against
a live datatable resource:

1. **JS `Date`** fell into the typeof "object" branch and was tagged
   `::JSON`. It worked accidentally for `${date}::timestamptz` via PG's
   `json → text → timestamptz` implicit cast chain, but `${date}` against
   a `timestamptz` column without a user-supplied cast bound the value
   as a JSON string and the comparison `timestamptz = json` failed. Now:
   `inferSqlType` recognises `Date` and tags `::TIMESTAMPTZ`;
   `serializeArgValue` emits `Date.toISOString()` so the executor's
   `Value::String → TIMESTAMPTZ` arm parses it cleanly.

2. **JS `NaN` / `±Infinity`** silently became NULL. `JSON.stringify(NaN)`
   returns `"null"` per the JS spec, so the value reached the executor as
   JSON null — the SDK's `::DOUBLE PRECISION` tag then bound a NULL
   double. Fix: detect non-finite numbers in `serializeArgValue` and
   stringify them as `"NaN" / "Infinity" / "-Infinity"`. The executor's
   `Value::String → FLOAT8` arm (`f64::from_str`) accepts these literals
   directly, and the result-side already renders the values as JSON
   strings (matching round-trip).

SDK unit tests grow from 42 → 44 passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(pg): integration coverage for multi-word arrays + stringified array elements

Locks in the two array fixes from the previous commit
(`fix(pg): multi-word PG types with [] suffix lost the array-ness`)
with end-to-end cases in `test_postgresql_arg_type_combinations`:

- `double precision[]`, `character varying[]`, `timestamp without time
  zone[]` — verifies the parser keeps the `[]` suffix after multi-word
  alias resolution.
- `numeric[]` / `int[]` / `bool[]` from stringified primitives — verifies
  the array arms of `convert_vec_val` apply the same string-coercion
  the scalar arms do.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* style: fix indentation drift on otyp_inferred lines

cargo fmt cleanup of leftover indentation where the perl-driven sweep
that introduced the otyp_inferred field landed at the wrong column.
No behaviour change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
 2026-05-01 17:08:59 +00:00
Ruben Fiszel 0317d5891c feat: add powershell common parameters support (#8683) 
* feat: add powershell common parameters support (-Verbose, -Debug, -ErrorAction, -WhatIf)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add powershell common params to script editor test panel

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: detect CmdletBinding from code instead of schema in script editor

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: ignore commented-out CmdletBinding in powershell detection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use preference variables for -Verbose/-Debug instead of CLI args

Verbose/Debug output goes to PowerShell stream 4/5 which isn't captured
by the 2>&1 redirect. Setting $VerbosePreference/$DebugPreference in the
wrapper scope propagates to child scripts and output flows through the
host to stderr, which Windmill captures as logs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use *>&1 to capture all powershell streams including verbose/debug

The previous 2>&1 only captured error stream. Verbose (stream 4) and
debug (stream 5) output was silently lost. Using *>&1 redirects all
streams to success stream so they flow through Tee-Object into logs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use targeted stream redirects (4>&1 5>&1 2>&1) instead of *>&1

*>&1 breaks $PSCmdlet.ShouldProcess() by redirecting internal streams.
Only redirect verbose (4), debug (5), and error (2) to success stream.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert to 2>&1 redirect — stream 4/5 redirects break powershell

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use 4>&1 5>&1 for verbose/debug capture, remove WhatIf support

Stream 4/5 redirects capture verbose/debug in the pipeline. WhatIf is
removed because $PSCmdlet.ShouldProcess() doesn't work when scripts
are invoked through Windmill's wrapper.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: redirect verbose/debug to files to keep result pipeline clean

Verbose (4) and debug (5) streams are redirected to separate log files
during script execution, then output via Write-Host after the script
completes. This keeps them out of the Tee-Object pipeline (used for
result extraction) while still showing them in the job logs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: output verbose/debug to stderr via Console.Error for log capture

Write-Host goes to stdout which gets mixed with result output and
truncated by OSS log threshold. Using [Console]::Error.WriteLine()
writes to stderr which Windmill captures separately as logs, with
VERBOSE:/DEBUG: prefixes for clarity.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: redirect script output to file only, send verbose/debug to stdout

The OSS log storage has a 9KB threshold. Previously, Tee-Object sent
the full JSON result to both stdout (logs) and the pipe file, eating
the log budget. Now script output goes only to the pipe file (> $pipe),
and only verbose/debug messages go to stdout for the log viewer.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: preserve original Tee-Object behavior, append verbose/debug after

Keep the original wrapper behavior (Tee-Object to stdout + pipe file).
Only add 4>verbose.log 5>debug.log to capture those streams, and
output them at the end of logs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: inject preference vars into main.ps1 instead of CLI args

Passing -Verbose/-Debug as CLI args causes PowerShell module loading
to emit verbose noise. Instead, inject $VerbosePreference/$DebugPreference
inside main.ps1's try block so they only affect user code. Stream 4/5
are still redirected to files in the wrapper for log output.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: restore common param toggles from previous job args on Run Again

Extract _wm_ps_* keys from loaded args and initialize the toggle
states in PowerShellCommonParams. Also strip them from main args
so they don't appear as unknown schema form inputs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: show active common param badges when section is collapsed

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: inject ErrorAction as preference variable instead of CLI arg

-ErrorAction as a CLI arg only affects the caller, not the script's
internal error handling. Setting $ErrorActionPreference inside main.ps1
correctly overrides the default 'Stop' behavior for the user's code.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: ensure full backward compatibility with existing powershell scripts

- Only filter common param names when [CmdletBinding()] is present
  (without it, $Verbose etc. are regular user-defined parameters)
- Only add 4>verbose.log 5>debug.log and log output lines when common
  params are actually enabled — original wrapper is unchanged otherwise

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: lighter styling for common params section

Replaced heavy Section component with a subtle inline chevron toggle
labeled "Common parameters". Smaller text, secondary color, indented
options. Badges still show when collapsed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: rename section to CmdletBinding parameters

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add ..Default::default() to windmill-parser-r (new parser from main)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: missing comma in graphql parser test + merge main

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add missing commas before ..Default::default() in parser tests

Merge from main brought test constructors with formatting issues
from the original automated script (missing comma between last field
and ..Default::default()).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: restore comment markers in nu parser test that script broke

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review — ignore commented CmdletBinding, clear stale params

1. Parser: strip comment lines before detecting [CmdletBinding()] to
   avoid false positives from commented-out attributes
2. RunForm: always assign psCommonParams (not just when non-empty) so
   stale settings from a previous run don't leak into later runs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 13:03:22 +00:00
Ruben Fiszel 31d6660d56 feat: script module mode with CLI sync, preview, and WAC UI improvements (#8380) 
* feat: add script module mode with folder model for Bun and Python

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add missing modules field to RawCode in bun_executor

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* sqlx

* feat: enrich WAC templates with checkpoint and replay semantics

Add prominent comments explaining that all computation must happen
inside task/step/taskScript or it will be replayed on resume/retry.
Clarify that waitForApproval does not hold a worker and that
approve/reject URLs are available in the timeline step details.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(cli): script module sync idempotency, per-module hash tracking, and preview support

- Fix pull→push idempotency: use `??` instead of `||` for module lock
  field so empty strings are preserved (matches API's `lock: ""`)
- Add per-module hash tracking in wmill-lock.yaml following the flow
  inline script pattern (SCRIPT_TOP_HASH + per-module subpath hashes)
- Selective module lock regeneration: only regenerate locks for modules
  whose content actually changed, not all modules
- Use unfiltered rawWorkspaceDependencies for module hashes to match
  what updateModuleLocks passes to fetchScriptLock
- Show changed module names in stale script output for clarity
- Add module support to `script preview` command: read modules from
  __mod/ folder and pass them in the preview API request
- Add preview tests for taskScript pattern (flat and folder layout)
- Update test assertion for module stale detection output

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(frontend): WAC UI improvements — reorder templates, module tab rename, import consolidation

- Reorder WAC template buttons: TypeScript before Python in
  ScriptBuilder, CreateActionsScript, and CreateActionsFlow
- Remove dropdown items from +Script button (simplify to direct link)
- Move "Import Workflow-as-Code" to +Flow dropdown with dedicated drawer
- Add module tab rename: pencil icon on hover opens popover with
  validation, fixed-width icon container prevents layout shift

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: remaining module-mode changes from working branch

- Backend parser updates for WAC detection
- CLI sync/types updates for raw app path and module support
- Frontend UI polish (Dev.svelte, ScriptRow, script hash page)
- Test fixture updates

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(cli): add test for module modification detection in generate-metadata

Verifies that modifying a single module file re-triggers stale
detection and only the changed module is listed, not all modules.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): critical fixes from PR review

- Fix hardcoded dev path in bun_executor.rs WAC v2 wrapper — use
  "windmill-client" import instead of absolute filesystem path
- Fix missed no_main_func → auto_kind rename in parser TS test
- Add modules column to clone_script SQL (windmill-common and
  windmill-api-workspaces) so cloned scripts retain their modules
- Add modules: None to RawCode structs in worker tests
- Restore complete sqlx cache (merge main's cache + our new queries)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): fix clone warning treated as error in CI

Change `.clone()` on double reference to `*k` dereference in
scripts.rs hash implementation. Update sqlx cache with new query
hashes from modified clone_script SQL.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): use published parser wasm versions for CI build

The local file:// paths for windmill-parser-wasm-py and
windmill-parser-wasm-ts don't exist in the Cloudflare Pages build
environment. Revert to published npm versions (1.655.0).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): update parser wasm packages to 1.657.2

Use newly published windmill-parser-wasm-ts and windmill-parser-wasm-py
v1.657.2 which include auto_kind/WAC detection changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): regenerate package-lock.json for npm ci compatibility

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): use main's lockfile as base, update only parser wasm packages

Regenerating package-lock.json from scratch pulled different dependency
versions causing svelte-check type errors. Instead, start from main's
lockfile and only update the two changed packages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): add modules column to fetch_script_for_update query

The Script<SR> struct has a modules field (FromRow), but
fetch_script_for_update didn't SELECT modules, causing a runtime
error "no column found for name: modules" when the worker processed
dependency jobs. This was the root cause of the relock_skip test
timeout.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): fix script module execution for Python and Bun

- Fix modules not passed through job queue: inject _MODULES into
  PushArgs.extra when pushing Code jobs so worker can extract them
- Fix Python module imports: use relative imports (from .helper)
  and add sys.path.insert for module directory in wrapper
- Fix Python tests: use relative imports and empty lock to prevent
  pip from resolving module names as packages
- Add local file check in Bun loader for module resolution
- Ignore Bun module test (bundle mode loader integration tracked
  separately)
- Add missing modules column to fetch_script_for_update query

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): remove unnecessary empty lock in Python module tests

Relative imports (from .helper) are not parsed as pip packages,
so the empty lock workaround is not needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): fix module execution for Python and Bun — all tests pass

Python modules:
- Use relative imports (from .helper import greet) since scripts run
  as packages
- Add sys.path.insert for module directory in wrapper to ensure local
  modules take precedence over pip packages with same name

Bun modules:
- Use bundled output (./out/main.js) as wrapper import when modules
  are present — the bundled output has module content inlined by
  Bun.build, avoiding runtime loader resolution issues
- Add local file check in loader.bun.js onResolve to short-circuit
  API URL resolution for module files on disk

Job queue:
- Inject _MODULES into PushArgs.extra when pushing Code jobs so
  the worker can extract them at execution time

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: address PR review — simplify, fix correctness, remove dead code

Critical fixes:
- Replace all CLI `no_main_func` references with `auto_kind` (string)
  to match the backend migration and API changes
- Remove duplicated `compute_python_module_dir` in worker.rs, use
  the canonical version from python_executor.rs

High priority:
- Auto-create `__init__.py` in intermediate directories for nested
  Python modules so imports like `from .utils.math import add` work
  without users manually creating __init__.py files
- Remove redundant `sys_path_insert` — relative imports use Python's
  package system, not sys.path

Medium:
- Fix lock file base name extraction: use regex to strip only the
  final extension (`.replace(/\.[^.]+$/, '')`) instead of `indexOf(".")`
  which breaks for files like `helper.test.ts`

Simplification:
- Remove dead `{#if false}` Popover block in ScriptEditor.svelte
- Guard loader.bun.js local file check to only run for relative paths
  (matching the Windows loader pattern)
- Add clarifying comment on Bun dual mechanism (build + run phases)
- Add maintenance comment on manual Hash impl for NewScript

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: final review fixes — stale cleanup, baseName, auto_kind export

- Fix sync.ts baseName extraction using indexOf(".") → regex
  (same fix as script.ts/metadata.ts, missed this instance)
- Add stale module file cleanup in writeModulesToDisk: removes files
  from __mod/ that are no longer in the modules map before writing,
  fixing the pull→push cycle that couldn't delete modules
- Log warning when _MODULES serialization fails in job push instead
  of silently dropping modules
- Use strict equality (===) for auto_kind comparison
- Exclude auto_kind from workspace export — it is auto-detected by
  the parser at deploy time from script content

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): remove auto_kind from push, comparison, and metadata

auto_kind is auto-detected by the parser at deploy time, so the CLI
should not send it, compare it, or write it to script.yaml.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove erroneously added backend/backend/.sqlx directory

Duplicate .sqlx cache was committed at the wrong nested path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review feedback + fix CI dead_code warning

Frontend (ScriptEditor.svelte):
- Fix switchToMain() missing lastSyncedCode update — prevents stale
  code sync on external changes while editing a module tab
- Fix formatAction saving module code to main script's localStorage
  draft — now saves main code when on a module tab
- Fix non-null assertion on inferModuleLang in renameModule — fall
  back to original language instead of force unwrap
- Remove redundant activeModuleTab truthy check in runTest

CLI (script.ts):
- Clean up empty directories after removing stale module files in
  writeModulesToDisk

Backend:
- Add path traversal guard in write_module_files — reject module
  paths containing ".."
- Fix dead_code warning on auto_kind field in workspace export struct

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(frontend): improve auto_kind UX + address review findings

- Rename "Include without main function" toggle to "Include library
  scripts" in script list (ItemsList.svelte)
- Update NoMainFuncBadge: "No main" → "Library" with clearer tooltip
- Filter module file extensions by main script language — Python
  scripts only allow .py modules, TypeScript only .ts, etc.
- Split flushModuleState into flushModuleContent (no UI side-effect)
  and flushModuleState (flush + reset tab), reducing duplication
- Dynamic placeholder and hint text in add module popover based on
  main script language

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 01:20:09 +00:00
dieriba 10befb995d feat: local type references parsing support for main function args (#5995) 
* add base struct

* feat resolve interface and type declarion in entrypoint param's function

* nits

* fix reset dependencies

* update package

* fix handle infinite recursion

* add depth level and handle enum for referenced type

* nits

* nits

* nits

* perf

* fix

* done

* fix schema form cache inconsistency

* fix default type and nits

* remove

* update Object typ for parser

* one level ref from from parent when resolving types and use format for resource

* update cli and use resource type

* nits

* update parsers

* fix: use specific parser versions

---------

Co-authored-by: HugoCasa <hugo@casademont.ch>
 2025-07-23 10:05:32 +02:00
pyranota 59740c0478 feat: add java support (#5458) 
* feat: add nu (nushell) support

* add worker tests

* deactivate tables and non-any types below top-level

full support will come in V1
for V0 it's better to keep things minimal and simple

* add syntax highlighting

used python's grammar, since nushell isn't supported by monaco nor svelte-highlights

for V1 nu will get it`s own grammar

* add logo

* partially implement plugin support

* change logo + ability to deploy + nsjail draft

* static variables + get_resource + get_variable

* lsp/dev.nu + initial nu lsp (not working yet)

* make it work with nsjail

* nullguard

* Much more flexible signature parsing and better error-messages

* add init script

* rename nulsp to nu

* install nu to dockerfile

* fix merge

* implement Default for MainArgSignature

* stage NU_CACHE_DIR

* improve dockerfiles

* dev.nu for parser-wasm + flake.nix

* update code for windows

* add nushell to flake

* upload Cargo.lock

* make build.sh work on nixos

* build wasm cli parsers

* add docs to README_DEV.md

* add helper script docker/dev.nu

* improve docker/dev.nu

* fix windows

* commit frontend/package(lock).json

* update cargo.lock

* correctly update cargo.lock

* remove lsp

* update flake.nix to include svelte server and nushell

* Revert base.sql to main

* remove PLUGIN_USE_RE

* make CARGO_PATH private

* add nu to cli

* Change flags to build wasm-nu-parser

* remove flake.nix from parser-wasm

* update wasm-build target

* remove unused import

* add cli support for nu

* update github workflows

* wasm-build 0.17 -> 0.19

* update build script

* update cargo.lock

* Fix typographical error

* start working on java

* do java boilerplate

* implement parser for java

* update Cargo.lock

* update ENV_SETTINGS

* use published nu parser

* update package.lock

* java is S3 + Caching enabled

* install nsjail backup

* commit v0

* fix nsjail

* v0.1

* rewrite parser in tree-sitter

* implement parser from scratch

* polishing

* change init script to match new parser

* fix imports

* fix cli build

* fix cli build

* refactor install phase

* implement .valid.windmill atomic verification

* implement java init functionality

* remove quick-xml

* fix windows not recognizing 'mvn'

* create empty settings.xml if there is no config provided

* clean up

* change default settings.xml

* change classpath format for windows

* docs to helper

* java copy bin cache instead of symlink

* remove comments

* merge

* fix package.json

* fix package.json 2

* minor fixing

* migrate to Coursier

* update misc

* Http(s) Proxy + CA certs

* remove unused .wasm

* make requirements insensitive to spaces

* update handle_child refs

* rework save_cache for directories

* fix s3 bug

* compile .wasm for cli

* remove uuid import

* fix compilation

* use reference

* fix zero-dep failure

* removing unsafe stuff

* remove unneeded imports

* revert: we still need winapi

* remove nix store from nsjail

* do not create cache_nomount

* add java to dnt

* remove duplicated dependency in init script

* fix typos

* fix CI

* use published parser
 2025-03-27 15:12:55 +01:00