mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-20 08:01:35 +00:00
fix/wac-python-cache-install-race
54 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ruben Fiszel
|
aedf369174 |
fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe (#8999)
* fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(pg): wrap encoder errors with arg context, add fallback test Followups on #8999 review: - Wrap rust-postgres "error serializing parameter N" failures with the arg name, JSON value kind, and asserted Postgres type plus a hint about an explicit cast — so users see actionable context instead of an opaque WrongType. - Drift-prevention meta-test: assert otyp_to_pg_type and convert_val agree on the Type for every recognised arg_t when the JSON value matches its natural Rust kind. Catches future drift if either side changes. - Integration test for the prepare + query_raw fallback path: confirms unrecognised arg_t (custom enum) is routed through prepare and the server-resolved type appears in the failure surface — flips into a test failure if a regression accidentally routes unrecognised types through query_typed_raw. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): add otyp_inferred flag + regex-based placeholder renumbering Two follow-ups from the review of #8999: 1. **Issue #1 (Number/Bool + explicit text decl in WHERE)** Add `Arg::otyp_inferred: bool` to the parser. The PG SQL parser sets it `true` only at the "no info → fall back to text" site (bare `$N`, no inline cast, no `-- $N (TYPE)` decl). All other arg sources keep it `false`. In `convert_val` this flag distinguishes: - explicit text-like target (`-- $1 (text)` or `$1::text`) — coerce `Bool`/`Number` → `Box<String>` so `WHERE text_col = $1` works (`text = text` operator). Pre-#8988 behaviour, restored. - parser-default text (bare `$N`) — bind the value's natural Rust type so the regression case (`Value::Bool` against a real `bool` column via `CAST AS bool`) keeps working. `Arg` is in `windmill-parser`; the new field has `#[serde(default)]` so persisted signatures stay backward-compatible. 2. **Issue #4 ($5/$50 substring rewrite collision)** Replace the per-index `String::replace` chain (which turned `$50` into `$10` when oidx=5 was processed first) with a single regex pass. `\d+` is greedy, so `$5` and `$50` match as distinct units; indices outside the mapping are left intact. 3. Tests: - parser: `test_parse_pgsql_otyp_inferred_flag` covers bare/inline- cast/decl/mixed shapes. - executor unit: `convert_val_bool_against_every_arg_t` and `convert_val_*_number_*` split each text-like target into explicit vs inferred expectations. - executor unit: `renumber_sparse_placeholders_no_collision`. - integration: `test_postgresql_arg_type_combinations` adds 4 cases covering decl(text)+Number/Bool in WHERE, bare $1+Bool, and sparse positional args ($5/$50). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg+sdk): enum support, extended String arms, position-aware $N rewrite, SDK quality Backend: 1. **`AnyTextValue` ToSql/FromSql wrapper**: vanilla `tokio_postgres`'s `ToSql for String` / `FromSql for String` reject `Kind::Enum` and `Kind::Domain` even though the wire format is plain UTF-8. The wrapper accepts those kinds in both directions. End result: explicit `$1::my_enum` / `CAST($1 AS my_enum)` casts now round-trip without the ugly `CAST($1::text AS my_enum)` workaround, AND `SELECT enum_col` results come back as JSON strings instead of erroring at the FromSql layer. 2. **#10 — Value::String → numeric/real/double/oid/bool**. Without these arms, a string-encoded value (`"3.14"`, `"true"`) for a non-text / non-temporal arg_t fell through to `Box<String> + TEXT`, which then failed at the server (no implicit cast text→numeric in expression context). Now strings are parsed into the matching native type with clear error messages on parse failure. 3. **Position-aware `$N` rewrite**: replaces the regex-based renumbering (which fixed the `$5/$50` substring collision but still walked through string literals and comments, mangling `'price: $5'` etc.) with a walk over `parse_pg_statement_arg_positions` — the same string/comment/dollar-quote-aware tokenizer used for index discovery. Adds `parse_pg_statement_arg_positions` to the parser's public API. SDK: 4. **BigInt support**: `JSON.stringify(BigInt)` throws. The SDK now stringifies bigints before serialisation; the executor accepts numeric strings into BIGINT arg slots via the existing `Value::String → INT8` parsing arm. SDK-side `inferSqlType` is split so `BigInt` always resolves to `BIGINT` (was reaching `Number.isInteger(BigInt)` which returns false → wrong default). 5. **Homogeneous array auto-tag**: `${[1,2,3]}` against an `int[]` column now emits `$1::BIGINT[]` instead of `$1::JSON`. Detection covers primitive types only (number / bigint / string / boolean); mixed or nested arrays still fall back to JSON. Mixed int/float widens to `DOUBLE PRECISION[]`. 6. **`.query()` positional bug**: previously the `.query()` method abused the template-tag builder, which appended `$N::TYPE` after the user's literal SQL string instead of binding by position (`SELECT $1, $2` became `SELECT $1, $2$1::BIGINT`). Now `.query()` builds the executor-shaped content directly: a `-- $N argN (TYPE)` declaration block followed by the user's SQL verbatim. Tests: - Parser: `test_parse_pg_statement_arg_positions_skips_strings_and_comments` asserts string literals, comments, and dollar-quoted blocks don't produce positions (so renumbering doesn't mangle them). - Executor unit: `renumber_sparse_placeholders_no_collision_no_string_mangling` uses the new position-aware path and includes string-literal + comment + `$$…$$` cases. Existing convert_val tests grow to cover new String→numeric/real/double/oid/bool arms. - Integration: `test_postgresql_arg_type_combinations` adds 13 cases (enum round-trip both directions, string→numeric/real/double/bool/oid, string-literal `$N` non-mangling). The prepare-fallback test now asserts SUCCESS (not failure) for enum encoding via AnyTextValue. - SDK: new `typescript-client/tests/sqlUtils.test.ts` (42 tests) exhaustively covering inferSqlType primitives + arrays, parseTypeAnnotation, datatable() template tag (with all the new shapes — BigInt, homogeneous arrays, RawSql, schema preamble), datatable().query() positional, and ducklake() shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): replace DISCARD ALL with curated reset (preserves typeinfo cache) Found while exhaustively probing custom-type DX: every cached-connection reuse was running `DISCARD ALL`, whose included `DEALLOCATE ALL` deallocates *all* prepared statements server-side — including the typeinfo statements that tokio_postgres caches per-Client to resolve custom enum / domain Oids. tokio_postgres still held `Statement` objects whose names the server had forgotten, so the next custom-type query failed with intermittent "prepared statement \"sN\" does not exist" errors. The failure was easy to reproduce: any sequence that forced typeinfo lookup for two different custom-type kinds on the same cached connection (e.g. enum followed by domain) would hit it. Replace `DISCARD ALL` with a curated reset that explicitly targets the state we actually care about, *without* touching prepared statements: RESET ALL — GUC parameters (search_path, application _name, statement_timeout, …) RESET SESSION AUTHORIZATION — undoes both `SET SESSION AUTHORIZATION` and `SET ROLE` (RESET ALL does NOT — these aren't GUC parameters, so without this an elevated role from a previous job would silently leak) UNLISTEN * — drops LISTEN registrations CLOSE ALL — closes open cursors Trade-off: temp tables, advisory locks (session-scoped), and user-created PREPARE statements may persist across cached-connection reuse — rare in datatable / PG-script workloads. tokio_postgres's typeinfo cache survives intact, so custom enum / domain queries are fast on subsequent reuse. Tests: - `test_postgresql_custom_types_on_cached_connection` — runs 10× alternating enum + domain queries on a cached connection. Pre-fix this failed with `prepared statement "sN" does not exist` after the first reuse; post-fix passes. - `test_postgresql_set_role_does_not_leak_across_cached_connection` — switches `SET ROLE` and `SET SESSION AUTHORIZATION` to a non-postgres role, then runs a follow-up job and asserts current_user/session_user are restored. Specifically catches the case where someone might switch back to `RESET ALL` alone (which doesn't cover SET ROLE / SESSION AUTHORIZATION) and silently introduce a permission-leak vector. - All existing session-isolation tests (`test_postgresql_cached_connection_resets_session`, `test_postgresql_single_worker_session_isolation`, `test_postgresql_100_jobs_cached`) continue to pass. Found via end-to-end probing of datatable / PG-script DX, not previously covered: the existing isolation tests only did `SET ROLE postgres`, the connecting user, so the leak was invisible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): address PR #8999 review (cubic + claude) cubic (P1, real bug): - `convert_vec_val` for `timetz` array asserted `Type::TIMETZ_ARRAY`, but chrono `NaiveTime` only encodes for TIME (same caveat as the scalar arm). Switch to `Type::TIME_ARRAY`; rely on PG's implicit `time→timetz` assignment cast at the column site. Add an explicit unit test. claude (#1, silent failure → explicit error): - `Bool` + explicit `(char)` / `(character)` decl previously silently bound BOOL, hoping the server would cast at the use site — but PG has no implicit `bool→char` and the resulting error ("operator does not exist: bool = char") was opaque. Now error at bind time with an actionable hint to use `bool` decl or pass the value as a "t"/"f" string. claude (#2, asymmetry doc): - Object/Array still coerce to text on `matches!(typ, Typ::Str(_))` (covers both explicit AND inferred-default text), unlike Bool/Number which key on `explicit_text_target`. The asymmetry is intentional (no implicit `jsonb → text` cast in expression context vs PG having implicit `bool/int → text` casts) — added a body comment so future maintainers don't try to "align" them. claude (#3, perf): - `parse_pg_statement_arg_indices` and `parse_pg_statement_arg_positions` walked the SQL tokenizer twice. Fold into a single pass that derives the index set from the position list. claude (#4, fmt drift): - `cargo fmt` over the parser crates I touched with perl scripts in the earlier commit (windmill-parser-{sql,bash,ts,go,php,java,csharp,nu,py, rust,graphql,yaml,r}). Net cosmetic. claude (#5, parseTypeAnnotation): - One-line caveat in the SDK's `parseTypeAnnotation` that the returned string is presence-only (e.g. `${x}::DOUBLE PRECISION` returns `"DOUBLE"`, `CAST(${x} AS int)` returns `"int)"` — neither matches a real PG type, but the only consumer just checks `!== undefined`). While here — discovered + fixed independently while exhaustively probing DX: - **Replace `DISCARD ALL` with curated reset** (`RESET ALL; RESET SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;`). DISCARD's `DEALLOCATE ALL` killed tokio_postgres' typeinfo cache, producing intermittent `prepared statement "sN" does not exist` errors on custom-type queries after cached-conn reuse. New regression tests: `test_postgresql_custom_types_on_cached_connection` and `test_postgresql_set_role_does_not_leak_across_cached_connection` (the latter catches the case where someone might switch back to `RESET ALL` alone and silently introduce a permission-leak vector — RESET ALL doesn't cover SET ROLE / SET SESSION AUTHORIZATION). - **ISO-8601 timestamp results** (`pg_cell_to_json_value`). Pre-fix `TIMESTAMP` was rendered with a space separator ("2024-01-15 10:30:00") and `TIMESTAMPTZ` with " UTC" suffix ("2024-01-15 10:30:00 UTC") — neither parseable by `date-fns parseISO`, JavaScript `new Date()` is lenient enough to handle them but several frontend `App*Input.svelte` components use parseISO and fail silently. Switched to ISO-8601 with `T` separator and `+00:00` offset; arg-parsing path still accepts the legacy " UTC" suffix for back-compat. Test coverage: - 17/17 unit (`pg_executor::tests`) - 9/9 integration (`backend/tests/worker.rs`, `test_postgresql_*`) - 27/27 parser (`windmill-parser-sql`) - 42/42 SDK (`typescript-client/tests/sqlUtils.test.ts`) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): bounded one-shot warning on numeric precision loss + ISO-8601 + NaN handling Found while probing PG-script DX with millions of numeric cells: 1. **Numeric precision-loss warning**: `numeric` results are still serialised as JSON Number (back-compat — switching to JSON String would silently break user code doing arithmetic on results), but we now detect `Decimal -> f64 -> Decimal` round-trip failure and emit a single job-log warning recommending a `::text` cast in the SQL. Bounded by `NUMERIC_PRECISION_CHECK_BUDGET = 256` cells per query (one atomic load + one fetch_sub on the hot path; first lossy value short-circuits to a single load thereafter). Worst-case overhead on a 1M-cell numeric-heavy query: ~25µs of checks + 5ns × N atomic loads (vs. ~100ms unbounded). 2. **ISO-8601 timestamps**: `pg_cell_to_json_value` previously returned `"2024-01-15 10:30:00"` (TIMESTAMP) and `"2024-01-15 10:30:00 UTC"` (TIMESTAMPTZ) — neither parseable by date-fns `parseISO`, which is what the apps `App*Input.svelte` components use, so timestamp values silently failed to round-trip into date pickers. Switch to ISO-8601 (`T` separator + `+00:00` offset) on the result side; arg-parser continues to accept the legacy `" UTC"`-suffixed format for back-compat. 3. **Float NaN / Infinity results**: `Number::from_f64` returns None for NaN / ±Inf, which `pg_cell_to_json_value` was raising as "invalid json-float" — failing the *entire* query if any cell held one of these special values. Now serialise them as JSON strings ("NaN", "Infinity", "-Infinity") and let the rest of the row come through. Arg-side: `s.parse::<f64>()` already accepts the same strings. Tests: - `decimal_fits_f64_losslessly_predicate` — covers fits / doesn't-fit cases for the precision-loss predicate. - `precision_check_budget_caps_per_query_overhead` — locks in the budget cap and the loss-flag short-circuit. - All 9 PG integration tests + 17 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): add pg_advisory_unlock_all to reset; warn on missing args; honor decl defaults While probing PG-script DX further found three more frictions: 1. **Advisory lock leak** (cubic P2): switching from `DISCARD ALL` to `RESET ALL; RESET SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;` meant session-scoped advisory locks (`pg_advisory_lock`) leaked across cached-connection reuse. Add `SELECT pg_advisory_unlock_all()` to the chain — `DISCARD ALL` covered this implicitly via `DISCARD PLANS / DEALLOCATE / pg_advisory_unlock_all` and we lost it in the switch. 2. **Missing-arg silent NULL**: an arg declared in the SQL (e.g. `-- $1 amount (numeric)`) but not provided in the args object was bound as NULL with no error / warning. Misspelling the key in the args object silently produced a row of NULLs — a notorious DX debugging trap. Now: collect the names of declared-but-missing args during dispatch and emit a single one-shot warning to the job logs at end-of-query naming each one. Bound NULL is preserved for back-compat. 3. **Declaration defaults ignored**: `-- $1 a (int) = 5` carries `arg.default = Some(Number(5))`, but the dispatch fell straight to NULL when the arg was missing. Now: respect the default — user-supplied value > declaration default > NULL. Also fixes the warning logic above (only warn for args that *don't* have a default). Tests: existing 19 unit + 9 integration pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): multi-word PG types with [] suffix lost the array-ness; array arms accept stringified values Two more frictions found while probing SDK end-to-end against a real datatable resource: 1. **Multi-word array types lose the [] suffix in the parser**. `transform_types_with_spaces` recognises aliases for "double precision", "character varying", "timestamp with time zone", etc. but its return type was `&'a str` — only the bare alias, never with a trailing `[]`. The `RE_CODE_PGSQL` regex's `\w+` captures stop at the first space, so the regex's own `(?:\[\])?` array-suffix branch sees only `"double"` (not `"double precision[]"`); the `[]` was silently lost. Result: `$1::double precision[]` (which the SDK now emits for homogeneous float arrays via the new auto-tag) routed through `Value::Array → Type::JSONB` and the server failed with "cannot cast type jsonb to double precision[]". Fix: switch `transform_types_with_spaces` to return `Cow<'a, str>` and re-check the trailing bytes after a multi-word match. If they start with `[]`, return `format!("{alias}[]")` — Owned. Single-word types and the no-match path keep returning Borrowed slices, so no allocation in the hot path. 2. **Array arms in `convert_vec_val` rejected stringified values for numeric / int* / bool / oid / real / double**. The scalar `convert_val` already parses strings into the matching native type for these arg_ts, but the array variant only accepted JSON-native counterparts. Sending `["1.5", "2.5", "3.5"]` against `$1::numeric[]` (e.g. via `unnest` for bulk loading, or `JSON.stringify(BigInt[])` round-trip) failed with "Mixed types in array". Now the array arms mirror the scalar ones — `as_<native>().or_else(|| as_str().and_then(parse))` — so both shapes round-trip cleanly. Tests: 19 unit + 9 integration pass; existing parser tests cover the multi-word array forms (the regex-cap behaviour didn't break for single-word types, and Cow plumbing is transparent to all callers). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(parsers): add otyp_inferred field to Arg literals in tests + 3 missed src files CI failures: the perl-driven sweep that added `otyp_inferred: false` to every `Arg { ... }` literal when I introduced the field in the parser schema covered `src/lib.rs` files but missed: - parsers/windmill-parser-bash/src/lib.rs (mass-edited but a later format pass un-applied a few sites) - parsers/windmill-parser-go/src/lib.rs (same) - parsers/windmill-parser-graphql/src/lib.rs (same) - parsers/windmill-parser-nu/tests/tests.rs (test file — not swept the first time) - parsers/windmill-parser-ts/tests/tests.rs (test file — same) Also tightened the regex to handle `oidx: None` without the trailing comma (some test files had the field as the last initialiser line). `cargo build --features <CI feature combo> --workspace --all-targets` is clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(sdk): Date → TIMESTAMPTZ; NaN / ±Infinity → string Two more frictions found while running the actual SDK end-to-end against a live datatable resource: 1. **JS `Date`** fell into the typeof "object" branch and was tagged `::JSON`. It worked accidentally for `${date}::timestamptz` via PG's `json → text → timestamptz` implicit cast chain, but `${date}` against a `timestamptz` column without a user-supplied cast bound the value as a JSON string and the comparison `timestamptz = json` failed. Now: `inferSqlType` recognises `Date` and tags `::TIMESTAMPTZ`; `serializeArgValue` emits `Date.toISOString()` so the executor's `Value::String → TIMESTAMPTZ` arm parses it cleanly. 2. **JS `NaN` / `±Infinity`** silently became NULL. `JSON.stringify(NaN)` returns `"null"` per the JS spec, so the value reached the executor as JSON null — the SDK's `::DOUBLE PRECISION` tag then bound a NULL double. Fix: detect non-finite numbers in `serializeArgValue` and stringify them as `"NaN" / "Infinity" / "-Infinity"`. The executor's `Value::String → FLOAT8` arm (`f64::from_str`) accepts these literals directly, and the result-side already renders the values as JSON strings (matching round-trip). SDK unit tests grow from 42 → 44 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(pg): integration coverage for multi-word arrays + stringified array elements Locks in the two array fixes from the previous commit (`fix(pg): multi-word PG types with [] suffix lost the array-ness`) with end-to-end cases in `test_postgresql_arg_type_combinations`: - `double precision[]`, `character varying[]`, `timestamp without time zone[]` — verifies the parser keeps the `[]` suffix after multi-word alias resolution. - `numeric[]` / `int[]` / `bool[]` from stringified primitives — verifies the array arms of `convert_vec_val` apply the same string-coercion the scalar arms do. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: fix indentation drift on otyp_inferred lines cargo fmt cleanup of leftover indentation where the perl-driven sweep that introduced the otyp_inferred field landed at the wrong column. No behaviour change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
 hugocasa
|
c0eeea9c83 |
feat: support S3Object input args in native SQL scripts (#8954)
* feat: support S3Object input args in native SQL scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: review fixes from local-review Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * update parser --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
hugocasa
|
53badf1a8c |
fix: track dollar-quoted strings in SQL block splitter (#8891)
* fix: track dollar-quoted strings in SQL block splitter Queries like `CREATE FUNCTION ... AS $$ ... ; ... $$ LANGUAGE plpgsql;` were being shredded on every `;` inside the function body because the SQL splitter's state machine didn't recognize PostgreSQL dollar-quoted strings. Add an `InDollarQuote(tag)` state so `$$ ... $$` and `$tag$ ... $tag$` regions are treated as a single quoted span. Opt-in via a new `track_dollar_quotes` flag on `parse_sql_blocks`; enabled for PostgreSQL and DuckDB, disabled for MySQL/Oracle/BigQuery/ Snowflake which don't support the syntax. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: make windmill-parser-wasm a self-contained workspace The wasm parser crate is excluded from the backend workspace (its nightly-only `cargo-features = ["panic-immediate-abort"]` would break stable cargo on the whole workspace), but its manifest still used `.workspace = true` inheritance — which fails with "failed to find a workspace root" once the parent no longer considers it a member. Declare the crate as its own workspace by adding `[workspace]`, `[workspace.package]`, and `[workspace.dependencies]` tables. Mirror the relevant entries from the parent `backend/Cargo.toml` (same version specs, same path targets) so resolution stays byte-identical to what the parent would have produced. Also: - Teach `.github/change-versions.sh` (+ mac variant) to update this crate's own `Cargo.toml` version and bulk-bump the `windmill-*` entries in its `Cargo.lock` on each release. - Bump the frontend's pinned `windmill-parser-wasm-regex` to 1.688.0 to match the freshly-built package, and refresh `package-lock.json`. - Regenerate the wasm crate's `Cargo.lock` from scratch (first build under the new workspace re-resolves the full graph; target-gated deps from sibling crates like `windmill-parser-py-imports` are now recorded in the lockfile but not compiled when targeting wasm32). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
0317d5891c |
feat: add powershell common parameters support (#8683)
* feat: add powershell common parameters support (-Verbose, -Debug, -ErrorAction, -WhatIf) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add powershell common params to script editor test panel Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: detect CmdletBinding from code instead of schema in script editor Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: ignore commented-out CmdletBinding in powershell detection Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use preference variables for -Verbose/-Debug instead of CLI args Verbose/Debug output goes to PowerShell stream 4/5 which isn't captured by the 2>&1 redirect. Setting $VerbosePreference/$DebugPreference in the wrapper scope propagates to child scripts and output flows through the host to stderr, which Windmill captures as logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use *>&1 to capture all powershell streams including verbose/debug The previous 2>&1 only captured error stream. Verbose (stream 4) and debug (stream 5) output was silently lost. Using *>&1 redirects all streams to success stream so they flow through Tee-Object into logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use targeted stream redirects (4>&1 5>&1 2>&1) instead of *>&1 *>&1 breaks $PSCmdlet.ShouldProcess() by redirecting internal streams. Only redirect verbose (4), debug (5), and error (2) to success stream. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert to 2>&1 redirect — stream 4/5 redirects break powershell Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use 4>&1 5>&1 for verbose/debug capture, remove WhatIf support Stream 4/5 redirects capture verbose/debug in the pipeline. WhatIf is removed because $PSCmdlet.ShouldProcess() doesn't work when scripts are invoked through Windmill's wrapper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: redirect verbose/debug to files to keep result pipeline clean Verbose (4) and debug (5) streams are redirected to separate log files during script execution, then output via Write-Host after the script completes. This keeps them out of the Tee-Object pipeline (used for result extraction) while still showing them in the job logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: output verbose/debug to stderr via Console.Error for log capture Write-Host goes to stdout which gets mixed with result output and truncated by OSS log threshold. Using [Console]::Error.WriteLine() writes to stderr which Windmill captures separately as logs, with VERBOSE:/DEBUG: prefixes for clarity. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: redirect script output to file only, send verbose/debug to stdout The OSS log storage has a 9KB threshold. Previously, Tee-Object sent the full JSON result to both stdout (logs) and the pipe file, eating the log budget. Now script output goes only to the pipe file (> $pipe), and only verbose/debug messages go to stdout for the log viewer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: preserve original Tee-Object behavior, append verbose/debug after Keep the original wrapper behavior (Tee-Object to stdout + pipe file). Only add 4>verbose.log 5>debug.log to capture those streams, and output them at the end of logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: inject preference vars into main.ps1 instead of CLI args Passing -Verbose/-Debug as CLI args causes PowerShell module loading to emit verbose noise. Instead, inject $VerbosePreference/$DebugPreference inside main.ps1's try block so they only affect user code. Stream 4/5 are still redirected to files in the wrapper for log output. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restore common param toggles from previous job args on Run Again Extract _wm_ps_* keys from loaded args and initialize the toggle states in PowerShellCommonParams. Also strip them from main args so they don't appear as unknown schema form inputs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: show active common param badges when section is collapsed Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: inject ErrorAction as preference variable instead of CLI arg -ErrorAction as a CLI arg only affects the caller, not the script's internal error handling. Setting $ErrorActionPreference inside main.ps1 correctly overrides the default 'Stop' behavior for the user's code. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: ensure full backward compatibility with existing powershell scripts - Only filter common param names when [CmdletBinding()] is present (without it, $Verbose etc. are regular user-defined parameters) - Only add 4>verbose.log 5>debug.log and log output lines when common params are actually enabled — original wrapper is unchanged otherwise Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: lighter styling for common params section Replaced heavy Section component with a subtle inline chevron toggle labeled "Common parameters". Smaller text, secondary color, indented options. Badges still show when collapsed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: rename section to CmdletBinding parameters Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add ..Default::default() to windmill-parser-r (new parser from main) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: missing comma in graphql parser test + merge main Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add missing commas before ..Default::default() in parser tests Merge from main brought test constructors with formatting issues from the original automated script (missing comma between last field and ..Default::default()). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: restore comment markers in nu parser test that script broke Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review — ignore commented CmdletBinding, clear stale params 1. Parser: strip comment lines before detecting [CmdletBinding()] to avoid false positives from commented-out attributes 2. RunForm: always assign psCommonParams (not just when non-empty) so stale settings from a previous run don't leak into later runs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
 Pyra
|
9643006f1e |
feat(cli): better stale scripts detection #3 (#8480)
* fix Signed-off-by: pyranota <pyra@duck.com> * reduce tests Signed-off-by: pyranota <pyra@duck.com> * update Signed-off-by: pyranota <pyra@duck.com> * fix Signed-off-by: pyranota <pyra@duck.com> * update Signed-off-by: pyranota <pyra@duck.com> * WIP: stash changes after merge with origin/main * Delete backend/parsers/windmill-parser-wasm/Cargo.lock * reset cargo.toml * feat(cli): integrate dependency tree into generate-metadata command - Add isDirectlyStale field to DependencyNode for staleness tracking - Update addScript to accept itemType, folder, isRawApp, isDirectlyStale - Update propagateStaleness to use isDirectlyStale field instead of parameter - Handlers now determine staleness and pass it to tree.addScript - generate-metadata calls propagateStaleness() and populates staleItems from tree - Pass legacyBehaviour=false and tree to handlers during generation phase 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): store originalPath in tree for correct handler invocation Scripts need the path with extension to be passed to the handler. Added originalPath field to DependencyNode to track this. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix parsers Signed-off-by: pyranota <pyra@duck.com> * rever sqlx removal * update sqlx * feat: make py-imports parser WASM-compatible and add as separate WASM package Gate heavy deps (sqlx, windmill-common, async-recursion, toml, pep440_rs, tracing) behind cfg(not(wasm32)). Make parse_code_for_imports, parse_relative_imports, NImport, and ImportPin public. Remove duplicate import_parser from parser-py (reset to origin/main). Add py-imports-parser feature to windmill-parser-wasm and py-imports target to build.nu. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * safer return * update * fix: CLI metadata fixes - folder filter, staleness detection, WASM py-imports setup - Fix lazy_static cfg gating for WASM compatibility (split into separate blocks) - Fix folder argument filter to match specific file paths (not just directories) - Fix staleness detection to use checkHash with conf (includes module hashes) - Convert relative_imports_skip tests from Deno to bun APIs - Add windmill-parser-wasm-py-imports to CLI and build-npm dependencies - Relax module stale test to not require per-module change detail in output Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: restore temp_script_refs parameter in parse_python_imports Re-adds the temp_script_refs parameter that was lost when resetting py-imports crate to origin/main. This enables resolving relative imports from not-yet-deployed scripts during CLI lock generation. * fixes * extend testsuit * update ee repo ref * fix: diff endpoint bytea cast, upload only mismatched scripts - Add POST /scripts/raw_temp/diff endpoint to batch-compare local content hashes against deployed versions using Postgres sha256() - Use convert_to(content, 'UTF8') instead of content::bytea to avoid failure on scripts containing backslash sequences (e.g. \n) - CLI now diffs all scripts against deployed, uploads only mismatched ones - propagateStaleness no longer deletes non-stale nodes (needed for diff) - Suppress verbose log.info messages during metadata generation - Add E2E tests for locally modified and unpushed helper scripts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * rework * sqlx * fixes * add index * expand tests * fix flows * archive script before executing * disable tests for ci * skip Python-dependent E2E tests on CI Tests requiring the python backend feature are skipped when CI_MINIMAL_FEATURES=true since CI builds with zip-only features. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: make flow fixture lock optional and reset nonDottedPaths after tests Flow fixtures no longer emit an empty lock file by default. The lockContent parameter controls whether a lock: "!inline ..." line appears in flow.yaml. This prevents flows from appearing "up-to-date" when they should be processed by generate-metadata. Also adds afterAll to reset setNonDottedPaths(false) so global state doesn't leak between test files when run together. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * debug: add error logging in withTestBackend to diagnose CI failures Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * debug: add --bail 1 to CI test runner to show full error on first failure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * debug: include CLI stdout/stderr in assertion message for workspace deps test Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: set WMDEBUG_FORCE_V0_WORKSPACE_DEPENDENCIES in test backend The workspace deps feature requires workers to report their version, but in test/CI there are no separate workers (standalone mode). The version check fails because workers haven't had time to ping yet. Setting this env var bypasses the version check. Also reverts --bail 1 from CI workflow now that the root cause is fixed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * debug: add --bail 1 to Windows CI and assertion messages for Windows failure diagnosis Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: replace TEMP_SCRIPT_REFS_PLACEHOLDER in bun builder tests The loader.bun.js now includes a TEMP_SCRIPT_REFS_PLACEHOLDER that must be replaced before execution. The builder tests were missing this replacement, causing all 6 bun_builder_tests to fail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use cdirFwd in Windows loader filterLoad regex Raw cdir (with backslashes) interpolated into RegExp causes \r to become carriage return and \w to become word-char, so filterLoad never matches main.ts. This prevents replaceRelativeImports from running, leaving bare relative imports like "./script_b" in the bundled output, which scanImports then misparses as package ".". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: Windows filterLoad regex + graceful fallback for old backends - Fix filterLoad in loader.bun.windows.js to match both native backslash and forward-slash paths from Bun's resolver by escaping cdir for regex - Wrap uploadScripts in try/catch so generate-metadata degrades gracefully when the backend lacks /raw_temp endpoints (locks use deployed versions) - Add TODO for missing TEMP_SCRIPT_REFS support in Windows loader Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * debug: add loader/builder debug logging for Windows CI diagnosis Temporary console.log statements to understand: - What path Bun passes to onLoad for main.ts - Whether filterLoad regex matches - Whether replaceRelativeImports fires - What the bundled output contains - What imports scanImports extracts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: trigger CI for cli path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: trigger CI via workflow file change Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add TEMP_SCRIPT_REFS to Windows loader, use .ts extensions in test imports - Add TEMP_SCRIPT_REFS_PLACEHOLDER support to loader.bun.windows.js (mirrors loader.bun.js) so CLI lock generation can resolve imports from locally-modified scripts on Windows - Use .ts extensions in all test relative imports to work around the Windows filterLoad regex bug (replaceRelativeImports doesn't fire on Windows, so extensionless imports fail) - Remove unused uploadSucceeded variable Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove debug logging from loader_builder.bun.js Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove windmill-parser-wasm-py-imports from frontend package.json This dependency is only needed by the CLI, not the frontend. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * debug: add temp_script_refs logging for Windows CI investigation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: remove --bail 1 from Windows CLI tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: normalize backslashes in folder filter treePath lookup (Windows) On Windows, item.path (originalPath) uses backslashes but tree keys use forward slashes. The isRelevant filter's touchesFolder call passed the unnormalized path to traverseTransitive, which couldn't find the node. This caused cross-folder importers to be excluded from generate-metadata when a folder argument was specified. Also removes debug logging from previous commit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update cli-tests.yml * fix: normalize backslashes in strict-folder-boundaries warning message (Windows) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: update ee-repo-ref to fe8f0d1d7448464c98474d994e6492c0a45e8e38 This commit updates the EE repository reference after PR #467 was merged in windmill-ee-private. Previous ee-repo-ref: 03e6eaf950776c96b9581848a583af9ad735be60 New ee-repo-ref: fe8f0d1d7448464c98474d994e6492c0a45e8e38 Automated by sync-ee-ref workflow. * revert cli-tests.yml --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
31d6660d56 |
feat: script module mode with CLI sync, preview, and WAC UI improvements (#8380)
* feat: add script module mode with folder model for Bun and Python Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add missing modules field to RawCode in bun_executor Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * sqlx * feat: enrich WAC templates with checkpoint and replay semantics Add prominent comments explaining that all computation must happen inside task/step/taskScript or it will be replayed on resume/retry. Clarify that waitForApproval does not hold a worker and that approve/reject URLs are available in the timeline step details. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): script module sync idempotency, per-module hash tracking, and preview support - Fix pull→push idempotency: use `??` instead of `||` for module lock field so empty strings are preserved (matches API's `lock: ""`) - Add per-module hash tracking in wmill-lock.yaml following the flow inline script pattern (SCRIPT_TOP_HASH + per-module subpath hashes) - Selective module lock regeneration: only regenerate locks for modules whose content actually changed, not all modules - Use unfiltered rawWorkspaceDependencies for module hashes to match what updateModuleLocks passes to fetchScriptLock - Show changed module names in stale script output for clarity - Add module support to `script preview` command: read modules from __mod/ folder and pass them in the preview API request - Add preview tests for taskScript pattern (flat and folder layout) - Update test assertion for module stale detection output Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(frontend): WAC UI improvements — reorder templates, module tab rename, import consolidation - Reorder WAC template buttons: TypeScript before Python in ScriptBuilder, CreateActionsScript, and CreateActionsFlow - Remove dropdown items from +Script button (simplify to direct link) - Move "Import Workflow-as-Code" to +Flow dropdown with dedicated drawer - Add module tab rename: pencil icon on hover opens popover with validation, fixed-width icon container prevents layout shift Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: remaining module-mode changes from working branch - Backend parser updates for WAC detection - CLI sync/types updates for raw app path and module support - Frontend UI polish (Dev.svelte, ScriptRow, script hash page) - Test fixture updates Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(cli): add test for module modification detection in generate-metadata Verifies that modifying a single module file re-triggers stale detection and only the changed module is listed, not all modules. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): critical fixes from PR review - Fix hardcoded dev path in bun_executor.rs WAC v2 wrapper — use "windmill-client" import instead of absolute filesystem path - Fix missed no_main_func → auto_kind rename in parser TS test - Add modules column to clone_script SQL (windmill-common and windmill-api-workspaces) so cloned scripts retain their modules - Add modules: None to RawCode structs in worker tests - Restore complete sqlx cache (merge main's cache + our new queries) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix clone warning treated as error in CI Change `.clone()` on double reference to `*k` dereference in scripts.rs hash implementation. Update sqlx cache with new query hashes from modified clone_script SQL. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): use published parser wasm versions for CI build The local file:// paths for windmill-parser-wasm-py and windmill-parser-wasm-ts don't exist in the Cloudflare Pages build environment. Revert to published npm versions (1.655.0). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): update parser wasm packages to 1.657.2 Use newly published windmill-parser-wasm-ts and windmill-parser-wasm-py v1.657.2 which include auto_kind/WAC detection changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): regenerate package-lock.json for npm ci compatibility Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): use main's lockfile as base, update only parser wasm packages Regenerating package-lock.json from scratch pulled different dependency versions causing svelte-check type errors. Instead, start from main's lockfile and only update the two changed packages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): add modules column to fetch_script_for_update query The Script<SR> struct has a modules field (FromRow), but fetch_script_for_update didn't SELECT modules, causing a runtime error "no column found for name: modules" when the worker processed dependency jobs. This was the root cause of the relock_skip test timeout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix script module execution for Python and Bun - Fix modules not passed through job queue: inject _MODULES into PushArgs.extra when pushing Code jobs so worker can extract them - Fix Python module imports: use relative imports (from .helper) and add sys.path.insert for module directory in wrapper - Fix Python tests: use relative imports and empty lock to prevent pip from resolving module names as packages - Add local file check in Bun loader for module resolution - Ignore Bun module test (bundle mode loader integration tracked separately) - Add missing modules column to fetch_script_for_update query Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): remove unnecessary empty lock in Python module tests Relative imports (from .helper) are not parsed as pip packages, so the empty lock workaround is not needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix module execution for Python and Bun — all tests pass Python modules: - Use relative imports (from .helper import greet) since scripts run as packages - Add sys.path.insert for module directory in wrapper to ensure local modules take precedence over pip packages with same name Bun modules: - Use bundled output (./out/main.js) as wrapper import when modules are present — the bundled output has module content inlined by Bun.build, avoiding runtime loader resolution issues - Add local file check in loader.bun.js onResolve to short-circuit API URL resolution for module files on disk Job queue: - Inject _MODULES into PushArgs.extra when pushing Code jobs so the worker can extract them at execution time Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: address PR review — simplify, fix correctness, remove dead code Critical fixes: - Replace all CLI `no_main_func` references with `auto_kind` (string) to match the backend migration and API changes - Remove duplicated `compute_python_module_dir` in worker.rs, use the canonical version from python_executor.rs High priority: - Auto-create `__init__.py` in intermediate directories for nested Python modules so imports like `from .utils.math import add` work without users manually creating __init__.py files - Remove redundant `sys_path_insert` — relative imports use Python's package system, not sys.path Medium: - Fix lock file base name extraction: use regex to strip only the final extension (`.replace(/\.[^.]+$/, '')`) instead of `indexOf(".")` which breaks for files like `helper.test.ts` Simplification: - Remove dead `{#if false}` Popover block in ScriptEditor.svelte - Guard loader.bun.js local file check to only run for relative paths (matching the Windows loader pattern) - Add clarifying comment on Bun dual mechanism (build + run phases) - Add maintenance comment on manual Hash impl for NewScript Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: final review fixes — stale cleanup, baseName, auto_kind export - Fix sync.ts baseName extraction using indexOf(".") → regex (same fix as script.ts/metadata.ts, missed this instance) - Add stale module file cleanup in writeModulesToDisk: removes files from __mod/ that are no longer in the modules map before writing, fixing the pull→push cycle that couldn't delete modules - Log warning when _MODULES serialization fails in job push instead of silently dropping modules - Use strict equality (===) for auto_kind comparison - Exclude auto_kind from workspace export — it is auto-detected by the parser at deploy time from script content Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): remove auto_kind from push, comparison, and metadata auto_kind is auto-detected by the parser at deploy time, so the CLI should not send it, compare it, or write it to script.yaml. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove erroneously added backend/backend/.sqlx directory Duplicate .sqlx cache was committed at the wrong nested path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review feedback + fix CI dead_code warning Frontend (ScriptEditor.svelte): - Fix switchToMain() missing lastSyncedCode update — prevents stale code sync on external changes while editing a module tab - Fix formatAction saving module code to main script's localStorage draft — now saves main code when on a module tab - Fix non-null assertion on inferModuleLang in renameModule — fall back to original language instead of force unwrap - Remove redundant activeModuleTab truthy check in runTest CLI (script.ts): - Clean up empty directories after removing stale module files in writeModulesToDisk Backend: - Add path traversal guard in write_module_files — reject module paths containing ".." - Fix dead_code warning on auto_kind field in workspace export struct Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(frontend): improve auto_kind UX + address review findings - Rename "Include without main function" toggle to "Include library scripts" in script list (ItemsList.svelte) - Update NoMainFuncBadge: "No main" → "Library" with clearer tooltip - Filter module file extensions by main script language — Python scripts only allow .py modules, TypeScript only .ts, etc. - Split flushModuleState into flushModuleContent (no UI side-effect) and flushModuleState (flush + reset tab), reducing duplication - Dynamic placeholder and hint text in add module popover based on main script language Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
 Diego Imbert
|
577484d06a |
Separate asset parsers (#8321)
* Refactor asset_parser * package update * package lock |
||
|
Diego Imbert
|
1d78589940 |
fix: Database studio fixes (#8251)
* disable dynamic fields for db studio config * Fix SQL safe interpolated arg * Fix db studio not passing AppEditorContext to modal * Fix db studio modal grid not being able to move/resize components |
||
|
Diego Imbert
|
762fd3d993 |
Fix python datatable client requiring explicit types (#8086)
* Support arg type decl in postgres * Python datatable client no longer requires explicit arg typing * compilation fix * Set correct type in statement exec * reset to main * Explicit pg arg types * remove code duplication * update parser js * FLOAT8 doesn't have space --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
Diego Imbert
|
095505136c |
fix: Handle CTEs and local tables in SQL asset parser (#8131)
* Handle CTEs and local tables in SQL asset parser * also handle CREATE VIEW * Update package regex version |
||
|
Diego Imbert
|
87f3de9ae5 |
feat: Support column detection on S3 objects in DuckDB (#8018)
* Support column detection on S3 objects in DuckDB * Compilation fix * support direct s3 path without read_parquet() * package update * npm i |
||
|
Ruben Fiszel
|
6bf544f507 |
refactor: extract object store into dedicated crate with filesystem backend (#7996)
* refactor: extract object store code into windmill-object-store crate with filesystem backend Consolidate all object_store-dependent code from windmill-common into a new windmill-object-store crate. Add a filesystem-backed object store implementation using LocalFileSystem for dev/testing without cloud credentials. Includes 30 comprehensive tests covering render_endpoint, lfs_to_object_store_resource, duckdb_connection_settings, error mapping, and filesystem-backed integration tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * all * all * all * fix: fix raw_app hardcoded path, add missing ObjectStoreResource import, and add tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: move S3ModeFormat to windmill-types, make windmill-parser-sql optional, restore debug logs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Diego Imbert
|
3476ef4b9c |
fix: fix DB Manager not working with db resources with 4+ path segments (#7809)
* support more than 3 path segments * Fix explore db resource not working with 4+ path segments * don't assume 3 segments * ?table= syntax impl * update parsers * more nit fixes * fix sql query * claude nit * Update SQLx metadata --------- Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Diego Imbert
|
0caa533fbd |
feat: column-level asset tracking for ducklake and datatables (#7774)
* Refactor 1
* claude tmp1
* fixes1
* support for insert and update
* Fix returning
* docs: add nuanced mutex selection guidance to Rust backend skill (#7737)
Add "Mutex Selection in Async Code" section explaining when to use
std::sync::Mutex vs tokio::sync::Mutex based on official Tokio docs.
std::sync::Mutex is preferred for data protection as it's faster;
tokio::sync::Mutex only needed when holding locks across .await points.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix(cli): revert findCodebase change that broke ../shared codebases (#7740)
* fix(cli): revert findCodebase relative_path check that broke ../shared codebases
The previous change added a check to ensure script paths start with the
codebase's relative_path. However, this broke cases where relative_path
uses parent directory references (e.g., "../shared") because:
1. path.join normalizes paths, so "/project/../shared/f/script.ts" becomes
"/shared/f/script.ts"
2. FSFSElement strips the cwd prefix, resulting in "f/script.ts"
3. The check "f/script.ts".startsWith("../shared/") failed
The original behavior was correct - relative_path indicates where to find
codebase files, while includes/excludes patterns match against the normalized
paths that get passed during sync.
Fixes regression reported in #7729 comments.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test(cli): add preview test for codebase with imports
Tests that codebase bundling correctly includes imported modules,
which is the key functionality needed for ../shared codebases.
The test creates a helper module and a main script that imports
from it, then verifies the bundled script executes correctly.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* chore(main): release 1.621.2 (#7735)
* chore(main): release 1.621.2
* Apply automatic changes
---------
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
* feat: workspace dedicated workers (#7741)
* feat: workspace dedicated workers
* ref
* chore: update ee-repo-ref to a18ac31062ac092cb9a5fc87629e217d97f4911d
This commit updates the EE repository reference after PR #398 was merged in windmill-ee-private.
Previous ee-repo-ref: 98cfe3fef764d9d815d326d5056c734a03689d33
New ee-repo-ref: a18ac31062ac092cb9a5fc87629e217d97f4911d
Automated by sync-ee-ref workflow.
* fix(frontend): workspace script in flow steps
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix: remove uuid-ossp extension requirement for RDS compatibility
The uuid-ossp extension was created in the first migration but never
actually used - the codebase uses gen_random_uuid() which is built-in
to PostgreSQL 13+. This allows Windmill to run on AWS RDS where
application users may not have CREATE SCHEMA privileges.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: indexer build error (#7744)
* fix: indexer build error
* prepare sqlx
* Remove changes from Cargo.toml
* refactor: remove seed parameter from AI chat completions (#7745)
* better claude
* refactor: remove seed parameter from AI chat completions
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: visibility bug on deployment UI (issue when renaming items) + add tracking of folders and resource types (#7739)
* fix: Raw apps deployment UI (and merge UI)
* Add folders and resource tpyes to merge UI
* claude first pass on adding the new arg for h_deploy_metadata
* Add missing argument to handle_deployment_metadata in all its calls
* Add support for folders and resource types in merge UI
* Update eereporef for CI
* Update ee repo
* Add migration to reset cached diff with potential artifacts
* fix type in frontend
* Preapare sqlx
* Remove unused import and logs
* update ee-repo
* Update eerepo
* chore: update ee-repo-ref to aca38475afd2cafaf63f4bbffc65be9437d57d86
This commit updates the EE repository reference after PR #397 was merged in windmill-ee-private.
Previous ee-repo-ref: 19c64cf8c61d83f45047b37660054b29658cd403
New ee-repo-ref: aca38475afd2cafaf63f4bbffc65be9437d57d86
Automated by sync-ee-ref workflow.
* Make integration test for workspace comparisons
* Update SQLx metadata
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix: require AGENT_TOKEN and BASE_INTERNAL_URL for agent mode
- Add AgentConfig struct to validate required env vars on startup
- Change build_agent_http_client to require explicit token and URL
- Remove DEFAULT_BASE_INTERNAL_URL fallback (no more silent localhost:8000)
- Exit immediately if agent cannot connect to server on initial load
- Update integration tests to use dynamic port for BASE_INTERNAL_URL
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: forward teams error to client (#7746)
* fix: forward teams error to client
* chore: update ee-repo-ref to 9a3d71f2c6a41ed4d17111a8c05d8e1d4933898d
This commit updates the EE repository reference after PR #400 was merged in windmill-ee-private.
Previous ee-repo-ref: 25d35a8de1cd70e281dc876e51cd30402580b5c0
New ee-repo-ref: 9a3d71f2c6a41ed4d17111a8c05d8e1d4933898d
Automated by sync-ee-ref workflow.
* fix
* fix
* fix
* al
* sqlx
* sqlx
* all
* all
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* feat: add token usage tracking to AI agent output (#7738)
* feat: add token usage tracking to AI agent output
Add TokenUsage struct to track input/output/cache tokens from AI providers.
Currently implemented for Bedrock provider, with infrastructure in place
for other providers. Usage is included in the AI agent result alongside
output and messages when available.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add token usage extraction for Anthropic provider
Extract usage from message_delta SSE event and convert to TokenUsage.
Includes input_tokens, output_tokens, cache_read_input_tokens, and
cache_creation_input_tokens (mapped to cache_write_input_tokens).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add token usage extraction for Google AI/Gemini provider
Extract usage from usageMetadata in Gemini SSE events and convert to TokenUsage.
Maps promptTokenCount -> input_tokens, candidatesTokenCount -> output_tokens,
totalTokenCount -> total_tokens.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add token usage extraction for OpenAI Responses API provider
Extract usage from response.completed SSE event and convert to TokenUsage.
Maps input_tokens, output_tokens, and total_tokens directly.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add token usage extraction for Azure OpenAI / Chat Completions API
Add stream_options.include_usage to request and parse usage from final
SSE chunk for providers using the standard OpenAI Chat Completions API
(Azure OpenAI, Mistral, DeepSeek, Groq, TogetherAI, CustomAI).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: cleanup token usage tracking - remove unused Image usage field and accumulate across iterations
- Remove unused `usage` field from ParsedResponse::Image variant
- Add TokenUsage::accumulate() method to sum usage across agent iterations
- Accumulate input/output/total/cache tokens instead of replacing with last iteration
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: remove verbose debug logging from AI providers
Remove tracing::info!("[debug] ...") statements that were too verbose
for production. These logged raw events on every streaming event.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: add retry mechanism for OpenAI-compatible providers without stream_options support
Some OpenAI-compatible providers don't support the stream_options parameter
for usage tracking. This adds a retry mechanism that:
- First attempts the request with stream_options.include_usage
- If it fails with 400 and error mentions stream_options/include_usage,
automatically retries without the parameter
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: remove unnecessary text parsing overhead in image response handlers
Revert debugging changes that read response as text before parsing JSON.
Using response.json() directly is more efficient.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor: centralize TokenUsage conversion with constructor methods
Add new(), from_input_output(), and with_cache() constructors to TokenUsage
to eliminate duplicate conversion logic across providers. Also fixes potential
truncation in Bedrock cache token conversion by using i32::try_from with
fallback to i32::MAX.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: simplify Anthropic usage extraction and add Default derive
- Use idiomatic `if let` pattern instead of `is_some()` check for usage extraction
- Add Default derive to OpenAIChatUsage for consistency with other usage structs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: use saturating_add to prevent overflow in token accumulation
In long-running agents with many iterations, token counts could
potentially overflow. Using saturating_add ensures values cap at
i32::MAX instead of wrapping around.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* better claude
* nit
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* chore(main): release 1.622.0 (#7742)
* chore(main): release 1.622.0
* Apply automatic changes
---------
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
* fix npm check
* fix: add schema compatibility layer for MCP clients like n8n (#7747)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* nit ui nextcloud triggers (#7749)
* feat: add PocketID OAuth provider support (#7318)
* feat(oauth): add Pocket-ID OAuth provider component
- Implements PocketIdSetting.svelte following Keycloak pattern
- Configures OIDC endpoints for Pocket-ID (/authorize, /api/oidc/*)
- Supports standard OIDC scopes (openid, profile, email)
- Uses passkey-only authentication via Pocket-ID
Refs #5678
* feat(oauth): register Pocket-ID in SSO provider list
- Import PocketIdSetting component
- Add Pocket-ID to provider list in SSO tab
- Update exclusion filter to prevent duplicate custom entries
Refs #5678
* fix(oauth): add missing PocketID icon and fix component integration
- Create PocketIdIcon.svelte component with user profile icon
- Register pocket-id in APP_TO_ICON_COMPONENT mapping
- Fix PocketIdSetting to use IconedResourceType pattern matching other OAuth providers
This resolves the issue where PocketID toggle was not appearing in SSO settings.
Refs #5678
* refactor: migrate PocketIdSetting to Svelte 5 runes syntax
- Use $props() with $bindable() for reactive prop binding
- Use $state() for local reactive state
- Use $derived() for computed values
- Use $effect() for reactive side effects
- Replace on:change with onchange event handler
- Pre-populate base URL from existing config when editing
- Clean up bracket notation to dot notation for value properties
Addresses reviewer feedback
* fix: rename pocket-id to pocketid for naming convention compliance
Change identifier from 'pocket-id' to 'pocketid' to match Windmill's naming convention.
No OAuth provider uses hyphens - all custom SSO providers (keycloak, authentik, authelia,
kanidm, zitadel) use no separator.
Changes:
- AuthSettings.svelte: oauths['pocket-id'] → oauths['pocketid'] (2 locations)
- PocketIdSetting.svelte: name={'pocket-id'} → name={'pocketid'}
- icons/index.ts: 'pocket-id': PocketIdIcon → pocketid: PocketIdIcon
Note: PocketID does not need oauth_connect.json entry as it's a custom SSO provider
with user-configured endpoints, similar to Keycloak/Authentik.
Addresses reviewer feedback
* fix: use TextInput component for consistency
---------
Co-authored-by: hugocasa <hugo@casademont.ch>
* fix: preserve script envs field during sync push
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* nit frontend fix
* chore(main): release 1.623.0 (#7748)
* chore(main): release 1.623.0
* Apply automatic changes
---------
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
* fix: resolve infinite effect loop in PocketIdSetting component (#7753)
* fix: prevent retention cleanup from deleting jobs of active flows (#7755)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* chore(main): release 1.623.1 (#7754)
* chore(main): release 1.623.1
* Apply automatic changes
---------
Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
* feat: default to quickjs on ce for flow eval (#7756)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: runtime assets (#7656)
* Runtime assets
* Nits
* Revert "Nits"
This reverts commit
|
||
|
Diego Imbert
|
635a24f82c |
feat: runtime assets (#7656)
* Runtime assets
* Nits
* Revert "Nits"
This reverts commit
|
||
|
Diego Imbert
|
3affbb3321 |
feat: type-checked data tables v0 (#7381)
* data tables settings ui * install runed * zod 4 fixes * use new toJSONSchema * Migrate ducklake catalogs to more generic custom instance databases * fix compilation * Safety conversion for old duckdb ffi * data tables settings * ts client basis * inline run works * datatables work * Revert "datatables work" This reverts commit |
||
|
Diego Imbert
|
75fdc2cdc9 |
feat: data table schemas (#7353)
* data tables settings ui * install runed * zod 4 fixes * use new toJSONSchema * Migrate ducklake catalogs to more generic custom instance databases * fix compilation * Safety conversion for old duckdb ffi * data tables settings * ts client basis * inline run works * datatables work * Revert "datatables work" This reverts commit |
||
|
Diego Imbert
|
c6c7f3415a |
Specific asset tables (#7323)
* data tables settings ui * install runed * zod 4 fixes * use new toJSONSchema * Migrate ducklake catalogs to more generic custom instance databases * fix compilation * Safety conversion for old duckdb ffi * data tables settings * ts client basis * inline run works * datatables work * Revert "datatables work" This reverts commit |
||
|
Diego Imbert
|
9bbab3321e |
feat: Data tables (#7226)
* data tables settings ui * install runed * zod 4 fixes * use new toJSONSchema * Migrate ducklake catalogs to more generic custom instance databases * fix compilation * Safety conversion for old duckdb ffi * data tables settings * ts client basis * inline run works * datatables work * Revert "datatables work" This reverts commit |
||
|
hugocasa
|
4b26def0cd | chore(parsers): publish bash parser with CRLF handling (#6905) | ||
|
Diego Imbert
|
59cdb141c3 |
NULL Toggle in InsertRow drawer (#6729)
* NULL toggle in InsertRow * fix long type parsing in postgres * nits * graphite catch * lazy_static * support for time/timestamp/tz long forms in pg parser * graphite suggestion |
||
|
Diego Imbert
|
673b4d2a4c |
Advanced S3 permissionning (#6617)
* Advanced permission rules UI * stash * first iteration for s3 rule parsing * Move to glob based approach * cache expiry * fix popover positioning * ee * unused imports * forgot windmill_uploads * Check S3 permissions for apps * nit * typo * ee repo ref * forgot get_workspace_s3_resource_and_check_paths in oss |
||
|
Diego Imbert
|
e6f1211d31 |
feat: Ducklake native support (#6268)
* upgrade duckdb * basic ducklake works * ducklake works with custom db catalogs * fix: pwsh skip already installed modules outside of cache (#6037) * improve query performance of user stats * separate ducklake_catalog db * ducklake settings * DucklakeSettings frontend * Ducklake ws settings saved in backend * fetch ducklake catalog resource * Ducklake works with configured s3 storage * Ducklake as asset * ducklake asset icon * Fix duckdb array and object args not working properly (#6254) * Fix bug with comments in duckdb * Avoid multiple queries when doing ATTACH ducklake * trunc sig no longer needed now that comments are trimmed * cache DuckdbConnectionSettingsResponse * duplicated code * transform_attach_ducklake contributes to duckdb_connection_settings_cache * eliminate the need for used_storages * nit * cleaner management of the bigquery credentials file * DBManagerDrawer refactor to prepare for Ducklake * get ducklake schema * implement delete for ducklake * load column metadata for ducklake * Select query works for ducklake, basic db explorer works ! * duckdb count query * Support all db ops for ducklake * clean migrations * SQL repl for Ducklake * fix broken database studio * nit * assert function * Ducklake in Editor Bar * default ducklake syntax + allow extra args * DucklakeCatalogWizard UI * nit + remove extra $ * modal when databases do not exist * cannot be windmill * Ducklake works safely with instance database * Avoid sending instance db credentials on network * resource leak security * remove fetch_attach_db_conn_str * prevent instance pg password leak * hide asset usage count when not available * case unsensitivity duckdb * warnings * disable instance catalog * use shorthand syntax when inserting with EditorBar * Instance ducklake catalog is now safe to use * use safer argon2 pwd * update package json parsers * update package json * better msgs * tooltips * disable explore button until saved * nit * fix warnings * better ducklake_user password management * nit * Sanitize passwords from errors in ducklake * DisplayResult broken in job result * remove superadmin requirement to check databases_exist * duckdb_connection_settings_v2_inner * Ducklake works on agent worker (finally) * ci * #[allow(dead_code)] * fix openapi missing response * Separate +Database button for DuckDB in EditorBar * Fix dropdown in ducklake settings * Attempt to fix migration race condition in CI * update sqlx failing for some offline queries * avoid temp password for ducklake_user * nits * ducklake settings nits * update duckdb default script * fix sql repl resetting text on refresh * avoid pgcrypto extension --------- Co-authored-by: HugoCasa <hugo@casademont.ch> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
 dieriba
|
10befb995d |
feat: local type references parsing support for main function args (#5995)
* add base struct * feat resolve interface and type declarion in entrypoint param's function * nits * fix reset dependencies * update package * fix handle infinite recursion * add depth level and handle enum for referenced type * nits * nits * nits * perf * fix * done * fix schema form cache inconsistency * fix default type and nits * remove * update Object typ for parser * one level ref from from parent when resolving types and use format for resource * update cli and use resource type * nits * update parsers * fix: use specific parser versions --------- Co-authored-by: HugoCasa <hugo@casademont.ch> |
||
|
Diego Imbert
|
2062a634f6 | Fix duckdb array and object args not working properly (#6254) | ||
|
Diego Imbert
|
433341b295 |
feat: assets as a primary concept (#6125)
* assets migration * parse assets (duckdb) * iterate on assets * S3 object Preview * remove pagination * filterText * better occurence list * tweak * assets in JobPreview * clone impl * AssetsDetectedBadge * improve DbManagerButton + asset dropdown button * edit resource btn * warning when incorrect resource * +Resource in DuckDB * +S3 Object editor bar * nit fix rename * flow asset badge * More Generic OnChange * Highlight assets used in modules * Show occurence count in flow * Better UX, avoid moving parts * nit * Asset nodes * move to dedicated Asset ctx * fix layoutNodes not handling first assetsMap * explore asset btn in flow asset node * correct offset * single computeAssetNodes function * Fix y positioning of nodes with assets * resource editor * write mode node (ui) * accessType in ctx + fix insert button positioning * right positioning when mixing read and write nodes * right positioning when mixing R and W assets * Better layout fix algorithm * listAssetsByUsage and asset nodes on transitive usages * refactor + remove linkAssets * Refactor to allow for custom R/W modes * AssetsDropdownButton in flow script editor * R/W/RW selection and changes node pos in flow * layoutNodes doesnt need recompute now * fix wrong assumption that nodes recompute when assets change * r/w/rw multi toggle * MultiToggle cool animation + clearable * rename + 1px nit * remove mini toggle button group, use ToggleButtonGroup * Combinator parser that detects R / W asset context * nit fix missing flex-1 * missing order by * better ui indication for access type * special x offset case when only one asset node for clarity * parse getResource in TS with swc ecma parser * support load and write s3 detection in TS * Python asset parser * support wmill api calls without special $res: or s3:// syntax * detect out of context asset uris python * do not use access type override when not ambiguous in flow graph * parse_assets match case in rust * AsRef<str> refactor * From impl * Save flow assets * Save script asset usages + fixes + save fallback access types * asset sub icon * max total asset node width to avoid overlap * small refactor * don't parse comments in duckdb assets * fix assets clearing on parse error * fix script asset save in wrong place * load initial asset fallback access types * support variables * ui fixes * Support S3Object as URI in TS client * support new syntax in python client * Support +S3Object in EditorBar for TS and python * Reduce resource requests in assets page * import windmill client when necessary * update s3Types.d.ts * nit fix * Show input resources and s3 objects as assets * improve asset icons * DarkModeObserver refactor * asset page tabs * Moved resource variables and s3object pages to assets tabs * fetch resource usages * Get variables usages * move assets usage dropdown to component * Revert "move assets usage dropdown to component" This reverts commit |
||
|
Diego Imbert
|
fdefd4be93 | feat: duckdb sql lang support (#5761) | ||
|
Diego Imbert
|
c7886ea07a |
feat: sql jobs outputting to s3 + streaming for high-number of rows (#5704)
* stream to s3 boilerplate * S3 works with new syntax * snowflake s3 streaming support * postgres s3 support * fix postgres stream format * mysql s3 streaming * mssql s3 streaming * new s3 mode syntax * optional folder param * rename folder to prefix * json_stream_arr_values * cargo toml rollback * convert_ndjson with datafusion * format conversion kinda works * Fixed not finishing the datafusion writer * support for pg and mssql * fix file ext * bigquery conversion and works with s3 streaming * fix s3 flag parser * snowflake s3 streaming support * factor out duplicate code * remove anyhow * Err case for parse s3 mode * Send error to mpsc * bigquery s3 streaming fix for huge queries * remove extra stuff * snowflake s3 streaming support * small regex mistake * cfg(not(feature = "parquet")) * fix CI (unused import) * error handling fix (graphite) |
||
 wendrul
|
38ee0183aa |
feat: unsafe parameters for sql queries (table names, column names) (#5488)
* Make schema validation struct Schema Validation rules that are constructed from the schema or from the MainArgSig(TODO). * Make other validator builder * Fail dependency job like with lockfile failing for schema validator * Add last types + tests * Remove unused dependency * fix typos * Migration ID was colliding with another, changed it manually * Add Oneof + other fixes * fix: cache for querying scripts correclty handles ScriptMetadata * Add cache for schema validation from main arg sig * Prepare sqlx * Remove default features * Feature flags * WIP: unsafe sql params for sql langauges * Fix down migration table name * cleanup: put validation logic inside a function * Refactor to cache the should_validate boolean Changed the schemavalidators cache to take in an Option<SchemaValidator>, effectively storing the `should_validate_schema` information. Also pass the schema when avaialble to construct the schema validator * Add other job kinds to u8 cache key just in case * Change sql languages to all get arguments as Values instead of RawValue * Only cache if not preview * Add last sql languages and some CI fixes * Rename after typo on `sanitized` * Finish rename * Remove unused import * Fix wrong test * Add newly published regex parser version * Remove default features from cargo.toml * Change to a cleaner syntax for the interpolated args * Update republished parser |
||
|
wendrul
|
77d825540f | feat: add oracle db support on ee (#5062) | ||
|
HugoCasa
|
8c2f2ebb1e | fix: update ms sql template (#5059) | ||
|
HugoCasa
|
ae8f29a4cb |
feat: http routing (#4339)
* feat: http routing * all * feat: improve UI * final stuff * fix: sqlx * fix: nit * fix: nits * fix: error handler display * fix: routes panel perms * all * fix: improve ability to paste from macos in vscode extension * fix lock-write for deno * all * cleaning * fix * cli preprocessor * nits * nits --------- Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com> |
||
|
wendrul
|
858f63344f |
split script argument and dependency parser packages to lighten initial load of the script editor (#4287)
* Add feature flags to split parsers into different pkgs * Split wasm parser imports * Use regex-lite, reorganize the parser split * Update imports to the new wasm parser split * Remove panic system on wasm and simplify snake case convert logic * Adapt new imports * Fix to_snake_case + fix tests * Adapt wasm test dependencies * Add publish script * Fix publish script * Publish script relative to script location * pkg diff + publish * Fix TS WASM import + add pakcage lock * Fix lint --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com> |
||
|
HugoCasa
|
1897ef2f13 | fix: mysql params starting with underscore (#4201) | ||
|
HugoCasa
|
6b41a3473b | fix: mysql support for underscore in named param (#4200) | ||
|
HugoCasa
|
2b9b4eedc3 |
feat: multi sql statement with pg fix (#4134)
* Revert "Revert "feat: multi statement sql (#4104)" (#4133)"
This reverts commit
|
||
|
Ruben Fiszel
|
c578f05e2d |
Revert "feat: multi statement sql (#4104)" (#4133)
This reverts commit
|
||
|
HugoCasa
|
5bc0e96171 |
feat: multi statement sql (#4104)
* feat: multi statement pg * fix: add other flavors * feat: make pg params start at 1 and sequential * fix: improve sql statement parsing * add tests * fix: allow no semi in last statement * fix: merge conflict * fix: minor improvement * fix: parser version |
||
|
HugoCasa
|
6da5e15903 |
feat: add support for bytea in pg (#3926)
* feat: add support for bytea in pg * fix: editor nits |
||
|
HugoCasa
|
a79e09d65f |
feat: improve parsers when no main func (#3805)
* feat: improve parsers when no main func * chore: update parser version |
||
|
HugoCasa
|
e8905b2734 | feat: pg add json support (#3620) | ||
|
HugoCasa
|
b1ae732df0 |
fix: support all pg types from db studio (#3613)
* fix: support all pg types from db studio * chore: parser pkg frontend update |
||
 Ruben Fiszel
|
c08eb0abc8 | fix: remove requirement on full wasm parser for row insert of db studio | ||
|
HugoCasa
|
c03c73797b | fix: pg timstamptz param (#3364) | ||
|
HugoCasa
|
1ee45a1f7d | feat: allow to pin database in sql scripts (#3304) | ||
|
Ruben Fiszel
|
e485811309 | fix: improve sql default arg parsing + auto invite | ||
|
HugoCasa
|
825448e1f1 |
feat: add mysql datetime (#2808)
* feat: mysql datetime support * feat: nits |
||
|
HugoCasa
|
6e138528dd | feat: mysql named params (#2805) | ||
|
HugoCasa
|
2fe623cf6b |
feat: add sql server (#2604)
* feat: add sql server * feat: mssql test, db schema, AI * chore: update to latest parser * fix: clean mssql executor |
||
|
HugoCasa
|
9002a0e05a |
fix: add numeric, array and date types (#2379)
* fix: add numeric, array and date types * fix: update parser npm |