* feat(cli): add datatable list and run commands
* feat(cli): render datatable query results as a table
* feat(cli): serve datatables as a postgres-wire endpoint
* feat(cli): add 'datatable psql' to launch psql against the proxy
* feat(cli): route datatable serve by client-supplied database name
* override database list + password option
* fix: support extended queries in datatable serve
* fix: correct cloud size threshold log and parse CLI descriptions with parens/trailing comma
* refactor: extract raw_output envelope encoding into pg_raw_output module
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
* feat(nsjail): make tmpfs size configurable via instance setting
Adds a new `nsjail_tmpfs_size_mb` instance setting that overrides the
size of the `/tmp` tmpfs mount inside the nsjail sandbox across all
languages. When unset, the existing per-language defaults (500MB or
800MB) continue to apply, so no behavior change for existing
deployments.
The setting is exposed under Settings → Jobs and is read at job
execution time, so changes take effect on the next job without a
restart.
Fixes WIN-1963
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(nsjail): unify default tmpfs size to 800MB
Previously each executor passed its own per-language default (500MB or
800MB) to resolve_nsjail_tmpfs_size. Unify on a single
DEFAULT_NSJAIL_TMPFS_SIZE_BYTES constant (800MB) so the placeholder
behavior is consistent across languages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(nsjail): resolve tmpfs size outside ruby download closure
The download.ruby config render runs inside a sync closure passed to
par_install_language_dependencies_seq, so `.await` on
resolve_nsjail_tmpfs_size() was a compile error under the `ruby`
feature. Resolve the size once before the closure and capture the
string instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(nsjail): rename resolver to *_bytes and clarify fallback
Addresses CI review feedback:
- Rename `resolve_nsjail_tmpfs_size` to `resolve_nsjail_tmpfs_size_bytes`
so the returned unit is unambiguous at the call site (cubic P2).
- Fix the `NSJAIL_TMPFS_SIZE_MB` doc comment that still said "per-language
default" — there is no per-language fallback anymore, all unset
values resolve to the unified 800MB `DEFAULT_NSJAIL_TMPFS_SIZE_BYTES`
(codex/pi P2).
- Expand the resolver doc to call out that `Some(0)` and negative values
also fall back, since the match arm is `Some(mb) if mb > 0`.
No behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: thread temp_script_refs into preview jobs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: resolve python preview relative imports from temp script refs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: use local relative imports in wmill script preview
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: use local relative imports in wmill flow preview
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add temp_script_refs to Preview and FlowPreview openapi schemas
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: pass temp_script_refs to bun lockfile gen for no-lock preview
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: route script preview through shared buildPreviewTempScriptRefs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: resolve local relative imports in wmill app dev inline scripts
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address cubic review — bundle cache key, preview-mode gate, error masking
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* perf: skip dep-tree build when previewed script has no relative imports; narrow old-backend classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address review issues (preview-only gate, bundle preview, app dev cwd)
Three P1s flagged in repeated codex/pi reviews on PR #9233:
- Gate _TEMP_SCRIPT_REFS extraction on JobKind::Preview (bun + python
executors) and propagation in worker_flow on JobKind::FlowPreview. job.args
includes caller-controlled request args, so honoring this key on deployed
runs would let a caller swap import resolution to local content uploaded
via /raw_temp.
- run_bundle_preview_script now injects temp_script_refs into PushArgs.extra,
mirroring run_preview_script — closes the silent data drop for the bundle
preview path.
- wmill app dev chdirs to the wmill.yaml root before buildPreviewTempScriptRefs
and restores after, so the `cd <app>__raw_app && wmill app dev` invocation
(cwd is the raw_app folder, no app_folder arg) still walks sibling workspace
scripts like f/lib.ts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ee): bump ee-repo-ref to 5b347d6 (handle_python_deps arity fix)
Picks up the EE arity fix so cargo_test + check_ee_full compile cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ee): bump ee-repo-ref to 52e273d (agent-workers bundle path arity fix)
Picks up windmill-ee-private 52e273d which adds the missing &None arg to
compute_bundle_local_and_remote_path in windmill-api-agent-workers/src/ee.rs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ee): bump ee-repo-ref to 2d6ffd3 (EE main merged in)
Previous bump pinned an older EE commit, missing the audit-log object-store
export module (EE PR #579, commit ec3cd35) and other EE main updates. The
CE backend's `crate::ee_oss::anchor_audit_logs_s3_checkpoint_env_var` and
`export_audit_logs_to_object_store` references need the new EE definitions.
Merged origin/main into the EE branch and pinned the merge commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to b0c87b1272c25dca4aa9148c87fb024a9d9ef322
This commit updates the EE repository reference after PR #583 was merged in windmill-ee-private.
Previous ee-repo-ref: 2d6ffd32c99bd93e79cf78675cb89499a81b17e1
New ee-repo-ref: b0c87b1272c25dca4aa9148c87fb024a9d9ef322
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(snowflake): derive public key from private key when omitted (WIN-1959)
Snowflake key-pair auth needs a SHA256 fingerprint of the public key for
the JWT iss claim, but the public key is mathematically derivable from
the RSA private key. Other tools (e.g. Power BI) only require the
private key, so requiring users to supply both is redundant. When
public_key is missing, fall back to deriving it from private_key (PKCS#8
or PKCS#1 PEM) instead of erroring out.
Fixes WIN-1959
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(snowflake): treat empty public_key/private_key as missing
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses local-review findings before PR:
- P1: record_and_check_join_slot was a non-atomic check-then-act on a
pooled connection; concurrent completion of a subscriber's last two
partition-bearing inputs on different workers could double-dispatch.
Now one transaction guarded by a tx-scoped advisory lock keyed on
(workspace, subscriber, partition) so the gate fires exactly once.
- P2: the preprocessed-args overwrite in result_processor replaced args
wholesale, dropping a partition resolved by resolve_partition_for_job;
the UPDATE now preserves an existing persisted partition key.
- P2: gate resolve_partition_for_job on a cheap code.contains check so
non-pipeline script jobs skip the annotation scan on the hot path.
- P2: remove 40 scratch screenshot PNGs, a flicker-debug script and a
local scheduler lock accidentally committed; gitignore the lock.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Stage C: in handle_code_execution_job, once the script content is loaded,
parse the // partitioned annotation (free here) and resolve the concrete
partition once — schedule fire-time (scheduled_for anchor, not wall-clock)
for time kinds, triggering payload for dynamic. The value is injected
into the in-memory args the body sees (via a shadowed job clone) and
persisted back to v2_job.args so dispatch_asset_triggers propagates the
same value down the cascade. Already-set (explicit/backfill/cascade)
partitions are never re-resolved (run identity immutable); unresolvable
partitioned runs fail with a clear error. Integration test exercises the
full worker loop + cascade propagation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix: atomic bundle cache writes to prevent parallel cold-load race
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: trust-but-replace in atomic_publish_dir to never trust a stale partial cache dir
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: simplify atomic_publish_dir and add content-addressed rename-failure fallback
Revert the destroy-then-recreate dir swap (introduced concurrent-publisher
edge cases: spurious Err under a real herd, EACCES masking a stale partial)
back to a single atomic rename. Add the content-addressed exists-fallback to
atomic_write_file_bytes/atomic_copy_file so the loser of a publish race (and
Windows, where rename cannot replace an open/existing destination) treats the
already-published identical cache as success instead of failing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: aggregate wait time should target the true root job, not flow_innermost_root_job
* refactor: reuse get_root_job_id helper for wait-time aggregation
Instead of duplicating the root_job → flow_innermost_root_job →
parent_job fallback chain inline, call the existing get_root_job_id()
helper (the same one used by push_next_flow_job) and filter out the
self-id case so standalone scripts still skip aggregate insertion.
Behaviorally identical, single source of truth.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: windmill-internal-app[bot] <1429786+windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(python): preserve strings containing Infinity/NaN in result JSON
* test(python): add sanity checks for Infinity/NaN in results
* refactor(python): use string-aware regex callback for single-pass cleanup
* refactor(python): compact regex + handle backslash-escape parity
* perf(python): short-circuit cleanup when no Infinity/NaN/NUL in result
* fix(bun): pass --preserve-symlinks on unbundled execution
Bun 1.2/1.3 moved its global package cache to a content-addressed
layout and the installer now creates a single directory symlink from
node_modules/<pkg> to the cache entry. Without --preserve-symlinks,
Bun resolves modules from each file's realpath, so any require/import
inside an installed package walks up from cache_nomount/bun/... and
never finds the sibling deps living under <job_dir>/node_modules/.
This manifested as e.g.
ENOENT while resolving package 'zod/v3' from
'/tmp/windmill/cache_nomount/bun/@langchain/core@1.1.44@@@1/dist/...'
on //nobundling scripts that pull @langchain/core, even though zod is
correctly installed alongside it in node_modules.
The bundled execution path already had --preserve-symlinks since #4132
(needed because we symlink the cached bundle file into the job dir).
The unbundled path didn't, because at the time Bun installed via per-
file hardlinks and the realpath of node_modules entries was the job
dir itself. The Bun installer's layout change made the flag necessary
on the unbundled path as well.
Add the flag to all three unbundled `bun run` invocations:
- nsjail unbundled path
- non-nsjail unbundled path
- dedicated worker (always unbundled)
This also fixes a latent bug on the first run of any bun script that
imports a package whose internals reference siblings (the build_cache
path runs unbundled this round while it builds the bundle for next
time).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(bun): regression test for nobundling + transitive require resolution
Adds an integration test that mirrors the original failure: a //nobundling
script importing @langchain/core, which (in its CJS internals) does
require('zod/v3'). Before --preserve-symlinks was added to the unbundled
bun run invocations, this failed with:
ENOENT while resolving package 'zod/v3' from
'.../cache_nomount/bun/@langchain/core@<ver>@@@1/dist/runnables/base.js'
The test covers the non-nsjail unbundled path. Reproducibility of the
pre-fix failure depends on Bun's installer choosing the directory-symlink
layout for the node_modules entry (the default on Bun 1.2/1.3+ with the
new content-addressed global cache that produced the user's error).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: refine windmill ai refactor plan
* refactor: move ai sse plumbing to windmill-ai
* refactor: remove ai re-export shims
* fix: update ee ai memory ref
* chore: update ee-repo-ref to d3bc7fa85195b46b7a38d43c2f806520bf8b5454
This commit updates the EE repository reference after PR #560 was merged in windmill-ee-private.
Previous ee-repo-ref: ff35bf7cc198e13884b33654e1d6dbd8a8b314d3
New ee-repo-ref: d3bc7fa85195b46b7a38d43c2f806520bf8b5454
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
The Python per-package dependency cache could persist an incomplete wheel
extraction with `.valid.windmill` set, then propagate that broken artifact
to every worker through the object store. Customer hit this on
argon2-cffi==25.1.0 (missing argon2/_utils.py), and previously on
botocore/httpx (truncated tars). Symptom is a runtime ImportError that
looks like a missing dependency declaration rather than a Windmill bug.
Three changes that together stop the propagation:
1. After `pull_from_tar`, parse the wheel's `<dist-info>/RECORD` and
confirm every listed path exists on disk before writing
`.valid.windmill`. On failure, wipe the directory and fall through
to a fresh local install — the next install also self-heals the
broken object-store entry by pushing a fresh tar.
2. After `uv pip install` succeeds, run the same RECORD check before
queuing the piptar upload or writing `.valid.windmill`. A bad install
never becomes the source of a broken tar in the object store.
3. Finalize the tar (`drop(tar.into_inner()?)`) before reading its bytes
for upload, so we never push an unfinalized archive (no end-of-archive
marker) to the object store.
Verified with a 60-package end-to-end integration test (first-fill →
clear-local-cache → re-pull-from-objectstore → corrupt-objectstore-tar
→ detect-and-self-heal). All 27 packages on the live test pulled cleanly,
and the deliberately corrupted argon2-cffi tar was caught with the exact
expected log line ("wheel RECORD lists files missing on disk: argon2/_utils.py")
and replaced with a fresh tar.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: parse windmill_failure field in job result to tag run as failure
* feat: preserve top-level fields when windmill_failure tags a run as failure
* fix: address review findings on windmill_manual_failure
* refactor: rename windmill_manual_failure to wm_failure and add wm_* aliases
* fix: prefer injected ManualFailure error over sibling name/message in OTel
The Python per-package dependency cache could persist an incomplete wheel
extraction with `.valid.windmill` set, then propagate that broken artifact
to every worker through the object store. Customer hit this on
argon2-cffi==25.1.0 (missing argon2/_utils.py), and previously on
botocore/httpx (truncated tars). Symptom is a runtime ImportError that
looks like a missing dependency declaration rather than a Windmill bug.
Three changes that together stop the propagation:
1. After `pull_from_tar`, parse the wheel's `<dist-info>/RECORD` and
confirm every listed path exists on disk before writing
`.valid.windmill`. On failure, wipe the directory and fall through
to a fresh local install — the next install also self-heals the
broken object-store entry by pushing a fresh tar.
2. After `uv pip install` succeeds, run the same RECORD check before
queuing the piptar upload or writing `.valid.windmill`. A bad install
never becomes the source of a broken tar in the object store.
3. Finalize the tar (`drop(tar.into_inner()?)`) before reading its bytes
for upload, so we never push an unfinalized archive (no end-of-archive
marker) to the object store.
Verified with a 60-package end-to-end integration test (first-fill →
clear-local-cache → re-pull-from-objectstore → corrupt-objectstore-tar
→ detect-and-self-heal). All 27 packages on the live test pulled cleanly,
and the deliberately corrupted argon2-cffi tar was caught with the exact
expected log line ("wheel RECORD lists files missing on disk: argon2/_utils.py")
and replaced with a fresh tar.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: parse windmill_failure field in job result to tag run as failure
* feat: preserve top-level fields when windmill_failure tags a run as failure
* fix: address review findings on windmill_manual_failure
* refactor: rename windmill_manual_failure to wm_failure and add wm_* aliases
* fix: prefer injected ManualFailure error over sibling name/message in OTel
* fix(flows): inherit flow_env in sub-flow predicates
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(flows): align flow_env lookup with get_root_job_id and tighten gate
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(flows): drop recursive CTE, root_job propagation suffices
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(flows): walk via flow_innermost_root_job to respect imported-flow scope
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(flows): remove flow_env API endpoint, dead code from deno_core era
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(flows): don't bubble error when continue_on_error is on the last step
When the last step of a flow (or branch/forloop) failed with continue_on_error
or skip_failures enabled, should_continue_flow resolved to false (because the
flow was at its last step), and the flow was completed with success=false.
This made parent flows / subflows treat the run as a failure even though the
user explicitly asked to continue past errors.
Detect this case and set success=true so the failure is captured in the
result but not propagated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: explain why success is overridden post should_continue_flow
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: distinguish job-already-completed from execution failure on OTLP span
handle_queued_job's bool return type conflated two distinct Ok(false)
cases: a real race with another worker (Error::AlreadyCompleted) and any
job execution that returned an error via process_result. The outer "job"
span recorded "job already completed by another worker" for both, so
every failed Python/bun/etc. script ended up with that misleading
otel.status_message even though the real error was correctly recorded
on the inner job_postprocessing span.
Replace the bool with a JobOutcome enum (Completed / Failed { description }
/ AlreadyCompleted). Failed carries the truncated error string, so the
outer span's Status.message now reflects the actual cause.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: extract user-facing error from error_value for span description
Cubic flagged that capturing e.to_string() before the match meant
ExitStatus failures (the most common failure mode for script jobs)
ended up with the generic "exit status: …" string rather than the
script error extracted from job logs by extract_error_value.
Move the description capture to after error_value is built and
deserialise it as ErrorMessage to pull out the structured message.
Falls back to "Job failed" if the value isn't shaped that way.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: use permissive description extraction for OTLP status message
Cubic flagged that strict ErrorMessage parsing downgraded real failures
to a generic "Job failed" whenever the result wasn't shaped as
{message, name} — e.g. agent-worker's "See logs for more details" raw
string, or runtime-written result.json files with a different shape.
Switch to parsing as serde_json::Value and pulling .message out if it's
a string, falling back to the whole value if it's a bare string. Only
fall back to "Job failed" when neither is available.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* feat: support assigning a worker tag to app/raw-app inline scripts
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: omit empty tag field from inline script raw_code payload
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* style: shrink tag popover width
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(pg): wrap encoder errors with arg context, add fallback test
Followups on #8999 review:
- Wrap rust-postgres "error serializing parameter N" failures with the arg
name, JSON value kind, and asserted Postgres type plus a hint about an
explicit cast — so users see actionable context instead of an opaque
WrongType.
- Drift-prevention meta-test: assert otyp_to_pg_type and convert_val agree
on the Type for every recognised arg_t when the JSON value matches its
natural Rust kind. Catches future drift if either side changes.
- Integration test for the prepare + query_raw fallback path: confirms
unrecognised arg_t (custom enum) is routed through prepare and the
server-resolved type appears in the failure surface — flips into a
test failure if a regression accidentally routes unrecognised types
through query_typed_raw.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pg): add otyp_inferred flag + regex-based placeholder renumbering
Two follow-ups from the review of #8999:
1. **Issue #1 (Number/Bool + explicit text decl in WHERE)**
Add `Arg::otyp_inferred: bool` to the parser. The PG SQL parser sets
it `true` only at the "no info → fall back to text" site (bare `$N`,
no inline cast, no `-- $N (TYPE)` decl). All other arg sources keep
it `false`.
In `convert_val` this flag distinguishes:
- explicit text-like target (`-- $1 (text)` or `$1::text`) — coerce
`Bool`/`Number` → `Box<String>` so `WHERE text_col = $1` works
(`text = text` operator). Pre-#8988 behaviour, restored.
- parser-default text (bare `$N`) — bind the value's natural Rust
type so the regression case (`Value::Bool` against a real `bool`
column via `CAST AS bool`) keeps working.
`Arg` is in `windmill-parser`; the new field has `#[serde(default)]`
so persisted signatures stay backward-compatible.
2. **Issue #4 ($5/$50 substring rewrite collision)**
Replace the per-index `String::replace` chain (which turned `$50`
into `$10` when oidx=5 was processed first) with a single regex
pass. `\d+` is greedy, so `$5` and `$50` match as distinct units;
indices outside the mapping are left intact.
3. Tests:
- parser: `test_parse_pgsql_otyp_inferred_flag` covers bare/inline-
cast/decl/mixed shapes.
- executor unit: `convert_val_bool_against_every_arg_t` and
`convert_val_*_number_*` split each text-like target into explicit
vs inferred expectations.
- executor unit: `renumber_sparse_placeholders_no_collision`.
- integration: `test_postgresql_arg_type_combinations` adds 4 cases
covering decl(text)+Number/Bool in WHERE, bare $1+Bool, and
sparse positional args ($5/$50).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pg+sdk): enum support, extended String arms, position-aware $N rewrite, SDK quality
Backend:
1. **`AnyTextValue` ToSql/FromSql wrapper**: vanilla `tokio_postgres`'s
`ToSql for String` / `FromSql for String` reject `Kind::Enum` and
`Kind::Domain` even though the wire format is plain UTF-8. The wrapper
accepts those kinds in both directions. End result: explicit
`$1::my_enum` / `CAST($1 AS my_enum)` casts now round-trip without the
ugly `CAST($1::text AS my_enum)` workaround, AND `SELECT enum_col`
results come back as JSON strings instead of erroring at the FromSql
layer.
2. **#10 — Value::String → numeric/real/double/oid/bool**. Without these
arms, a string-encoded value (`"3.14"`, `"true"`) for a non-text /
non-temporal arg_t fell through to `Box<String> + TEXT`, which then
failed at the server (no implicit cast text→numeric in expression
context). Now strings are parsed into the matching native type with
clear error messages on parse failure.
3. **Position-aware `$N` rewrite**: replaces the regex-based renumbering
(which fixed the `$5/$50` substring collision but still walked through
string literals and comments, mangling `'price: $5'` etc.) with a
walk over `parse_pg_statement_arg_positions` — the same
string/comment/dollar-quote-aware tokenizer used for index discovery.
Adds `parse_pg_statement_arg_positions` to the parser's public API.
SDK:
4. **BigInt support**: `JSON.stringify(BigInt)` throws. The SDK now
stringifies bigints before serialisation; the executor accepts
numeric strings into BIGINT arg slots via the existing
`Value::String → INT8` parsing arm. SDK-side `inferSqlType` is split
so `BigInt` always resolves to `BIGINT` (was reaching
`Number.isInteger(BigInt)` which returns false → wrong default).
5. **Homogeneous array auto-tag**: `${[1,2,3]}` against an `int[]` column
now emits `$1::BIGINT[]` instead of `$1::JSON`. Detection covers
primitive types only (number / bigint / string / boolean); mixed or
nested arrays still fall back to JSON. Mixed int/float widens to
`DOUBLE PRECISION[]`.
6. **`.query()` positional bug**: previously the `.query()` method
abused the template-tag builder, which appended `$N::TYPE` after the
user's literal SQL string instead of binding by position
(`SELECT $1, $2` became `SELECT $1, $2$1::BIGINT`). Now `.query()`
builds the executor-shaped content directly: a `-- $N argN (TYPE)`
declaration block followed by the user's SQL verbatim.
Tests:
- Parser: `test_parse_pg_statement_arg_positions_skips_strings_and_comments`
asserts string literals, comments, and dollar-quoted blocks don't
produce positions (so renumbering doesn't mangle them).
- Executor unit: `renumber_sparse_placeholders_no_collision_no_string_mangling`
uses the new position-aware path and includes string-literal + comment
+ `$$…$$` cases. Existing convert_val tests grow to cover new
String→numeric/real/double/oid/bool arms.
- Integration: `test_postgresql_arg_type_combinations` adds 13 cases
(enum round-trip both directions, string→numeric/real/double/bool/oid,
string-literal `$N` non-mangling). The prepare-fallback test now
asserts SUCCESS (not failure) for enum encoding via AnyTextValue.
- SDK: new `typescript-client/tests/sqlUtils.test.ts` (42 tests)
exhaustively covering inferSqlType primitives + arrays,
parseTypeAnnotation, datatable() template tag (with all the new
shapes — BigInt, homogeneous arrays, RawSql, schema preamble),
datatable().query() positional, and ducklake() shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pg): replace DISCARD ALL with curated reset (preserves typeinfo cache)
Found while exhaustively probing custom-type DX: every cached-connection
reuse was running `DISCARD ALL`, whose included `DEALLOCATE ALL`
deallocates *all* prepared statements server-side — including the typeinfo
statements that tokio_postgres caches per-Client to resolve custom enum /
domain Oids. tokio_postgres still held `Statement` objects whose names
the server had forgotten, so the next custom-type query failed with
intermittent "prepared statement \"sN\" does not exist" errors. The
failure was easy to reproduce: any sequence that forced typeinfo lookup
for two different custom-type kinds on the same cached connection (e.g.
enum followed by domain) would hit it.
Replace `DISCARD ALL` with a curated reset that explicitly targets the
state we actually care about, *without* touching prepared statements:
RESET ALL — GUC parameters (search_path, application
_name, statement_timeout, …)
RESET SESSION AUTHORIZATION — undoes both `SET SESSION AUTHORIZATION`
and `SET ROLE` (RESET ALL does NOT —
these aren't GUC parameters, so without
this an elevated role from a previous
job would silently leak)
UNLISTEN * — drops LISTEN registrations
CLOSE ALL — closes open cursors
Trade-off: temp tables, advisory locks (session-scoped), and user-created
PREPARE statements may persist across cached-connection reuse — rare in
datatable / PG-script workloads. tokio_postgres's typeinfo cache survives
intact, so custom enum / domain queries are fast on subsequent reuse.
Tests:
- `test_postgresql_custom_types_on_cached_connection` — runs 10×
alternating enum + domain queries on a cached connection. Pre-fix this
failed with `prepared statement "sN" does not exist` after the first
reuse; post-fix passes.
- `test_postgresql_set_role_does_not_leak_across_cached_connection` —
switches `SET ROLE` and `SET SESSION AUTHORIZATION` to a non-postgres
role, then runs a follow-up job and asserts current_user/session_user
are restored. Specifically catches the case where someone might switch
back to `RESET ALL` alone (which doesn't cover SET ROLE / SESSION
AUTHORIZATION) and silently introduce a permission-leak vector.
- All existing session-isolation tests
(`test_postgresql_cached_connection_resets_session`,
`test_postgresql_single_worker_session_isolation`,
`test_postgresql_100_jobs_cached`) continue to pass.
Found via end-to-end probing of datatable / PG-script DX, not previously
covered: the existing isolation tests only did `SET ROLE postgres`, the
connecting user, so the leak was invisible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pg): address PR #8999 review (cubic + claude)
cubic (P1, real bug):
- `convert_vec_val` for `timetz` array asserted `Type::TIMETZ_ARRAY`, but
chrono `NaiveTime` only encodes for TIME (same caveat as the scalar
arm). Switch to `Type::TIME_ARRAY`; rely on PG's implicit `time→timetz`
assignment cast at the column site. Add an explicit unit test.
claude (#1, silent failure → explicit error):
- `Bool` + explicit `(char)` / `(character)` decl previously silently
bound BOOL, hoping the server would cast at the use site — but PG has
no implicit `bool→char` and the resulting error
("operator does not exist: bool = char") was opaque. Now error at
bind time with an actionable hint to use `bool` decl or pass the
value as a "t"/"f" string.
claude (#2, asymmetry doc):
- Object/Array still coerce to text on `matches!(typ, Typ::Str(_))`
(covers both explicit AND inferred-default text), unlike Bool/Number
which key on `explicit_text_target`. The asymmetry is intentional
(no implicit `jsonb → text` cast in expression context vs PG having
implicit `bool/int → text` casts) — added a body comment so future
maintainers don't try to "align" them.
claude (#3, perf):
- `parse_pg_statement_arg_indices` and `parse_pg_statement_arg_positions`
walked the SQL tokenizer twice. Fold into a single pass that derives
the index set from the position list.
claude (#4, fmt drift):
- `cargo fmt` over the parser crates I touched with perl scripts in the
earlier commit (windmill-parser-{sql,bash,ts,go,php,java,csharp,nu,py,
rust,graphql,yaml,r}). Net cosmetic.
claude (#5, parseTypeAnnotation):
- One-line caveat in the SDK's `parseTypeAnnotation` that the returned
string is presence-only (e.g. `${x}::DOUBLE PRECISION` returns
`"DOUBLE"`, `CAST(${x} AS int)` returns `"int)"` — neither matches a
real PG type, but the only consumer just checks `!== undefined`).
While here — discovered + fixed independently while exhaustively probing
DX:
- **Replace `DISCARD ALL` with curated reset** (`RESET ALL; RESET
SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;`). DISCARD's
`DEALLOCATE ALL` killed tokio_postgres' typeinfo cache, producing
intermittent `prepared statement "sN" does not exist` errors on
custom-type queries after cached-conn reuse. New regression tests:
`test_postgresql_custom_types_on_cached_connection` and
`test_postgresql_set_role_does_not_leak_across_cached_connection`
(the latter catches the case where someone might switch back to
`RESET ALL` alone and silently introduce a permission-leak vector —
RESET ALL doesn't cover SET ROLE / SET SESSION AUTHORIZATION).
- **ISO-8601 timestamp results** (`pg_cell_to_json_value`). Pre-fix
`TIMESTAMP` was rendered with a space separator ("2024-01-15 10:30:00")
and `TIMESTAMPTZ` with " UTC" suffix ("2024-01-15 10:30:00 UTC") —
neither parseable by `date-fns parseISO`, JavaScript `new Date()` is
lenient enough to handle them but several frontend `App*Input.svelte`
components use parseISO and fail silently. Switched to ISO-8601 with
`T` separator and `+00:00` offset; arg-parsing path still accepts the
legacy " UTC" suffix for back-compat.
Test coverage:
- 17/17 unit (`pg_executor::tests`)
- 9/9 integration (`backend/tests/worker.rs`, `test_postgresql_*`)
- 27/27 parser (`windmill-parser-sql`)
- 42/42 SDK (`typescript-client/tests/sqlUtils.test.ts`)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pg): bounded one-shot warning on numeric precision loss + ISO-8601 + NaN handling
Found while probing PG-script DX with millions of numeric cells:
1. **Numeric precision-loss warning**: `numeric` results are still serialised
as JSON Number (back-compat — switching to JSON String would silently
break user code doing arithmetic on results), but we now detect
`Decimal -> f64 -> Decimal` round-trip failure and emit a single
job-log warning recommending a `::text` cast in the SQL. Bounded by
`NUMERIC_PRECISION_CHECK_BUDGET = 256` cells per query (one atomic
load + one fetch_sub on the hot path; first lossy value
short-circuits to a single load thereafter). Worst-case overhead on
a 1M-cell numeric-heavy query: ~25µs of checks + 5ns × N atomic
loads (vs. ~100ms unbounded).
2. **ISO-8601 timestamps**: `pg_cell_to_json_value` previously returned
`"2024-01-15 10:30:00"` (TIMESTAMP) and `"2024-01-15 10:30:00 UTC"`
(TIMESTAMPTZ) — neither parseable by date-fns `parseISO`, which is
what the apps `App*Input.svelte` components use, so timestamp values
silently failed to round-trip into date pickers. Switch to ISO-8601
(`T` separator + `+00:00` offset) on the result side; arg-parser
continues to accept the legacy `" UTC"`-suffixed format for
back-compat.
3. **Float NaN / Infinity results**: `Number::from_f64` returns None for
NaN / ±Inf, which `pg_cell_to_json_value` was raising as
"invalid json-float" — failing the *entire* query if any cell held
one of these special values. Now serialise them as JSON strings
("NaN", "Infinity", "-Infinity") and let the rest of the row come
through. Arg-side: `s.parse::<f64>()` already accepts the same
strings.
Tests:
- `decimal_fits_f64_losslessly_predicate` — covers fits / doesn't-fit
cases for the precision-loss predicate.
- `precision_check_budget_caps_per_query_overhead` — locks in the
budget cap and the loss-flag short-circuit.
- All 9 PG integration tests + 17 unit tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pg): add pg_advisory_unlock_all to reset; warn on missing args; honor decl defaults
While probing PG-script DX further found three more frictions:
1. **Advisory lock leak** (cubic P2): switching from `DISCARD ALL` to
`RESET ALL; RESET SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;`
meant session-scoped advisory locks (`pg_advisory_lock`) leaked
across cached-connection reuse. Add `SELECT pg_advisory_unlock_all()`
to the chain — `DISCARD ALL` covered this implicitly via
`DISCARD PLANS / DEALLOCATE / pg_advisory_unlock_all` and we lost it
in the switch.
2. **Missing-arg silent NULL**: an arg declared in the SQL (e.g.
`-- $1 amount (numeric)`) but not provided in the args object was
bound as NULL with no error / warning. Misspelling the key in the
args object silently produced a row of NULLs — a notorious DX
debugging trap. Now: collect the names of declared-but-missing
args during dispatch and emit a single one-shot warning to the job
logs at end-of-query naming each one. Bound NULL is preserved for
back-compat.
3. **Declaration defaults ignored**: `-- $1 a (int) = 5` carries
`arg.default = Some(Number(5))`, but the dispatch fell straight to
NULL when the arg was missing. Now: respect the default —
user-supplied value > declaration default > NULL. Also fixes the
warning logic above (only warn for args that *don't* have a default).
Tests: existing 19 unit + 9 integration pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pg): multi-word PG types with [] suffix lost the array-ness; array arms accept stringified values
Two more frictions found while probing SDK end-to-end against a real
datatable resource:
1. **Multi-word array types lose the [] suffix in the parser**.
`transform_types_with_spaces` recognises aliases for "double
precision", "character varying", "timestamp with time zone", etc.
but its return type was `&'a str` — only the bare alias, never with
a trailing `[]`. The `RE_CODE_PGSQL` regex's `\w+` captures stop at
the first space, so the regex's own `(?:\[\])?` array-suffix branch
sees only `"double"` (not `"double precision[]"`); the `[]` was
silently lost. Result: `$1::double precision[]` (which the SDK now
emits for homogeneous float arrays via the new auto-tag) routed
through `Value::Array → Type::JSONB` and the server failed with
"cannot cast type jsonb to double precision[]".
Fix: switch `transform_types_with_spaces` to return `Cow<'a, str>`
and re-check the trailing bytes after a multi-word match. If they
start with `[]`, return `format!("{alias}[]")` — Owned. Single-word
types and the no-match path keep returning Borrowed slices, so no
allocation in the hot path.
2. **Array arms in `convert_vec_val` rejected stringified values for
numeric / int* / bool / oid / real / double**. The scalar `convert_val`
already parses strings into the matching native type for these arg_ts,
but the array variant only accepted JSON-native counterparts. Sending
`["1.5", "2.5", "3.5"]` against `$1::numeric[]` (e.g. via `unnest` for
bulk loading, or `JSON.stringify(BigInt[])` round-trip) failed with
"Mixed types in array". Now the array arms mirror the scalar ones —
`as_<native>().or_else(|| as_str().and_then(parse))` — so both shapes
round-trip cleanly.
Tests: 19 unit + 9 integration pass; existing parser tests cover the
multi-word array forms (the regex-cap behaviour didn't break for
single-word types, and Cow plumbing is transparent to all callers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(parsers): add otyp_inferred field to Arg literals in tests + 3 missed src files
CI failures: the perl-driven sweep that added `otyp_inferred: false` to
every `Arg { ... }` literal when I introduced the field in the parser
schema covered `src/lib.rs` files but missed:
- parsers/windmill-parser-bash/src/lib.rs (mass-edited but a
later format pass un-applied a few sites)
- parsers/windmill-parser-go/src/lib.rs (same)
- parsers/windmill-parser-graphql/src/lib.rs (same)
- parsers/windmill-parser-nu/tests/tests.rs (test file — not
swept the first time)
- parsers/windmill-parser-ts/tests/tests.rs (test file — same)
Also tightened the regex to handle `oidx: None` without the trailing
comma (some test files had the field as the last initialiser line).
`cargo build --features <CI feature combo> --workspace --all-targets`
is clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sdk): Date → TIMESTAMPTZ; NaN / ±Infinity → string
Two more frictions found while running the actual SDK end-to-end against
a live datatable resource:
1. **JS `Date`** fell into the typeof "object" branch and was tagged
`::JSON`. It worked accidentally for `${date}::timestamptz` via PG's
`json → text → timestamptz` implicit cast chain, but `${date}` against
a `timestamptz` column without a user-supplied cast bound the value
as a JSON string and the comparison `timestamptz = json` failed. Now:
`inferSqlType` recognises `Date` and tags `::TIMESTAMPTZ`;
`serializeArgValue` emits `Date.toISOString()` so the executor's
`Value::String → TIMESTAMPTZ` arm parses it cleanly.
2. **JS `NaN` / `±Infinity`** silently became NULL. `JSON.stringify(NaN)`
returns `"null"` per the JS spec, so the value reached the executor as
JSON null — the SDK's `::DOUBLE PRECISION` tag then bound a NULL
double. Fix: detect non-finite numbers in `serializeArgValue` and
stringify them as `"NaN" / "Infinity" / "-Infinity"`. The executor's
`Value::String → FLOAT8` arm (`f64::from_str`) accepts these literals
directly, and the result-side already renders the values as JSON
strings (matching round-trip).
SDK unit tests grow from 42 → 44 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(pg): integration coverage for multi-word arrays + stringified array elements
Locks in the two array fixes from the previous commit
(`fix(pg): multi-word PG types with [] suffix lost the array-ness`)
with end-to-end cases in `test_postgresql_arg_type_combinations`:
- `double precision[]`, `character varying[]`, `timestamp without time
zone[]` — verifies the parser keeps the `[]` suffix after multi-word
alias resolution.
- `numeric[]` / `int[]` / `bool[]` from stringified primitives — verifies
the array arms of `convert_vec_val` apply the same string-coercion
the scalar arms do.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: fix indentation drift on otyp_inferred lines
cargo fmt cleanup of leftover indentation where the perl-driven sweep
that introduced the otyp_inferred field landed at the wrong column.
No behaviour change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
tracing-opentelemetry only recognizes otel.status_code and
otel.status_message as fields that map to the OTLP Status proto.
The previously-used otel.status_description fell through to the
generic attribute recorder, leaving Status.message unset and
preventing OTLP consumers from filtering spans on error status.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: ansible delegate_to_git_repo install_requirements, dynamic fields, --limit
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: include .yaml variants in collections/roles requirements lookup
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Ruben Fiszel
Diego Imbert
windmill-internal-app[bot]
centdix
hugocasa