* fix: bound postgres result collection so it cannot OOM the worker
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: render the sql result limit exactly so the error can be set verbatim
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: point the fraction rationale at the renderer that still emits them
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf: stop re-parsing every collected row to rebuild it as a RawValue
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* style: drop a dangling doc line and an unrelated rustfmt reflow
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: bound duckdb result collection so an oversized result cannot OOM the worker
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: keep the duckdb cap a worker-survival limit rather than a cloud product one
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: refuse an oversized blob before it expands to one json value per byte
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: share one expansion budget across a row's values, nested ones included
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: bound the row's own serialization so escaping cannot outgrow the budget
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: charge a json column before parsing it into a value tree
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: trim the json budget rationale and name what the budget does not cover
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: gate the sql result size limit on the duckdb feature
Its only consumer is the duckdb executor, so the minimal build compiled it
as dead code and failed under -D warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: bound how much disk a single duckdb job can spill
* fix: name the env var and correct duckdb's unreachable spill-cap advice
* fix: do not blame an unset env var for duckdb's default spill cap
* style: keep the duckdb spill-cap invariant comments within four lines
* docs: size the duckdb spill cap against the disk cloud pods actually use
* fix: explain duckdb failures caused by job isolation
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: apply the isolation policy to the schema-sync pre-pass
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: bump ee ref for the out-of-memory hint wording
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: bump the bundled DuckDB engine to 1.5.5
The 1.5.5 duckdb crate no longer hands back a 96-bit `rust_decimal`, so a
DECIMAL wider than that renders instead of panicking inside an `extern "C"`
frame — which, being unable to unwind, aborted the whole worker process and
left the job running as a zombie. `SELECT
'1234567890123456789012345678.9012345678'::DECIMAL(38, 10)` was enough.
Adapting to the crate's API: `Value` is now `#[non_exhaustive]` and gained
`UHugeInt` and `Geometry`, and `rust_decimal` became an optional feature that
the `decimal`/`numeric` argument path still needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: address review findings on the duckdb bump
Run the FFI crate's own tests in CI: it is excluded from the workspace, so the
`cargo test --all` in backend-test never reached them and the new guard against
the worker-aborting DECIMAL would not have run. build_dev.sh now honors a
caller-pinned CARGO_TARGET_DIR so the test build reuses that compile instead of
building the bundled engine a second time.
Also pin UHUGEINT rendering, and correct the rust_decimal rationale —
`Decimal::new` is public without the feature, so the reason is that the feature
reproduces the exact binding the crate used to derive, not that nothing else can.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: address review nits on the duckdb bump
Name the unsupported DuckDB type rather than dumping the value, which may be
arbitrarily large or hold data that does not belong in an error message, and
say which column it came from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: pin the ee ref to the narrowed duckdb extension allowlist
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: keep duckdb spilling behind the local-filesystem fence
* chore: repin the duckdb fork after adding the reset-test exclusion
* docs: stop claiming the duckdb patch has been filed upstream
* docs: point the backend duckdb bullet at the fork's rationale
* fix: place lock_temp_directory so no existing struct member moves
* fix: skip the extension-load guard when the repo is unreachable
* refactor: trim the fork comments and fail the extension guard in CI
* chore: repin the duckdb fork onto upstream duckdb-rs main
* fix: keep the engine patch applying on a CRLF checkout
* docs: link the upstream issue tracking the underlying problem
* chore: repin the duckdb fork onto the patch as filed upstream
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 88568d11162ffa11723e7955e613224bab4f0568
This commit updates the EE repository reference after PR #720 was merged in windmill-ee-private.
Previous ee-repo-ref: 22f075c1164d9dd5a3ba92d682905aabd071d273
New ee-repo-ref: 88568d11162ffa11723e7955e613224bab4f0568
Automated by sync-ee-ref workflow.
* chore: repin the duckdb fork onto the cmake/fmt build fix
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: pin the immutability half of lock_temp_directory
The spill test proves the exemption works; nothing proved the lock that makes
it sound. A rebase could drop the refusals and leave every other tripwire green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix: bump the bundled DuckDB engine to 1.5.5
The 1.5.5 duckdb crate no longer hands back a 96-bit `rust_decimal`, so a
DECIMAL wider than that renders instead of panicking inside an `extern "C"`
frame — which, being unable to unwind, aborted the whole worker process and
left the job running as a zombie. `SELECT
'1234567890123456789012345678.9012345678'::DECIMAL(38, 10)` was enough.
Adapting to the crate's API: `Value` is now `#[non_exhaustive]` and gained
`UHugeInt` and `Geometry`, and `rust_decimal` became an optional feature that
the `decimal`/`numeric` argument path still needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: address review findings on the duckdb bump
Run the FFI crate's own tests in CI: it is excluded from the workspace, so the
`cargo test --all` in backend-test never reached them and the new guard against
the worker-aborting DECIMAL would not have run. build_dev.sh now honors a
caller-pinned CARGO_TARGET_DIR so the test build reuses that compile instead of
building the bundled engine a second time.
Also pin UHUGEINT rendering, and correct the rust_decimal rationale —
`Decimal::new` is public without the feature, so the reason is that the feature
reproduces the exact binding the crate used to derive, not that nothing else can.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: address review nits on the duckdb bump
Name the unsupported DuckDB type rather than dumping the value, which may be
arbitrarily large or hold data that does not belong in an error message, and
say which column it came from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: pin the ee ref to the narrowed duckdb extension allowlist
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: pin the ee ref to the verified duckdb extension allowlist
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: pin the ee ref to the allowlist regression test
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 04dd9c5c352f04995cd0470400a877261f956561
This commit updates the EE repository reference after PR #716 was merged in windmill-ee-private.
Previous ee-repo-ref: fe7eb440a5bbae37774d3a96b69ab5c46c0b8936
New ee-repo-ref: 04dd9c5c352f04995cd0470400a877261f956561
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* wip: partial work before earlyoom-recovery relaunch
* fix(pipelines): scaffold the strftime {partition} filter idiom (frontend-only)
The DuckDB materialize scaffold and the AI pipeline prompt now teach the
grain-agnostic `WHERE strftime(<ts_col>, '<fmt>') = {partition}` filter instead
of the naive `= TIMESTAMP {partition}` cast. `{partition}` substitutes to the
partition IDENTITY string (`2026-07-05T23`, `2026-W27`, `2026-07`), which is not
a valid DuckDB TIMESTAMP literal for any non-daily grain — so the naive form
raises a `Conversion Error` for hourly/weekly/monthly (only daily parses).
Adds a frontend unit test asserting the hourly scaffold emits the strftime
idiom (`%Y-%m-%dT%H`) for every grain and never scaffolds the naive TIMESTAMP
cast as executable SQL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): scope strftime partition idiom to time grains
Review nit: `dynamic` partitioning's identity is a caller-supplied key, not a
timestamp, so `strftime` doesn't apply. Scope the scaffold + AI prompt claim to
time grains and add a `dynamic` example that filters on the user's own key.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): wm_partition macro for grain-agnostic partition filters
The materialize runtime now injects a `wm_partition(ts)` temp macro as the first
setup statement of a time-partitioned script, so filtering the source to the
active slice is one grain-agnostic line — `WHERE wm_partition(<ts_col>) =
{partition}` — instead of a hand-written `strftime` format the author must keep
in lockstep with the resolver, or the `= TIMESTAMP {partition}` cast that only
parses for daily and Conversion-Errors for hourly/weekly/monthly.
The macro's format comes from `PartitionKind::default_time_format` in
windmill-parser, the same source the EE resolver reads to stamp the `{partition}`
identity, so the two can't drift. `dynamic` partitions get no macro (their
identity is a caller-supplied key → `WHERE <key_col> = {partition}`).
Replaces the earlier 9-line strftime comment block in the scaffold with the
single macro line; AI pipeline prompt and design doc updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(pipelines): verify wm_partition strftime parity vs chrono through real DuckDB
Runs the bundled DuckDB engine in-memory and asserts strftime renders every
grain format (daily/hourly/weekly `%G-W%V`/monthly) byte-for-byte identically to
chrono — the engine the resolver uses to stamp the `{partition}` identity —
across ISO-week year boundaries (2027-01-01 → 2026-W53 etc.). Also proves the
injected `wm_partition` macro buckets the whole slice and that the naive
`TIMESTAMP '<weekly|monthly identity>'` cast Conversion-Errors.
Closes the one cross-engine assumption the pure-Rust/frontend tests couldn't
reach (flagged by CI review for weekly ISO-week rendering).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 0de2412ff0734b11e12ba378c9bcc373ff9ae800
This commit updates the EE repository reference after PR #649 was merged in windmill-ee-private.
Previous ee-repo-ref: ad6c6685689d7741058e7d2c9ecbe95d982e6268
New ee-repo-ref: 0de2412ff0734b11e12ba378c9bcc373ff9ae800
Automated by sync-ee-ref workflow.
* fix(pipelines): classify CREATE TEMP MACRO as a DuckDB prepare-path setup statement
The FFI prepare/diagnostics pass only EXECUTES statements recognized by
is_setup_statement (ATTACH/USE/INSTALL/…); everything else is merely prepared.
`CREATE [OR REPLACE] TEMP MACRO` wasn't recognized, so on a `-- prepare` run of a
partitioned materialize the injected `wm_partition` macro was never created on
the connection, and the later generated `CREATE TABLE … SELECT … WHERE
wm_partition(...)` failed to bind ("function does not exist"). The same latent
gap affected the workspace-macro splicer, which injects TEMP MACRO blocks too.
Classify CREATE [OR REPLACE] TEMP|TEMPORARY MACRO as setup so it's executed
before dependent blocks and excluded from the PrepareQueryResult count
(persistent CREATE MACRO stays a user statement). Adds a prepare-path test that
fails without the fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
The `duckdb` bundled feature compiles the whole DuckDB C++ library from
source (~2min), which dominates the FFI crate's build. A fresh git
worktree had an empty target dir and paid that cost every time.
build_dev.sh now builds into a per-user cache shared across worktrees,
keyed by Cargo.lock + build.rs so distinct DuckDB versions don't collide.
Uncommitted changes to the crate source fall back to an isolated
per-worktree ./target so active FFI development neither disturbs nor is
disturbed by the shared cache. Add a crate .gitignore for /target, and
note the shared cache in the AGENTS.md / backend CLAUDE.md build steps.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): capture violating-row samples for data tests
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): byte-accurate sample cap and leaf-level payload sanitize
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump ee-repo-ref to WAP guard probe adaptation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: WAP failures are counts-only — samples exist only on commit-then-test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: qualify where sample row data appears — job result and failed-job log line
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: error handlers receive the full result incl. samples, like any failed job
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: update ee-repo-ref to 80d309edebb899e36a3bdcdf4ea73c4db070534d
This commit updates the EE repository reference after PR #646 was merged in windmill-ee-private.
Previous ee-repo-ref: 16e916bf11f26381920560b55771fce693e668c6
New ee-repo-ref: 80d309edebb899e36a3bdcdf4ea73c4db070534d
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix(frontend): keep pipeline graph layered when lineage has cycles
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): fit pipeline graph to visible canvas on initial load
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): style pipeline minimap so it reads as a minimap
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(parser): don't infer s3 reads from bare string-literal mentions in sql
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(duckdb): render temporal values as ISO strings in job results
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): key pipeline viewport fit on the loaded graph's folder
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(parser): treat list/named read-fn arguments as definitive s3 reads
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(security): sanitize dependency names & connection strings against command/SQL injection
Follow-up to the PowerShell module-name injection fix (#9587, CWE-78): the
same audit surfaced the identical "secondary identifier interpolated into an
interpreter/SQL command without escaping" pattern in a few other executors.
- R executor (the real twin, HIGH): package name/version parsed from a
user-supplied renv.lock were interpolated raw into an `Rscript -e
"...renv::install(\"{pkg}@{version}\"...)"` eval string. A double-quote in
the name/version broke out → arbitrary R (unsandboxed under DISABLE_NSJAIL /
non-Linux). Now validated in parse_renv_lock (charset) and escaped at the
sink as defense-in-depth (also escapes the lib path, which holds backslashes
on Windows).
- DuckDB ATTACH (MED): the connection string built from resource fields
(host/db/user/password) is embedded in a single-quoted DuckDB literal; escape
quotes so a field value can't break out of the ATTACH statement.
- PgDatabase::to_uri: URL-encode host and dbname (user/password already were),
so '@'/'/'/'?'/'&' can't reshape the parsed URI (feeds live PG connect and
DuckDB ATTACH).
- DuckDB CREATE SECRET (FFI): wrap the interpolated S3 key/secret/endpoint in
the existing sql_single_quote() helper, consistent with the resource-limits
setup right above it.
- PowerShell: also escape the configured private repo URL/PAT in the install
template (same sink as the module names; the escape landed after #9587 was
squash-merged so it was not in the merged change).
Adds unit tests for the R validation/escaping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(security): escape ducklake/snowflake/bigquery identifiers; preserve IPv6 host
- to_uri: don't percent-encode bracketed IPv6 literal hosts ([::1]) — encoding
their brackets/colons would stop them parsing as a host (review fix).
- duckdb ducklake ATTACH: the catalog conn string, storage and data_path are
embedded in single-quoted DuckDB literals; escape quotes so a resource field
can't break out (the ducklake path bypassed the ATTACH escape added earlier).
- snowflake: validate account_identifier (it forms the request hostname).
- bigquery: validate project_id (it forms a request URL path segment).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf: re-export GetOptions and GetRange from object_store
Needed by S3 proxy to use get_opts with range for single-request
range fetches instead of HEAD + get_range.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Avoid logging S3 proxy requests as info
* Revert "Avoid logging S3 proxy requests as info"
This reverts commit b6359a7a03.
* Don't log s3 proxy
* Revert "Don't log s3 proxy"
This reverts commit 2b21ee3c78.
* Update duckdb
* AUTOMATIC_MIGRATION for ducklake
* ee repo ref
* wrong comment
* chore: update ee-repo-ref to 41b0d1cb312919109407640fc4bd7060cfe0e107
This commit updates the EE repository reference after PR #505 was merged in windmill-ee-private.
Previous ee-repo-ref: 9b97a1c563365006657c4c6cde6e7df31c5173c3
New ee-repo-ref: 41b0d1cb312919109407640fc4bd7060cfe0e107
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Ruben FiszelandClaude Opus 5
Diego Imbert
