* feat: upgrade the bundled DuckDB engine to 2.0.0 LTS Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: never repeat a ducklake attach option, duckdb 2.0 rejects duplicates Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: dedupe quoted attach option names and drop comments from the list Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: pin duckdb 2.0 from crates.io and revert more 2.0 defaults that change results Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: fail duckdb jobs with a clear message on x86 cpus without avx2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: strip sql comments from lake attach args before appending fork options Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: scan E'' escape-string literals when stripping attach option comments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
10 KiB
How DuckDB jobs run
A DuckDB job runs in windmill_duckdb_runner, a binary built from
backend/windmill-duckdb-ffi-internal (src/bin) and shipped next to the windmill binary.
The worker spawns one runner per DuckDB connection a job opens: the job itself, and the
on_schema_change=sync pre-pass. It never loads the engine into its own process.
The runner gets the same job isolation as every other language. duckdb_runner::runner_mode
decides it per job:
| worker isolates with | runner runs | what it can see |
|---|---|---|
| nsjail | inside run.duckdb.config.proto |
read-only /usr, /lib, /etc; its own /tmp; its job dir read-only, except duckdb/ (spill + result); a read-only extension cache |
| unshare | under build_command_with_isolation |
the worker user's file system, its own PID namespace, a cleared environment |
| nothing | as a plain child process | the worker user's file system, a cleared environment |
There is no SQL-level policy on top. A script can read and write what the jail lets it and load any extension DuckDB will sign, as a Python job can read the same files and load native code. Under nsjail its disk use is capped from the worker (below). Network egress is open in every mode, as it is for the other languages: the S3 proxy, metadata endpoints and internal hosts are all reachable.
What crosses to the runner
spawn_runner writes one JSON request to the runner's stdin, so nothing secret is put on argv or
into the environment:
- the blocks, already rewritten by the worker: resource
ATTACHes, DuckLake and datatable attaches with their credentials inlined, workspace macros; - the job args, the job token,
base_internal_urland the workspace id. The runner derives the S3/GCS secrets for the workspace's S3 proxy from the token, which it reaches over the network; memory_limit(DUCKDB_MEMORY_LIMIT, or 80% of the worker's cgroup), the temp directory (the job dir), and the result-size cap (MAX_SQL_RESULT_SIZE).
The environment is cleared except PATH, HOME (outside nsjail), DUCKDB_MAX_TEMP_DIRECTORY_SIZE,
GOOGLE_APPLICATION_CREDENTIALS (BigQuery ATTACH, whose credentials file nsjail mounts) and the
proxy variables. In particular a script cannot use the worker's own AWS_* credential chain.
Under nsjail the job dir is mounted read-only at its host path, so every path the worker wrote
into the blocks (Postgres root certificates) resolves unchanged. The one writable place on the
host is {job dir}/duckdb/: the spill directory and the result file, which the worker opens
without following symlinks, since the script can write there.
The runner writes its answer to a file in that directory: a first line with the last statement's
column order, then either the collected results or ERROR <json-encoded message>.
finish_run / finish_prepare in duckdb_executor.rs turn that into the job result.
Failure, cancellation and memory
- A crash fails the job, never the worker. A runner that exits without writing its result
becomes
The DuckDB engine process … The worker is unaffected.with the tail of its stderr. An enginestd::terminate(for example a remotetemp_directorythat has to spill) and a panic in result rendering both land here. - Cancellation and timeouts kill the engine. The executor drops the job future, and
KillGroupOnDropkills the runner's whole process group. Killing the spawned process alone is not enough:unshare --forkleaves its child running. - The kernel OOM-kills the runner, not the worker.
JOB_OOM_SCORE_ADJis written inpre_exec, before nsjail orunshareis exec'd, so the runner inherits it. Written after spawn it would race their fork, and the jail's/procis read-only. Within the job, DuckDB bounds itself throughmemory_limitand spills past it. - Runner output is bounded. stdout/stderr are drained keeping only the last 4 KB: a script
controls how much the engine prints (
CALL enable_logging(storage='stdout')).
Extensions
Outside nsjail the runner uses the worker user's ~/.duckdb/extensions and installs whatever a
script names.
Under nsjail it gets its own cache, ~/.duckdb/windmill_jail_extensions, mounted read-only.
Only the worker writes to it: before a jailed run it starts the runner unjailed in install
mode, with only INSTALL statements it built itself, for httpfs, azure, parquet, json
and every extension a block installs by name. That runs no script SQL and loads nothing, so no
extension code runs outside the jail. It happens once per extension per worker process, except
for FORCE INSTALL, which is honored every time and becomes a plain INSTALL inside the jail.
A job can therefore neither add to nor replace what the next job loads from that cache. An
extension used without being named and outside those four (icu behind SET TimeZone, say)
fails on the read-only mount: name it with INSTALL.
A script can still point extension_directory somewhere writable and install into it. That only
affects its own connection, and is no more than a Python job could do.
The engine
The runner links duckdb-rs pinned exactly (=) so cargo update cannot move the engine. The engine loads the prebuilt extensions from extensions.duckdb.org,
compiled for the exact release it reports, so an engine bump is a version change in Cargo.toml
and nothing else. The reported version comes from the vendored sources' pragma_version.cpp: a
build that calls itself vX.Y.Z-devN downloads from a per-commit path that does not exist, and
every extension INSTALL 404s. Check SELECT version() after a bump.
The engine is built from source with baseline flags, but the prebuilt 2.0 linux_amd64 extensions
are compiled for x86-64-v3: on a CPU without AVX2 (pre-Haswell, or a VM CPU model such as
Proxmox's x86-64-v2-AES or a low VMware EVC baseline) LOAD httpfs dies with SIGILL, and
every job loads httpfs. duckdb_runner::run therefore fails the job up front on such a CPU, with
a message naming the VM settings to change, rather than letting the runner crash. linux_arm64
extensions run on Cortex-A53 and later.
What a bump can break is user scripts and shared state, so a major one keeps the previous major's behavior and leaves the new one opt-in:
setup_duckdb_connectionreverts the defaults DuckDB 2.0 changed that would make a query that worked on 1.x return something else:x -> …lambdas,NULLfor integer///%by zero, untypedNULLcolumns typed INTEGER (the 2.0 NULL type makes duckdb-rs panic on read), whole-string~,.duckdbfiles written at a version 1.x can open, and the 800GB S3 upload cap. A script canSETany of them back. DuckDB 2.1 removeslambda_syntax, and the setting has to go then.ieee_floating_point_opsstays at 2.0's default on purpose. Turning it off does not restore 1.x:/by zero then returnsNULLwhere 1.x returned infinity. Left on, the only change is thatsqrt(-1),ln(0)and the like return NaN instead of erroring.- MySQL resource attaches turn off
mysql_enable_remote_pushdown, which is read when the ATTACH runs. Left on, binary-collated text comes back as BLOB and grouping follows MySQL's collation. - What no setting restores: auto-generated names of unaliased expressions (
main.prefixes dropped), unnamed structs rendered aselement1… instead of a single""key, JSON auto-detection typing offset timestamps as TIMESTAMPTZ,GROUP BY ALLover anUNNESTerroring,X'..'literals typed BLOB, and Parquet columns from constants written REQUIRED. transform_attach_ducklakeattaches lakes withDUCKLAKE_VERSION '1.0'unless the lake's extra args name one. Windmill also passesAUTOMATIC_MIGRATION TRUE, which unpinned migrates to the newest catalog format. A migrated catalog cannot be downgraded and older engines cannot read it, so a mixed-version worker fleet or a rollback would lose the lake. A lake opts into 1.1 withDUCKLAKE_VERSION '1.1'in its extra args, once no worker runs DuckDB 1.x.
Confinement belongs to the job isolation, not to engine settings. DuckDB spills through the same
file system a query reads with, so disabled_filesystems='LocalFileSystem' also stops every query
that outgrows memory_limit, and no combination of stock settings gives "spill, but no other local
file access". Fencing the engine at the SQL level again would need a patched engine to keep
spilling.
Bounding disk use
The cap is DUCKDB_MAX_TEMP_DIRECTORY_SIZE, or 6GiB on a CLOUD_HOSTED worker that does not
set it. The cloud runs one worker per pod on nodes with no per-pod ephemeral-storage limit, and a
worker pod already holds several GiB of runtime caches, so the cap is sized against a mostly full
node; re-measure before raising it. Unset elsewhere, DuckDB's own default applies: 90% of the
temp volume's free space.
It is applied twice:
- DuckDB's
max_temp_directory_size, set on every connection, so an ordinary query that outgrows it fails with the engine'sfailed to offload data block … max_temp_directory_sizeerror and cleans up its temp files. - A worker-side watchdog, under nsjail. A script can raise the engine setting, so on its own
it is only a default. The worker sums the allocated size of
{job dir}/duckdb/once a second and kills the runner when it passes the cap, whatever wrote the bytes: spill,COPY … TO, or the result file. With a disk-backed jail/tmp(NSJAIL_TMP_BACKING=disk) that/tmplives inside the same directory, so it counts too; a RAM-backed one is bounded by its tmpfs size. Enforcement lags by up to a second of writing. Outside nsjail the script can write anywhere the worker user can, as other languages can, so there the cap stays a default.
The value is DuckDB's spelling, not Kubernetes': KiB/MiB/GiB/TiB for 1024-based units,
KB/MB/GB/TB for 1000-based. An unparseable value fails every DuckDB job on the worker
rather than being ignored. DUCKDB_MEMORY_LIMIT is spelled the same way, and the two are set in
separate statements so a parse error names the right variable.