Files
windmill/docs/duckdb-execution.md
T

10 KiB

How DuckDB jobs run

A DuckDB job runs in windmill_duckdb_runner, a binary built from backend/windmill-duckdb-ffi-internal (src/bin) and shipped next to the windmill binary. The worker spawns one runner per DuckDB connection a job opens: the job itself, and the on_schema_change=sync pre-pass. It never loads the engine into its own process.

The runner gets the same job isolation as every other language. duckdb_runner::runner_mode decides it per job:

worker isolates with runner runs what it can see
nsjail inside run.duckdb.config.proto read-only /usr, /lib, /etc; its own /tmp; its job dir read-only, except duckdb/ (spill + result); a read-only extension cache
unshare under build_command_with_isolation the worker user's file system, its own PID namespace, a cleared environment
nothing as a plain child process the worker user's file system, a cleared environment

There is no SQL-level policy on top. A script can read and write what the jail lets it and load any extension DuckDB will sign, as a Python job can read the same files and load native code. Under nsjail its disk use is capped from the worker (below). Network egress is open in every mode, as it is for the other languages: the S3 proxy, metadata endpoints and internal hosts are all reachable.

What crosses to the runner

spawn_runner writes one JSON request to the runner's stdin, so nothing secret is put on argv or into the environment:

  • the blocks, already rewritten by the worker: resource ATTACHes, DuckLake and datatable attaches with their credentials inlined, workspace macros;
  • the job args, the job token, base_internal_url and the workspace id. The runner derives the S3/GCS secrets for the workspace's S3 proxy from the token, which it reaches over the network;
  • memory_limit (DUCKDB_MEMORY_LIMIT, or 80% of the worker's cgroup), the temp directory (the job dir), and the result-size cap (MAX_SQL_RESULT_SIZE).

The environment is cleared except PATH, HOME (outside nsjail), DUCKDB_MAX_TEMP_DIRECTORY_SIZE, GOOGLE_APPLICATION_CREDENTIALS (BigQuery ATTACH, whose credentials file nsjail mounts) and the proxy variables. In particular a script cannot use the worker's own AWS_* credential chain.

Under nsjail the job dir is mounted read-only at its host path, so every path the worker wrote into the blocks (Postgres root certificates) resolves unchanged. The one writable place on the host is {job dir}/duckdb/: the spill directory and the result file, which the worker opens without following symlinks, since the script can write there.

The runner writes its answer to a file in that directory: a first line with the last statement's column order, then either the collected results or ERROR <json-encoded message>. finish_run / finish_prepare in duckdb_executor.rs turn that into the job result.

Failure, cancellation and memory

  • A crash fails the job, never the worker. A runner that exits without writing its result becomes The DuckDB engine process … The worker is unaffected. with the tail of its stderr. An engine std::terminate (for example a remote temp_directory that has to spill) and a panic in result rendering both land here.
  • Cancellation and timeouts kill the engine. The executor drops the job future, and KillGroupOnDrop kills the runner's whole process group. Killing the spawned process alone is not enough: unshare --fork leaves its child running.
  • The kernel OOM-kills the runner, not the worker. JOB_OOM_SCORE_ADJ is written in pre_exec, before nsjail or unshare is exec'd, so the runner inherits it. Written after spawn it would race their fork, and the jail's /proc is read-only. Within the job, DuckDB bounds itself through memory_limit and spills past it.
  • Runner output is bounded. stdout/stderr are drained keeping only the last 4 KB: a script controls how much the engine prints (CALL enable_logging(storage='stdout')).

Extensions

Outside nsjail the runner uses the worker user's ~/.duckdb/extensions and installs whatever a script names.

Under nsjail it gets its own cache, ~/.duckdb/windmill_jail_extensions, mounted read-only. Only the worker writes to it: before a jailed run it starts the runner unjailed in install mode, with only INSTALL statements it built itself, for httpfs, azure, parquet, json and every extension a block installs by name. That runs no script SQL and loads nothing, so no extension code runs outside the jail. It happens once per extension per worker process, except for FORCE INSTALL, which is honored every time and becomes a plain INSTALL inside the jail. A job can therefore neither add to nor replace what the next job loads from that cache. An extension used without being named and outside those four (icu behind SET TimeZone, say) fails on the read-only mount: name it with INSTALL.

A script can still point extension_directory somewhere writable and install into it. That only affects its own connection, and is no more than a Python job could do.

The engine

The runner links duckdb-rs pinned exactly (=) so cargo update cannot move the engine. The engine loads the prebuilt extensions from extensions.duckdb.org, compiled for the exact release it reports, so an engine bump is a version change in Cargo.toml and nothing else. The reported version comes from the vendored sources' pragma_version.cpp: a build that calls itself vX.Y.Z-devN downloads from a per-commit path that does not exist, and every extension INSTALL 404s. Check SELECT version() after a bump.

The engine is built from source with baseline flags, but the prebuilt 2.0 linux_amd64 extensions are compiled for x86-64-v3: on a CPU without AVX2 (pre-Haswell, or a VM CPU model such as Proxmox's x86-64-v2-AES or a low VMware EVC baseline) LOAD httpfs dies with SIGILL, and every job loads httpfs. duckdb_runner::run therefore fails the job up front on such a CPU, with a message naming the VM settings to change, rather than letting the runner crash. linux_arm64 extensions run on Cortex-A53 and later.

What a bump can break is user scripts and shared state, so a major one keeps the previous major's behavior and leaves the new one opt-in:

  • setup_duckdb_connection reverts the defaults DuckDB 2.0 changed that would make a query that worked on 1.x return something else: x -> … lambdas, NULL for integer // / % by zero, untyped NULL columns typed INTEGER (the 2.0 NULL type makes duckdb-rs panic on read), whole-string ~, and the 800GB S3 upload cap. A script can SET any of them back. DuckDB 2.1 removes lambda_syntax, and the setting has to go then.
  • .duckdb files are written at 2.0's storage version, which DuckDB 1.x cannot open. Windmill keeps none of its own state in them, so it only reaches a script that persists one outside its job dir for a 1.x reader; that script can SET storage_compatibility_version.
  • ieee_floating_point_ops stays at 2.0's default on purpose. Turning it off does not restore 1.x: / by zero then returns NULL where 1.x returned infinity. Left on, the only change is that sqrt(-1), ln(0) and the like return NaN instead of erroring.
  • MySQL resource attaches turn off mysql_enable_remote_pushdown, which is read when the ATTACH runs. Left on, binary-collated text comes back as BLOB and grouping follows MySQL's collation.
  • What no setting restores: auto-generated names of unaliased expressions (main. prefixes dropped), unnamed structs rendered as element1… instead of a single "" key, JSON auto-detection typing offset timestamps as TIMESTAMPTZ, GROUP BY ALL over an UNNEST erroring, X'..' literals typed BLOB, and Parquet columns from constants written REQUIRED.
  • transform_attach_ducklake attaches lakes with DUCKLAKE_VERSION '1.0' unless the lake's extra args name one. Windmill also passes AUTOMATIC_MIGRATION TRUE, which unpinned migrates to the newest catalog format. A migrated catalog cannot be downgraded and older engines cannot read it, so a mixed-version worker fleet or a rollback would lose the lake. A lake opts into 1.1 with DUCKLAKE_VERSION '1.1' in its extra args, once no worker runs DuckDB 1.x.

Confinement belongs to the job isolation, not to engine settings. DuckDB spills through the same file system a query reads with, so disabled_filesystems='LocalFileSystem' also stops every query that outgrows memory_limit, and no combination of stock settings gives "spill, but no other local file access". Fencing the engine at the SQL level again would need a patched engine to keep spilling.

Bounding disk use

The cap is DUCKDB_MAX_TEMP_DIRECTORY_SIZE, or 6GiB on a CLOUD_HOSTED worker that does not set it. The cloud runs one worker per pod on nodes with no per-pod ephemeral-storage limit, and a worker pod already holds several GiB of runtime caches, so the cap is sized against a mostly full node; re-measure before raising it. Unset elsewhere, DuckDB's own default applies: 90% of the temp volume's free space.

It is applied twice:

  • DuckDB's max_temp_directory_size, set on every connection, so an ordinary query that outgrows it fails with the engine's failed to offload data block … max_temp_directory_size error and cleans up its temp files.
  • A worker-side watchdog, under nsjail. A script can raise the engine setting, so on its own it is only a default. The worker sums the allocated size of {job dir}/duckdb/ once a second and kills the runner when it passes the cap, whatever wrote the bytes: spill, COPY … TO, or the result file. With a disk-backed jail /tmp (NSJAIL_TMP_BACKING=disk) that /tmp lives inside the same directory, so it counts too; a RAM-backed one is bounded by its tmpfs size. Enforcement lags by up to a second of writing. Outside nsjail the script can write anywhere the worker user can, as other languages can, so there the cap stays a default.

The value is DuckDB's spelling, not Kubernetes': KiB/MiB/GiB/TiB for 1024-based units, KB/MB/GB/TB for 1000-based. An unparseable value fails every DuckDB job on the worker rather than being ignored. DUCKDB_MEMORY_LIMIT is spelled the same way, and the two are set in separate statements so a parse error names the right variable.