mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-11-17 00:20:51 +00:00
155 lines
10 KiB
Markdown
155 lines
10 KiB
Markdown
# How DuckDB jobs run
|
|
|
|
A DuckDB job runs in `windmill_duckdb_runner`, a binary built from
|
|
`backend/windmill-duckdb-ffi-internal` (`src/bin`) and shipped next to the `windmill` binary.
|
|
The worker spawns one runner per DuckDB connection a job opens: the job itself, and the
|
|
`on_schema_change=sync` pre-pass. It never loads the engine into its own process.
|
|
|
|
The runner gets the same job isolation as every other language. `duckdb_runner::runner_mode`
|
|
decides it per job:
|
|
|
|
| worker isolates with | runner runs | what it can see |
|
|
|---|---|---|
|
|
| nsjail | inside `run.duckdb.config.proto` | read-only `/usr`, `/lib`, `/etc`; its own `/tmp`; its job dir read-only, except `duckdb/` (spill + result); a read-only extension cache |
|
|
| unshare | under `build_command_with_isolation` | the worker user's file system, its own PID namespace, a cleared environment |
|
|
| nothing | as a plain child process | the worker user's file system, a cleared environment |
|
|
|
|
There is no SQL-level policy on top. A script can read and write what the jail lets it and load
|
|
any extension DuckDB will sign, as a Python job can read the same files and load native code.
|
|
Under nsjail its disk use is capped from the worker (below). Network egress is open in every
|
|
mode, as it is for the other languages: the S3 proxy, metadata endpoints and internal hosts are
|
|
all reachable.
|
|
|
|
## What crosses to the runner
|
|
|
|
`spawn_runner` writes one JSON request to the runner's stdin, so nothing secret is put on argv or
|
|
into the environment:
|
|
|
|
- the blocks, already rewritten by the worker: resource `ATTACH`es, DuckLake and datatable
|
|
attaches with their credentials inlined, workspace macros;
|
|
- the job args, the job token, `base_internal_url` and the workspace id. The runner derives the
|
|
S3/GCS secrets for the workspace's S3 proxy from the token, which it reaches over the network;
|
|
- `memory_limit` (`DUCKDB_MEMORY_LIMIT`, or 80% of the worker's cgroup), the temp directory (the
|
|
job dir), and the result-size cap (`MAX_SQL_RESULT_SIZE`).
|
|
|
|
The environment is cleared except `PATH`, `HOME` (outside nsjail), `DUCKDB_MAX_TEMP_DIRECTORY_SIZE`,
|
|
`GOOGLE_APPLICATION_CREDENTIALS` (BigQuery `ATTACH`, whose credentials file nsjail mounts) and the
|
|
proxy variables. In particular a script cannot use the worker's own `AWS_*` credential chain.
|
|
|
|
Under nsjail the job dir is mounted read-only at its host path, so every path the worker wrote
|
|
into the blocks (Postgres root certificates) resolves unchanged. The one writable place on the
|
|
host is `{job dir}/duckdb/`: the spill directory and the result file, which the worker opens
|
|
without following symlinks, since the script can write there.
|
|
|
|
The runner writes its answer to a file in that directory: a first line with the last statement's
|
|
column order, then either the collected results or `ERROR <json-encoded message>`.
|
|
`finish_run` / `finish_prepare` in `duckdb_executor.rs` turn that into the job result.
|
|
|
|
## Failure, cancellation and memory
|
|
|
|
- **A crash fails the job, never the worker.** A runner that exits without writing its result
|
|
becomes `The DuckDB engine process … The worker is unaffected.` with the tail of its stderr.
|
|
An engine `std::terminate` (for example a remote `temp_directory` that has to spill) and a
|
|
panic in result rendering both land here.
|
|
- **Cancellation and timeouts kill the engine.** The executor drops the job future, and
|
|
`KillGroupOnDrop` kills the runner's whole process group. Killing the spawned process alone is
|
|
not enough: `unshare --fork` leaves its child running.
|
|
- **The kernel OOM-kills the runner, not the worker.** `JOB_OOM_SCORE_ADJ` is written in
|
|
`pre_exec`, before nsjail or `unshare` is exec'd, so the runner inherits it. Written after spawn
|
|
it would race their fork, and the jail's `/proc` is read-only. Within the job, DuckDB bounds
|
|
itself through `memory_limit` and spills past it.
|
|
- **Runner output is bounded.** stdout/stderr are drained keeping only the last 4 KB: a script
|
|
controls how much the engine prints (`CALL enable_logging(storage='stdout')`).
|
|
|
|
## Extensions
|
|
|
|
Outside nsjail the runner uses the worker user's `~/.duckdb/extensions` and installs whatever a
|
|
script names.
|
|
|
|
Under nsjail it gets its own cache, `~/.duckdb/windmill_jail_extensions`, mounted read-only.
|
|
Only the worker writes to it: before a jailed run it starts the runner unjailed in `install`
|
|
mode, with only `INSTALL` statements it built itself, for `httpfs`, `azure`, `parquet`, `json`
|
|
and every extension a block installs by name. That runs no script SQL and loads nothing, so no
|
|
extension code runs outside the jail. It happens once per extension per worker process, except
|
|
for `FORCE INSTALL`, which is honored every time and becomes a plain `INSTALL` inside the jail.
|
|
A job can therefore neither add to nor replace what the next job loads from that cache. An
|
|
extension used without being named and outside those four (`icu` behind `SET TimeZone`, say)
|
|
fails on the read-only mount: name it with `INSTALL`.
|
|
|
|
A script can still point `extension_directory` somewhere writable and install into it. That only
|
|
affects its own connection, and is no more than a Python job could do.
|
|
|
|
## The engine
|
|
|
|
The runner links duckdb-rs pinned exactly (`=`) so `cargo update` cannot move the engine. The engine loads the **prebuilt** extensions from `extensions.duckdb.org`,
|
|
compiled for the exact release it reports, so an engine bump is a version change in `Cargo.toml`
|
|
and nothing else. The reported version comes from the vendored sources' `pragma_version.cpp`: a
|
|
build that calls itself `vX.Y.Z-devN` downloads from a per-commit path that does not exist, and
|
|
every extension `INSTALL` 404s. Check `SELECT version()` after a bump.
|
|
|
|
The engine is built from source with baseline flags, but the prebuilt 2.0 linux_amd64 extensions
|
|
are compiled for x86-64-v3: on a CPU without AVX2 (pre-Haswell, or a VM CPU model such as
|
|
Proxmox's `x86-64-v2-AES` or a low VMware EVC baseline) `LOAD httpfs` dies with SIGILL, and
|
|
every job loads httpfs. `duckdb_runner::run` therefore fails the job up front on such a CPU, with
|
|
a message naming the VM settings to change, rather than letting the runner crash. linux_arm64
|
|
extensions run on Cortex-A53 and later.
|
|
|
|
What a bump can break is user scripts and shared state, so a major one keeps the previous major's
|
|
behavior and leaves the new one opt-in:
|
|
|
|
- `setup_duckdb_connection` reverts the defaults DuckDB 2.0 changed that would make a query that
|
|
worked on 1.x return something else: `x -> …` lambdas, `NULL` for integer `//` / `%` by zero,
|
|
untyped `NULL` columns typed INTEGER (the 2.0 NULL type makes duckdb-rs panic on read),
|
|
whole-string `~`, and the 800GB S3 upload cap. A script can `SET` any of them back. DuckDB 2.1
|
|
removes `lambda_syntax`, and the setting has to go then.
|
|
- `.duckdb` files are written at 2.0's storage version, which DuckDB 1.x cannot open. Windmill
|
|
keeps none of its own state in them, so it only reaches a script that persists one outside
|
|
its job dir for a 1.x reader; that script can `SET storage_compatibility_version`.
|
|
- `ieee_floating_point_ops` stays at 2.0's default on purpose. Turning it off does not restore
|
|
1.x: `/` by zero then returns `NULL` where 1.x returned infinity. Left on, the only change is
|
|
that `sqrt(-1)`, `ln(0)` and the like return NaN instead of erroring.
|
|
- MySQL resource attaches turn off `mysql_enable_remote_pushdown`, which is read when the
|
|
ATTACH runs. Left on, binary-collated text comes back as BLOB and grouping follows MySQL's
|
|
collation.
|
|
- What no setting restores: auto-generated names of unaliased expressions (`main.` prefixes
|
|
dropped), unnamed structs rendered as `element1`… instead of a single `""` key, JSON
|
|
auto-detection typing offset timestamps as TIMESTAMPTZ, `GROUP BY ALL` over an `UNNEST`
|
|
erroring, `X'..'` literals typed BLOB, and Parquet columns from constants written REQUIRED.
|
|
- `transform_attach_ducklake` attaches lakes with `DUCKLAKE_VERSION '1.0'` unless the lake's
|
|
extra args name one. Windmill also passes `AUTOMATIC_MIGRATION TRUE`, which unpinned migrates to
|
|
the newest catalog format. A migrated catalog cannot be downgraded and older engines cannot read
|
|
it, so a mixed-version worker fleet or a rollback would lose the lake. A lake opts into 1.1
|
|
with `DUCKLAKE_VERSION '1.1'` in its extra args, once no worker runs DuckDB 1.x.
|
|
|
|
Confinement belongs to the job isolation, not to engine settings. DuckDB spills through the same
|
|
file system a query reads with, so `disabled_filesystems='LocalFileSystem'` also stops every query
|
|
that outgrows `memory_limit`, and no combination of stock settings gives "spill, but no other local
|
|
file access". Fencing the engine at the SQL level again would need a patched engine to keep
|
|
spilling.
|
|
|
|
## Bounding disk use
|
|
|
|
The cap is `DUCKDB_MAX_TEMP_DIRECTORY_SIZE`, or `6GiB` on a `CLOUD_HOSTED` worker that does not
|
|
set it. The cloud runs one worker per pod on nodes with no per-pod ephemeral-storage limit, and a
|
|
worker pod already holds several GiB of runtime caches, so the cap is sized against a mostly full
|
|
node; re-measure before raising it. Unset elsewhere, DuckDB's own default applies: 90% of the
|
|
temp volume's free space.
|
|
|
|
It is applied twice:
|
|
|
|
- **DuckDB's `max_temp_directory_size`**, set on every connection, so an ordinary query that
|
|
outgrows it fails with the engine's `failed to offload data block … max_temp_directory_size`
|
|
error and cleans up its temp files.
|
|
- **A worker-side watchdog**, under nsjail. A script can raise the engine setting, so on its own
|
|
it is only a default. The worker sums the allocated size of `{job dir}/duckdb/` once a second
|
|
and kills the runner when it passes the cap, whatever wrote the bytes: spill, `COPY … TO`, or
|
|
the result file. With a disk-backed jail `/tmp` (`NSJAIL_TMP_BACKING=disk`) that `/tmp` lives
|
|
inside the same directory, so it counts too; a RAM-backed one is bounded by its tmpfs size.
|
|
Enforcement lags by up to a second of writing. Outside nsjail the script can
|
|
write anywhere the worker user can, as other languages can, so there the cap stays a default.
|
|
|
|
The value is DuckDB's spelling, not Kubernetes': `KiB`/`MiB`/`GiB`/`TiB` for 1024-based units,
|
|
`KB`/`MB`/`GB`/`TB` for 1000-based. An unparseable value fails every DuckDB job on the worker
|
|
rather than being ignored. `DUCKDB_MEMORY_LIMIT` is spelled the same way, and the two are set in
|
|
separate statements so a parse error names the right variable.
|