starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-09-10 08:07:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: document feature set + gotchas for running DuckLake pipelines from source (#9940)

Browse Source
This commit is contained in:
Ruben Fiszel
2026-07-05 23:16:14 +02:00
committed by GitHub
parent 55451db009
commit b40b504513
2 changed files with 36 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+1
View File
@@ -25,6 +25,7 @@ Open-source platform for internal tools, workflows, API integrations, background
- **Backend**: `cargo run` from `backend/` (API at http://localhost:8000)
- **DuckDB local jobs**: before running DuckDB scripts locally, build the FFI shared library with `cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`. Re-run it after clean builds or when `backend/target/debug/libwindmill_duckdb_ffi_internal.*` is missing.
- **Data pipelines (DuckLake) from source**: a plain `cargo run` (even `--features quickjs`) advertises a `duckdb` worker tag but **cannot** execute DuckDB scripts and has **no** working S3 proxy (DuckLake writes 404). Build CE DuckLake with `cargo run --features quickjs,duckdb,parquet,private` (add `,python` for Python scripts, `,enterprise,license` for EE) **and** build the FFI (bullet above). See `backend/CLAUDE.md` → "Running data pipelines (DuckLake) from source" for the exact feature sets and the two feature-gate gotchas.
- **Frontend**: `REMOTE=http://localhost:8000 npm run dev` from `frontend/` (port 3000+)
- **DB**: `psql postgres://postgres:changeme@localhost:5432/windmill`
- **Login**: `admin@windmill.dev` / `changeme`
backend/CLAUDE.md
+35
View File
@@ -11,6 +11,41 @@
cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh
```
Re-run after clean builds or when `target/debug/libwindmill_duckdb_ffi_internal.*` is missing.
- **Running data pipelines (DuckLake) from source**: see the section below — a plain build
advertises the `duckdb` tag but cannot execute DuckDB scripts and has no working S3 proxy.
## Running data pipelines (DuckLake) from source
DuckLake pipelines need **both** the right cargo features **and** the prebuilt DuckDB FFI. A
plain `cargo run` (or `cargo run --features quickjs`) does **not** suffice, and the failure modes
are silent-ish, so agents lose time. Verify feature names against `backend/Cargo.toml` `[features]`.
**Feature sets** (run from `backend/`):
| Goal | Command |
|---|---|
| CE DuckLake (DuckDB scripts + S3 proxy) | `cargo run --features quickjs,duckdb,parquet,private` |
| + Python scripts | add `,python` |
| EE features (WAP, partitioning, forks, …) | add `,enterprise,license` |
`enterprise` already pulls in `license`, but list both when you want the license-gated paths.
`quickjs` is for JS eval, not DuckLake per se — keep it if your baseline build had it.
**Before running any DuckDB script**, build the FFI (see the bullet above):
`cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`.
**Two gotchas that a wrong feature set produces:**
1. **`duckdb` tag advertised, feature missing.** The `duckdb` worker tag is in the *unconditional*
default tag list (`windmill-common/src/worker.rs`, `DEFAULT_TAGS`), so a worker advertises it even
without the `duckdb` feature. Jobs then dispatch but fail at execution with
`"Duck DB requires the duckdb feature to be enabled"` (`windmill-worker/src/worker.rs`). Fix:
compile with `--features duckdb`.
2. **DuckLake writes 404 (no S3 proxy).** The workspace S3 proxy (`/w/{ws}/s3_proxy/*`) that
DuckLake uses for reads/writes only mounts the real service under
`#[cfg(all(feature = "private", feature = "parquet"))]` (`windmill-api/src/s3_proxy_oss.rs`);
otherwise it's an empty router and every proxied request 404s. Fix: compile with **both**
`private` and `parquet`.
## Cloud vs self-hosted gating