mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-21 00:02:30 +00:00
Ruben FiszelandClaude Opus 4.8
234a7254d5
The `duckdb` bundled feature compiles the whole DuckDB C++ library from source (~2min), which dominates the FFI crate's build. A fresh git worktree had an empty target dir and paid that cost every time. build_dev.sh now builds into a per-user cache shared across worktrees, keyed by Cargo.lock + build.rs so distinct DuckDB versions don't collide. Uncommitted changes to the crate source fall back to an isolated per-worktree ./target so active FFI development neither disturbs nor is disturbed by the shared cache. Add a crate .gitignore for /target, and note the shared cache in the AGENTS.md / backend CLAUDE.md build steps. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
63 lines
3.9 KiB
Markdown
63 lines
3.9 KiB
Markdown
# Backend (Rust)
|
|
|
|
- **Coding patterns**: MUST use the `rust-backend` skill when writing Rust code
|
|
- **Validation**: `docs/validation.md` — which `cargo check` flags to use
|
|
- **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
|
|
- **DB schema**: `backend/summarized_schema.txt`
|
|
- **API routes entry point**: `windmill-api/src/lib.rs`
|
|
- **OpenAPI spec**: `windmill-api/openapi.yaml`
|
|
- **DuckDB local jobs**: build the dynamic FFI library before running DuckDB scripts locally:
|
|
```bash
|
|
cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh
|
|
```
|
|
Re-run after clean builds or when `target/debug/libwindmill_duckdb_ffi_internal.*` is missing.
|
|
The bundled DuckDB compile (~2min) is cached in a per-user dir shared across
|
|
worktrees (keyed by the crate's `Cargo.lock`), so a fresh worktree reuses it and
|
|
the build is near-instant — you don't pay the full compile per worktree. Editing
|
|
the FFI crate's own source falls back to an isolated per-worktree `./target`.
|
|
- **Running data pipelines (DuckLake) from source**: see the section below — a plain build
|
|
advertises the `duckdb` tag but cannot execute DuckDB scripts and has no working S3 proxy.
|
|
|
|
## Running data pipelines (DuckLake) from source
|
|
|
|
DuckLake pipelines need **both** the right cargo features **and** the prebuilt DuckDB FFI. A
|
|
plain `cargo run` (or `cargo run --features quickjs`) does **not** suffice, and the failure modes
|
|
are silent-ish, so agents lose time. Verify feature names against `backend/Cargo.toml` `[features]`.
|
|
|
|
**Feature sets** (run from `backend/`):
|
|
|
|
| Goal | Command |
|
|
|---|---|
|
|
| CE DuckLake (DuckDB scripts + S3 proxy) | `cargo run --features quickjs,duckdb,parquet,private` |
|
|
| + Python scripts | add `,python` |
|
|
| EE features (WAP, partitioning, forks, …) | add `,enterprise,license` |
|
|
|
|
`enterprise` already pulls in `license`, but list both when you want the license-gated paths.
|
|
`quickjs` is for JS eval, not DuckLake per se — keep it if your baseline build had it.
|
|
|
|
**Before running any DuckDB script**, build the FFI (see the bullet above):
|
|
`cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`.
|
|
|
|
**Two gotchas that a wrong feature set produces:**
|
|
|
|
1. **`duckdb` tag advertised, feature missing.** The `duckdb` worker tag is in the *unconditional*
|
|
default tag list (`windmill-common/src/worker.rs`, `DEFAULT_TAGS`), so a worker advertises it even
|
|
without the `duckdb` feature. Jobs then dispatch but fail at execution with
|
|
`"Duck DB requires the duckdb feature to be enabled"` (`windmill-worker/src/worker.rs`). Fix:
|
|
compile with `--features duckdb`.
|
|
2. **DuckLake writes 404 (no S3 proxy).** The workspace S3 proxy (`/w/{ws}/s3_proxy/*`) that
|
|
DuckLake uses for reads/writes only mounts the real service under
|
|
`#[cfg(all(feature = "private", feature = "parquet"))]` (`windmill-api/src/s3_proxy_oss.rs`);
|
|
otherwise it's an empty router and every proxied request 404s. Fix: compile with **both**
|
|
`private` and `parquet`.
|
|
|
|
## Cloud vs self-hosted gating
|
|
|
|
The `cloud` cargo feature is compiled into **all** EE builds, so `#[cfg(feature = "cloud")]` is **not** a "cloud-only" runtime gate — it only means the code is present. The real gate for behavior specific to the managed cloud (app.windmill.dev) is the runtime flag `*CLOUD_HOSTED` (`windmill_common::worker::CLOUD_HOSTED`, from the `CLOUD_HOSTED` env var; note it's loaded from `.env` via `dotenv`, so it won't show in `/proc/<pid>/environ` — check the running behavior, not the exec env).
|
|
|
|
Cloud-only logic must be behind `if *CLOUD_HOSTED { ... }`: feature-gate the helper so it compiles, then **runtime-gate the call**. `#[cfg(feature = "cloud")]` on its own is only sufficient for:
|
|
- pure helper/struct definitions (they only run when a gated caller invokes them),
|
|
- code already inside an `if *CLOUD_HOSTED { ... }` block,
|
|
- handlers that early-return on `!*CLOUD_HOSTED`,
|
|
- idempotent no-ops that are harmless off-cloud (e.g. cache invalidation).
|