Commit Graph
8 Commits
Author SHA1 Message Date
18ae0bdfbf feat: keep duckdb spilling behind the local-filesystem fence (#10607)
* fix: explain duckdb failures caused by job isolation

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: apply the isolation policy to the schema-sync pre-pass

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: bump ee ref for the out-of-memory hint wording

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: bump the bundled DuckDB engine to 1.5.5

The 1.5.5 duckdb crate no longer hands back a 96-bit `rust_decimal`, so a
DECIMAL wider than that renders instead of panicking inside an `extern "C"`
frame — which, being unable to unwind, aborted the whole worker process and
left the job running as a zombie. `SELECT
'1234567890123456789012345678.9012345678'::DECIMAL(38, 10)` was enough.

Adapting to the crate's API: `Value` is now `#[non_exhaustive]` and gained
`UHugeInt` and `Geometry`, and `rust_decimal` became an optional feature that
the `decimal`/`numeric` argument path still needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: address review findings on the duckdb bump

Run the FFI crate's own tests in CI: it is excluded from the workspace, so the
`cargo test --all` in backend-test never reached them and the new guard against
the worker-aborting DECIMAL would not have run. build_dev.sh now honors a
caller-pinned CARGO_TARGET_DIR so the test build reuses that compile instead of
building the bundled engine a second time.

Also pin UHUGEINT rendering, and correct the rust_decimal rationale —
`Decimal::new` is public without the feature, so the reason is that the feature
reproduces the exact binding the crate used to derive, not that nothing else can.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: address review nits on the duckdb bump

Name the unsupported DuckDB type rather than dumping the value, which may be
arbitrarily large or hold data that does not belong in an error message, and
say which column it came from.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: pin the ee ref to the narrowed duckdb extension allowlist

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: keep duckdb spilling behind the local-filesystem fence

* chore: repin the duckdb fork after adding the reset-test exclusion

* docs: stop claiming the duckdb patch has been filed upstream

* docs: point the backend duckdb bullet at the fork's rationale

* fix: place lock_temp_directory so no existing struct member moves

* fix: skip the extension-load guard when the repo is unreachable

* refactor: trim the fork comments and fail the extension guard in CI

* chore: repin the duckdb fork onto upstream duckdb-rs main

* fix: keep the engine patch applying on a CRLF checkout

* docs: link the upstream issue tracking the underlying problem

* chore: repin the duckdb fork onto the patch as filed upstream

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to 88568d11162ffa11723e7955e613224bab4f0568

This commit updates the EE repository reference after PR #720 was merged in windmill-ee-private.

Previous ee-repo-ref: 22f075c1164d9dd5a3ba92d682905aabd071d273

New ee-repo-ref: 88568d11162ffa11723e7955e613224bab4f0568

Automated by sync-ee-ref workflow.

* chore: repin the duckdb fork onto the cmake/fmt build fix

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test: pin the immutability half of lock_temp_directory

The spill test proves the exemption works; nothing proved the lock that makes
it sound. A rebase could drop the refusals and leave every other tripwire green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-08-11 13:38:09 +00:00
4fafe59371 fix: bump the bundled DuckDB engine to 1.5.5 (#10588)
* fix: bump the bundled DuckDB engine to 1.5.5

The 1.5.5 duckdb crate no longer hands back a 96-bit `rust_decimal`, so a
DECIMAL wider than that renders instead of panicking inside an `extern "C"`
frame — which, being unable to unwind, aborted the whole worker process and
left the job running as a zombie. `SELECT
'1234567890123456789012345678.9012345678'::DECIMAL(38, 10)` was enough.

Adapting to the crate's API: `Value` is now `#[non_exhaustive]` and gained
`UHugeInt` and `Geometry`, and `rust_decimal` became an optional feature that
the `decimal`/`numeric` argument path still needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: address review findings on the duckdb bump

Run the FFI crate's own tests in CI: it is excluded from the workspace, so the
`cargo test --all` in backend-test never reached them and the new guard against
the worker-aborting DECIMAL would not have run. build_dev.sh now honors a
caller-pinned CARGO_TARGET_DIR so the test build reuses that compile instead of
building the bundled engine a second time.

Also pin UHUGEINT rendering, and correct the rust_decimal rationale —
`Decimal::new` is public without the feature, so the reason is that the feature
reproduces the exact binding the crate used to derive, not that nothing else can.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: address review nits on the duckdb bump

Name the unsupported DuckDB type rather than dumping the value, which may be
arbitrarily large or hold data that does not belong in an error message, and
say which column it came from.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: pin the ee ref to the narrowed duckdb extension allowlist

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: pin the ee ref to the verified duckdb extension allowlist

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: pin the ee ref to the allowlist regression test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to 04dd9c5c352f04995cd0470400a877261f956561

This commit updates the EE repository reference after PR #716 was merged in windmill-ee-private.

Previous ee-repo-ref: fe7eb440a5bbae37774d3a96b69ab5c46c0b8936

New ee-repo-ref: 04dd9c5c352f04995cd0470400a877261f956561

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-08-11 09:34:27 +02:00
Ruben FiszelandClaude Opus 4.8 234a7254d5 chore: share DuckDB FFI build cache across worktrees (#9954)
The `duckdb` bundled feature compiles the whole DuckDB C++ library from
source (~2min), which dominates the FFI crate's build. A fresh git
worktree had an empty target dir and paid that cost every time.

build_dev.sh now builds into a per-user cache shared across worktrees,
keyed by Cargo.lock + build.rs so distinct DuckDB versions don't collide.
Uncommitted changes to the crate source fall back to an isolated
per-worktree ./target so active FFI development neither disturbs nor is
disturbed by the shared cache. Add a crate .gitignore for /target, and
note the shared cache in the AGENTS.md / backend CLAUDE.md build steps.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 08:55:02 +02:00
Ruben Fiszel 00221128cb fix: cgroup-aware DuckDB memory_limit + allocator memory release (#9245) 2026-05-20 14:05:16 +00:00
Ruben Fiszel 94f9175cb8 add target debug 2025-12-09 21:37:59 +00:00
Ruben Fiszel 350a1e7e70 fix test 2025-11-19 11:15:31 +00:00
Ruben Fiszel 2c04e04bf0 fix: handle better alias types in duckdb 2025-11-15 20:42:05 +00:00
Diego Imbert 304f953710 Separate duckdb crate to fix c++ build/link issues (#6551)
* call ffi

* remove duckdb dep

* rename windmill_duckdb_ffi_internal

* static lib

* ci

* back to dylib, bug isn't fixed in static

* feature flag and copy dynamic lib

* fix dynlib in docker

* load libwindmill_duckdb_ffi_internal at runtime on usage

* lazy static deadlocks

* Cache dynamic library handles

* update auto s3 path insert from editor bar

* Fix duckdb S3 freezing worker because of blocking task in tokio async context

* build dll windows GH workflow

* try fix windows build

* revert build.rs

* nit fixes CI

* Dockerfile update (not tested yet

* build dev sh for duckdb lib

* mistake

* attach windmill_duckdb_ffi_internal.so artefact

* rhel9

* docker fixes

* fix dockerfile

* better err msg

* forgot lib prefix .so

* add column_order

* fix column_order
2025-09-10 14:43:46 +00:00