starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-10-17 08:03:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: run duckdb jobs out of process under job isolation, drop the engine fork, bump to 1.5.6 (#11780)

Browse Source 
* feat: prototype running duckdb jobs out of process inside nsjail

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: bound runner output, honor force install outside the jail, set oom score before exec

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: gate the duckdb runner bin behind a feature and decode pre-install errors

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: allowlist-only policy inside the duckdb jail, with its own extension cache

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: run duckdb jobs out of process under job isolation and drop the engine fork

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: runner everywhere, stock duckdb engine, no sql fence

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: update ee-repo-ref to 24bdc6615b3f651141fb740e2cacf9c3c2d81252

This commit updates the EE repository reference after PR #884 was merged in windmill-ee-private.

Previous ee-repo-ref: 73452bbc280cc9ca33cf0fe325d26c2fbf3aace4

New ee-repo-ref: 24bdc6615b3f651141fb740e2cacf9c3c2d81252

Automated by sync-ee-ref workflow.

* fix: pass the windows system env and TZ to the duckdb runner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: cap a jailed duckdb job's disk use from the worker, 6GiB by default on cloud

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a disk-backed jail tmp under the duckdb disk cap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: drop the benchmark scaffolding and bump duckdb to 1.5.6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
Ruben Fiszel
2026-10-15 16:49:50 +02:00
committed by GitHub
co-authored by Claude Opus 5.5 windmill-internal-app[bot]
parent 8c0185d55a
commit f9513e82ee
27 changed files with 1266 additions and 1124 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/backend-test-windows.yml
+2 -2
View File
@@ -139,8 +139,8 @@ jobs:
run: |
cargo build --release -p windmill_duckdb_ffi_internal
New-Item -ItemType Directory -Path ..\target\debug -Force
Copy-Item target\release\windmill_duckdb_ffi_internal.dll ..\target\debug\
# duckdb is bundled (~2GB of build artifacts); the DLL is the only
Copy-Item target\release\windmill_duckdb_runner.exe ..\target\debug\
# duckdb is bundled (~2GB of build artifacts); the runner is the only
# thing we need from this excluded-crate target dir.
Remove-Item -Recurse -Force -ErrorAction SilentlyContinue target
Get-PSDrive C | Select-Object Used,Free | Format-Table -AutoSize
.github/workflows/build-publish-rh-image.yml
+6 -6
View File
@@ -83,17 +83,17 @@ jobs:
# Extract arm64 binary (include deps/ for hard link resolution)
mkdir -p /tmp/arm64
crane export --platform linux/arm64 ${{ steps.meta-ee-public.outputs.tags }} - \
| tar -xf - -C /tmp/arm64 windmill/target/release/ usr/src/app/libwindmill_duckdb_ffi_internal.so
| tar -xf - -C /tmp/arm64 windmill/target/release/ usr/src/app/windmill_duckdb_runner
cp /tmp/arm64/windmill/target/release/windmill extracted/windmill-ee-arm64-rhel9
cp /tmp/arm64/usr/src/app/libwindmill_duckdb_ffi_internal.so extracted/libwindmill_duckdb_ffi_internal-arm64.so
cp /tmp/arm64/usr/src/app/windmill_duckdb_runner extracted/windmill_duckdb_runner-arm64
rm -rf /tmp/arm64
# Extract amd64 binary
mkdir -p /tmp/amd64
crane export --platform linux/amd64 ${{ steps.meta-ee-public.outputs.tags }} - \
| tar -xf - -C /tmp/amd64 windmill/target/release/ usr/src/app/libwindmill_duckdb_ffi_internal.so
| tar -xf - -C /tmp/amd64 windmill/target/release/ usr/src/app/windmill_duckdb_runner
cp /tmp/amd64/windmill/target/release/windmill extracted/windmill-ee-amd64-rhel9
cp /tmp/amd64/usr/src/app/libwindmill_duckdb_ffi_internal.so extracted/libwindmill_duckdb_ffi_internal-amd64.so
cp /tmp/amd64/usr/src/app/windmill_duckdb_runner extracted/windmill_duckdb_runner-amd64
rm -rf /tmp/amd64
- uses: actions/upload-artifact@v4
@@ -109,9 +109,9 @@ jobs:
- uses: actions/upload-artifact@v4
with:
name: RHEL9-arm64 dynamic libraries build
path: extracted/libwindmill_duckdb_ffi_internal-arm64.so
path: extracted/windmill_duckdb_runner-arm64
- uses: actions/upload-artifact@v4
with:
name: RHEL9-amd64 dynamic libraries build
path: extracted/libwindmill_duckdb_ffi_internal-amd64.so
path: extracted/windmill_duckdb_runner-amd64
.github/workflows/build-publish-rh8-image.yml
+2 -2
View File
@@ -103,7 +103,7 @@ jobs:
id: extract-duckdb-ffi-internal
with:
image: ${{ steps.meta-ee-public.outputs.tags}}-amd64
path: "/usr/src/app/libwindmill_duckdb_ffi_internal.so"
path: "/usr/src/app/windmill_duckdb_runner"
# - uses: shrink/actions-docker-extract@v3
# id: extract-ee-arm64
@@ -124,7 +124,7 @@ jobs:
- uses: actions/upload-artifact@v4
with:
name: RHEL8-amd64 dynamic libraries build
path: ${{ steps.extract-duckdb-ffi-internal.outputs.destination }}/libwindmill_duckdb_ffi_internal.so
path: ${{ steps.extract-duckdb-ffi-internal.outputs.destination }}/windmill_duckdb_runner
# - uses: actions/upload-artifact@v4
# with:
.github/workflows/build_windows_worker_.yml
+2 -2
View File
@@ -81,5 +81,5 @@ jobs:
- name: Upload dynamic libraries artifact
uses: actions/upload-artifact@v4
with:
name: windmill_duckdb_ffi_internal.dll
path: ./backend/windmill-duckdb-ffi-internal/target/release/windmill_duckdb_ffi_internal.dll
name: windmill_duckdb_runner.exe
path: ./backend/windmill-duckdb-ffi-internal/target/release/windmill_duckdb_runner.exe
.github/workflows/docker-image.yml
+1 -1
View File
@@ -218,7 +218,7 @@ jobs:
id: extract-duckdb-ffi-internal
with:
image: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ needs.build.outputs.digest }}
path: "/usr/src/app/libwindmill_duckdb_ffi_internal.so"
path: "/usr/src/app/windmill_duckdb_runner"
- uses: shrink/actions-docker-extract@v3
id: extract-ee
.github/workflows/publish_windows_worker.yml
+2 -2
View File
@@ -72,8 +72,8 @@ jobs:
files: |
./backend/target/release/windmill-ee.exe
- name: Attach dynamic libraries to release
- name: Attach the DuckDB runner to release
uses: softprops/action-gh-release@v2
with:
files: |
./backend/windmill-duckdb-ffi-internal/target/release/windmill_duckdb_ffi_internal.dll
./backend/windmill-duckdb-ffi-internal/target/release/windmill_duckdb_runner.exe