Commit Graph

21 Commits

Author SHA1 Message Date
Ruben Fiszel aff0a4ec18 fix(security): sanitize dependency names & connection strings against command/SQL injection (#9590)
* fix(security): sanitize dependency names & connection strings against command/SQL injection

Follow-up to the PowerShell module-name injection fix (#9587, CWE-78): the
same audit surfaced the identical "secondary identifier interpolated into an
interpreter/SQL command without escaping" pattern in a few other executors.

- R executor (the real twin, HIGH): package name/version parsed from a
  user-supplied renv.lock were interpolated raw into an `Rscript -e
  "...renv::install(\"{pkg}@{version}\"...)"` eval string. A double-quote in
  the name/version broke out → arbitrary R (unsandboxed under DISABLE_NSJAIL /
  non-Linux). Now validated in parse_renv_lock (charset) and escaped at the
  sink as defense-in-depth (also escapes the lib path, which holds backslashes
  on Windows).

- DuckDB ATTACH (MED): the connection string built from resource fields
  (host/db/user/password) is embedded in a single-quoted DuckDB literal; escape
  quotes so a field value can't break out of the ATTACH statement.

- PgDatabase::to_uri: URL-encode host and dbname (user/password already were),
  so '@'/'/'/'?'/'&' can't reshape the parsed URI (feeds live PG connect and
  DuckDB ATTACH).

- DuckDB CREATE SECRET (FFI): wrap the interpolated S3 key/secret/endpoint in
  the existing sql_single_quote() helper, consistent with the resource-limits
  setup right above it.

- PowerShell: also escape the configured private repo URL/PAT in the install
  template (same sink as the module names; the escape landed after #9587 was
  squash-merged so it was not in the merged change).

Adds unit tests for the R validation/escaping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(security): escape ducklake/snowflake/bigquery identifiers; preserve IPv6 host

- to_uri: don't percent-encode bracketed IPv6 literal hosts ([::1]) — encoding
  their brackets/colons would stop them parsing as a host (review fix).
- duckdb ducklake ATTACH: the catalog conn string, storage and data_path are
  embedded in single-quoted DuckDB literals; escape quotes so a resource field
  can't break out (the ducklake path bypassed the ATTACH escape added earlier).
- snowflake: validate account_identifier (it forms the request hostname).
- bigquery: validate project_id (it forms a request URL path segment).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 19:06:00 +02:00
Ruben Fiszel 00221128cb fix: cgroup-aware DuckDB memory_limit + allocator memory release (#9245) 2026-05-20 14:05:16 +00:00
Diego Imbert 7f2486bdba fix: Update duckdb to 1.5.2 (Ducklake 1.0.0) (#8848) 2026-04-16 06:23:57 -07:00
Diego Imbert 0cfa462c37 fix: optimize S3 proxy performance (#8685)
* perf: re-export GetOptions and GetRange from object_store

Needed by S3 proxy to use get_opts with range for single-request
range fetches instead of HEAD + get_range.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Avoid logging S3 proxy requests as info

* Revert "Avoid logging S3 proxy requests as info"

This reverts commit b6359a7a03.

* Don't log s3 proxy

* Revert "Don't log s3 proxy"

This reverts commit 2b21ee3c78.

* Update duckdb

* AUTOMATIC_MIGRATION for ducklake

* ee repo ref

* wrong comment

* chore: update ee-repo-ref to 41b0d1cb312919109407640fc4bd7060cfe0e107

This commit updates the EE repository reference after PR #505 was merged in windmill-ee-private.

Previous ee-repo-ref: 9b97a1c563365006657c4c6cde6e7df31c5173c3

New ee-repo-ref: 41b0d1cb312919109407640fc4bd7060cfe0e107

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2026-04-03 11:53:10 +00:00
Diego Imbert 53caecf1da feat: Ducklake typechecker (#8118)
* Typedchecked ducklake queries

* Display script preview error as SQL error

* Fix duplication

* fix replacer

* Revert "fix replacer"

This reverts commit c5492033c8.

* Don't recompile regex every call

* nit OOB

* avoid potential panic

* Apply suggestions from code review

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* safety throw

* Update backend/windmill-worker/src/duckdb_executor.rs

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* Try catch individual chunks in prepareDatatableQueries

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* format

* nit comment

* Revert "Try catch individual chunks in prepareDatatableQueries"

This reverts commit ae64a8ad27.

* Correct try catch

* better error messages

* nit unused variable

* comment

* handle non describable queries

* npm i

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
2026-03-04 10:46:08 +00:00
Diego Imbert 0bee3c1197 Update duckdb (#8110)
* Update duckdb

* nit
2026-02-26 10:34:10 +00:00
Diego Imbert 3d4f9334c5 rollback lib (#7557) 2026-01-14 10:41:49 +00:00
Ruben Fiszel 729aac2d45 also revert duckdb version 2026-01-14 09:40:19 +00:00
Ruben Fiszel f879246cfd restore duckdb-ffi-internal to previous lock 2026-01-14 09:05:00 +00:00
Diego Imbert f0fd1c5e1a Improve alter table query speed + duckdb nits (#7538)
* Fetch alter table metadata much faster

* Upgrade duckdb to 1.4.3

* Disable transactional DDL for Ducklake (bug on their side)
2026-01-10 01:24:02 +01:00
Ruben Fiszel 94f9175cb8 add target debug 2025-12-09 21:37:59 +00:00
Ruben Fiszel 350a1e7e70 fix test 2025-11-19 11:15:31 +00:00
Ruben Fiszel d062d5e390 duckdb nits 2025-11-15 21:06:19 +00:00
Ruben Fiszel 2c04e04bf0 fix: handle better alias types in duckdb 2025-11-15 20:42:05 +00:00
Diego Imbert 6426ebf8cb fix: temporary fix for duckdb type_aliases causing issues (#7148) 2025-11-15 18:10:05 +00:00
Diego Imbert d3fc459b40 fix: DuckDB FFI crash fix (#7145) 2025-11-14 22:14:28 +00:00
Diego Imbert c079155e3e SQL result_collection flag (#7041)
* Return array of results duckdb

* Migration script to add result_collection=legacy

* migration script fixes

* app_version_lite not necessary

* New annotations macro that supports custom types

* pass unit tests

* nit style

* result_collection almost works for postgres

* fix last_statement

* frontend suggesitons

* fix column_order making columns disappear

* added version check for duckdb FFI lib to avoid crashes when changing FFI interface on Windows agent workers

* result_collection for duckdb

* Correct legacy behavior in DuckDB

* mysql result_collection

* mssql collection_strategy

* result_collection for oracle

* snowflake result_collection

* fix errors

* mistake, .clone() causing deadlock

* fix frontend errors on oracle / mssql

* fix legacy behavior on mssql

* add exception catch in migration

* missing app_script update

* change cache layout to avoid worker pulling old scripts
2025-11-12 15:32:07 +00:00
Diego Imbert 0cbb0dacb1 parse duckdb json query results (#7040)
* parse duckdb json query results

* don't pass alias recursively
2025-11-04 15:29:56 +00:00
Diego Imbert e17440bcc3 upgrade duckdb rs (#6770) 2025-10-08 06:46:50 +00:00
Diego Imbert 4828926be6 Fix SEGFAULT with duckdb and jemalloc (#6680) 2025-09-26 13:57:28 +00:00
Diego Imbert 304f953710 Separate duckdb crate to fix c++ build/link issues (#6551)
* call ffi

* remove duckdb dep

* rename windmill_duckdb_ffi_internal

* static lib

* ci

* back to dylib, bug isn't fixed in static

* feature flag and copy dynamic lib

* fix dynlib in docker

* load libwindmill_duckdb_ffi_internal at runtime on usage

* lazy static deadlocks

* Cache dynamic library handles

* update auto s3 path insert from editor bar

* Fix duckdb S3 freezing worker because of blocking task in tokio async context

* build dll windows GH workflow

* try fix windows build

* revert build.rs

* nit fixes CI

* Dockerfile update (not tested yet

* build dev sh for duckdb lib

* mistake

* attach windmill_duckdb_ffi_internal.so artefact

* rhel9

* docker fixes

* fix dockerfile

* better err msg

* forgot lib prefix .so

* add column_order

* fix column_order
2025-09-10 14:43:46 +00:00