* feat: support S3Object input args in native SQL scripts
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: review fixes from local-review
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* update parser
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: track dollar-quoted strings in SQL block splitter
Queries like `CREATE FUNCTION ... AS $$ ... ; ... $$ LANGUAGE plpgsql;`
were being shredded on every `;` inside the function body because the
SQL splitter's state machine didn't recognize PostgreSQL dollar-quoted
strings. Add an `InDollarQuote(tag)` state so `$$ ... $$` and
`$tag$ ... $tag$` regions are treated as a single quoted span.
Opt-in via a new `track_dollar_quotes` flag on `parse_sql_blocks`;
enabled for PostgreSQL and DuckDB, disabled for MySQL/Oracle/BigQuery/
Snowflake which don't support the syntax.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: make windmill-parser-wasm a self-contained workspace
The wasm parser crate is excluded from the backend workspace (its
nightly-only `cargo-features = ["panic-immediate-abort"]` would break
stable cargo on the whole workspace), but its manifest still used
`.workspace = true` inheritance — which fails with "failed to find a
workspace root" once the parent no longer considers it a member.
Declare the crate as its own workspace by adding `[workspace]`,
`[workspace.package]`, and `[workspace.dependencies]` tables. Mirror
the relevant entries from the parent `backend/Cargo.toml` (same
version specs, same path targets) so resolution stays byte-identical
to what the parent would have produced.
Also:
- Teach `.github/change-versions.sh` (+ mac variant) to update this
crate's own `Cargo.toml` version and bulk-bump the `windmill-*`
entries in its `Cargo.lock` on each release.
- Bump the frontend's pinned `windmill-parser-wasm-regex` to 1.688.0
to match the freshly-built package, and refresh `package-lock.json`.
- Regenerate the wasm crate's `Cargo.lock` from scratch (first build
under the new workspace re-resolves the full graph; target-gated
deps from sibling crates like `windmill-parser-py-imports` are
now recorded in the lockfile but not compiled when targeting
wasm32).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extract the MSSQL s3 ingest+upload logging pattern into a reusable
`s3_stream_and_upload_with_logs` helper and apply it to the PostgreSQL,
MySQL, OracleDB, BigQuery, and Snowflake executors. Each s3 streamed
query now emits periodic progress lines, an ingest-done line, and an
upload+transcode-done line to the job output, matching MSSQL.
`convert_json_line_stream` now returns `BoxStream<'static, _>` so the
output stream can be forwarded to `s3.upload` from inside the generic
helper without lifetime gymnastics; the two existing callers already
boxed the result, so behavior is unchanged.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* perf: speed up mssql s3 ingest and add phase logs to job output
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: drop mssql s3 progress interval to 10s for better visibility
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: drop transient tests that compared against removed code path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: extract object store code into windmill-object-store crate with filesystem backend
Consolidate all object_store-dependent code from windmill-common into a new
windmill-object-store crate. Add a filesystem-backed object store implementation
using LocalFileSystem for dev/testing without cloud credentials. Includes 30
comprehensive tests covering render_endpoint, lfs_to_object_store_resource,
duckdb_connection_settings, error mapping, and filesystem-backed integration tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* all
* all
* all
* fix: fix raw_app hardcoded path, add missing ObjectStoreResource import, and add tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: move S3ModeFormat to windmill-types, make windmill-parser-sql optional, restore debug logs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: extract windmill-dep-map crate for parallel api/worker compilation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve WebhookShared type mismatch and missing enterprise propagation
- Make windmill-api webhook_util re-export from windmill-common instead of
duplicating types, fixing Extension<WebhookShared> mismatch between
windmill-store and windmill-api
- Add windmill-api-jobs/enterprise to windmill-trigger enterprise feature
so check_license_key_valid is available when trigger subcrates enable
enterprise on windmill-trigger
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: stop trigger features from unconditionally enabling enterprise
Move enterprise propagation for all trigger subcrates from individual
trigger feature definitions to the enterprise feature itself, so
enterprise is only enabled when explicitly requested.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: remove unused pub use re-exports and disable CI cargo cache
- Remove unused re-exports from windmill-worker/src/lib.rs:
trigger_dependents_to_recompute_dependencies, handle_job_error,
and unused bun/otel items
- Fix callers to use direct module paths instead
- Add windmill-dep-map as dev-dependency for tests
- Disable cargo cache in backend-check CI (faster from-scratch builds)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: restore bun re-exports used by tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* chore: re-enable cargo cache for check_ee_full CI job
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When updating rows in the Database Manager with MariaDB tables containing
DATE columns with non-null values, the update query was failing silently
because the date-only format (YYYY-MM-DD) was not recognized by the date
parsing regex. This caused the WHERE clause to use '0000-00-00' instead
of the actual date value. Added a new regex pattern to handle the date-only format that MariaDB
returns for DATE columns, ensuring proper date comparison in UPDATE
queries.
Co-authored-by: Diego Imbert <70353967+diegoimbert@users.noreply.github.com>
* backend
* iterate
* all
* all
* all
* iterate
* revert
* all
* add tracing to get of authed client
* all
* all
* lal
* all
* update
* fix
* push
* all
* all
* revert
* frontend
* fix checks
* avoid deadlock
* safer
* fix
* fix
* stream to s3 boilerplate
* S3 works with new syntax
* snowflake s3 streaming support
* postgres s3 support
* fix postgres stream format
* mysql s3 streaming
* mssql s3 streaming
* new s3 mode syntax
* optional folder param
* rename folder to prefix
* json_stream_arr_values
* cargo toml rollback
* convert_ndjson with datafusion
* format conversion kinda works
* Fixed not finishing the datafusion writer
* support for pg and mssql
* fix file ext
* bigquery conversion and works with s3 streaming
* fix s3 flag parser
* snowflake s3 streaming support
* factor out duplicate code
* remove anyhow
* Err case for parse s3 mode
* Send error to mpsc
* bigquery s3 streaming fix for huge queries
* remove extra stuff
* snowflake s3 streaming support
* small regex mistake
* cfg(not(feature = "parquet"))
* fix CI (unused import)
* error handling fix (graphite)
* Make schema validation struct
Schema Validation rules that are constructed from the schema or from the
MainArgSig(TODO).
* Make other validator builder
* Fail dependency job like with lockfile failing for schema validator
* Add last types + tests
* Remove unused dependency
* fix typos
* Migration ID was colliding with another, changed it manually
* Add Oneof + other fixes
* fix: cache for querying scripts correclty handles ScriptMetadata
* Add cache for schema validation from main arg sig
* Prepare sqlx
* Remove default features
* Feature flags
* WIP: unsafe sql params for sql langauges
* Fix down migration table name
* cleanup: put validation logic inside a function
* Refactor to cache the should_validate boolean
Changed the schemavalidators cache to take in an
Option<SchemaValidator>, effectively storing the `should_validate_schema` information.
Also pass the schema when avaialble to construct the schema validator
* Add other job kinds to u8 cache key just in case
* Change sql languages to all get arguments as Values instead of RawValue
* Only cache if not preview
* Add last sql languages and some CI fixes
* Rename after typo on `sanitized`
* Finish rename
* Remove unused import
* Fix wrong test
* Add newly published regex parser version
* Remove default features from cargo.toml
* Change to a cleaner syntax for the interpolated args
* Update republished parser
* Refactor annotation system
* Move tests to different location
* Limit annotations per line (15)
* Write initial benchmark
* Optimize
* More optimizations
* Rewrite with 3x speed!
* Remove benchmarks
They were in wrong place and for development
* Polishing
* Remove unused leftover
* Integrate new annotation system
* Add comment explanation
* Remove `hashe-based matching`
Turned to be it is impure and non-deterministic method
* Add `tracing::error` for `// native` without `deno_core`
If you build backend without `deno_core` feature flag, and add //native
annotation to ts script, it would just silently exit.
This commit prints error to logs
* add more tests
* Update annotations.rs
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* feat: multi statement pg
* fix: add other flavors
* feat: make pg params start at 1 and sequential
* fix: improve sql statement parsing
* add tests
* fix: allow no semi in last statement
* fix: merge conflict
* fix: minor improvement
* fix: parser version
hugocasaandClaude Opus 4.5
Devdatta TaleleandDiego Imbert
wendrul
Ruben Fiszel
pyranotaandRuben Fiszel
