1203 Commits

Author SHA1 Message Date
Ruben Fiszel beb658757f nit error 2025-05-17 10:56:16 +02:00
Ruben Fiszel 59afa493fa fix: improve handling of custom concurrency key/tag with preprocessors (#5762)
* fix

* rm spurrious diff

* handle failing preprocessors

* nits

* update

* update
2025-05-17 10:52:28 +02:00
Diego Imbert b29c6e7636 fix: S3 sql mode returns S3Object (#5764)
* fix upload_s3_file not checking response

* Return S3Object instead of path string
2025-05-16 19:28:57 +02:00
Ruben Fiszel e87d4f3c1a fix: avoid deadlocks in sending completed job to result processors (#5742)
* buffer

* unbounded and bounded

* fix

* buffer

* impl clone

* flume it all

* update

* update

* update
2025-05-15 09:13:09 +02:00
Diego Imbert 669a95c3be oracle support for s3 streaming + misc (#5738)
* changed SQL templates to include S3

* oracle support for s3 streaming
2025-05-14 15:45:00 +02:00
Ruben Fiszel 5b26498e09 add more duration warnings to logs 2025-05-14 10:13:14 +02:00
Ruben Fiszel d35a7d22f9 fix: add support for log compaction on docker jobs (#5732)
* improve docker compact

* improve docker compact

* update ref

* update

* update agent workers
2025-05-13 21:42:15 +02:00
wendrul f73c90c751 fix: Ansible lockfile back compatibility issue (#5731)
* fix: Ansible lockfile back compatibility issue

* Replace condition with more readable alternative
2025-05-13 16:19:18 +02:00
Diego Imbert c7886ea07a feat: sql jobs outputting to s3 + streaming for high-number of rows (#5704)
* stream to s3 boilerplate

* S3 works with new syntax

* snowflake s3 streaming support

* postgres s3 support

* fix postgres stream format

* mysql s3 streaming

* mssql s3 streaming

* new s3 mode syntax

* optional folder param

* rename folder to prefix

* json_stream_arr_values

* cargo toml rollback

* convert_ndjson with datafusion

* format conversion kinda works

* Fixed not finishing the datafusion writer

* support for pg and mssql

* fix file ext

* bigquery conversion and works with s3 streaming

* fix s3 flag parser

* snowflake s3 streaming support

* factor out duplicate code

* remove anyhow

* Err case for parse s3 mode

* Send error to mpsc

* bigquery s3 streaming fix for huge queries

* remove extra stuff

* snowflake s3 streaming support

* small regex mistake

* cfg(not(feature = "parquet"))

* fix CI (unused import)

* error handling fix (graphite)
2025-05-13 10:19:44 +02:00
Ruben Fiszel b51568c166 fix: make ansible more resilient to invalid lockfiles 2025-05-12 11:40:13 +02:00
dieriba 54b4815df2 nits: use generic deserializer for empty data (#5727)
* use generic deserializer for empty data

* update repo ref

* fix import
2025-05-12 11:00:55 +02:00
Ruben Fiszel a23a4c0faf nit compile 2025-05-11 11:52:42 +02:00
Ruben Fiszel a3c76fb10c fix: improve error display of nativets exceptions 2025-05-11 11:35:31 +02:00
Ruben Fiszel 7c69959853 fix: improve agents workers handling of WHITELIST_ENVS 2025-05-11 08:38:46 +02:00
HugoCasa 254c3cf8ef feat: preprocessor refactor (#5629)
* feat: preprocessor refactor

* nit

* oneOf kind support + new capture format + fix build

* fix build

* fix build

* update preprocessor templates

* cache runnable version info

* update ee ref

* update hub scripts && ee ref
2025-05-09 09:50:01 +02:00
dieriba 242a565428 feat: raise error if end early in flow (#5653)
* update front

* handle err_msg display if enabled when stop condition is met

* nits

* Update backend/windmill-worker/src/worker_flow.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update openflow.openapi.yaml

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update worker_flow.rs

* Update worker_flow.rs

* update flow test

* update condition

* remove println

* nits

* fix flow test

* nits and missing property on struct definition

* nits

* fix unsaved state

* update .sqlx

* cleanup sqlx

---------

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-05-07 18:23:40 +02:00
Ruben Fiszel 9bdd301f52 fix: ansible in agent mode can use inventory.ini 2025-05-06 11:57:08 +02:00
Ruben Fiszel 7eb9d7d46c fix: do not track relative deps for scripts with raw defined deps from CLI (#5696) 2025-05-02 00:37:28 +02:00
Ruben Fiszel 46a3c29c9f improve agent workers 2025-04-29 20:15:49 +02:00
Diego Imbert 35b69da25c fix: skip_flow_update for dependency tracking table (#5670) 2025-04-28 17:57:35 +02:00
Ruben Fiszel a7b2b51444 fix: improve skip failure on parallel branchall 2025-04-27 00:37:25 +02:00
wendrul fdd1642ce1 feat: Ansible improvements (vault, roles and git repos) (#5655)
* Ansible vault + roles

* Clone arbitrary repos

* Fix cloning logic after merge

* Make function for cloning without history any commit

* Cloning repos and lockfile on the commit

* Improve error messages

* Create lockfile for roles and collections

* Simplify ansible ssh identity interface

* Ansible vault password: pass just a variable instead of 2 step approach

* Lock lockfiles for roles and collections

* fix typo

* Change git ssh identity section name

* Rename variable

* Update init script for ansible

* Suppress error when no roles

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-04-24 16:04:28 +02:00
Ruben Fiszel e316dbd9bd fix: track relative imports in python and ts even if lockfile is provided 2025-04-23 08:40:22 +02:00
Diego Imbert 5bca8f60e9 fix: improve MySQL datetime parser timezone handling (WIN-1155) (#5645)
* fix mysql datetime parser for non tz dates

* static regexes
2025-04-21 14:46:28 +02:00
Ruben Fiszel 3c68fef2ab ui code builder v0 (at secret path) (#4964) 2025-04-20 01:19:02 +02:00
Ruben Fiszel 87dbb14656 improve no_proxy value by default 2025-04-17 00:05:47 +02:00
Ruben Fiszel 24e893b8c5 fix: flow editor workspace script test use actual workspace script hash 2025-04-16 00:44:49 +00:00
Ruben Fiszel 496960d349 improve last_job_suspended_history 2025-04-12 15:02:43 +00:00
Ruben Fiszel 8c6e620f5c nit benchmarks 2025-04-12 12:54:05 +00:00
Ruben Fiszel 12ba15c928 improve benchmarks 2025-04-12 11:09:21 +00:00
pyranota 0b6d017fed feat(python): per import requirement pin (#5520)
* implement single line pin

* make panic-safe

* use pin even if multiple modules imported withing single statement

* add repins and make imports respect pins

* keep all pins

* Allow multiple pins

* add comments + handle stuff more safely

* fix fully qualified imports

* remove ignore

* sort nested

* apply unique to output requirements list

* fix typo

* remove mut

* update sqlx

* sort imports

* sort imports

* fix formatter and format

* refactor

* fix comptime error

* write tests

* perf: do not capture if string is empty
2025-04-11 21:31:51 +00:00
Ruben Fiszel d5186da271 allow multiple workers on agent mode (#5607) 2025-04-11 21:19:29 +00:00
HugoCasa b9e879618b feat: signed s3 objects (#5593)
* feat: accept signed s3 objects for s3 file keys in apps + sign endpoint and helpers

* Update backend/windmill-api/openapi.yaml

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix build

* Update python-client/wmill/wmill/client.py

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* nti

* fix build

* fix build

* presigned

* Update frontend/src/lib/components/apps/editor/settingsPanel/InputsSpecEditor.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/apps/editor/settingsPanel/InputsSpecEditor.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix build

* b

* fix sqlx

* nit

* typo

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-04-11 18:03:40 +02:00
dieriba 6339775404 feat: add gcp trigger (#5501)
* update: add migration for gcp_trigger table, add cli method gcp and add gcp_trigger args for github ci

* update: add gcp module and routing it in lib.rs

* update: added gcp type to TRIGGER KIND type in db, add new feature condition to try_get_fn and added gcp_trigger feature in cargo file

* update: add ui for gcp trigger, added missing triggers type for job kin, add gcp trigger in deploy to functionallity

* adding google cloud crate

* update: handle pull and push delivery

* update: handle pull and push delivery

* update: handle push event, changed front ui

* update: ui

* fix: capture for gcp

* update: automatically manage pub sub subscription and refactoring

* capture done

* update cli types

* fix: wrong func argument and type openapi

* fix: missing import

* update .sqlx

* update: svg color and remove pulse button for capture push gcp

* update: handle deployto

* update script helper and link to hub gcp script/flow

* update gcp representation

* update: add confirmation modal

* add unique index to mitigate same subscriber id, fix `zombie worker`, update test connection logic

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerEditorInner.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update cli/gen/services.gen.ts

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* nits

* update repo ref

* fix feature function

* update dependencies

* update .sqlx

* fix missing import

* update: handle existing subscription and creating new one or update it

* fix ee build

* handle existing config properly

* update .sqlx

* update .sqlx

* remove unused

* update documentation link fix ci

* use on instance of empty_string_as_none fn

* update refo ref

* updat script_helper

* update ui

* update migration

* add missing arguments

* fix and nits

* update repo ref

* remove unused

* nits

* add doc links

* nits

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerPanel.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerEditorInner.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* use trigger path as default route path

* update .sqlx

* update repo ref

* update ref

* update route path

* update types

* fix deploy to

* nits

* update repo ref

* update .sqlx

* Update frontend/src/lib/components/triggers/gcp/GcpTriggerEditorConfigSection.svelte

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* nits

* nits

* nits

* remove dupliacte fn and update repo ref

* remove unused import

* fix missing import

* fix: wong name var

* update script helper and template script

* Update frontend/src/lib/script_helpers.ts

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update ee-repo-ref.txt

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
2025-04-10 20:39:01 +00:00
Ruben Fiszel 1f1164199b nit 2025-04-10 01:00:29 +00:00
Ruben Fiszel 63fa499015 feat: agent workers v2 using http (#5588) 2025-04-10 00:56:11 +00:00
HugoCasa b4f8c88c19 fix: mssql ca_cert deserializing (#5587) 2025-04-08 17:12:20 +02:00
Piyush मिश्र 1007f37f4b function takes 13 arguments but 14 arguments were supplied (#5577) 2025-04-06 20:38:39 +00:00
Piyush मिश्र 1f3e7d9029 fix: Dynamic select does not work with tag //native (#5576)
closes #5490
2025-04-05 20:43:53 +00:00
pyranota f3ed108e43 nit: replace KJQXZ with more meaningful notation (#5547)
* nit: replace `KJQXZ` with more meaningful notation

Originally this string is located in all places, where modification is needed in order to add new language support

* relative -> related

* revert shebang in substitue.sh

* remove '}'
2025-04-02 21:03:04 +00:00
Alexander Petric e5519f79aa feat: mssql support cert configuration (#5559)
* feat: mssql support cert configuration

* use tokio async

* using jobdir

* use std
2025-04-02 20:10:51 +00:00
Alexander Petric 3fd7a5ce9c fix: aad_token can be empty string (#5557)
* fix: mssql aad_token can be empty string

* simplify deserializer
2025-04-02 19:26:32 +02:00
Alexander Petric dd30692617 fix: mssql aad_token can be empty string (#5556) 2025-04-02 18:51:02 +02:00
Alexander Petric c230e2aed9 feat: ms sql aad authentication support (#5539)
* feat: ms sql aad authentication support

* format

* format
2025-03-31 04:05:08 +00:00
pyranota c020af5a98 nit: fix windows build (#5531)
rror[E0425]: cannot find function `pull_from_tar` in module `crate::global_cache`
    --> windmill-worker\src\common.rs:1214:43
     |
1214 |                 Some(crate::global_cache::pull_from_tar(
     |                                           ^^^^^^^^^^^^^ not found in `crate::global_cache`
     |
note: found an item that was configured out
    --> windmill-worker\src\global_cache.rs:87:14
     |
87   | pub async fn pull_from_tar(
     |              ^^^^^^^^^^^^^
note: the item is gated here
    --> windmill-worker\src\global_cache.rs:86:1
     |
86   | #[cfg(all(feature = "enterprise", feature = "parquet", unix))]
     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-03-28 17:17:23 +01:00
Ruben Fiszel 2e74595a37 nit, add scheduled_for in logs 2025-03-27 21:28:21 +01:00
pyranota 59740c0478 feat: add java support (#5458)
* feat: add nu (nushell) support

* add worker tests

* deactivate tables and non-any types below top-level

full support will come in V1
for V0 it's better to keep things minimal and simple

* add syntax highlighting

used python's grammar, since nushell isn't supported by monaco nor svelte-highlights

for V1 nu will get it`s own grammar

* add logo

* partially implement plugin support

* change logo + ability to deploy + nsjail draft

* static variables + get_resource + get_variable

* lsp/dev.nu + initial nu lsp (not working yet)

* make it work with nsjail

* nullguard

* Much more flexible signature parsing and better error-messages

* add init script

* rename nulsp to nu

* install nu to dockerfile

* fix merge

* implement Default for MainArgSignature

* stage NU_CACHE_DIR

* improve dockerfiles

* dev.nu for parser-wasm + flake.nix

* update code for windows

* add nushell to flake

* upload Cargo.lock

* make build.sh work on nixos

* build wasm cli parsers

* add docs to README_DEV.md

* add helper script docker/dev.nu

* improve docker/dev.nu

* fix windows

* commit frontend/package(lock).json

* update cargo.lock

* correctly update cargo.lock

* remove lsp

* update flake.nix to include svelte server and nushell

* Revert base.sql to main

* remove PLUGIN_USE_RE

* make CARGO_PATH private

* add nu to cli

* Change flags to build wasm-nu-parser

* remove flake.nix from parser-wasm

* update wasm-build target

* remove unused import

* add cli support for nu

* update github workflows

* wasm-build 0.17 -> 0.19

* update build script

* update cargo.lock

* Fix typographical error

* start working on java

* do java boilerplate

* implement parser for java

* update Cargo.lock

* update ENV_SETTINGS

* use published nu parser

* update package.lock

* java is S3 + Caching enabled

* install nsjail backup

* commit v0

* fix nsjail

* v0.1

* rewrite parser in tree-sitter

* implement parser from scratch

* polishing

* change init script to match new parser

* fix imports

* fix cli build

* fix cli build

* refactor install phase

* implement .valid.windmill atomic verification

* implement java init functionality

* remove quick-xml

* fix windows not recognizing 'mvn'

* create empty settings.xml if there is no config provided

* clean up

* change default settings.xml

* change classpath format for windows

* docs to helper

* java copy bin cache instead of symlink

* remove comments

* merge

* fix package.json

* fix package.json 2

* minor fixing

* migrate to Coursier

* update misc

* Http(s) Proxy + CA certs

* remove unused .wasm

* make requirements insensitive to spaces

* update handle_child refs

* rework save_cache for directories

* fix s3 bug

* compile .wasm for cli

* remove uuid import

* fix compilation

* use reference

* fix zero-dep failure

* removing unsafe stuff

* remove unneeded imports

* revert: we still need winapi

* remove nix store from nsjail

* do not create cache_nomount

* add java to dnt

* remove duplicated dependency in init script

* fix typos

* fix CI

* use published parser
2025-03-27 15:12:55 +01:00
Ruben Fiszel 8129672d9e fix: clean job dirs between flow locks 2025-03-26 23:23:04 +01:00
Ruben Fiszel 5573d88695 perf: cache workspace premium check 2025-03-25 02:00:51 +01:00
Diego Imbert e868fe2bf5 feat: list references upon renaming a script or a flow (#5487)
* Refactored flow_workspace_runnables to more generic workspace_runnable_dependencies

* list flows referencing an item upon renaming it

* Refactor with two exclusive columns to avoid breaking FK constraints

* Show apps depending on item upon renaming

* sqlx prepare

* list-disc instead of •

* on delete and on update cascade

* displayPathChangedWarning oneOf check instead noneOf

* combine migrations + add "on update cascade" to flow fk

* unique index on app dependencies to avoid duplicates

* create new workspace_runnable_dependencies instead of renaming old table

* Add "looking for references" loading msg

* Revert "create new workspace_runnable_dependencies instead of renaming old table"

This reverts commit 015c38ca8f.

* flow_workspace_runnables view for backwards compatibility

* Add warning for script imports on rename

* support import dependency tracking in deno

* number of using scripts / flows / apps tooltip

* forgot sqlx prepare

* delete app-related rows in down migration
2025-03-24 22:08:19 +01:00