Commit Graph

1224 Commits

Author SHA1 Message Date
Ruben Fiszel 556a4118e9 fix: improve concurrent job parallelism performance 2025-06-09 16:59:07 +02:00
pyranota cb7d23ab0b nit: fix windows build (#5896) 2025-06-07 03:35:54 +02:00
pyranota 5ab4eca2f0 fix: never consider minor version for global site packages (#5893) 2025-06-07 03:35:28 +02:00
pyranota 7fcffc6375 fix(python): account instance version when cli deploy and local lockfile (#5894)
* fix: do not ignore instance python version with cli deploy and local lockfile

* ellipsis
2025-06-06 17:49:59 +00:00
pyranota 68f179ab13 fix: cannot parse INSTANCE_PYTHON_VERSION (#5874) 2025-06-05 16:56:30 +02:00
pyranota a382ed0b2d fix(python): assign PATCH version to python runtime only when needed (#5866)
* fix

* clean up

* nit

* fix integration tests

* nit

* another nit

* remove dublicate test
2025-06-04 17:15:46 +02:00
dieriba 5b534be168 feat: ssh repl like direct to workers hosts machine (#5809)
* feat(backend): add repl for worker

* feat(frontend): add repl to interact with worker and missing packages

* nits

* Update backend/windmill-worker/src/result_processor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/result_processor.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update frontend/src/lib/components/WorkerRepl.svelte

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* fix typo

* fix cd

* nits

* refactoring

* feat: handle other agent worker to also access repl feature, fix bugs

* update repo ref

* nits and match function args

* nits

* typo

* remove struct AgentWorkerData

* update repo ref

* nits

* impl new for authed client and revert to async

* update ref

* updage ee repo ref

* fix missing method/imports small bugs

* update ee repo ref

* update .sqlx

* test

* clean wait for interactive_shell future

* free call stack

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-06-03 18:41:14 +02:00
Diego Imbert 7467833239 EE Refactor (#5844)
* app compiles with every ee substituted

* Replace all oss files content

* Revert "Replace all oss files content"

This reverts commit ea4017d59f.

* delete all ee

* hide all _ee files under private flag

* hide every oss stuff when private flag set

* pub use *

* gitignore and substitute script

* pub mod for ee needed for ee repo

* small mistakes

* remove oidc_oss impl

* ee ref (temp)

* ee ref

* fix --all-features selecting private in OSS CI

* ee repo ref

* allow unused
2025-06-02 22:12:33 +02:00
pyranota 631ccea793 feat(rust): shared build directory (#5610)
* feat: rust incremental compilation

* update dockerfiles

* fix compilation error

* add windows flags

* init

* polishing

* update

* return mount-point

* Update backend/windmill-worker/src/rust_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* share worker dir if nsjail off

* final refactor

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-05-29 22:11:35 +02:00
Ruben Fiszel b0074da69e duckdb optional dep 2025-05-29 14:54:56 +02:00
Diego Imbert d5e9213fd5 feat: duckdb sql lang support (#5761) 2025-05-27 15:52:57 +02:00
Ruben Fiszel e15b94692d feat: add aws oidc support for instance s3 storage (#5810)
* backend

* iterate

* all

* all

* all

* iterate

* revert

* all

* add tracing to get of authed client

* all

* all

* lal

* all

* update

* fix

* push

* all

* all

* revert

* frontend

* fix checks

* avoid deadlock

* safer

* fix

* fix
2025-05-25 14:03:38 +02:00
pyranota 2833d6da3f feat(python): inline script metadata (PEP 723) (#5712)
* make resolver

* more updates

* fix build

* fix raw_dependencies job type

* compat with http agent workers

* refactor

* rename

* more refactor

* cleanup

* more tests

* fix s3

* small fixes

* more fixing

* fix endpoint

* nit: update comment

* update ee ref

* update ee ref

* update ee ref

* implement safer `list_available_python_versions`

* add tracing to get of authed client

* internal: Trigger claude when commenting with /aider (#5783)

* add claude instructions files

* call claude too when using aider

* fix

* add draft for linear claude integration

* fix build

* update ee ref

* ignore versions <=3.9

* fix windows build

* correct versions filter

* fix windows build (this time for real)

* inject error to debug CI

* update CI

* undo debug of CI

* fix tests

* remove outdated comment

* update ee repo ref

* Update ee-repo-ref.txt

* Update backend/parsers/windmill-parser-py-imports/src/lib.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update InstanceSetting.svelte

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: centdix <40307056+centdix@users.noreply.github.com>
Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-05-24 02:13:47 +02:00
Ruben Fiszel d7d67f3e62 improve err message 2025-05-22 19:48:26 +02:00
Ruben Fiszel 81dfc0abdb fix: improve docker mode unexpected exit handling 2025-05-22 19:37:47 +02:00
Ruben Fiszel ab632ba92a reduce the pip resolution cache duration 2025-05-22 18:47:02 +02:00
Ruben Fiszel a268bb87e4 fix 2025-05-22 14:12:10 +02:00
Ruben Fiszel 729d90f61e add more labels to traces 2025-05-22 12:07:21 +02:00
Ruben Fiszel 32b473c6ad more verbose docker wait errors 2025-05-22 01:07:29 +02:00
HugoCasa 23cce4ddb0 fix: workspace preprocessor improvements (#5784)
* add tracing to get of authed client

* internal: Trigger claude when commenting with /aider (#5783)

* add claude instructions files

* call claude too when using aider

* fix

* add draft for linear claude integration

* fix: workspace preprocessor fixes

* tmp ee ref

* fix build

* update ee ref

* fix: hub script preprocessor handling

* fix build

* good ref

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: centdix <40307056+centdix@users.noreply.github.com>
2025-05-21 17:49:21 +02:00
pyranota 14fd6458a8 feat(python): add annotation to skip result post-processing (#5769)
* feat(python): add annotation to skip result post-processing

Typically windmill will replace all NaN, Infinity and -Infinity from resulting string.
We do it because JSON specification does not support these types as well as DB.

However it will substitute also the cases when any of those words are used within the string.
E.g. script returning "To Infinity and Beyond" will be postprocessed to "To null and Beyond".
Current behaviour is done for the performance sake and now can be disabled with `#skip_result_postprocessing` annotation.

* add comments

* remove extra comments
2025-05-19 22:31:46 +02:00
Ruben Fiszel 985ef9cd9a nit error 2025-05-17 10:56:16 +02:00
Ruben Fiszel a499f9dffe fix: improve handling of custom concurrency key/tag with preprocessors (#5762)
* fix

* rm spurrious diff

* handle failing preprocessors

* nits

* update

* update
2025-05-17 10:52:28 +02:00
Diego Imbert e88954a217 fix: S3 sql mode returns S3Object (#5764)
* fix upload_s3_file not checking response

* Return S3Object instead of path string
2025-05-16 19:28:57 +02:00
Ruben Fiszel 123657fa63 fix: avoid deadlocks in sending completed job to result processors (#5742)
* buffer

* unbounded and bounded

* fix

* buffer

* impl clone

* flume it all

* update

* update

* update
2025-05-15 09:13:09 +02:00
Diego Imbert d9916318b1 oracle support for s3 streaming + misc (#5738)
* changed SQL templates to include S3

* oracle support for s3 streaming
2025-05-14 15:45:00 +02:00
Ruben Fiszel ef068ab5ef add more duration warnings to logs 2025-05-14 10:13:14 +02:00
Ruben Fiszel 95cf9c6814 fix: add support for log compaction on docker jobs (#5732)
* improve docker compact

* improve docker compact

* update ref

* update

* update agent workers
2025-05-13 21:42:15 +02:00
wendrul 4662cfd67d fix: Ansible lockfile back compatibility issue (#5731)
* fix: Ansible lockfile back compatibility issue

* Replace condition with more readable alternative
2025-05-13 16:19:18 +02:00
Diego Imbert 25bbdba1d1 feat: sql jobs outputting to s3 + streaming for high-number of rows (#5704)
* stream to s3 boilerplate

* S3 works with new syntax

* snowflake s3 streaming support

* postgres s3 support

* fix postgres stream format

* mysql s3 streaming

* mssql s3 streaming

* new s3 mode syntax

* optional folder param

* rename folder to prefix

* json_stream_arr_values

* cargo toml rollback

* convert_ndjson with datafusion

* format conversion kinda works

* Fixed not finishing the datafusion writer

* support for pg and mssql

* fix file ext

* bigquery conversion and works with s3 streaming

* fix s3 flag parser

* snowflake s3 streaming support

* factor out duplicate code

* remove anyhow

* Err case for parse s3 mode

* Send error to mpsc

* bigquery s3 streaming fix for huge queries

* remove extra stuff

* snowflake s3 streaming support

* small regex mistake

* cfg(not(feature = "parquet"))

* fix CI (unused import)

* error handling fix (graphite)
2025-05-13 10:19:44 +02:00
Ruben Fiszel c31aedabab fix: make ansible more resilient to invalid lockfiles 2025-05-12 11:40:13 +02:00
dieriba 00faf1eefe nits: use generic deserializer for empty data (#5727)
* use generic deserializer for empty data

* update repo ref

* fix import
2025-05-12 11:00:55 +02:00
Ruben Fiszel 4bf2c38cbf nit compile 2025-05-11 11:52:42 +02:00
Ruben Fiszel c76bc4a9cc fix: improve error display of nativets exceptions 2025-05-11 11:35:31 +02:00
Ruben Fiszel 5061358c89 fix: improve agents workers handling of WHITELIST_ENVS 2025-05-11 08:38:46 +02:00
HugoCasa d2787cfd78 feat: preprocessor refactor (#5629)
* feat: preprocessor refactor

* nit

* oneOf kind support + new capture format + fix build

* fix build

* fix build

* update preprocessor templates

* cache runnable version info

* update ee ref

* update hub scripts && ee ref
2025-05-09 09:50:01 +02:00
dieriba b2c7bdff13 feat: raise error if end early in flow (#5653)
* update front

* handle err_msg display if enabled when stop condition is met

* nits

* Update backend/windmill-worker/src/worker_flow.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* Update openflow.openapi.yaml

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update worker_flow.rs

* Update worker_flow.rs

* update flow test

* update condition

* remove println

* nits

* fix flow test

* nits and missing property on struct definition

* nits

* fix unsaved state

* update .sqlx

* cleanup sqlx

---------

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-05-07 18:23:40 +02:00
Ruben Fiszel 46b8ac93ae fix: ansible in agent mode can use inventory.ini 2025-05-06 11:57:08 +02:00
Ruben Fiszel ed6313258b fix: do not track relative deps for scripts with raw defined deps from CLI (#5696) 2025-05-02 00:37:28 +02:00
Ruben Fiszel 0e0c9dd579 improve agent workers 2025-04-29 20:15:49 +02:00
Diego Imbert e1ead53ce5 fix: skip_flow_update for dependency tracking table (#5670) 2025-04-28 17:57:35 +02:00
Ruben Fiszel 33380a8cd9 fix: improve skip failure on parallel branchall 2025-04-27 00:37:25 +02:00
wendrul cfe05fa1b8 feat: Ansible improvements (vault, roles and git repos) (#5655)
* Ansible vault + roles

* Clone arbitrary repos

* Fix cloning logic after merge

* Make function for cloning without history any commit

* Cloning repos and lockfile on the commit

* Improve error messages

* Create lockfile for roles and collections

* Simplify ansible ssh identity interface

* Ansible vault password: pass just a variable instead of 2 step approach

* Lock lockfiles for roles and collections

* fix typo

* Change git ssh identity section name

* Rename variable

* Update init script for ansible

* Suppress error when no roles

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-04-24 16:04:28 +02:00
Ruben Fiszel 77bd5b2458 fix: track relative imports in python and ts even if lockfile is provided 2025-04-23 08:40:22 +02:00
Diego Imbert b8a328a870 fix: improve MySQL datetime parser timezone handling (WIN-1155) (#5645)
* fix mysql datetime parser for non tz dates

* static regexes
2025-04-21 14:46:28 +02:00
Ruben Fiszel 031e522a6a ui code builder v0 (at secret path) (#4964) 2025-04-20 01:19:02 +02:00
Ruben Fiszel 55063e28c7 improve no_proxy value by default 2025-04-17 00:05:47 +02:00
Ruben Fiszel c4ceb51bdb fix: flow editor workspace script test use actual workspace script hash 2025-04-16 00:44:49 +00:00
Ruben Fiszel f79ce8b5bf improve last_job_suspended_history 2025-04-12 15:02:43 +00:00
Ruben Fiszel 4f6ef903d0 nit benchmarks 2025-04-12 12:54:05 +00:00