Commit Graph

970 Commits

Author SHA1 Message Date
Ruben Fiszel b868e446fc improve error messages on windows 2024-11-15 09:43:54 +01:00
Ruben Fiszel 029462bc57 nits logs 2024-11-15 01:15:51 +01:00
Ruben Fiszel 089826e5b5 delete venv folder if pip install didn't succeed 2024-11-15 01:11:17 +01:00
pyranota f240d1322a feat: Handle pip install by uv (#4517)
* feat: Handle `pip install` by `uv`

Dirty and untested, but already something working

* Integrate with NSJAIL and prepare fallbacks

* Refactor fallback
no_uv disable compile and install
where no_uv_install and no_uv_compile are a bit more specific

* Remove `--disable-pip-version-check`
Reason:
   warning: pip's `--disable-pip-version-check` has no effect

* Fix backend compilation error

* Pip fallback overwrite UV's cache

* Initially refactor cache (No S3)

* Support S3

* Remove unused import

* Handle flags for NSJAIL

* Return deleted flag

* Update Dockerfile

* Update docker-image.yml

* Update docker-image.yml

* Add --link-mode=copy and remove -v

* Fix NSJAIL INDEX_URL

* Fix flock and windows

* Update python_executor.rs

* Remove line from Dockerfile

We dont need it and to trigger build

* fixing for windows

* Dont pin python to specific version

* Change TMP for windows

* Revert docker-image.yml

* Disable UV for ansible

Will be enabled later.
Needs proper testing and its better to split onto 2 PRs with first modifying python and second ansible

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Alexander Petric <petric.al@gmail.com>
2024-11-15 00:12:56 +01:00
Lucas Abel 4fdca87de9 nit: cleanup raw_flow usage (#4707)
* nit: cleanup `raw_flow` usage

* nit: refactor two queries into one
2024-11-14 12:23:27 +01:00
Lucas Abel 2d572695ef nit: generalize usage of has_failure_module (#4706) 2024-11-14 08:49:12 +01:00
Ruben Fiszel 84bd06e906 fix: fix intempestive expr type change in flow transform 2024-11-14 00:24:02 +01:00
Ruben Fiszel 4b8f3580a4 add LOGS_TO_STDOUT to ee 2024-11-13 23:30:52 +01:00
Ruben Fiszel ce01fa1677 split more ee log logic 2024-11-13 18:49:41 +01:00
Ruben Fiszel 50861fccc0 fix: support multiple pip-extra-index-url with commas 2024-11-12 23:31:00 +01:00
Ruben Fiszel 8c6f5a320b propagate exit code in error result when relevant 2024-11-08 16:37:58 +01:00
Ruben Fiszel 8bc9a021a8 fix(bash): correctly propagate exit errors 2024-11-08 16:33:01 +01:00
HugoCasa dc8bd6d2b5 feat: http custom routes for static assets (#4666)
* feat: http custom routes for static assets

* update ee ref + fix build

* fix build

* nit
2024-11-07 20:16:14 +01:00
Ruben Fiszel 771d740701 fix: clarify error messages when job timeout or cancelled with more details 2024-11-06 19:18:34 +01:00
Ruben Fiszel 7c0d9901e0 fix main 2024-11-06 16:34:09 +01:00
HugoCasa 7578cebaf9 fix: display logs in native mode when script fails (#4655)
* fix: display logs in native mode when script fails

* oups

* oups
2024-11-06 16:26:23 +01:00
Alexander Petric 693b7a4fd4 feat(oauth): add snowflake oauth support (#4622)
* feat(oauth): add snowflake oauth support

* fixes

* fix keypair auth

* avoid loop when changing settings

* including account id doc link in the settings ui

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-11-04 23:35:27 +01:00
Guilhem 22ab51e991 feat(frontend): improve and simplify scheduled poll flows (#4560) 2024-11-04 23:18:09 +01:00
Ruben Fiszel e11cc13c92 fix: apply NO_PROXY and HTTP_PROXY, HTTPS_PROXY more consistently 2024-11-02 13:57:30 +01:00
Ruben Fiszel 05ac6ba094 fix(prometheus): fix incorrect worker_busy set to 1 2024-11-01 16:31:32 +01:00
Ruben Fiszel 557b949efe fix(s3): align s3 handler additional creds providers 2024-11-01 00:51:28 +01:00
pyranota ef37928865 Add rollback for multipython (#4610) 2024-10-31 14:27:34 +01:00
HugoCasa 921e3217c6 improve array params handling in postgres (#4605) 2024-10-30 14:44:42 +01:00
Ruben Fiszel 692b0568fc fix(bun): handle bun lockfile created with windows (#4602) 2024-10-29 14:09:38 +01:00
Ruben Fiszel 71f84dee08 handle better empty tag overrides 2024-10-29 09:01:02 +01:00
Ruben Fiszel 99400ebbf3 improve test this step for flow tag overide 2024-10-29 08:19:17 +01:00
Ruben Fiszel ab38215c86 fix(backend): in flows, workspace scripts should use their set tags instead of the default one 2024-10-29 07:29:39 +01:00
Ruben Fiszel 85de44cb93 feat: autoscaling v0 (#4593)
* all

* all

all

all

all

all

all

all

all

all

* all

* all

* nits

* nits
2024-10-28 00:47:19 +01:00
Ruben Fiszel 2cbf027653 fix: do not update created_at of scripts on lockfile generation 2024-10-25 17:44:41 +02:00
wendrul 058d48ccab Make file_resources that are filled with a variable in ansible (#4584)
* Make file_resources that are filled with a variable in ansible

* Rename file_resources section to files

* Add variable example to init code
2024-10-25 14:18:28 +02:00
Stephen Beckstrand 09a8a0361b chore: Update references to windmill-ee-full image (#4578)
* chore: Update windmill-ee-full image references

The messaging in a few backend executors references the need to
use the full enterprise image rather than `windmill-ee`. These
messages reference the `windmill-full-ee` image but its the image
is actually named `windmill-ee-full` under the available Packages.

* chore: Update refs to rust to ansible in ansible executor
2024-10-24 17:40:59 +02:00
Ruben Fiszel 2cd46cc55f fix(bash): correctly propagate sigterm for cancelled bash scripts 2024-10-24 14:42:37 +02:00
Ruben Fiszel 7024404bb3 feat(typescript-bun): support relative imports without the .ts extension 2024-10-18 12:26:57 +02:00
pyranota c3ba832444 Annotations refactor (#4512)
* Refactor annotation system

* Move tests to different location

* Limit annotations per line (15)

* Write initial benchmark

* Optimize

* More optimizations

* Rewrite with 3x speed!

* Remove benchmarks

They were in wrong place and for development

* Polishing

* Remove unused leftover

* Integrate new annotation system

* Add comment explanation

* Remove `hashe-based matching`

Turned to be it is impure and non-deterministic method

* Add `tracing::error` for `// native` without `deno_core`

If you build backend without `deno_core` feature flag, and add //native
annotation to ts script, it would just silently exit.

This commit prints error to logs

* add more tests

* Update annotations.rs

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-17 09:14:31 +02:00
Ruben Fiszel d89877e0ba fix(nsjail): improve memory reading when using nsjail 2024-10-17 07:42:30 +02:00
Ruben Fiszel f3ddd76159 fix: add extra args support for exception to bun scripts 2024-10-17 01:19:25 +02:00
Ruben Fiszel fac41fd700 fix: fix script persistence in url + add support for extra error args in python 2024-10-17 01:07:56 +02:00
Ruben Fiszel f8d1a60b08 fix deno lockall tests 2024-10-12 17:59:05 +02:00
Ruben Fiszel 98e2580bbb fix: fix deno cache --allow-import on deno 2 2024-10-12 17:18:46 +02:00
Ruben Fiszel b603f4a6ba allow python multiprocessing on nsjail 2024-10-11 01:40:09 +02:00
Ruben Fiszel 3330727ac0 add --allow-import to nsjail deno 2024-10-10 15:29:49 +02:00
Ruben Fiszel 196dfc3978 nit deno_core 2024-10-10 15:13:55 +02:00
Ruben Fiszel 9e31cceb99 fix: improve handling of empty lock files on deno 2.0 2024-10-10 04:39:57 +02:00
Ruben Fiszel 52edd395cf deno 2.0 use frozen for cache 2024-10-10 04:06:27 +02:00
Ruben Fiszel 5c51d86b6a fix: update internal deno runtime to latest (deno 2.0) 2024-10-10 03:42:28 +02:00
Alexander Petric c070df12e8 improving pwsh error handling (#4504)
* making pwsh error handling more robust

* cargo fmt
2024-10-09 05:20:46 +02:00
Ruben Fiszel 7c0950d9da fix: fix uv path 2024-10-03 22:42:34 +02:00
Ruben Fiszel c2c2a494bd fix: flow picker of flows + precache hub scripts as bundles 2024-10-03 20:48:26 +02:00
pyranota 1e7a37ac8b feat: Replace pip-compile with uv (#4460)
* Update `shell.nix`

- Replace pip-compile with uv packages
- Pin rust version
- Add var to trigger windmill print more info in stdout

* Replace `pip-compile` with `uv` (dirty + untested)

* Fix arguments passed to uv

Some of the flags are included by default in UV and can be safely removed:
- --resolver=backtracking
- --no-emit-index-url

Also uv does not support `--pip-args` and suggests to directly pass args to uv.

* Remove extra `dbg!`

* Replace 'pip-compile' with 'uv' in Dockerfile

* Add fallback option to `pip-compile` (Disabled)

* Add `uv` to `docker/DockerfileSlim*`

* Add `get_annotation_python` and rename `get_annotation` to `get_annotation_ts`

* Add option to fallback to pip-compile

Put `# no_uv` on top the file for specific python script

Or set `USE_PIP_COMPILE` variable to `true`

* Put back `pip-tools` into shell.nix

* Make sure lockfile resolves again if `#no_uv` used

Add #no_uv to the end of requirements (requirements.in)
That way if something breaks for customer, then they put #no_uv and new lockfile will be resolved

* Put `pip install pip-tools` in original spot

* Fix compilation error

* Fix EE compilation error

error[E0658]: attributes on expressions are experimental
   --> windmill-worker/src/python_executor.rs:144:5
    |
144 |     #[cfg(feature = "enterprise")]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: see issue #15701 <https://github.com/rust-lang/rust/issues/15701> for more information

* Add `no_cache` annotation

Will force recalculation of lockfile
And block uv from using cached values

* Target uv cache to /tmp/windmill/cache

* Prohibit uv from managing python

* Add uv to DockerfileBackendTests

* Pin uv version to 0.4.18 in Dockerfiles

* Dont put `#no_uv` in requirements.in

Instead postfix hash for requirements.in with `-no_uv`

* Push Warning to logs if fallbacked to pip-compile

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 16:00:35 +02:00
Alexander Petric 6eb7f8ce81 feat(worker): support workers to run natively on windows (#4446)
* minimal code change to get windmill worker on windows for bun and python + rustfmt

* adding support for powershell

* compiling error on unix

* rust linting comments

* comments hugo: PSModulePath

* comments ruben, refactor to simplify

* adding build workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* skip migration env, ee fixes

* improvements powershell

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* killing process tree in windows

* sqlx_offline

* install openssl for github windows runner

* used pre-installed openssl

* used pre-installed openssl

* build ee

* build ee

* build ee

* build ee

* adding commented out steps for artifact publishing

* build on tag matchinv v* pattern

* ren instead of mv on Windows

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix merging issue

* gate imports for windows

* fixing default cargo home path...

* fixing default cargo home path...

* comments ruben

* make pwsh default modules loading more robust on unix (#4448)

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 15:59:12 +02:00