Commit Graph

937 Commits

Author SHA1 Message Date
Ruben Fiszel 649872ab0d all 2024-10-17 15:18:03 +02:00
Ruben Fiszel b7ad19bb75 fix(nsjail): improve memory reading when using nsjail 2024-10-17 07:42:30 +02:00
Ruben Fiszel 1466da3999 fix: add extra args support for exception to bun scripts 2024-10-17 01:19:25 +02:00
Ruben Fiszel 3174024d8e fix: fix script persistence in url + add support for extra error args in python 2024-10-17 01:07:56 +02:00
Ruben Fiszel a323d2b29a fix deno lockall tests 2024-10-12 17:59:05 +02:00
Ruben Fiszel 42fe31f804 fix: fix deno cache --allow-import on deno 2 2024-10-12 17:18:46 +02:00
Ruben Fiszel 0915968eba allow python multiprocessing on nsjail 2024-10-11 01:40:09 +02:00
Ruben Fiszel 7db74eecb9 add --allow-import to nsjail deno 2024-10-10 15:29:49 +02:00
Ruben Fiszel b133f19333 nit deno_core 2024-10-10 15:13:55 +02:00
Ruben Fiszel 7ca5bf2fae fix: improve handling of empty lock files on deno 2.0 2024-10-10 04:39:57 +02:00
Ruben Fiszel 0e61f8db78 deno 2.0 use frozen for cache 2024-10-10 04:06:27 +02:00
Ruben Fiszel c3a5736641 fix: update internal deno runtime to latest (deno 2.0) 2024-10-10 03:42:28 +02:00
Alexander Petric f831b9b1fe improving pwsh error handling (#4504)
* making pwsh error handling more robust

* cargo fmt
2024-10-09 05:20:46 +02:00
Ruben Fiszel 19c62ba195 fix: fix uv path 2024-10-03 22:42:34 +02:00
Ruben Fiszel c84e6fd05d fix: flow picker of flows + precache hub scripts as bundles 2024-10-03 20:48:26 +02:00
pyranota b54c9ee657 feat: Replace pip-compile with uv (#4460)
* Update `shell.nix`

- Replace pip-compile with uv packages
- Pin rust version
- Add var to trigger windmill print more info in stdout

* Replace `pip-compile` with `uv` (dirty + untested)

* Fix arguments passed to uv

Some of the flags are included by default in UV and can be safely removed:
- --resolver=backtracking
- --no-emit-index-url

Also uv does not support `--pip-args` and suggests to directly pass args to uv.

* Remove extra `dbg!`

* Replace 'pip-compile' with 'uv' in Dockerfile

* Add fallback option to `pip-compile` (Disabled)

* Add `uv` to `docker/DockerfileSlim*`

* Add `get_annotation_python` and rename `get_annotation` to `get_annotation_ts`

* Add option to fallback to pip-compile

Put `# no_uv` on top the file for specific python script

Or set `USE_PIP_COMPILE` variable to `true`

* Put back `pip-tools` into shell.nix

* Make sure lockfile resolves again if `#no_uv` used

Add #no_uv to the end of requirements (requirements.in)
That way if something breaks for customer, then they put #no_uv and new lockfile will be resolved

* Put `pip install pip-tools` in original spot

* Fix compilation error

* Fix EE compilation error

error[E0658]: attributes on expressions are experimental
   --> windmill-worker/src/python_executor.rs:144:5
    |
144 |     #[cfg(feature = "enterprise")]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: see issue #15701 <https://github.com/rust-lang/rust/issues/15701> for more information

* Add `no_cache` annotation

Will force recalculation of lockfile
And block uv from using cached values

* Target uv cache to /tmp/windmill/cache

* Prohibit uv from managing python

* Add uv to DockerfileBackendTests

* Pin uv version to 0.4.18 in Dockerfiles

* Dont put `#no_uv` in requirements.in

Instead postfix hash for requirements.in with `-no_uv`

* Push Warning to logs if fallbacked to pip-compile

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 16:00:35 +02:00
Alexander Petric f5c4727274 feat(worker): support workers to run natively on windows (#4446)
* minimal code change to get windmill worker on windows for bun and python + rustfmt

* adding support for powershell

* compiling error on unix

* rust linting comments

* comments hugo: PSModulePath

* comments ruben, refactor to simplify

* adding build workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* skip migration env, ee fixes

* improvements powershell

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* killing process tree in windows

* sqlx_offline

* install openssl for github windows runner

* used pre-installed openssl

* used pre-installed openssl

* build ee

* build ee

* build ee

* build ee

* adding commented out steps for artifact publishing

* build on tag matchinv v* pattern

* ren instead of mv on Windows

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix merging issue

* gate imports for windows

* fixing default cargo home path...

* fixing default cargo home path...

* comments ruben

* make pwsh default modules loading more robust on unix (#4448)

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 15:59:12 +02:00
Ruben Fiszel a4a59ef21e deno_core is an optional feature flag (#4473)
* full

* all

* add deno_core as features

* all

* remove warnings

* all
2024-10-03 13:11:58 +02:00
HugoCasa b70ca6e3ed skip step better UI (#4465) 2024-10-01 12:28:46 +02:00
HugoCasa 0df169e3f9 feat: flow step skipping (#4461) 2024-09-30 16:30:59 +02:00
Ruben Fiszel fcc51bc5a4 optimize flow update parallel queries 2024-09-29 13:17:11 +02:00
Ruben Fiszel 0df35e5787 add urandom to nsjail 2024-09-29 12:31:55 +02:00
Ruben Fiszel 004bcb2b07 quote req and target for nsjail python download 2024-09-29 12:13:54 +02:00
Ruben Fiszel 8fb3637183 optimize early stop query 2024-09-29 01:19:06 +02:00
Ruben Fiszel 5acfc83521 optimize skip failure query 2024-09-29 00:38:43 +02:00
Ruben Fiszel 30d9c40193 optimize more queries 2024-09-28 17:34:37 +02:00
Ruben Fiszel 363e5e1727 improve suspend_first frequency behavior 2024-09-28 15:50:12 +02:00
Ruben Fiszel b5e226b977 fix: improve suspend_first behavior and frequency 2024-09-28 15:46:50 +02:00
HugoCasa 0b9ec83036 fix: allow preprocessor to write to args.json on nsjail (#4455) 2024-09-27 21:12:38 +02:00
Ruben Fiszel cfaea38e92 improve benchmarking tools (#4450) 2024-09-27 10:48:04 +02:00
Ruben Fiszel 9f91b1995a feat(cli): add queues, workers and worker-groups commands (#4439)
* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all
2024-09-26 11:27:37 +02:00
HugoCasa 3ce5587faa feat: add return_last_result annotation to sql (#4443) 2024-09-25 18:11:11 +02:00
wendrul 398a09b741 feat: add static variable and resources support to ansible (#4435)
* Add static variable and resources to ansible

* Add wasm parser to cli

* Fix pulling of resources, rs and playbooks

* Remove unused import, add publish pkg-yaml

* Update package.json

* Update generate-metadata

* Fix null fields being serialized in script metadata

* npm install
2024-09-25 15:05:53 +02:00
HugoCasa 590321fd3c feat: add tag filtering to external JWT authentication (#4425)
* feat: tag filtering jwt ext auth

* move tags to scopes

* fix symlink

* update ee ref
2024-09-25 11:21:19 +02:00
Ruben Fiszel 93fa6a0e97 improve init script error handling 2024-09-24 19:27:00 +02:00
HugoCasa 61f27acbd5 fix: time handling in bun and deno (#4423)
* fix: time handling in bun and deno

* put dates conversion in try catch
2024-09-23 18:00:36 +02:00
wendrul a500994cc2 fix: various improvements for ansible (#4419)
* Improve error messages, fix dependency job issue

* Add nsjail support for ansible

* Remove invalid comment syntax

* Change nsjail config and ansible controller config to be ephemeral

* Move ansible cfg to later to overwrite possible files named ansible.cfg

* Fix ansible.cfg env var

* Fix tmp dir issue
2024-09-23 16:44:37 +02:00
HugoCasa 304dac3447 feat: http routing (#4339)
* feat: http routing

* all

* feat: improve UI

* final stuff

* fix: sqlx

* fix: nit

* fix: nits

* fix: error handler display

* fix: routes panel perms

* all

* fix: improve ability to paste from macos in vscode extension

* fix lock-write for deno

* all

* cleaning

* fix

* cli preprocessor

* nits

* nits

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-20 15:38:38 +02:00
Ruben Fiszel 69b8754aef fix(bun): never pre-bundle BASE_URL env variable 2024-09-20 09:48:00 +02:00
Ruben Fiszel 623ece8c67 fix(go): put shared directory in job dir and not go parent 2024-09-19 10:05:08 +02:00
wendrul 6855b8da9a feat: ansible playbook support (#4399)
* Ansible execution and parsing

* Working collections and pip dependencies

* Remove unused vars

* File resources logic for ansible

* Make frontend for ansible and resource file

* Format file

* Change naming of file resource and inventory fields

* Add autocomplete for file extension resource creation

* Add endpoint to list file formats for resource types

* Add CLI functionality to pull/push file resources

* Add beta tag to ansible

* Add Full image dockerfiles containing ansible

* Prepare sqlx

* Update cargo.lock

* Update ansible path

* Remove unused imports

* Add back import removed by rust-analyzer

* Improve ansible init code

* Prepare sqlx

* Change dockerfile to make the full windmill image

* Remove old dockerfile

* Improve autocomplete file resource extension select

* Remove comment

* Validate file extension

* Add icons to text file resources

* Remove editability of file resource types

* Remove unused import

* Missing space

* Add yaml parser
2024-09-19 00:21:15 +02:00
Ruben Fiszel 37d152feeb fix: postgres scripts that take longer than 20s do not timeout anymore 2024-09-18 14:32:16 +02:00
pyranota d6d4756b7a feat: Allow setProgress and getProgress from within the script (#4400)
* Allow setting progress explicitly from script body.

This feature exposes:
 * `getProgress`
 * `setProgress`
 * `incProgress`

API in TypeScript client (python is coming soon).

NOTE: Progress cannot be out of range 0..100 and cannot decrease.

With exposed APIs there is also UI changes, so progress can be shown for individual jobs as well.

For optimization reasons, jobs start to ask for progress only after N-seconds of execution.

* feat: Add `shell.nix`

If you dont have anything but nix, dont worry, run nix-shell in root, or activate with direnv and get all needed dependencies

NOTE: You will still need docker

* feat: Add `dev.nu` to typescript client

Little helper function, allowing developer to work on ts client easier.

To use:
  `./dev.nu watch`

Now add import of windmill in body of your script and `//nobundle` on top of the file

Edit ts client in your favourite editor and hit save. Script will do the rest.

* Cleanup files

* Fix: Failed to deserialize query string: missing field `get_progress`

* perf: Implement non-naive polling mechanism for getting job progress

* Add independant delay for getProgress

Problem in `TestJobLoader`:

There should be 2 delays:

    One until we find our first progress (every 5s)
    Once we found our first progress, we can do it every second

* nit: Use `query_scalar!` instead of `query_as`

* Fix: Sql error, no rows returned by a query that expected to return at least one row

* refactor: Remove global CSS for JobProgressBar

* Change UI for progress of flow subjobs

* Replace `Step 1` with `Running` in ProgressBar for individual jobs

* Remove `incProgress`

incProgress is not very usefull and error-prone

* perf: Set metric only for jobs that are actually using it

(https://github.com/windmill-labs/windmill/pull/4373#discussion_r1759843773)

* Offload registering progress from clients to server

* Add `jobId?` argument to typescript-client's `setProgress` and `getProgress`

Allows to set progress of other jobs and flows,
if jobId specified, than flow id will be inferred automatically.

Could be used by SDK.

* Add `Error::MetricNotFound` for better error handling

* Fix: Make `JobProgressBar` display in red when failed

* Add persistant progress bar

Now you can reload the page after job is done and progress will be still there

* Allow succeeded individual job's progress bar stick to 100%

* Add python support 

* nit: Remove usage of undefined variable in python-client

* Add `async` in ts client (for error handling)

* nit(frontend): Remove unused import

* Dont load JobProgressBar when it is not needed

* nit: npm check fix

* cargo sqlx prepare

* fix sqlx

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-18 01:16:19 +02:00
Ruben Fiszel f7454e621c fix lock-write for deno 2024-09-18 00:57:53 +02:00
Ruben Fiszel 277d085dbe fix(deno): replace lock-write with frozen=false 2024-09-17 08:33:33 +02:00
Ruben Fiszel d57b139df2 fix: cannot create duplicate apps and raw_apps 2024-09-16 21:15:29 +02:00
HugoCasa de78f6c192 feat: failed key renewal alert + renew on start if no recent renewal (#4387)
* feat: failed key renewal alert + renew on start if no recent renewal

* Update ee-repo-ref.txt
2024-09-16 16:54:35 +02:00
Ruben Fiszel 396258f637 fix: improve first time setup experience 2024-09-15 21:02:36 +02:00
Ruben Fiszel 4dc9ca7f4e fix: timeout in pg executor on postgresql connection after 20s 2024-09-12 18:49:43 +02:00
Ruben Fiszel 0e64380473 fix: add tag_override for script in flows 2024-09-11 23:02:45 +02:00