70 Commits

Author SHA1 Message Date
pyranota 59740c0478 feat: add java support (#5458)
* feat: add nu (nushell) support

* add worker tests

* deactivate tables and non-any types below top-level

full support will come in V1
for V0 it's better to keep things minimal and simple

* add syntax highlighting

used python's grammar, since nushell isn't supported by monaco nor svelte-highlights

for V1 nu will get it`s own grammar

* add logo

* partially implement plugin support

* change logo + ability to deploy + nsjail draft

* static variables + get_resource + get_variable

* lsp/dev.nu + initial nu lsp (not working yet)

* make it work with nsjail

* nullguard

* Much more flexible signature parsing and better error-messages

* add init script

* rename nulsp to nu

* install nu to dockerfile

* fix merge

* implement Default for MainArgSignature

* stage NU_CACHE_DIR

* improve dockerfiles

* dev.nu for parser-wasm + flake.nix

* update code for windows

* add nushell to flake

* upload Cargo.lock

* make build.sh work on nixos

* build wasm cli parsers

* add docs to README_DEV.md

* add helper script docker/dev.nu

* improve docker/dev.nu

* fix windows

* commit frontend/package(lock).json

* update cargo.lock

* correctly update cargo.lock

* remove lsp

* update flake.nix to include svelte server and nushell

* Revert base.sql to main

* remove PLUGIN_USE_RE

* make CARGO_PATH private

* add nu to cli

* Change flags to build wasm-nu-parser

* remove flake.nix from parser-wasm

* update wasm-build target

* remove unused import

* add cli support for nu

* update github workflows

* wasm-build 0.17 -> 0.19

* update build script

* update cargo.lock

* Fix typographical error

* start working on java

* do java boilerplate

* implement parser for java

* update Cargo.lock

* update ENV_SETTINGS

* use published nu parser

* update package.lock

* java is S3 + Caching enabled

* install nsjail backup

* commit v0

* fix nsjail

* v0.1

* rewrite parser in tree-sitter

* implement parser from scratch

* polishing

* change init script to match new parser

* fix imports

* fix cli build

* fix cli build

* refactor install phase

* implement .valid.windmill atomic verification

* implement java init functionality

* remove quick-xml

* fix windows not recognizing 'mvn'

* create empty settings.xml if there is no config provided

* clean up

* change default settings.xml

* change classpath format for windows

* docs to helper

* java copy bin cache instead of symlink

* remove comments

* merge

* fix package.json

* fix package.json 2

* minor fixing

* migrate to Coursier

* update misc

* Http(s) Proxy + CA certs

* remove unused .wasm

* make requirements insensitive to spaces

* update handle_child refs

* rework save_cache for directories

* fix s3 bug

* compile .wasm for cli

* remove uuid import

* fix compilation

* use reference

* fix zero-dep failure

* removing unsafe stuff

* remove unneeded imports

* revert: we still need winapi

* remove nix store from nsjail

* do not create cache_nomount

* add java to dnt

* remove duplicated dependency in init script

* fix typos

* fix CI

* use published parser
2025-03-27 15:12:55 +01:00
pyranota a3faea16e7 feat: add nu-lang support (#5217)
* feat: add nu (nushell) support

* add worker tests

* deactivate tables and non-any types below top-level

full support will come in V1
for V0 it's better to keep things minimal and simple

* add syntax highlighting

used python's grammar, since nushell isn't supported by monaco nor svelte-highlights

for V1 nu will get it`s own grammar

* add logo

* partially implement plugin support

* change logo + ability to deploy + nsjail draft

* static variables + get_resource + get_variable

* lsp/dev.nu + initial nu lsp (not working yet)

* make it work with nsjail

* nullguard

* Much more flexible signature parsing and better error-messages

* add init script

* rename nulsp to nu

* install nu to dockerfile

* fix merge

* implement Default for MainArgSignature

* stage NU_CACHE_DIR

* improve dockerfiles

* dev.nu for parser-wasm + flake.nix

* update code for windows

* add nushell to flake

* upload Cargo.lock

* make build.sh work on nixos

* build wasm cli parsers

* add docs to README_DEV.md

* add helper script docker/dev.nu

* improve docker/dev.nu

* fix windows

* commit frontend/package(lock).json

* update cargo.lock

* correctly update cargo.lock

* remove lsp

* update flake.nix to include svelte server and nushell

* Revert base.sql to main

* remove PLUGIN_USE_RE

* make CARGO_PATH private

* add nu to cli

* Change flags to build wasm-nu-parser

* remove flake.nix from parser-wasm

* update wasm-build target

* remove unused import

* add cli support for nu

* update github workflows

* wasm-build 0.17 -> 0.19

* update build script

* update cargo.lock

* Fix typographical error

* update Cargo.lock

* update ENV_SETTINGS

* use published nu parser

* update package.lock

* rewrite parser in tree-sitter

* implement parser from scratch

* polishing

* change init script to match new parser

* fix imports

* fix cli build

* fix cli build

* merge

* update wasm

* use MiniPulledJob

* update cli

* change cli wasm schema

* change cli

* update deno.json

* make wasm modules load lazily

* regenerate parsers

* remove leftover

* update cargo.lock

* clean up dnt.ts

* add docs to cli/test.nu

* add schema validation option

* add Nu to try_validate_schema

* reference frontend to new parser version
2025-03-20 16:45:21 +01:00
Ruben Fiszel 6b31a6dee5 fix ansible for nsjail 2025-02-21 17:06:15 +01:00
Ruben Fiszel 3188bee46e fix: disable toggle is more consistently applied on arg inputs 2025-02-21 14:07:44 +01:00
pyranota 4ad654fcf0 feat: remove pip fallback option for python and ansible (#5186) 2025-02-19 17:33:11 +01:00
Ruben Fiszel d4f61f13fd fix(bash): allow process substitution on nsjail 2025-02-18 16:46:42 +01:00
wendrul 6357ed3d5e fix: Remove cache dir mount and mount only the cache executable (Rust, C#) (#5270) 2025-02-11 16:08:19 +01:00
HugoCasa 3bb19dd87c fix: remove unnecessary rw on cache for powershell in nsjail (#5236) 2025-02-06 18:49:21 +01:00
Ruben Fiszel de1e1563ed remove go cache 2025-02-06 17:41:47 +01:00
Ruben Fiszel abf094bcbc fix uv install for nsjail (#5222) 2025-02-06 00:00:33 +01:00
Ruben Fiszel 6d33cd24e9 remove rw access to cache dir in php 2025-02-05 22:17:27 +01:00
Ruben Fiszel 1769d62c96 removing rw access to cache dir in uv pip install 2025-02-05 22:08:38 +01:00
Ruben Fiszel f8cc732ce0 removing rw access to cache dir in uv pip install 2025-02-05 22:06:52 +01:00
Ruben Fiszel 3c6d1029d1 remove rw access to bun cache in nsjail mode 2025-02-05 21:45:37 +01:00
pyranota c60f8dacf1 feat(python): add custom_wheels directory to PYTHONPATH (#5169)
* fix(python): fix uv can't find ssl certificates

- Add `PY_NATIVE_CERT` flag, forces UV to use native tls
- Rename `PIP_INDEX_CERT` to `PY_INDEX_CERT`
- Rename `PIP_TRUSTED_HOST` to `PY_TRUSTED_HOST`

For backwards compatibility PIP* variables are still accessible

* feat(python): add `custom_wheels` directory to PYTHONPATH

Add global directory by path `<CACHE_DIR>/python_xyz/custom_wheels`

For example for scripts running python 3.11, in every execution
`<CACHE_DIR>/python_311/custom_wheels` will be accessible and all wheels placed there could be imported and used.

This is usefull for preinstalling wheels before runtime

* Make it work with Nsjail

* Rework and make custom_wheels optional

* Remove `create_dir_all` from imports

* Use sync version of metadata

* Rename `custom_wheels` to `global-site-packages`
2025-01-29 19:13:47 +01:00
pyranota e47dd697f9 feat(python): Multiple runtime versions (#4579)
* feat: Handle `pip install` by `uv`

Dirty and untested, but already something working

* Integrate with NSJAIL and prepare fallbacks

* Refactor fallback
no_uv disable compile and install
where no_uv_install and no_uv_compile are a bit more specific

* Remove `--disable-pip-version-check`
Reason:
   warning: pip's `--disable-pip-version-check` has no effect

* Fix backend compilation error

* Pip fallback overwrite UV's cache

* Initially refactor cache (No S3)

* Support S3

* Remove unused import

* Handle flags for NSJAIL

* Return deleted flag

* Remove verbose mode and enable link-mode=copy

* Granural migration of lockfiles

Before i realized we dont need it :)

* Initial draft (not-working)

* Add fallback

* Fix bug preventing uv from installing deps

'\n' - Love it

* Add verbosity indicator

* Iterate on feature
- Added instance python version
- Rework logic

* Fix EE build error
error[E0599]: no method named `iter` found for tuple `(PyVersion, std::vec::Vec<std::string::String>)` in the current scope

* Support S3

* Support NSJAIL

* Refactor `get_python`

* Make NSJAIL work [Unsafe]

config file missed /proc mount causing install phase to fail

* Trigger CI

* Clean up

* Make Actions build it

* Trigger CI #2

* Update Dockerfile and clean up

* Change fallbacks
now there is only no_uv and NOUV

* Expose INSTANCE_PYTHON_VERSION through env variable

* Change namings

* Include py-version to requirements.in

Also add comments and make code much cleaner

* Use const for python installation dir

It was hardcoded before

* Pin preinstalled version

* Update python_executor.rs

* Up to date branch

* Create PYCACHE dirs

TODO: PY_TAR_DIRS

* Fix after merge

* Make it safer

* Implement USE_SYSTEM_PYTHON

* Implement latest_stable option

* Load INSTANCE_PYTHON_VERSION on startup

* Check for multiple annotations used

* Fix Latest Stable button not pressed if selected

* Proper error handling for conflict on multiple annotations

* Fix merge conflicts

* Preinstall 3.11 and Latest Stable

* Preinstall latest stable in non-blocking manner

* Fix Warning

* Gate preinstall logic behind "python" feature

* Handle raw_deps properly

* Make it work with nsjail

* Revert docker-image.yml

* Revert Dockerfile

* Cleanup + Fixing

* Add windows support

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-01-24 11:42:48 +01:00
pyranota fcfad69195 feat: Migrate to bun.lock (#5112)
* bun: Migrate to bun.lock

(In backwards compatible way)

Read more: https://bun.sh/blog/bun-lock-text-lockfile

* Clean up

* More clean up

* Mount bun.lock to jailed process
2025-01-22 12:32:17 +01:00
pyranota 5831822afe fix: improve uv dependency install reliability with an atomic success file
* Revert solution with Mutex

* Implement valid.windmill logic

* Remove unused import

* valid.windmill -> .valid.windmill

Just like .lock

* Dont delete wheels if cancelled/failed

Now we dont clean up requirement folder if it was failed.

This way we can fully utilize uv's flock system.

Also if we left wheel dir, but it was partially filled (resulting to invalid wheel)
we use --reinstall flag in order to overwrite any content of wheels

* Add comment

* Add --reinstall to nsjail
2025-01-20 16:21:10 +01:00
wendrul c85d2a4957 feat: add C# support (#4908)
* Add csharp stuff and merge after the rewrite

    * Add csharp, boilerplate + minimal execution
    * Add initial parser
    * Frontend + wasm export of the tree sitter parser
    * Arg spread and use cache

* More types and req parsing

* Add logo

* Parse async and void to fit the wrapper to the main sig

* Csharp editor resource and var buttons

* Lockfile generation

* Add default parsing using serde_json::from_str

* Update init code for c#

* Nsjail for c#

* Remove c_variadic feature

* Remove unused imports

* Add c_variadic only for wasm32 assembly

* Fix c# test

* Add dotnet to the images

* Update dockerfiles

* Update dockerfile again

* Touch Dockerfile

* npm install c# parser

* Add opt mount in nsjail

* Update ee repo ref

* Add csharp to backend image for tests

* Add setting to set nuget package

* Windows compatibility

* Disable csharp test

* ADd feature flag
2024-12-13 00:29:01 +01:00
pyranota f240d1322a feat: Handle pip install by uv (#4517)
* feat: Handle `pip install` by `uv`

Dirty and untested, but already something working

* Integrate with NSJAIL and prepare fallbacks

* Refactor fallback
no_uv disable compile and install
where no_uv_install and no_uv_compile are a bit more specific

* Remove `--disable-pip-version-check`
Reason:
   warning: pip's `--disable-pip-version-check` has no effect

* Fix backend compilation error

* Pip fallback overwrite UV's cache

* Initially refactor cache (No S3)

* Support S3

* Remove unused import

* Handle flags for NSJAIL

* Return deleted flag

* Update Dockerfile

* Update docker-image.yml

* Update docker-image.yml

* Add --link-mode=copy and remove -v

* Fix NSJAIL INDEX_URL

* Fix flock and windows

* Update python_executor.rs

* Remove line from Dockerfile

We dont need it and to trigger build

* fixing for windows

* Dont pin python to specific version

* Change TMP for windows

* Revert docker-image.yml

* Disable UV for ansible

Will be enabled later.
Needs proper testing and its better to split onto 2 PRs with first modifying python and second ansible

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Alexander Petric <petric.al@gmail.com>
2024-11-15 00:12:56 +01:00
Ruben Fiszel b603f4a6ba allow python multiprocessing on nsjail 2024-10-11 01:40:09 +02:00
Ruben Fiszel 827113c797 add urandom to nsjail 2024-09-29 12:31:55 +02:00
Ruben Fiszel c3546afa63 quote req and target for nsjail python download 2024-09-29 12:13:54 +02:00
HugoCasa 6fb936cdb6 fix: allow preprocessor to write to args.json on nsjail (#4455) 2024-09-27 21:12:38 +02:00
wendrul a7ef3ed293 fix: various improvements for ansible (#4419)
* Improve error messages, fix dependency job issue

* Add nsjail support for ansible

* Remove invalid comment syntax

* Change nsjail config and ansible controller config to be ephemeral

* Move ansible cfg to later to overwrite possible files named ansible.cfg

* Fix ansible.cfg env var

* Fix tmp dir issue
2024-09-23 16:44:37 +02:00
wendrul 91d328b132 feat: add rust (#4253)
* Add rust on frontend

* Add parser for rust

* Handle rust job execution

* Main has to return a result that is serializable

* Update parser: parse many rust types

* Rust parser dependencies

* Frontend wasm parser

* Add windmill parser rust to windmill parser wasm (temporarily)

* Add rust as a tag on worker__default

* Change init code for rust

* Cleanup rust_executor.rs

* Remove tree-sitter dep and fix unused imports

* Add lockfile and build logic

* Add cargo and rustup to windmill image

* Fix env var to work on docker image and dev env

* modify package.json

* Deps after parser publish

* Add stashed migration

* Unify rust versions

* Add error message when php or cargo are not present to use another image

* Error message conditionally on feature flag

* all ci/cd changes

* all ci/cd changes

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-08-30 17:01:04 +02:00
Ruben Fiszel 2f1d7349b9 fix: move bun cache to non mounted volume to benefit from cache optimization 2024-07-24 18:25:21 +02:00
Ruben Fiszel afed03565f make nsjsail mode compatible with arm64 by making lib64 non mandatory 2024-06-28 15:11:41 +02:00
HugoCasa f7f7a6d85f feat: add php (#3725)
* feat: add php

* fix: nsjail + frontend build
2024-05-15 08:12:11 +02:00
Ruben Fiszel 49469644b9 fix: bun loader with nsjail 2024-05-14 01:17:13 +02:00
Ruben Fiszel 1f64715043 feat: local typescript codebase as bundle (#3694)
* codebases

* codebases

* foo

* foo

* foo

* foo

* fix(frontend): Disable the insert button when required fields are empty strings (#3659)

* fix(frontend): use normal password mask for the sensitive fields of the resource editor

* handle super admins in password arg input

* chore(main): release 1.323.2 (#3660)

* chore(main): release 1.323.2

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>

* foo

* all

* s3_helpers move

* progress

* all

* progress

* progress

* progress

* progress

* codebase final

* update without ee

* sqlx

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* update

* all

* all

* all

---------

Co-authored-by: Faton Ramadani <faton.ramadani14@gmail.com>
Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2024-05-09 18:35:50 +02:00
Ruben Fiszel 8970009b44 fix powershell with nsjail 2024-03-26 23:11:26 +01:00
Ruben Fiszel 958df4f62c feat: refactor logging to improve performance by order of magnitude for longer jobs 2024-03-16 17:55:34 +01:00
Ruben Fiszel 1eff3fc9cc feat: experimental nodejs support 2024-01-14 18:53:43 +01:00
HugoCasa ff7834086d fix: powershell nsjail (#2856) 2023-12-14 13:34:40 +01:00
HugoCasa 297a31d468 fix: pwsh modules nsjail (#2845) 2023-12-13 14:16:08 +01:00
Ruben Fiszel d85217564e make bun cache rw on nsjail 2023-10-12 02:03:31 +02:00
Ruben Fiszel ff529e6c1a fix bun on nsjail 2023-10-11 23:13:10 +02:00
Ruben Fiszel e3d28b9507 feat: support pinned versions for bun in deployed scripts 2023-09-19 18:15:06 +02:00
Ruben Fiszel c2d766028e refactor worker-common 2023-08-20 16:44:22 +02:00
Ruben Fiszel 3024069427 improve BADF on cloud 2023-08-13 00:36:44 +02:00
Ruben Fiszel 70b7fe4989 improve BADF on cloud 2023-08-13 00:29:23 +02:00
Ruben Fiszel 7a66d03715 enable relative path under nsjail too 2023-08-12 20:54:35 +02:00
Ruben Fiszel 1778ec1117 feat: respect lockfiles for bun 2023-08-12 20:49:36 +02:00
Ruben Fiszel ea92fa1154 remove nltk_data binding 2023-08-03 11:30:36 +02:00
Ruben Fiszel 838266bb2f feat: lock depedency for the entire flow + dependency job depend on script/flow's tag 2023-07-26 21:14:14 +02:00
Ruben Fiszel 4317f065f5 feat: bun support (#1800)
* feat: bun support

* feat: bun support

* add bun support

* add bun support

* add bun support

* add bun
2023-07-06 21:19:07 +02:00
Ruben Fiszel 7b2d7b9af7 feat: increase timeout to 900 2023-05-10 19:08:18 +02:00
Ruben Fiszel 198dd9e8ff remove deno nsjail in favor of deno sandboxing 2023-04-08 15:25:10 +02:00
Ruben Fiszel 1f8513b937 remove unecessary deno lock file 2023-03-31 11:53:31 +02:00