Commit Graph

1106 Commits

Author SHA1 Message Date
wendrul 71c54024d3 feat: unsafe parameters for sql queries (table names, column names) (#5488)
* Make schema validation struct

Schema Validation rules that are constructed from the schema or from the
MainArgSig(TODO).

* Make other validator builder

* Fail dependency job like with lockfile failing for schema validator

* Add last types + tests

* Remove unused dependency

* fix typos

* Migration ID was colliding with another, changed it manually

* Add Oneof + other fixes

* fix: cache for querying scripts correclty handles ScriptMetadata

* Add cache for schema validation from main arg sig

* Prepare sqlx

* Remove default features

* Feature flags

* WIP: unsafe sql params for sql langauges

* Fix down migration table name

* cleanup: put validation logic inside a function

* Refactor to cache the should_validate boolean

Changed the schemavalidators cache to take in an
Option<SchemaValidator>, effectively storing the `should_validate_schema` information.

Also pass the schema when avaialble to construct the schema validator

* Add other job kinds to u8 cache key just in case

* Change sql languages to all get arguments as Values instead of RawValue

* Only cache if not preview

* Add last sql languages and some CI fixes

* Rename after typo on `sanitized`

* Finish rename

* Remove unused import

* Fix wrong test

* Add newly published regex parser version

* Remove default features from cargo.toml

* Change to a cleaner syntax for the interpolated args

* Update republished parser
2025-03-20 20:02:25 +01:00
pyranota f55cb420dd feat: add nu-lang support (#5217)
* feat: add nu (nushell) support

* add worker tests

* deactivate tables and non-any types below top-level

full support will come in V1
for V0 it's better to keep things minimal and simple

* add syntax highlighting

used python's grammar, since nushell isn't supported by monaco nor svelte-highlights

for V1 nu will get it`s own grammar

* add logo

* partially implement plugin support

* change logo + ability to deploy + nsjail draft

* static variables + get_resource + get_variable

* lsp/dev.nu + initial nu lsp (not working yet)

* make it work with nsjail

* nullguard

* Much more flexible signature parsing and better error-messages

* add init script

* rename nulsp to nu

* install nu to dockerfile

* fix merge

* implement Default for MainArgSignature

* stage NU_CACHE_DIR

* improve dockerfiles

* dev.nu for parser-wasm + flake.nix

* update code for windows

* add nushell to flake

* upload Cargo.lock

* make build.sh work on nixos

* build wasm cli parsers

* add docs to README_DEV.md

* add helper script docker/dev.nu

* improve docker/dev.nu

* fix windows

* commit frontend/package(lock).json

* update cargo.lock

* correctly update cargo.lock

* remove lsp

* update flake.nix to include svelte server and nushell

* Revert base.sql to main

* remove PLUGIN_USE_RE

* make CARGO_PATH private

* add nu to cli

* Change flags to build wasm-nu-parser

* remove flake.nix from parser-wasm

* update wasm-build target

* remove unused import

* add cli support for nu

* update github workflows

* wasm-build 0.17 -> 0.19

* update build script

* update cargo.lock

* Fix typographical error

* update Cargo.lock

* update ENV_SETTINGS

* use published nu parser

* update package.lock

* rewrite parser in tree-sitter

* implement parser from scratch

* polishing

* change init script to match new parser

* fix imports

* fix cli build

* fix cli build

* merge

* update wasm

* use MiniPulledJob

* update cli

* change cli wasm schema

* change cli

* update deno.json

* make wasm modules load lazily

* regenerate parsers

* remove leftover

* update cargo.lock

* clean up dnt.ts

* add docs to cli/test.nu

* add schema validation option

* add Nu to try_validate_schema

* reference frontend to new parser version
2025-03-20 16:45:21 +01:00
Ruben Fiszel debf68734b fix: improve memory metrics graph 2025-03-20 09:54:55 +01:00
wendrul 299c1430eb feat: backend arg schema validation (#5455)
* Make schema validation struct

Schema Validation rules that are constructed from the schema or from the
MainArgSig(TODO).

* Make other validator builder

* Fail dependency job like with lockfile failing for schema validator

* Add last types + tests

* Remove unused dependency

* fix typos

* Migration ID was colliding with another, changed it manually

* Add Oneof + other fixes

* fix: cache for querying scripts correclty handles ScriptMetadata

* Add cache for schema validation from main arg sig

* Prepare sqlx

* Remove default features

* Feature flags

* Fix down migration table name

* cleanup: put validation logic inside a function

* Refactor to cache the should_validate boolean

Changed the schemavalidators cache to take in an
Option<SchemaValidator>, effectively storing the `should_validate_schema` information.

Also pass the schema when avaialble to construct the schema validator

* Add other job kinds to u8 cache key just in case

* Only cache if not preview
2025-03-19 19:41:10 +01:00
Ruben Fiszel eefb77f445 nit move away from using views 2025-03-19 10:48:43 +01:00
Ruben Fiszel aaa4cde349 feat: filter by worker + backend perf opt (#5489)
* all

* all

* all

* refactor

* sqlx

* ref

* impr

* fix

* fix

* fix

* ee private
2025-03-19 09:55:53 +01:00
Ruben Fiszel 1dd9ca10fc pg nits 2025-03-14 12:13:12 +01:00
Ruben Fiszel f726b3af88 fix: avoid lock contention for native workers on cached connection (#5481) 2025-03-14 11:58:03 +01:00
Ruben Fiszel 5d2e5d5b48 fix: prevent monitoring task to die without sending killpill (#5472) 2025-03-12 11:51:53 +01:00
Ruben Fiszel 11b779ff41 display more complete error for failing to untar 2025-03-06 13:24:18 +01:00
Ruben Fiszel 4d23075cde allow ce to run codebase bundles preview 2025-03-04 22:21:57 +01:00
pyranota b705e4e415 fix(python): windows worker fails to install 3.10 (#5409)
```
execution error:\nFind python error: error: Failed to inspect Python interpreter from managed installations at `C:\\tmp\\windmill\\cache\\py_runtime\\cpython-3.10.16-windows-x86_64-none\\python.exe`\n  Caused by: Querying Python at `C:\\tmp\\windmill\\cache\\py_runtime\\cpython-3.10.16-windows-x86_64-none\\python.exe` failed with exit status exit code: 1\n\n[stderr]\nFatal Python error: _Py_HashRandomization_Init: failed to get random numbers to initialize Python\nPython runtime state: preinitialized\n"
```
2025-03-04 20:05:45 +01:00
Ruben Fiszel f9fcb711b2 local codebase in home 2025-03-04 19:30:13 +01:00
Ruben Fiszel 6504540fb7 fix codebase & bundles on EE 2025-03-04 18:53:06 +01:00
Ruben Fiszel 2cb3e5741c add standalone bundle support on CE 2025-03-04 18:09:12 +01:00
Ruben Fiszel c27ccbe15e all 2025-03-04 15:25:45 +01:00
pyranota 9c318c6fb9 nit: print more information in frontend if something goes wrong with uv (#5411)
* nit: print to frontend logs if there is fallback to system python

* print to logs if fallback
2025-03-04 00:36:11 +01:00
HugoCasa 40d4a4cf54 feat: app s3 input anonymous delete (#5401)
* feat: app s3 input anonymous delete

* tmp ee ref

* fix build
2025-02-27 16:28:56 +01:00
HugoCasa fd3459568f feat: track workspace runnables used in flows (#5369)
* feat: track workspace runnables used in flows

* track script hash

* weird

* do it with lock

* Revert "feat: add support for | None and Optional in python (#5361)"

This reverts commit 69a2d85f25.

* Revert "Revert "feat: add support for | None and Optional in python (#5361)""

This reverts commit bb8f709894.

* update openapi

* delete old in lock_modules + don't track hub scripts
2025-02-27 10:01:12 +01:00
HugoCasa d9b92a8a64 fix: disable bundling using env var (#5396) 2025-02-26 21:41:49 +01:00
pyranota ab95df4cc1 fix dedicated worker multipython (#5338)
* fix dedicated worker multipython

* fix typo

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix dedublicate logic

* fix ellipsis complaints

* more factorization

* pass slice instead of ref to vector

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-02-25 17:22:03 +01:00
Ruben Fiszel f7865229ef feat: limit the number of times a job can be restarted (3) after loss of pings 2025-02-25 16:35:29 +01:00
Ruben Fiszel 007bf252b9 fix: handle better optional chaining operator 2025-02-24 20:08:13 +01:00
Ruben Fiszel 19d23fb3f3 fix: add LOCALAPPDATA env variable to python execution on windows 2025-02-24 18:36:26 +01:00
Ruben Fiszel 3d6015e44a feat: make flow lock deployment error visible in UI 2025-02-22 15:02:16 +01:00
Ruben Fiszel e89b2feeac add PRINT_NATIVE_ERRORS env variable 2025-02-22 11:06:42 +01:00
Ruben Fiszel 5abc825e48 update deno_core implementation to latest 2.2.1 2025-02-22 11:04:54 +01:00
Ruben Fiszel 6359838d31 initialize an uv venv globally but skip it for python executor 2025-02-22 01:53:14 +01:00
Ruben Fiszel 7e58247e63 fix: init_script do not need to use nsjail even in nsjail mode 2025-02-22 01:04:33 +01:00
Ruben Fiszel 15c17c0a88 improve error message to run executable 2025-02-21 18:09:25 +01:00
Ruben Fiszel 546c9f3d96 fix ansible for nsjail 2025-02-21 17:06:15 +01:00
pyranota 8b6df77ce1 fix windows build (#5336) 2025-02-20 09:16:43 +01:00
Ruben Fiszel 50d75e358c fix: add proxy envs (http_proxy) to uv install 2025-02-20 03:10:09 +01:00
Ruben Fiszel aacd0930fe fix(python): PYTHON_PATH overrides python from uv 2025-02-20 02:54:51 +01:00
Ruben Fiszel 92965bba79 fix(cli): improve dependency job error message (logs in result) 2025-02-20 01:52:18 +01:00
pyranota ba187a0953 fix windows build (#5333) 2025-02-20 00:26:09 +01:00
pyranota c62728eaa9 feat: remove pip fallback option for python and ansible (#5186) 2025-02-19 17:33:11 +01:00
Ruben Fiszel b8032f71da feat: add ready endpoints for workers to enterprise 2025-02-18 21:49:35 +01:00
Ruben Fiszel 0462506065 back to pipe in bash for efficiency purposes 2025-02-18 20:08:25 +01:00
Ruben Fiszel 30390a1927 fix(bash): improve bash last line as result reliability using bash process substitution (#5321) 2025-02-18 15:13:28 +01:00
Ruben Fiszel a034c3ff28 fix: improve que job indices for faster performances 2025-02-17 21:26:46 +01:00
HugoCasa 88ae1dc990 feat: parse script for preprocessor/no_main_func on deploy (#5292)
* feat: parse script for preprocessor/no_main_func on deploy

* fix substitute for linux
2025-02-14 15:42:18 +01:00
Ruben Fiszel 8d8c36d27c fix: display branch chosen even if emoty branch 2025-02-14 04:23:25 +01:00
Ruben Fiszel ee839edd76 fix: improve handling of empty branches and loops 2025-02-14 04:12:55 +01:00
Ruben Fiszel 3afd2ed7e4 fix(bun): remove unecessary buntar in a bun bundle world 2025-02-13 10:27:48 +01:00
Ruben Fiszel 271ccca710 fix: better handling of null pre-processor return values 2025-02-12 22:10:53 +01:00
Ruben Fiszel a9d7116782 fix encrypted values passed from apps in flows 2025-02-12 20:07:27 +01:00
Lucas Abel ecbec1fe50 fix(backend): improve schedule queries plan to leverage indices better for performance (#5273)
* backend: fix schedule queries plan

* rework

* fix skipped as non success

* update sqlx

* update

* all

* fix npm run check

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-02-12 16:34:03 +01:00
wendrul 5022b0c596 fix: Remove cache dir mount and mount only the cache executable (Rust, C#) (#5270) 2025-02-11 16:08:19 +01:00
Lucas Abel 6092683983 backend: fix missing virtual modules when inserting flow version lite (#5257) 2025-02-10 18:55:31 +01:00