Commit Graph
27 Commits
Author SHA1 Message Date
wendrul 71c54024d3 feat: unsafe parameters for sql queries (table names, column names) (#5488)
* Make schema validation struct

Schema Validation rules that are constructed from the schema or from the
MainArgSig(TODO).

* Make other validator builder

* Fail dependency job like with lockfile failing for schema validator

* Add last types + tests

* Remove unused dependency

* fix typos

* Migration ID was colliding with another, changed it manually

* Add Oneof + other fixes

* fix: cache for querying scripts correclty handles ScriptMetadata

* Add cache for schema validation from main arg sig

* Prepare sqlx

* Remove default features

* Feature flags

* WIP: unsafe sql params for sql langauges

* Fix down migration table name

* cleanup: put validation logic inside a function

* Refactor to cache the should_validate boolean

Changed the schemavalidators cache to take in an
Option<SchemaValidator>, effectively storing the `should_validate_schema` information.

Also pass the schema when avaialble to construct the schema validator

* Add other job kinds to u8 cache key just in case

* Change sql languages to all get arguments as Values instead of RawValue

* Only cache if not preview

* Add last sql languages and some CI fixes

* Rename after typo on `sanitized`

* Finish rename

* Remove unused import

* Fix wrong test

* Add newly published regex parser version

* Remove default features from cargo.toml

* Change to a cleaner syntax for the interpolated args

* Update republished parser
2025-03-20 20:02:25 +01:00
Ruben Fiszel aaa4cde349 feat: filter by worker + backend perf opt (#5489)
* all

* all

* all

* refactor

* sqlx

* ref

* impr

* fix

* fix

* fix

* ee private
2025-03-19 09:55:53 +01:00
Ruben Fiszel dd3fb0f955 improve error messages for internal err 2025-02-05 16:32:01 +01:00
Ruben Fiszel 628ae617f7 feat: docker runtime 2024-12-09 18:02:18 +01:00
pyranotaandRuben Fiszel c3ba832444 Annotations refactor (#4512)
* Refactor annotation system

* Move tests to different location

* Limit annotations per line (15)

* Write initial benchmark

* Optimize

* More optimizations

* Rewrite with 3x speed!

* Remove benchmarks

They were in wrong place and for development

* Polishing

* Remove unused leftover

* Integrate new annotation system

* Add comment explanation

* Remove `hashe-based matching`

Turned to be it is impure and non-deterministic method

* Add `tracing::error` for `// native` without `deno_core`

If you build backend without `deno_core` feature flag, and add //native
annotation to ts script, it would just silently exit.

This commit prints error to logs

* add more tests

* Update annotations.rs

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-17 09:14:31 +02:00
Ruben Fiszel bd6ddc166b feat(cli): add queues, workers and worker-groups commands (#4439)
* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all
2024-09-26 11:27:37 +02:00
HugoCasa 97cdafac86 feat: add return_last_result annotation to sql (#4443) 2024-09-25 18:11:11 +02:00
HugoCasa 2b9b4eedc3 feat: multi sql statement with pg fix (#4134)
* Revert "Revert "feat: multi statement sql (#4104)" (#4133)"

This reverts commit c578f05e2d.

* fix: sort pg args
2024-07-26 09:23:39 +02:00
Ruben Fiszel c578f05e2d Revert "feat: multi statement sql (#4104)" (#4133)
This reverts commit 5bc0e96171.
2024-07-26 01:24:03 +02:00
HugoCasa 5bc0e96171 feat: multi statement sql (#4104)
* feat: multi statement pg

* fix: add other flavors

* feat: make pg params start at 1 and sequential

* fix: improve sql statement parsing

* add tests

* fix: allow no semi in last statement

* fix: merge conflict

* fix: minor improvement

* fix: parser version
2024-07-25 20:46:04 +02:00
HugoCasa 93f58056a0 feat: replace ephemeral tokens by jwt (#3908)
* feat: replace ephemeral tokens by jwt

* fix: migration + sqlx

* fix: tests, agent, ee ref

* fix: sqlx

* fix: tests

* fix: jwt prefix change + fix tests

* fix: nit

* fix: handle agent mode
2024-06-14 13:07:35 +02:00
Ruben Fiszel 6a332bc30f fix: make all error strings more verbose 2024-05-25 22:15:15 +02:00
HugoCasa c23fdc1585 fix: add mysql decimal support (#3557) 2024-04-15 14:26:17 +02:00
HugoCasaandRuben Fiszel c2d18a6d6f feat keep sql columns ordering (#3444)
* feat: keep pg columns ordering

* feat: pass column order through flow status

* feat: sql ordering for all sql langs

* fix: sqlx build on mac + rename to _metadata

* Update JobPreview.svelte

* Update +page.svelte

* Update +page.svelte

* Update LogPanel.svelte

* Update +page.svelte

* Update LogPanel.svelte

* Update JobPreview.svelte

* fix: build

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-03-22 10:34:32 +01:00
HugoCasa 1ee45a1f7d feat: allow to pin database in sql scripts (#3304) 2024-02-27 19:59:52 +01:00
HugoCasa bc9e2381b8 fix: add bigint support to mysql (#3294) 2024-02-26 16:59:36 +01:00
Ruben Fiszel d414e9fff9 refactor future wrapper for queries 2024-02-19 22:11:45 +01:00
Ruben Fiszel 487fa7f0c9 fix(mysql): support integer with float type 2024-02-16 18:33:21 +01:00
HugoCasa 825448e1f1 feat: add mysql datetime (#2808)
* feat: mysql datetime support

* feat: nits
2023-12-08 00:14:08 +01:00
HugoCasa 6e138528dd feat: mysql named params (#2805) 2023-12-07 18:55:14 +01:00
Ruben Fiszelandgbouv 2ee57f3f84 feat: refactor entirely json processing in favor or rawjson to handle larger payloads (#2446)
* raw json

* raw json

* raw json

* raw json

* the big rawjson rewrite

* resolve merge conflicts

* all

* progress

* progress

* all compile

* remove mut

* remove mut

* remove mut

* arc queuedjob

* finish

* finish

* finish

* fixes

* progress

* fix result

* small fixes

* small fixes

* fix tests

* fix tests

* fix tests

* optimize eval timeout

* optimize eval timeout

* optimize eval timeout

* handle big args and result in get jobs

* improve args

* fix

* fix

* fix tests

* fix tests

* done

* add bigquery to native picker

* add bigquery

---------

Co-authored-by: gbouv <guillaume@windmill.dev>
2023-10-16 10:46:55 +02:00
HugoCasa 8319f5d0e5 fix: add reserved variables in args (#2371)
* fix: add reserved variables in args

* fix: only get job when necessary
2023-10-03 18:14:37 +02:00
Ruben Fiszel 7982b75613 feat: submit result in background thread (unify architecture for dedicated worker) (#2226)
* merge

* all

* all
2023-09-04 01:56:23 +02:00
Ruben Fiszel c2d766028e refactor worker-common 2023-08-20 16:44:22 +02:00
Ruben Fiszel a4464b74ad refactor native job execution 2023-08-02 19:42:29 +02:00
Ruben Fiszel 55661faf60 trim docker image 2023-07-12 02:26:44 +02:00
Ruben Fiszel b7c7d564ca feat: add mysql as native integration (#1859) 2023-07-11 22:53:02 +02:00