Commit Graph

1030 Commits

Author SHA1 Message Date
pyranota 7398c1c09c Display memory peak while installing wheels with uv (#4889)
* fix no_uv not affecting deploy

Before this fix no_uv, no_uv_compile and no_uv_install were not affecting Dependency jobs

These jobs are only affected if used USE_PIP_COMPILE or USE_PIP_INSTALL env variables

To make it more consistant, no_uv should also affect dep jobs.

Also make ansible use uv by default

* Make it build

* Add no_uv_usage stats

* Provide build-env for Samael through shell.nix

* Run update_sqlx.sh

* Update ee-repo-ref

* Display memory peak while installing wheels with uv

* Invert DISABLE_NSJAIL

* Make it safer

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-12-12 11:26:02 +01:00
HugoCasa a6db5cddf8 add php and mysql feature flags (#4905) 2024-12-12 01:16:25 +01:00
pyranota 3618e26326 Add no_uv_usage stats (#4883)
* fix no_uv not affecting deploy

Before this fix no_uv, no_uv_compile and no_uv_install were not affecting Dependency jobs

These jobs are only affected if used USE_PIP_COMPILE or USE_PIP_INSTALL env variables

To make it more consistant, no_uv should also affect dep jobs.

Also make ansible use uv by default

* Make it build

* Add no_uv_usage stats

* Provide build-env for Samael through shell.nix

* Run update_sqlx.sh

* Update ee-repo-ref

* Update sqlx cache

* Update ee-repo-ref

* Update ee repo ref

* Update sqlx cache

* Update ee-repo-ref.txt
2024-12-12 00:32:03 +01:00
Ruben Fiszel a75c576a65 clear env for uv pip-compile 2024-12-11 17:06:05 +01:00
Lucas Abel 1b922f3ba9 fix s3_storage decl not being the same as in ee (#4901) 2024-12-11 14:39:45 +01:00
Lucas Abel af5cca1b00 feat(cache): re-work job results cache (#4898) 2024-12-11 12:01:48 +01:00
Lucas Abel e77ae1c25b fmt: re-format Rust leftovers (#4897) 2024-12-11 08:57:00 +01:00
Ruben Fiszel bcd2cfe674 fix(frontend): fix newly deployed apps with frontend scripts 2024-12-10 19:41:11 +01:00
Ruben Fiszel 5695767c2e handle empty requirement line in python 2024-12-10 16:40:12 +01:00
Ruben Fiszel c173e46a72 fix(python): do not follow symlinks when copying recursively in overlapping sites-package 2024-12-10 15:52:47 +01:00
Ruben Fiszel 1639bb2b35 improve debug logs 2024-12-10 12:35:56 +01:00
Ruben Fiszel 69195b71f3 add home in uv pip compile env 2024-12-09 20:53:43 +01:00
Ruben Fiszel b0f4caf376 backend tests 2024-12-09 18:41:13 +01:00
Ruben Fiszel 2c5d07f3d9 feat: docker runtime 2024-12-09 18:02:18 +01:00
Ruben Fiszel 81f35e7d21 update tokio and deno_core (#4871)
* all

* update
2024-12-08 16:10:46 +01:00
Ruben Fiszel 465ec09f96 improve otel 2024-12-08 11:06:35 +01:00
Ruben Fiszel fb510f33fc fix tracing 2024-12-07 17:17:25 +01:00
Ruben Fiszel 6d047449e2 feat: add otlp support (#4869)
* all

* all

* add otel

* add otel

* update docker-image

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update
2024-12-07 16:29:20 +01:00
Lucas Abel 691ef64688 feat(frontend): render new job kinds (#4864) 2024-12-06 15:11:57 +01:00
Lucas Abel 2bc4934c4f feat: add db storage for app inline scripts (#4837) 2024-12-06 13:37:45 +01:00
Lucas Abel f175158b9f fix: handle flow & workspace renames for flow_node (#4861) 2024-12-06 09:40:32 +01:00
Ruben Fiszel e5efefc7e8 improve retries max value handling 2024-12-06 01:03:46 +01:00
Lucas Abel 667167a022 fix: fix flow_node uniqueness (#4850)
`jsonb` comparison wasn't working as expected, and duplicated entries
were inserted within `flow_node`. To resolve this add a second hash
column, `hash_v2` with a unique default for uniqueness, and use this new
column to ensure unique entries. The previous hash column is left for
backward compatibility. Duplicated entries already insterted will remain
as is without breaking, and only new ones will preserve uniqueness.
2024-12-05 17:42:33 +01:00
Lucas Abel 96d4af0254 nit: format worker_lockfiles.rs (#4849) 2024-12-05 15:17:41 +01:00
Ruben Fiszel b19faa45cd fix compile 2024-12-05 15:17:24 +01:00
Ruben Fiszel 2c934cc3b8 improve windows worker 2024-12-05 14:45:23 +01:00
Ruben Fiszel 5404ec9b48 fix: pass USERPROFILE on windows 2024-12-05 10:12:44 +01:00
Ruben Fiszel db62a2c53c simplify warn_after_seconds 2024-12-04 18:28:28 +01:00
Ruben Fiszel bb9dd7d368 simplify warn_after_seconds 2024-12-04 18:24:31 +01:00
Ruben Fiszel b534ef6074 improve transparency for slow queries 2024-12-04 17:30:58 +01:00
Ruben Fiszel 71fc31c471 improve draft only delete security 2024-12-03 23:13:35 +01:00
Ruben Fiszel aa1c3702d7 fix empty branchall subflow 2024-11-29 19:02:47 +01:00
Ruben Fiszel 45d434c875 prettify cache msg 2024-11-29 13:01:54 +01:00
Ruben Fiszel b1603690f3 nit pip install 2024-11-29 12:37:52 +01:00
Ruben Fiszel d36e2a2647 add total time 2024-11-29 12:35:10 +01:00
Ruben Fiszel 9db86306a9 small uv s3 pull fix 2024-11-29 12:31:50 +01:00
Ruben Fiszel 0a95b6f9d2 fix: fix windows build 2024-11-29 09:53:05 +01:00
Lucas Abel 3fbb2bfc8a feat(cache): implement flow node caching (#4808)
* feat(cache): implement flow node caching

* feat(cache): implement script caching

* feat(cache): improve cache
2024-11-29 09:35:00 +01:00
Ruben Fiszel 8954b0b68c skip s3 python cache on windows and pro 2024-11-29 08:35:08 +01:00
pyranota 96371bc89d feat: Parallelize uv install (#4774)
* Implement MVP of Parallel uv installation

* Implement PY_CONCURRENT_DOWNLOADS

* Remove Flock for uv installs

* Make S3 pull/push parallel

* Refactor and allow to Cancel installation

* Dont print S3 in output if disabled

* Implement better error handling

* Polishing

* More polishing

* Implement error-handler for kill_tx_2.send()

* Fix and Format prev merge

* Presubscribe to all kill_tx's

We do it now before first event could fire
Meaning no events can be lost anymore

* Early print errors and safer error handling

* Return Err if installation failed

* Final changes

* Return error instead of just printing it

* Safer the way to acquire permit

* Fix compilation error

* Remove double error logs
2024-11-29 08:21:51 +01:00
Lucas Abel 390ee3113b fix: invalid null comparison while inserting flow node (#4815) 2024-11-28 17:29:16 +01:00
Lucas Abel 77937a494a fix: invalid jsonb comparison while inserting flow node (#4814) 2024-11-28 14:43:54 +01:00
pyranota a853c8d6a5 Remove flock (#4809) 2024-11-28 12:42:25 +01:00
Ruben Fiszel c998102cf3 simplify logic to check for license key on workers 2024-11-27 18:51:06 +01:00
Lucas Abel 2911775d0a feat(backend): store flow inner modules into flow_node table (#4778) 2024-11-27 16:08:31 +01:00
pyranota 34609a7d8c Fix tar prefix (#4781)
* Fix tar prefix

* Use TAR_PIP(PYTHON_311)_CACHE_DIR instead

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-11-27 15:54:47 +01:00
Lucas Abel d86ea30221 enable is_simple optimization for FlowScript (#4799)
* backend: remove some allocations in `worker_flow.rs`

* feat(backend): enable `is_simple` optimization for `FlowScript`

* fix: fix node retrieval for `flowscript` inner step
2024-11-27 10:06:19 +01:00
Lucas Abel 150ca33f0d fix: disable flow_version_lite in dedicated workers (#4800) 2024-11-27 10:02:44 +01:00
pyranota e876ae4f42 fix(python): Merge to site-packages packages with same name (#4793)
* (python) Initially implement merge

Iterate overall all python paths and if same folder has same name multiple times,
then merge the content and put to <job_dir>/site-packages

Solves problem with imports for some dependencies.

Default layout (/windmill/cache/):

dep==x.y.z
└── X
   └── A
dep-ext==x.y.z
└── X
   └── B

In this case python would be confused with finding B module.

This function will convert it to (/<job_id>):

site-packages
└── X
   ├── A
   └── B

This way python has no problems with finding correct module

* Better error-handling

* Optimize and fix bug

* Update backend/windmill-worker/src/python_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/python_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Polish

* Add path to PathBuf

* Add & before PathBuf

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2024-11-26 18:24:26 +01:00
Lucas Abel e4784e89da feat(backend): implement flow scripts (#4748)
* feat(backend): implement flow scripts

* conditionally resolve reduced flow value

* conditionally use flow scripts based on the min worker version

* grant all to `flow_{version_lite,node}`

* do not silently discard errors

* env: add flow script feature flag

* flow script on cloud automatically

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-11-25 23:20:51 +01:00