Commit Graph

106 Commits

Author SHA1 Message Date
Ruben Fiszel 7c0950d9da fix: fix uv path 2024-10-03 22:42:34 +02:00
pyranota 1e7a37ac8b feat: Replace pip-compile with uv (#4460)
* Update `shell.nix`

- Replace pip-compile with uv packages
- Pin rust version
- Add var to trigger windmill print more info in stdout

* Replace `pip-compile` with `uv` (dirty + untested)

* Fix arguments passed to uv

Some of the flags are included by default in UV and can be safely removed:
- --resolver=backtracking
- --no-emit-index-url

Also uv does not support `--pip-args` and suggests to directly pass args to uv.

* Remove extra `dbg!`

* Replace 'pip-compile' with 'uv' in Dockerfile

* Add fallback option to `pip-compile` (Disabled)

* Add `uv` to `docker/DockerfileSlim*`

* Add `get_annotation_python` and rename `get_annotation` to `get_annotation_ts`

* Add option to fallback to pip-compile

Put `# no_uv` on top the file for specific python script

Or set `USE_PIP_COMPILE` variable to `true`

* Put back `pip-tools` into shell.nix

* Make sure lockfile resolves again if `#no_uv` used

Add #no_uv to the end of requirements (requirements.in)
That way if something breaks for customer, then they put #no_uv and new lockfile will be resolved

* Put `pip install pip-tools` in original spot

* Fix compilation error

* Fix EE compilation error

error[E0658]: attributes on expressions are experimental
   --> windmill-worker/src/python_executor.rs:144:5
    |
144 |     #[cfg(feature = "enterprise")]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: see issue #15701 <https://github.com/rust-lang/rust/issues/15701> for more information

* Add `no_cache` annotation

Will force recalculation of lockfile
And block uv from using cached values

* Target uv cache to /tmp/windmill/cache

* Prohibit uv from managing python

* Add uv to DockerfileBackendTests

* Pin uv version to 0.4.18 in Dockerfiles

* Dont put `#no_uv` in requirements.in

Instead postfix hash for requirements.in with `-no_uv`

* Push Warning to logs if fallbacked to pip-compile

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 16:00:35 +02:00
Alexander Petric 6eb7f8ce81 feat(worker): support workers to run natively on windows (#4446)
* minimal code change to get windmill worker on windows for bun and python + rustfmt

* adding support for powershell

* compiling error on unix

* rust linting comments

* comments hugo: PSModulePath

* comments ruben, refactor to simplify

* adding build workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* skip migration env, ee fixes

* improvements powershell

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* killing process tree in windows

* sqlx_offline

* install openssl for github windows runner

* used pre-installed openssl

* used pre-installed openssl

* build ee

* build ee

* build ee

* build ee

* adding commented out steps for artifact publishing

* build on tag matchinv v* pattern

* ren instead of mv on Windows

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix merging issue

* gate imports for windows

* fixing default cargo home path...

* fixing default cargo home path...

* comments ruben

* make pwsh default modules loading more robust on unix (#4448)

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 15:59:12 +02:00
Ruben Fiszel bd6ddc166b feat(cli): add queues, workers and worker-groups commands (#4439)
* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all
2024-09-26 11:27:37 +02:00
HugoCasa ae8f29a4cb feat: http routing (#4339)
* feat: http routing

* all

* feat: improve UI

* final stuff

* fix: sqlx

* fix: nit

* fix: nits

* fix: error handler display

* fix: routes panel perms

* all

* fix: improve ability to paste from macos in vscode extension

* fix lock-write for deno

* all

* cleaning

* fix

* cli preprocessor

* nits

* nits

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-20 15:38:38 +02:00
Ruben Fiszel e1223a6f24 fix: cache hub scripts in more cases + pre-cache hub scripts deps in deno 2024-08-27 02:30:54 +02:00
Ruben Fiszel 2b3e86e341 feat: add vim support for monaco/webeditor 2024-08-26 22:31:23 +02:00
Ruben Fiszel 211e80b69e fix: make result_json path stable for python executors 2024-07-23 17:39:40 +02:00
Ruben Fiszel be25101377 fix: add WM_SCHEDULED_FOR to contextual variables and early stop of flows 2024-07-18 02:10:15 +02:00
Ruben Fiszel 2812792326 fix: memory optimization for flow with big args (#4019) 2024-07-04 10:08:42 +02:00
Ruben Fiszel 0ad858ee5c fix: improve support of relative paths in subflows and apps 2024-06-21 18:07:58 +02:00
Ruben Fiszel c8915063c4 feat: track dependency map for bun + inline script of flows 2024-06-08 16:43:48 +02:00
Ruben Fiszel 157e4d8e53 improve bun dedicated worker to use nodejs 2024-05-01 02:10:58 +02:00
Ruben Fiszel b5db06d742 add base_internal_url to dedicated python workers 2024-04-30 22:48:40 +02:00
Ruben Fiszel 1cac7bf239 more logs for dedicated workers 2024-04-30 17:19:34 +02:00
Ruben Fiszel 2cb1595cae fix(python): supports folders starting with numbers for execution 2024-04-30 11:23:48 +02:00
wendrul c3325c2cbf Instrument logs (#3566)
* Add spans to show worker_name and job_id

Always show worker_name and job_id when relevant, by creating spans that
live as long as the worker and the job.

* Remove redundant worker names from fn run_worker

Remove the now redundant worker_name tracing, as the information now
lives at the span level

* Use span when spawning new thread

* Instrument more pertinent functions

Move spans to more pertinent functions. Remove more redundant info, and
change worker_name to just worker

* Change tracing subscriber to log path:line_no

Change from logging target (module name usually) to file path and line
number
2024-04-18 01:11:14 +02:00
Ruben Fiszel d63f5ee26e fix: add support for azure storage on distributed cache/logs 2024-03-23 18:42:38 +01:00
HugoCasa 6b719fecea feat: add workspace environment vars (custom contextual vars) (#3455)
* feat: custom contextual variables

* chore: sqlx prepare

* fix: main merge
2024-03-23 11:40:55 +01:00
Ruben Fiszel 7714221f3e feat: large log disk and distributed storage compaction 2024-03-23 10:55:29 +01:00
Ruben Fiszel f8d3ba12af generalize cache object store 2024-03-21 08:50:10 +01:00
Ruben Fiszel d616735174 fix compile on non ee 2024-03-20 15:34:25 +01:00
Ruben Fiszel 7dafa3d62a fix compile on non ee 2024-03-20 15:09:28 +01:00
Ruben Fiszel a0a42067fb feat: s3 cache config added to instance settings + parallelized tar pulling 2024-03-20 14:19:34 +01:00
Ruben Fiszel ab18058405 fix: allow multiple PIP_SECRETS 2024-03-17 15:15:36 +01:00
Ruben Fiszel 958df4f62c feat: refactor logging to improve performance by order of magnitude for longer jobs 2024-03-16 17:55:34 +01:00
Ruben Fiszel 543d2a1577 fix: s3 cache is an instance settings (#3421)
* s3

* s3
2024-03-15 15:09:05 +01:00
Ruben Fiszel e5aec9c445 fix: improve pip_secret interpolation 2024-03-15 11:17:41 +01:00
Ruben Fiszel 14154d2e79 feat: workflow as code for typescript 2024-03-05 23:14:04 +01:00
Ruben Fiszel 426c2a87b5 feat: workflow as code with timelines 2024-03-05 12:45:24 +01:00
Ruben Fiszel ff3aa8ab06 feat: workflow as code v0 2024-03-04 20:19:34 +01:00
Ruben Fiszel d951d17b78 fix: add HOME to pip install 2024-02-26 15:27:56 +01:00
Ruben Fiszel 2ee069a1b8 fix(python): handle recursive python imports with loop 2024-02-20 19:42:11 +01:00
Ruben Fiszel 879a6cda00 chore: remove warnings 2024-02-18 14:03:00 +01:00
Ruben Fiszel ccbb1d9305 add support for pip index url 2024-02-17 15:58:20 +01:00
Ruben Fiszel 81bdd81817 fix compile 2024-02-10 00:40:53 +01:00
Ryan Doyle 5d792d3d6f fix: Properly handle pip index urls in pip-compile (#3192)
- URLs that contain simple auth seem to have that stripped before being
  passed to underlying pip if it needs to be called by pip-compile.
- This forces explicit passing of the complete, authenticated url
  string to pip, through pip-compile, via pip-compile's "--pip-args"
  argument.
2024-02-10 00:29:08 +01:00
Ruben Fiszel 5408eafdc1 add tracing debug to pip compile 2024-02-09 22:38:52 +01:00
Ruben Fiszel ec67ebf5cd add tracing debug to pip install command args 2024-02-09 22:35:59 +01:00
Ruben Fiszel d59d324d72 fix: handle better \u0000 in python result 2024-02-08 16:02:38 +01:00
Ruben Fiszel 1c2952a1e3 fix: add support for ephemeral tokens 2024-02-03 12:09:39 +01:00
Audrius Butkevicius 91f2b35c73 feat: Passing HOME env var through to python workers (#3092)
* feat: Allow passing through certain environment variables

* Lookup the env vars on startup

* Faceplam

* Update python_executor.rs
2024-01-27 15:49:29 +01:00
Ruben Fiszel 40f3bdcf48 fix(scim): improve get_user for scim 2024-01-24 20:23:42 +01:00
Ruben Fiszel 018cd79979 fix: no-emit-index-url on lockfiles 2024-01-24 12:47:19 +01:00
Ruben Fiszel 801eef301b fix build 2024-01-23 20:15:32 +01:00
Ruben Fiszel 4b4effe767 remove denogobuncache 2024-01-23 19:54:07 +01:00
Ruben Fiszel a24645ee00 feat: OIDC support (#3017)
* oidc

* update

* feat: add support for multiselect in python

* d

* d

* progress

* all

* all

* all
2024-01-16 23:32:26 +01:00
Ruben Fiszel eefce01f82 add step_id to errors in flows 2024-01-13 13:46:42 +01:00
Ruben Fiszel a94755072e replace tracing log 2024-01-10 13:54:58 +01:00
Ruben Fiszel ca4aac5eaa feat: add ability to use secrets in pip requirements 2024-01-10 13:12:20 +01:00