Commit Graph
832 Commits
Author SHA1 Message Date
pyranotaandRuben Fiszel d3e0b1c636 feat: Allow setProgress and getProgress from within the script (#4400)
* Allow setting progress explicitly from script body.

This feature exposes:
 * `getProgress`
 * `setProgress`
 * `incProgress`

API in TypeScript client (python is coming soon).

NOTE: Progress cannot be out of range 0..100 and cannot decrease.

With exposed APIs there is also UI changes, so progress can be shown for individual jobs as well.

For optimization reasons, jobs start to ask for progress only after N-seconds of execution.

* feat: Add `shell.nix`

If you dont have anything but nix, dont worry, run nix-shell in root, or activate with direnv and get all needed dependencies

NOTE: You will still need docker

* feat: Add `dev.nu` to typescript client

Little helper function, allowing developer to work on ts client easier.

To use:
  `./dev.nu watch`

Now add import of windmill in body of your script and `//nobundle` on top of the file

Edit ts client in your favourite editor and hit save. Script will do the rest.

* Cleanup files

* Fix: Failed to deserialize query string: missing field `get_progress`

* perf: Implement non-naive polling mechanism for getting job progress

* Add independant delay for getProgress

Problem in `TestJobLoader`:

There should be 2 delays:

    One until we find our first progress (every 5s)
    Once we found our first progress, we can do it every second

* nit: Use `query_scalar!` instead of `query_as`

* Fix: Sql error, no rows returned by a query that expected to return at least one row

* refactor: Remove global CSS for JobProgressBar

* Change UI for progress of flow subjobs

* Replace `Step 1` with `Running` in ProgressBar for individual jobs

* Remove `incProgress`

incProgress is not very usefull and error-prone

* perf: Set metric only for jobs that are actually using it

(https://github.com/windmill-labs/windmill/pull/4373#discussion_r1759843773)

* Offload registering progress from clients to server

* Add `jobId?` argument to typescript-client's `setProgress` and `getProgress`

Allows to set progress of other jobs and flows,
if jobId specified, than flow id will be inferred automatically.

Could be used by SDK.

* Add `Error::MetricNotFound` for better error handling

* Fix: Make `JobProgressBar` display in red when failed

* Add persistant progress bar

Now you can reload the page after job is done and progress will be still there

* Allow succeeded individual job's progress bar stick to 100%

* Add python support 

* nit: Remove usage of undefined variable in python-client

* Add `async` in ts client (for error handling)

* nit(frontend): Remove unused import

* Dont load JobProgressBar when it is not needed

* nit: npm check fix

* cargo sqlx prepare

* fix sqlx

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-09-18 01:16:19 +02:00
HugoCasa d3e8731be2 feat: failed key renewal alert + renew on start if no recent renewal (#4387)
* feat: failed key renewal alert + renew on start if no recent renewal

* Update ee-repo-ref.txt
2024-09-16 16:54:35 +02:00
Ruben Fiszel b5a9111d07 fix: improve runFlowAsync and run_flow_async default behavior + time formatting of scheduled for 2024-09-11 23:52:02 +02:00
Ruben Fiszel 7d29651d3e fix: graceful worker exits for same worker jobs (#4371)
* all

* fix compile
2024-09-11 18:32:08 +02:00
Ruben Fiszel 5817efc16f fix(cli): guide users to migrate to node version of the CLI 2024-09-07 21:27:36 +02:00
HugoCasa a009c8df23 feat: schedule success handler (#4346) 2024-09-06 19:03:09 +02:00
HugoCasa c487934e3b feat: add slack as a critical alert channel (#4319)
* feat: add slack as a critical alert channel

* update ee ref
2024-09-04 09:48:13 +02:00
Ruben Fiszel c93ccb6f55 improve service logs with json_fmt 2024-08-31 23:04:12 +02:00
Ruben Fiszel c62e152d27 feat: service logs (#4244) 2024-08-31 14:57:35 +02:00
HugoCasa c3ce68066c feat: min workers in worker group alert + zombie job critical alert (#4307)
* feat: min workers in worker group alert + zombie job critical alert

* updatee ee ref
2024-08-30 17:25:53 +02:00
wendrulandRuben Fiszel 91d328b132 feat: add rust (#4253)
* Add rust on frontend

* Add parser for rust

* Handle rust job execution

* Main has to return a result that is serializable

* Update parser: parse many rust types

* Rust parser dependencies

* Frontend wasm parser

* Add windmill parser rust to windmill parser wasm (temporarily)

* Add rust as a tag on worker__default

* Change init code for rust

* Cleanup rust_executor.rs

* Remove tree-sitter dep and fix unused imports

* Add lockfile and build logic

* Add cargo and rustup to windmill image

* Fix env var to work on docker image and dev env

* modify package.json

* Deps after parser publish

* Add stashed migration

* Unify rust versions

* Add error message when php or cargo are not present to use another image

* Error message conditionally on feature flag

* all ci/cd changes

* all ci/cd changes

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2024-08-30 17:01:04 +02:00
HugoCasa 304de14674 fix: items with starred info (#4298) 2024-08-28 11:45:50 +02:00
HugoCasa 9efad477da optional starred info on get item + update name of user instance (#4295) 2024-08-27 21:58:51 +02:00
Ruben Fiszel e1223a6f24 fix: cache hub scripts in more cases + pre-cache hub scripts deps in deno 2024-08-27 02:30:54 +02:00
HugoCasa 6a99d999a2 feat: improve early stop (#4257) 2024-08-21 00:47:34 +02:00
Ruben Fiszel 55b85b9979 ignore empty FORCE_WORKER_TAGS 2024-08-20 17:03:59 +02:00
Ruben Fiszel b80257720a fix: handle more gracefully worker without tags 2024-08-20 16:27:01 +02:00
Ruben Fiszel fa06d20380 fix: add FORCE_WORKER_TAGS & fix workers page when default worker group is missing 2024-08-20 12:15:23 +02:00
HugoCasa 904b5b4ea1 feat: cache common hub scripts in image (#4249)
* feat: cache common hub scripts in image

* Delete backend/src/hubPaths.json

* fix: missing file

* fix: dockerfile

* fix: cache by path

* fix

* fix

* precreate cache folder
2024-08-16 17:30:44 +02:00
Ruben Fiszel 1f7ece08c9 all 2024-08-14 14:34:58 +02:00
Ruben Fiszel 763a3062b3 add INSTANCE_IS_DEV 2024-08-13 21:44:14 +02:00
Ruben Fiszel 8b3a49e133 fix: add an option to disable bundling globally 2024-08-13 18:00:50 +02:00
Ruben Fiszel 1c79b27c37 fix compile 2024-08-12 10:37:51 +02:00
Ruben Fiszel 9d581920d5 support in list_flows, flows that have no last flow_version 2024-08-08 10:08:25 +02:00
HugoCasa 80a41669b2 feat: email triggers (#4163)
* feat: email triggers v0

* update docker compose to nginx with tcp reverse proxy + move smtp to private

* fix: open source build

* test: update ee ref for testing

* feat: use caddy with layer4

* fix: nit

* feat: configurable email domain

* fix: nit

* fix: nit

* fix: get l4 from main

* fix: default email domain to mail.domain

* update ee ref
2024-08-06 16:17:33 +02:00
Ruben Fiszel 4a295f7dd5 support tarbundle in vscode extension 2024-08-06 01:02:41 +02:00
wendrul af0e901954 feat: deployment UI filter deployable items (#4183)
* Add column to workspace settings table

* Add endpoint to change the deploy UI settings

* Make frontend page to edit UI settings

* Prepare sqlx

* Add deployment restrictions to frontend

* Fix function name

* Change dependency to minimatch, CE compatibility

* Remove default include_path

* Remove picomatch types (old dep)

* Use empty list instead of globstar as default

* All deployable if config is null
2024-08-05 13:27:13 +02:00
Ruben Fiszel 59e9a0889a feat: add support for assets using tar for codebase deploy 2024-08-04 23:02:51 +02:00
Ruben Fiszel 69e400563c feat: variables created by password fields expire after 7 days 2024-08-01 16:05:03 +02:00
Ruben Fiszel 553c95d71b improve queue indices 2024-07-31 23:44:40 +02:00
Ruben Fiszel 66ba60fee8 fix: database connections now scale linearly with number of subworkers 2024-07-30 17:45:03 +02:00
Ruben Fiszel 12ec75551e feat: remove nativets in favor of bun with native pragma 2024-07-29 19:20:00 +02:00
Ruben Fiszel dc89819178 fix: disable prebundling for nodejs mode scripts 2024-07-28 11:06:43 +02:00
Ruben Fiszel b45ec20b83 fix: add the nobundling option for bun 2024-07-28 10:56:04 +02:00
Ruben Fiszel 9b2c2c96b4 small nit BASE_URL 2024-07-28 10:25:33 +02:00
Ruben Fiszel 8777f95fb0 add loading timing 2024-07-26 21:05:48 +02:00
Ruben Fiszel 5a7d89ebc6 fix tests 2024-07-26 19:26:00 +02:00
Ruben Fiszel eedc39c100 feat: pre-bundle bun scripts (#4132) 2024-07-26 19:00:30 +02:00
Ruben Fiszel 248c7472e8 fix: support npmjs mode for raw deps of package.json from CLI 2024-07-25 09:53:18 +02:00
wendrul 200e22c026 feat: job search index backed up and loaded from s3 (#4100)
* Add fn to get logs from disk and store as string

* Update dependencies

* Add async

* Update dependecies

* Add missing async

* Move JOB_RETENTION_SECS to windmill_common

* Update ee repo ref
2024-07-23 19:35:29 +02:00
HugoCasa 609e5d00eb feat: apply workspace specific tags only to some workspaces (#4107) 2024-07-23 19:12:11 +02:00
HugoCasaandRuben Fiszel 9e5c5873c8 fix: cgroupv1 mem limit + granular memory reporting (#4119)
* fix: cgroupv1 mem limit + granular memory reporting

* update ee ref

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-07-23 16:24:22 +02:00
HugoCasa b970902275 fix: improve cgroup readings (#4030)
* fix: improve cgroup readings

* feat: add staging image action

* fix: link staging image on other repo

* fix: linking

* fix: remove useless label
2024-07-18 17:12:07 +02:00
Ruben Fiszel be25101377 fix: add WM_SCHEDULED_FOR to contextual variables and early stop of flows 2024-07-18 02:10:15 +02:00
HugoCasa ecf4706784 feat: rehydrate instance settings/configs/users from CLI (#4035)
* feat: rehydrate instance settings/configs/users from CLI

* fix: nits

* feat: encryption key, saved instances, include all workspaces

* preshot client version

* Update deps.ts
2024-07-15 19:38:44 +02:00
Ruben Fiszel 7f2cbb40f6 feat: improve flow status viewer (show branch chosen + all iterations in for loop) (#4074)
* all

* all

* all

* all

* all
2024-07-13 18:53:24 +02:00
wendrul 0b2b280113 Add indexer case to avoid 0 max_database_connection (#4073) 2024-07-13 00:18:56 +02:00
wendrul af444c8b36 feat: Full-text search on runs using tantivy and command palette for quick actions (#4046)
* Add indexer crate and files

* POC searcher

incomplete schema
only indexes at startup

* POC search component frontend

* Demo of the frontend element

* add Results and Args as text

* minimal functionality

* Make jump to scripts by name

also flows and apps

* Add button on sidebar to open search

* Update lock on indexer after merge

* Make arrow key navigation compatible with scrol

* Show empty result screen and log as a coming feat

* Add summary to script searchable items

* Catch `parts is undefined` error (uFuzzy)

* Index refreshing using tokio interval

* Fix JobLoader workspace being wrongly defined

* Fix click outside

* Add debouncing for completed run search

* Binary mode working + job index tracker

* Warning for no license + fix height scrollbars on content search

* Make it compile without EE files

* remove panic to use errors

* Move global search

* Cleanup UI, no more tab switcher but clear placeholders and actions

* Add tantivy feature flag for windmill-api

* Rework indexer mode

* Mac compatibility for shortcut

* Update test for new run_server

* Prepare sqlx

* Mac compatibility

* Fix openapi yaml

* Fix frontend

* Frontend api fix

* Update docker-compose.yml and caddyfile

With the (by default deactivated) container and reverse proxy to use the
windmill indexer

* fix feature flag for tests

* fix feature falg for running tests

* fix feature flag for running tests

* Make content search use search modal instead

* Add tantivy feature to ee build steps

* Remove old Content search

* change volume location for indexer

* Update dependencies

* Prepare sqlx

* Uncomment line on docker compose

* Add line between input and results

* Update ee repo ref
2024-07-11 11:13:27 +02:00
Ruben Fiszel 2812792326 fix: memory optimization for flow with big args (#4019) 2024-07-04 10:08:42 +02:00
HugoCasa 2ecf44f44f fix(backend): switch job run to user db (#4017) 2024-07-03 21:51:59 +02:00