Commit Graph

1785 Commits

Author SHA1 Message Date
Ruben Fiszel 96ff45ecb5 feat(backend): default parameters are used in python if missing from args 2023-05-05 09:44:53 +02:00
Ruben Fiszel f3861ebd2f fix(backend): fix python transformers 2023-05-03 23:27:14 +02:00
Ruben Fiszel e21208de31 feat(backend): add EXIT_AFTER_NO_JOB_FOR_SECS for ephemeral workers 2023-04-28 00:09:48 +02:00
Ruben Fiszel 1764c4114e feat(backend): expose tag in the job (#1486)
* fix

* update tag

* update tag

* update tag

* fix
2023-04-27 19:09:53 +02:00
Ruben Fiszel 1d1c812e36 feat(backend): add JOB_RETENTION_SECS to delete completed jobs completed after expiration period 2023-04-26 19:34:54 +02:00
Ruben Fiszel 02ba82064f feat(backend): worker groups (#1452)
* tagged scripts

* tagged scripts

* fix

* fix

* fix

* all

* update sqlx
2023-04-23 16:57:23 +02:00
Ruben Fiszel c9d6a47b31 fix(backend): avoid potential conflict between pull from tar and background sync 2023-04-21 23:49:14 +02:00
Ruben Fiszel a3487f18c4 fix(backend): global cache synco only start if all piptars have been downloaded 2023-04-21 22:21:04 +02:00
Ruben Fiszel 1ba0e24155 feat(backend): only run fully deployed scripts 2023-04-21 12:49:47 +02:00
Ruben Fiszel 394c6fa92d feat(backend): workers are instantly ready and sync with global cache in background 2023-04-21 09:45:25 +02:00
Ruben Fiszel d8ceaa5bcc improve log line for cache syncing 2023-04-21 01:54:58 +02:00
Ruben Fiszel dd6602284e fix untar_all_piptars 2023-04-21 01:02:51 +02:00
Ruben Fiszel ecca1e2c3e better rclone filters 2023-04-21 00:31:39 +02:00
Ruben Fiszel ee1599166c better rclone filters 2023-04-20 23:45:31 +02:00
Ruben Fiszel d8da7b2883 set relative folder for archive creation 2023-04-20 23:33:53 +02:00
Ruben Fiszel 8fda8bbaf8 remove deno/gen/windmill from tmp cache 2023-04-20 21:04:17 +02:00
Ruben Fiszel e74f8f8248 --s3-no-cache-buckets 2023-04-20 20:50:02 +02:00
Ruben Fiszel 242b2ae899 feat(backend): global cache refactor for pip using tar for each dependency (#1443)
* cache refactor

* exclude tar from being synced to bucket

* run

* update

* update
2023-04-20 20:05:12 +02:00
Ruben Fiszel ccda9850b6 fix python install on save with nsjail 2023-04-20 10:33:11 +02:00
Ruben Fiszel f0b939cd3a add barrier when num workers > 1 2023-04-19 17:00:03 +02:00
Ruben Fiszel 26c7e9e8c3 add barrier when num workers > 1 2023-04-19 12:46:25 +02:00
Ruben Fiszel fb0983bcb3 feat(ee): sync cache in background 2023-04-19 01:15:41 +02:00
Ruben Fiszel 89d0bea9fa feat(ee): sync cache in background 2023-04-19 00:55:51 +02:00
Ruben Fiszel 4934dc454f only cache on first worker 2023-04-18 21:23:29 +02:00
Ruben Fiszel 5e2ca89137 log how much time took each command child 2023-04-18 08:55:26 +02:00
Ruben Fiszel 66c532affc improve bash output buffering 2023-04-18 01:42:28 +02:00
Ruben Fiszel 732d61f190 performance: process completed job in background (#1396)
* compile

* improve
2023-04-15 11:06:51 +02:00
Ruben Fiszel 556dbce239 feat(backend): install python scripts on save 2023-04-13 22:57:53 +02:00
Ruben Fiszel c2b66fef4f do a test run on first iteration 2023-04-13 21:54:42 +02:00
Ruben Fiszel be41e45130 split worker.rs into more files 2023-04-13 17:44:47 +02:00
Ruben Fiszel 94098c23e0 fix refreshOn 2023-04-13 09:22:52 +02:00
Ruben Fiszel 90ce302f2d fix(backend): deno uses --no-check 2023-04-13 00:22:46 +02:00
Ruben Fiszel 4dc1773dc7 fix(backend): do not fail on schedule not existing anymore 2023-04-12 20:06:50 +02:00
Ruben Fiszel 53c4c744c4 exclude deno/gen/file from global cache 2023-04-12 19:18:48 +02:00
Ruben Fiszel e8611ba8c1 add GOPRIVATE and NETRC for private modules 2023-04-11 19:55:38 +02:00
Ruben Fiszel c4366b9ef4 fix relative url deno loading2 2023-04-11 15:57:59 +02:00
Ruben Fiszel df8236a03f fix(backend): nested deno relative imports 2023-04-11 15:16:55 +02:00
Kai Jellinghaus 63102b036c feat(backend): Redis based queue (#1324)
* Merge?

* Fix V8 breaking change

* WIP

* WIP

* Cleanup

* Move to git reference

* Fix Merge conflict

* update

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2023-04-11 12:24:13 +02:00
Ruben Fiszel 66d2ae813a fix tests + scheduled_for uses now instead of system time 2023-04-11 11:48:14 +02:00
Ruben Fiszel 613cd64003 improve metrics 2023-04-10 11:58:44 +02:00
Ruben Fiszel 621bb282af improve metrics 2023-04-10 11:53:32 +02:00
Ruben Fiszel 415be7c70f perf: parallelize more operations for deno jobs 2023-04-10 10:55:56 +02:00
Ruben Fiszel b7b26dc9c7 improve token lock 2023-04-10 08:00:41 +02:00
Ruben Fiszel 6ab4f30026 fetch token in the background 2023-04-10 01:16:15 +02:00
Ruben Fiszel a23a14f9f2 make create_token_for_owner not a transaction anymore 2023-04-09 21:47:05 +02:00
Ruben Fiszel bbb6a15da7 only register prometheus metrics if they are enabled 2023-04-09 15:39:25 +02:00
Ruben Fiszel ff22759a43 deno optimization v0 2023-04-09 14:10:56 +02:00
Ruben Fiszel 36e338c461 feat(backend): extend cached resolution for go 2023-04-09 10:58:45 +02:00
Ruben Fiszel 130ddbbf85 feat(python): cache dependency resolution 2023-04-08 23:25:46 +02:00
Ruben Fiszel a59cdeff11 show status code when possible 2023-04-08 20:09:27 +02:00