216 Commits
Author SHA1 Message Date
Ruben Fiszel 5707dfe4f1 add go-client mvp + stabilize openapi (#732) 2022-10-14 16:26:51 +02:00
Ruben Fiszel 444750d594 improve supercache 2022-10-13 03:04:55 +02:00
Ruben Fiszel 0cf9f0c8e2 improve supercache 2022-10-13 01:23:08 +02:00
Ruben Fiszel c1cc60045a improve supercache 2022-10-13 00:47:00 +02:00
Ruben Fiszel d116218a0e improve supercache 2022-10-13 00:15:21 +02:00
Ruben Fiszel 889965feb3 fix supercache in presence of other deps 2022-10-12 23:28:50 +02:00
Jakub Kołodziejczak 7e35d9989a feat(backend): supercache for python heavy dependencies in alpha 2022-10-12 23:02:27 +02:00
Ruben Fiszel 2cc0fe7bd7 fix handle shared path not existing because job was resumed on another worker 2022-10-12 20:09:07 +02:00
Ruben Fiszel 1438898617 fix suspend + sleep with same worker 2022-10-12 15:56:15 +02:00
Ruben Fiszel ff7fb0f6f3 feat: secure suspended resume event + configurable timeout (#721)
* fix suspend count for early message + delete resume_job

* all

* done everything

* done everything
2022-10-11 22:55:14 +02:00
Ruben Fiszel c7dfd0ebc4 fix suspend count for early message + delete resume_job 2022-10-10 20:17:49 +02:00
Ruben Fiszel f6e794cc5f add failed_jobs to retrystatus 2022-10-10 13:41:07 +02:00
Ruben Fiszel e6cad4df7b specify branch_chosen in openflow 2022-10-10 12:59:16 +02:00
Ruben Fiszel cc07a6b7e4 feat(backend): implement new OpenFlow module Branches (#692)
* branches

* iterate

* branches

* progress

* progress

* progress

* done

* adapt frontend

* adapt frontend

* sqlx
2022-10-09 23:12:05 +02:00
Ruben Fiszel 7bdbfec71a feat: support struct in Go as script parameters #705 2022-10-08 20:03:08 +02:00
Ruben Fiszel b99cb2e299 improve error messages 2022-10-08 14:36:21 +02:00
Ruben Fiszel 3f8f0eb878 improve error messages 2022-10-08 14:22:34 +02:00
Ruben Fiszel 6cb3fbc8b7 feat: add go LSP (#699)
* lsp

* progress

* 1.36.0
2022-10-08 13:28:46 +02:00
Ruben Fiszel 6ebedfc5fb feat(backend): separate properly logs from result 2022-10-07 19:54:44 +02:00
Ruben Fiszel 22c4a3b375 fix(backend): go lock dependency with no requirements 2022-10-07 16:11:45 +02:00
Ruben Fiszel a08436622b feat: sleep for arbitrary number of seconds statically or with a javascript expression (#691)
* progress

* progress

* fix serde

* fix frontend

* fix frontend
2022-10-07 04:09:32 +02:00
Ruben Fiszel feb5aadcc3 only stop wait on child on ok changes values 2022-10-06 23:37:10 +02:00
Ruben Fiszel 8accb59a8c fix: get info about kill reason 2022-10-06 23:25:49 +02:00
Ruben Fiszel b31e72a620 fix: get info about kill reason 2022-10-06 23:15:27 +02:00
Ruben Fiszel 2c5b32bdb7 fix(backend): same_worker uses the same folder even within loops 2022-10-06 21:39:07 +02:00
Ruben Fiszel 1b2994ac8a add GOMEMLIMIT to 2Gb 2022-10-06 05:13:06 +02:00
Ruben Fiszel f4caa4ffa6 feat: implement same_worker openflow attribute for running flow all in one go + sharing folder /shared (#689)
* init

* progress

* all

* sqlx

* remove unecessary test

* done

* frontend

* frontend
2022-10-06 04:59:09 +02:00
Ruben Fiszel 91ce66e7fe spurrious printlns 2022-10-04 14:44:56 +02:00
Ruben Fiszel e385eb54af progress 2022-10-04 08:40:01 +02:00
Ruben Fiszel 79636e6d86 progress 2022-10-04 08:40:01 +02:00
Ruben Fiszel 89a27837af minor UI fixes 2022-10-02 21:35:07 +02:00
Ruben Fiszel 612f7272a9 feat(backend): add WM_BASE_URL 2022-10-02 18:53:05 +02:00
Ruben Fiszel 6fe5b8d6b7 fix(backend): allow for now payload on resume GET 2022-10-02 05:14:36 +02:00
Ruben Fiszel 945a750c6b fix(go): make lines align with appended code 2022-10-02 01:28:14 +02:00
Ruben Fiszel eabd835807 fix(go): inner_main is in a separate file rather than wrapped 2022-10-02 01:22:33 +02:00
Ruben Fiszel d863b1ed90 feat(backend): add WM_FLOW_JOB_ID 2022-10-01 09:02:00 +02:00
Ruben Fiszel eaadad6143 hotfix trusted host for pip-compile 2022-09-28 09:17:08 +02:00
Ruben Fiszel ae3f86db11 feat: add private registries pip (#636)
* add private registries pip

* add private registries pip

* fix

* fix

* fix

* echo parameters
2022-09-28 08:47:43 +02:00
Ruben Fiszel b87a498087 do not restart flow as zombie job 2022-09-26 00:45:26 +02:00
Ruben Fiszel 4044edf8d6 refactor logs panel everywhere applicable (#625)
* init

* init

* progress

* progress

* refactor logs entirely

* fix sqlx
2022-09-25 18:03:51 +02:00
Ruben Fiszel 3a530c8566 fix cancel job 2022-09-25 16:36:32 +02:00
sqwishy e7a6c1b99f reorganize handle_child (#606)
* reorganize handle_child

There were a couple issues with the current implementation:

1. When reading stdout and stderr from the child, as soon as we hit EOF
   on one we would stop reading from both (line 1420). This could lead
   to the return value not being read from the job program.

2. Lines read from stdout and stderr are put into a channel and read
   elsewhere with `rx.recv()` (line 1497) but that channel isn't read
   until empty.  It is only read in the `while !done.load(...)` (line
   1449) loop and that loop can stop after any `.store(true, ...)`.
   Which happens when the child exits, when the job is cancelled, when
   either stdout or stderr reach EOF...

   This can be verified by putting `dbg!(rx.recv().await)` or a similar
   assertion after the while loop before returning from that function.
   It shows the channel still containing log lines on rare occasions.

I was pretty careful in this to maintain the current behaviour; adding
comments to express intention.

One difference in this is that some regular intervals (cancel check and
ping update) should be more regular?

Before...

> at 00ms wait for 10ms
> at 10ms do things for 3ms
> at 13ms wait again for *10ms*
> at 23ms do things again ...

With change...

> at 00ms wait for 10ms
> at 10ms do things for 3ms
> at 13ms wait again but for *7ms*
> at 20ms do things again ...

Which I'm guessing is preferable but I could be wrong.

* renames; interpolate values in log messages

* do `append_logs()` in tokio::task

* tokio::time::interval & close pipe after limit

* clean up comments
2022-09-24 22:55:52 +02:00
sqwishy 47fba21256 remove delay between spawning each worker (#614) 2022-09-21 21:05:19 -07:00
Ruben Fiszel e1f54832fa more details on error handling job 2022-09-18 10:11:21 +02:00
sqwishy 126dd24c71 feat(backend): flow suspend resume (#522)
Flow observes `suspend` setting and will wait for resume messages sent for the job before continuing to the next step in a flow.

Adds endpoints under workspaces at `/jobs/<cancel|resume>/<job-uuid>` to either cancel or resume the job with a payload. For POST requests to the endpoint, payload is a JSON document. For GET requests to the endpoints, the payload is a base64url encoded JSON document as the value of the payload query parameter.
2022-09-14 11:46:57 -07:00
Ruben Fiszelandsqwishy 39918a9bb1 feat: implement go support (#571)
* progress

* progress

* all in one

* frontend

* small nits

* go job test

* go.sum is optional

* add golang-go to backend test image

Co-authored-by: sqwishy <somebody@froghat.ca>
2022-09-13 21:14:21 +02:00
Ruben Fiszel 110a25f6f8 fix: iterator input transform is made more generic (#524)
* fix: iterator expr is standardized with regular input transforms

* fix: iterator expr is standardized with regular input transforms

* v2
2022-09-13 20:11:00 +02:00
ex0ns 6f09405c2d feat(job): run job by hash (#551)
It was possible to run them using a webhook, but not through an endpoint.
This PR aims to fix that so the user can target a specific version of
the script to run.
2022-09-11 16:39:48 +02:00
Ruben Fiszel e9abcffdd1 feat: is_trigger is just a type tag, soon to include failure and command (#523)
* script kind

* all

* init code flow

* kind: trigger

* kind: trigger
2022-09-05 09:55:04 +02:00
Ruben Fiszel 482dc808ae logs around fetching job 2022-09-04 13:30:48 +02:00