Compare commits

...
212 Commits
Author SHA1 Message Date
Ruben Fiszelandrubenfiszel 8d73c9276e chore(main): release 1.84.0 (#1336)
* chore(main): release 1.84.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-31 20:53:24 +02:00
Faton Ramadani 57f8dd9570 feat(frontend): Display frontend execution result in Debug Runs (#1341)
* feat(frontend): record frontend errors

* feat(frontend): display result

* feat(frontend): Fix name shadowing

* feat(frontend): fix typo

* fix(frontend): better display frontend errors
2023-03-31 15:20:16 +02:00
Ruben Fiszel 5a8e00d285 remove unecessary deno lock file 2023-03-31 11:53:31 +02:00
Ruben Fiszel f217a2c368 fix input_transform -> input_transforms tests 2023-03-31 11:27:37 +02:00
Ádám Kovács 2779891411 fix(frontend): Export python code as string (#1339) 2023-03-31 11:09:53 +02:00
Ryan Rich dfd2abc764 fix(backend): backend compatability on macos (#1340) 2023-03-31 09:00:54 +02:00
Ruben Fiszel 624279e568 fix compile 2023-03-31 00:30:10 +02:00
Ruben Fiszel cd1f9b6baa fix(frontend): improve app tables 2023-03-31 00:19:14 +02:00
Ruben Fiszel 8b4e828e64 add license 2023-03-30 19:48:18 +02:00
Ruben Fiszel 2e7871439a consistent sort + minor nits 2023-03-30 17:27:21 +02:00
Ruben Fiszel 82578ef836 minor ux nits 2023-03-30 16:15:02 +02:00
Ruben Fiszel b3254938fe fix(frontend): improve loading of big jobs in run form 2023-03-30 13:14:39 +02:00
Ruben Fiszel 71619acdfa fix(frontend): improve loading of big args in job details 2023-03-30 12:59:11 +02:00
Ruben Fiszel c7506e4dae feat(backend): reduce memory allocation for big forloops of flows 2023-03-30 11:52:54 +02:00
Ruben Fiszel 2368da2146 feat: add the ability to edit previous versions 2023-03-30 08:25:00 +02:00
Ruben Fiszel 7fc97e274e only decrypt secret value 2023-03-30 07:58:51 +02:00
Ruben Fiszel 4f75a5840a redeploy 2023-03-30 01:41:37 +02:00
Ruben Fiszel 8b80b3cb74 fix cancel job 2023-03-30 01:20:54 +02:00
Ruben Fiszel 31d0d102eb use requests for . imports 2023-03-30 01:07:24 +02:00
Ruben Fiszel fbe5c18da0 feat: add force cancel 2023-03-30 00:42:13 +02:00
Ruben Fiszel 8d0d996bbb fix flow connection 2023-03-29 22:01:12 +02:00
Ruben Fiszel 2b09fead4f Update deploy-to-s3.yml (#1338) 2023-03-29 20:31:09 +02:00
Ruben Fiszel fccb3d8088 truncate file too big 2023-03-29 20:04:31 +02:00
Ruben Fiszel 9aaeaf4ee0 truncate file too big 2023-03-29 20:03:17 +02:00
Ádám KovácsandFaton Ramadani 01564f0a1c feat(frontend): App component style input grouping (#1334)
* feat(frontend): App component style input grouping

* fix: default value

* fix

* update styling

* fix property toggling

---------

Co-authored-by: Faton Ramadani <faton.ramadani14@gmail.com>
2023-03-29 19:24:20 +02:00
Ruben Fiszel 78085a8a12 deploy main to s3 2023-03-29 17:09:56 +02:00
Ruben Fiszel 5abd9854ad deploy main to s3 2023-03-29 17:05:42 +02:00
Ruben Fiszel 85e9aa983b deploy main to s3 2023-03-29 16:50:30 +02:00
Ruben Fiszel 9853380df6 deploy main to s3 2023-03-29 16:46:36 +02:00
Ruben Fiszel 5aa14562a0 handle space and uppercase in script_path 2023-03-29 16:16:14 +02:00
Ruben Fiszel c07a17ff8a improve focusedGrid in tabs and drawers 2023-03-29 10:34:31 +02:00
Faton Ramadani 5ac646e859 feat(frontend): improve input connection UI (#1333)
* feat(frontend): improve input connection UI

* feat(frontend): prevent pointerup from bubbling + refactor the code

* feat(frontend): remove unnecessary alert + fix null display

* feat(frontend): restore hoveredComponent when connecting + properly open deeply nested component

* feat(frontend): fix ObjectViewer display
2023-03-29 09:05:50 +02:00
Ruben Fiszel bb61cef0e5 fix(backend): add a refresh button to workspace script/hub 2023-03-29 01:06:24 +02:00
Ruben Fiszelandrubenfiszel f73664759f chore(main): release 1.83.1 (#1335)
* chore(main): release 1.83.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-28 23:38:12 +02:00
Ruben Fiszel 569a55e45b fix(cli): plain secrets might be undefined 2023-03-28 23:32:41 +02:00
Ruben Fiszel 5d53967ba4 fix compile issue swc_common 2023-03-28 23:13:43 +02:00
Ruben Fiszelandrubenfiszel a24a3b4787 chore(main): release 1.83.0 (#1327)
* chore(main): release 1.83.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-28 22:50:27 +02:00
Ruben Fiszel 98d51e219d fix(cli): add --plain-secrets 2023-03-28 22:46:43 +02:00
Ruben Fiszel 11431a75f4 fix python imports 2023-03-28 21:51:12 +02:00
Ruben Fiszel 32ef18bffe fix tests 2023-03-28 21:40:36 +02:00
Ruben Fiszel 8497d1d1c0 fix autogen script names 2023-03-28 21:07:21 +02:00
Ruben Fiszel 015f8e893f fix autogen script names 2023-03-28 20:43:51 +02:00
Ruben Fiszel 07ab2dbb0a fix autogen script names 2023-03-28 20:39:58 +02:00
Ruben Fiszel a5500ea40a feat(backend): allow relative imports for python 2023-03-28 20:27:40 +02:00
Ruben Fiszel 5eab9431bd fix integer handling as field type 2023-03-28 17:36:21 +02:00
Ruben Fiszel 33c07d3e63 feat(frontend): add pagination 2023-03-28 16:58:43 +02:00
Ruben Fiszel 5f335d2464 more flow logs 2023-03-28 14:01:47 +02:00
Ruben Fiszel c0d72e2881 improve logging to trace job from root_job and fetching flow details from a separate workspace 2023-03-28 07:58:35 +02:00
Ruben Fiszel e40f16c969 minor backend fixes 2023-03-27 22:57:29 +02:00
Ruben Fiszel 1d63877a69 improve flow renaming 2023-03-27 21:16:43 +02:00
Ruben Fiszel 6764c519b2 add more sanity checks to api 2023-03-27 21:07:34 +02:00
Ruben Fiszel 3463bfe36f fix array list removal 2023-03-27 20:51:52 +02:00
Ruben Fiszel 2b31653a8a feat(frontend): add textareacomponent + fix multiselect style + select multi components 2023-03-27 18:33:50 +02:00
Ruben Fiszel 449d3ae5dd feat(frontend): add recompute as a primitive 2023-03-27 17:41:37 +02:00
Ruben Fiszel cfa3f9ce7c fix recomputeOthers 2023-03-27 15:49:24 +02:00
Ruben Fiszel 201aa6d088 improve app setup 2023-03-27 15:24:13 +02:00
Ruben Fiszel 021fa23f9f feat(backend): execute /bin/bash instead of /bin/sh for bash scripts 2023-03-27 13:08:34 +02:00
Ruben Fiszel b95afaa9bb feat(frontend): use rich json editor for arrays of objects and for object in ArgInput 2023-03-27 13:03:06 +02:00
Ruben Fiszel 83e982e84d minor fix 2023-03-27 11:54:30 +02:00
Ádám Kovács fa457bb709 fix(frontend): Revert app upload input (#1330)
* fix(frontend): Revert app upload input

* fix default value
2023-03-27 11:51:30 +02:00
Faton Ramadani 75306c8316 fix(frontend): Small app fixes (#1331)
* fix(frontend): wip

* fix(frontend): wip

* fix(frontend): merge main

* fix(frontend): fix build

* fix(frontend): revert css

* fix(frontend): fix alignement + remove id from  if deleted

* fix(frontend): fix connection height

* fix(frontend): done
2023-03-27 11:50:40 +02:00
Ruben Fiszel eaac598af3 feat(backend): improve relative importsfor deno 2023-03-27 08:45:35 +02:00
Ruben Fiszel 00b70d9aaa feat(backend): increase timeout for premium workspace 2023-03-26 18:56:40 +02:00
Ruben Fiszel 9b09fac27a add more variants to shadow 2023-03-26 18:17:47 +02:00
Ruben Fiszel 6ed7268258 many apps small fixes 2023-03-26 18:14:37 +02:00
Ruben Fiszel 014765c83b fix redraw issues 2023-03-25 20:35:28 +01:00
Ruben Fiszel 577dec5c57 feat(frontend): multiselect components for apps 2023-03-25 16:54:50 +01:00
Ruben Fiszel 9ab087a20c make cancel api an optauthed 2023-03-25 08:39:56 +01:00
Ruben Fiszel 1a4867302f fix(frontend): persist description for schemas 2023-03-25 08:28:27 +01:00
Ruben Fiszel 8e3d8acc80 fix(apps): improve app table actionButtons behavior under many clicks 2023-03-25 07:59:32 +01:00
Ádám Kovács ac2486219c feat(frontend): Add quick style settings to app editor (#1308)
* feat(frontend): Add app secondary settings menu

* refactor(frontend): Separate color picker

* save

* feat(frontend): Add quick style options

* fix(frontend): Handle overflow

* feat(frontend): Add suggestions to quick styles

* save wip

* feat(frontend): Add UI for quick styling

* fix(frontend): Handle multi value properties

* fix(frontend): Convenience updates

* feat(frontend): Add styling properties to components

* fix(frontend): Parse inner colors

* fix(frontend): Multi value sync

* fix(frontend): Correct unit handling

* fix(frontend): Correct multivalue handling

* remove comments

* fix color picker label

* feat(frontend): Add box-shadow property

* feat(frontend): Add concise unit selector

* feat(frontend): Update app labels

* remove unused imports

* fix width

* fix(frontend): App freezing

* fix(frontend): Remove unused imports

* fix(frontend): Conditional chaining

* fix

* revert

* minor updates

* feat(frontend): Add 'apply style to all' button

* fix(frontend): Update styling properties

* fix(frontend): Styling menu typography update
2023-03-24 18:26:16 +01:00
Faton Ramadani a527cb8222 fix(frontend): add a modal that is always mounted to make sure compon… (#1328)
* fix(frontend): add a modal that is always mounted to make sure component binding are properly set

* fix(frontend): remove uselss open prop
2023-03-24 17:30:04 +01:00
Ádám Kovács da24e9ab06 fix(frontend): Disable app keyboard navigation on focused inputs (#1326) 2023-03-24 16:48:34 +01:00
Ruben Fiszel 4dc00c2587 improve require super_admin check error 2023-03-24 14:55:56 +01:00
Ruben Fiszel 45c52f7723 make superadmin_secret work also for apis 2023-03-24 14:53:28 +01:00
Ruben Fiszel 70a7089352 make superadmin_email permission be inherited by ephemeral tokens 2023-03-24 13:22:56 +01:00
Ruben Fiszel 895609f0d2 fix backend compile 2023-03-24 12:54:20 +01:00
Ruben Fiszelandrubenfiszel dd06c05046 chore(main): release 1.82.0 (#1316)
* chore(main): release 1.82.0

* Apply automatic changes

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-24 12:33:33 +01:00
Ruben Fiszel a86fad6a9e fix change versions for cli 2023-03-24 12:25:30 +01:00
Ruben Fiszel f7c30b5d2f fix(backend): do not consider FlowPreview as potential zombie job 2023-03-24 12:12:51 +01:00
Ruben Fiszel b999c9894b fix(cli): improve diff speed + fix replacing cli 2023-03-24 09:28:45 +01:00
Ruben Fiszel a2de6c7d5f recompute others for modal work even if no script attached 2023-03-23 21:10:18 +01:00
Ruben Fiszel 15812b4cec fix compile 2023-03-23 21:05:51 +01:00
Faton Ramadani b22600e9c3 Fix settings panel v2 (#1325)
* fix(frontend): fix settings panel

* fix(frontend): fix settings panel

* fix(frontend): fix settings panel
2023-03-23 20:55:57 +01:00
Faton Ramadani c15bc8a7bf fix(frontend): Fix AppTable error display + clear errors when removing a component + properly detect that latest component run had an error (#1322) 2023-03-23 20:54:02 +01:00
Faton Ramadani 30b8e474df fix(frontend): fix settings panel (#1323)
* fix(frontend): fix settings panel

* fix(frontend): fix settings panel

* fix(frontend): fix settings panel
2023-03-23 20:53:07 +01:00
Ruben Fiszel f420999363 fix recomputeOthers 2023-03-23 17:55:21 +01:00
Ruben Fiszel 984c9a1191 fix backend timeout issues 2023-03-23 17:22:35 +01:00
Ruben Fiszel a2df650936 fix backend timeout issues 2023-03-23 15:59:28 +01:00
Ruben Fiszel c0076c652d fix same worker query 2023-03-23 15:57:52 +01:00
Faton Ramadani addabcceb0 fix(frontend): Fix transformer (#1321) 2023-03-23 14:20:15 +01:00
Ruben Fiszel 47a7f7163a feat(backend): introduce RESTART_ZOMBIE_JOBS and ZOMBIE_JOB_TIMEOUT 2023-03-23 12:47:49 +01:00
Ruben Fiszel 34e25f0f96 fix(backend): increase dynamic js timeout + improve client passing 2023-03-23 10:32:09 +01:00
Ruben Fiszel 93ce252954 fix caddyfile to re-enable lsp 2023-03-22 21:55:59 +01:00
Ruben Fiszel d3effe953b re-enable transformers 2023-03-22 21:53:28 +01:00
Ruben Fiszel d935dba28b fix add property not in portal 2023-03-22 21:47:59 +01:00
Ruben Fiszel 58167a16cd refresh in more cases 2023-03-22 21:37:49 +01:00
Ruben Fiszel 9e9683c6f7 refresh in more cases 2023-03-22 21:29:24 +01:00
Ruben Fiszel 7511f0b18e add red toggle 2023-03-22 20:06:38 +01:00
Ruben Fiszel c5d305bad8 fix runnableComponent even if not changed on inputs 2023-03-22 20:01:17 +01:00
Ruben Fiszel b4008e62fd clarify trigger list 2023-03-22 19:41:33 +01:00
Ruben Fiszel bb227b69c8 add keys to switch scripts 2023-03-22 19:33:37 +01:00
Ruben Fiszel 5518eab7b7 fix disappearing scripts 2023-03-22 19:30:11 +01:00
Faton Ramadani a47031a41e fix(frontend): fix refresh with manual dependencies (#1319)
* fix(frontend): fix refresh with manual dependencies

* fix(frontend): fix id generation

* fix(frontend): wip

* fix(frontend): wip

* fix(frontend): Fix binding + hide toggle for frontend scripts
2023-03-22 18:27:38 +01:00
Faton Ramadani e193a0bcdf fix(frontend): remove unnecessary div (#1318) 2023-03-22 14:40:29 +01:00
Ruben Fiszel 2df1373a69 fix compile error 2023-03-22 12:03:48 +01:00
Ruben Fiszelandrubenfiszel c6bf67605d chore(main): release 1.81.0 (#1314)
* chore(main): release 1.81.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-21 23:15:55 +01:00
Faton Ramadani 0086f99dcc fix(frontend): fix tabs height (#1315) 2023-03-21 23:15:22 +01:00
Ruben Fiszel bba09fdaeb fix(cli): make --skip-pull work even if no state is present 2023-03-21 23:14:56 +01:00
Ruben Fiszel 942d2b2244 remove do from acceptable ids 2023-03-21 21:20:21 +01:00
Ruben Fiszel 8a2e6365a0 improve load time for tabs 2023-03-21 20:56:54 +01:00
Ruben Fiszel 527f4b543a improve load time for text and button components 2023-03-21 19:55:35 +01:00
Ruben Fiszel 2593218cbf feat(apps): add action on form/button/formbutton 2023-03-21 18:22:20 +01:00
dependabot[bot] bfb5c1b5a4 chore(deps-dev): bump svelte2tsx from 0.6.1 to 0.6.10 in /frontend (#1311)
Bumps [svelte2tsx](https://github.com/sveltejs/language-tools) from 0.6.1 to 0.6.10.
- [Release notes](https://github.com/sveltejs/language-tools/releases)
- [Commits](https://github.com/sveltejs/language-tools/compare/svelte2tsx-0.6.1...svelte2tsx-0.6.10)

---
updated-dependencies:
- dependency-name: svelte2tsx
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-21 13:05:04 +01:00
Faton Ramadani a7c4f1a12e fix(frontend): Remove action outline on preview mode (#1313) 2023-03-21 13:04:43 +01:00
Ruben Fiszelandrubenfiszel 7cb363845e chore(main): release 1.80.1 (#1312)
* chore(main): release 1.80.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-21 02:13:02 +01:00
Ruben Fiszel 652d3c3889 fix initialization for app with forms 2023-03-21 02:11:34 +01:00
Ruben Fiszel 42f6d2e0ee fix(cli): add support for non metadataed scripts 2023-03-21 02:01:34 +01:00
Ruben Fiszelandrubenfiszel d2cccd98e0 chore(main): release 1.80.0 (#1305)
* chore(main): release 1.80.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-21 01:00:29 +01:00
Ruben Fiszel cc4d61b6bd small fix 2023-03-20 19:54:04 +01:00
Ruben Fiszel fbdda1a4dd make top bar togglable 2023-03-20 19:25:39 +01:00
Faton Ramadani 48413a78c5 feat(frontend): App set tab (#1307)
* feat(frontend): Set tab

* feat(frontend): Set tab

* feat(frontend): fix styling

* feat(frontend): update how set Tab is stored

* feat(frontend): clean up

* feat(frontend): update setTab to an array structure

* feat(frontend): revert

* feat(frontend): support all components

* feat(frontend): fix typing

* feat(frontend): group success side effects

* feat(frontend): add missing tooltip + remove duplicate code
2023-03-20 19:09:56 +01:00
Ruben Fiszel be1d987b41 fix table action input fields 2023-03-20 16:04:50 +01:00
Ruben Fiszel 0f64859961 fix minor app issues 2023-03-20 15:27:30 +01:00
Ruben Fiszel b762de1eae revert caddyfileremote change 2023-03-20 14:04:18 +01:00
Ruben Fiszel b00bde0a63 fix preview of ontextfocus 2023-03-20 14:01:34 +01:00
Ruben Fiszel f972e4bb06 improve drag for apps 2023-03-20 10:58:20 +01:00
Faton Ramadani 29b1cc6ff0 fix(frontend): add missing optional chaining (#1306) 2023-03-20 09:49:11 +01:00
Ruben Fiszel b51246411f use skeleton for loading in apps 2023-03-20 08:49:51 +01:00
Ruben Fiszel f26c7ff62b minor app improvements 2023-03-20 08:35:44 +01:00
Ruben Fiszel 211ad52edb minor app improvements 2023-03-20 08:26:02 +01:00
Ruben Fiszel 1392bebf87 improve move animation 2023-03-19 23:12:01 +01:00
Ruben Fiszel 874cf412a1 minor app fixes 2023-03-19 22:37:00 +01:00
Ruben Fiszel b497c3463f minor app fixes 2023-03-19 15:35:40 +01:00
Ruben Fiszel fbe2f0ca93 fix when loading is set to true 2023-03-19 15:25:46 +01:00
Ruben Fiszel 0abacac06c feat(apps): add transformers for data sources 2023-03-19 15:16:31 +01:00
Ruben Fiszel 8fab191a7f apps load when all outputs are initialized 2023-03-18 20:25:21 +01:00
Ruben Fiszel f8fad8326d more typescript type safety tricks 2023-03-18 18:15:15 +01:00
Ruben Fiszel 3b84672363 minor app fixes 2023-03-18 16:25:34 +01:00
Ruben Fiszel e45917c020 fix lock + reactivity on ctrl + fix optionValuesKeys 2023-03-18 15:52:46 +01:00
Ruben Fiszel d570ef58ac improve type definition of apps and sync them with static components + purge app content of unecessary data 2023-03-18 15:29:42 +01:00
Faton Ramadani cf2d031e8e fix(frontend): App button triggered by (#1304)
* feat(frontend): add trigger list

* feat(frontend): add trigger list

* feat(frontend): add support for refreshOn

* feat(frontend): add support for refreshOn

* feat(frontend): rework

* feat(frontend): rework

* feat(frontend): code cleanup

* fix(frontend): add support for triggerOnAppLoad
2023-03-17 21:55:40 +01:00
Ruben Fiszel 9657cc9c7e fix compile error after deno upgrade 2023-03-17 21:45:54 +01:00
Ruben Fiszelandrubenfiszel 200adec32f chore(main): release 1.79.0 (#1295)
* chore(main): release 1.79.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-17 21:10:44 +01:00
Ruben Fiszel 7625782038 fix apps bar not being persistent 2023-03-17 21:08:45 +01:00
Ruben Fiszel 4242f1bb63 fix cli failing on script with non json files 2023-03-17 21:01:44 +01:00
Ruben Fiszel f1e718e718 fix cli failing on script with non json files 2023-03-17 20:58:13 +01:00
Ruben Fiszel f3dfad5b94 fix loading nested resource in client fetched resources 2023-03-17 20:54:37 +01:00
Faton Ramadani 078cb1bf3e feat(frontend): App component triggers (#1303)
* feat(frontend): add trigger list

* feat(frontend): add trigger list

* feat(frontend): add support for refreshOn

* feat(frontend): add support for refreshOn

* feat(frontend): rework

* feat(frontend): rework

* feat(frontend): code cleanup
2023-03-17 20:51:00 +01:00
Ruben Fiszel 97e3bb4aa8 fix reactivity for static inputs 2023-03-17 20:50:34 +01:00
Faton Ramadani c1e43de4ea Fix sub grid (#1302)
* fix(frontend): fix horizontal splitpanes

* fix(frontend): fix subgrid height

* fix(frontend): fix subgrid height
2023-03-17 15:56:26 +01:00
Faton Ramadani ea3dab411b fix(frontend): fix horizontal splitpanes (#1301) 2023-03-17 15:46:45 +01:00
Ruben Fiszel a76f6f7bd9 apps rich configuration v0 + resource picker clear fix for apps 2023-03-17 12:25:03 +01:00
Ruben Fiszel 4305670d90 fix app inputs wrt to subgrids 2023-03-16 16:50:01 +01:00
Ruben Fiszel 597e38ef36 feat(frontend): add listeners for frontend scripts 2023-03-16 14:56:16 +01:00
Ruben Fiszel ca3e3624c0 fix key left navigation 2023-03-16 12:48:31 +01:00
Ádám KovácsandFaton Ramadani c1dd35c3f0 fix(frontend): App panel styling (#1284)
* fix(frontend): App panel styling

* remove

* revert

---------

Co-authored-by: Faton Ramadani <faton.ramadani14@gmail.com>
2023-03-16 12:42:41 +01:00
Faton Ramadani bd927a27ed feat(frontend): Component control (#1293)
* fix(frontend): fix app tabs

* fix(frontend): app controls

* fix(frontend): app controls

* fix(frontend): align output panel

* fix(frontend): clean up

* fix(frontend): refactor events

* fix(frontend): fix display

* fix(frontend): fix indentation

* fix(frontend): merge main
2023-03-16 12:42:10 +01:00
Faton Ramadani 00927210fd fix(frontend): fix map render (#1297)
* fix(frontend): fix map render

* fix(frontend): fix map render
2023-03-16 12:39:12 +01:00
Ádám Kovács bd3ee81b14 fix(frontend): Display app context search on top (#1300)
Fixing z-index issue of the search bar in the outputs panel
2023-03-16 12:38:35 +01:00
Ádám Kovács bac831b23c fix(frontend): Hide archive toggle with empty list (#1296) 2023-03-16 12:36:56 +01:00
Faton Ramadani c3ba1a6ab9 feat(frontend): add table actions navigation (#1298)
* feat(frontend): add table actions navigation

* feat(frontend): add table actions navigation

* feat(frontend): add table actions navigation
2023-03-16 12:36:33 +01:00
Ruben Fiszelandrubenfiszel 52157faf72 chore(main): release 1.78.0 (#1292)
* chore(main): release 1.78.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-16 03:11:38 +01:00
Ruben Fiszel a9e8aa0f1c update cli error message for folders 2023-03-16 03:10:45 +01:00
Ruben Fiszel 50c1c614ef context panel transitions 2023-03-16 03:00:01 +01:00
Ruben Fiszel 11567d6280 feat(frontend): app textcomponent editable + tooltip 2023-03-16 02:34:39 +01:00
Ruben Fiszel a6e1510405 fix minor flow UX 2023-03-16 01:18:11 +01:00
Ruben Fiszel b7d8fd1a4d apps minor fixes 2023-03-16 00:55:45 +01:00
Ruben Fiszel e27de7fb5d lightarginput for apps forms 2023-03-16 00:13:09 +01:00
Ruben Fiszel 99ec12e10c various minor fixes 2023-03-15 23:31:35 +01:00
Ruben Fiszel 9bfd471439 context panel refactor 2023-03-15 22:24:13 +01:00
Ruben Fiszel dbdfd62638 fix(frontend): remove staticOutputs from apps 2023-03-15 17:22:46 +01:00
Ruben Fiszel 6f890f2120 fix(frontend): improve rendering performance after component moving 2023-03-15 12:35:11 +01:00
Ruben Fiszel 183a4591df fix(backend): whitelist for include_header was ignored in some cases 2023-03-14 23:47:57 +01:00
Ruben Fiszelandrubenfiszel 646c0f23da chore(main): release 1.77.0 (#1286)
* chore(main): release 1.77.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-14 22:27:08 +01:00
Ádám KovácsandFaton Ramadani dea12e8870 fix(frontend): Update setting accordion (#1285)
* fix(frontend): Update setting accordion

* remove empty tooltip

---------

Co-authored-by: Faton Ramadani <faton.ramadani14@gmail.com>
2023-03-14 22:24:23 +01:00
Ryan Rich 944795f6ee feat(deno): add support for custom npm repo (#1291)
* feat(deno): add support for custom npm repo

* feat(deno): add support for custom npm repo

* feat(deno): fix comment being on wrong line

* feat(deno): review feedback
2023-03-14 22:23:44 +01:00
Ruben Fiszel 65d4bc519c add env build configuration 2023-03-14 22:20:45 +01:00
Ruben Fiszel 4d3507aec2 add .env.production to .gitignore 2023-03-14 22:19:08 +01:00
Ruben Fiszel 1d395ccc17 add env build configuration 2023-03-14 22:16:22 +01:00
hcourdent 3a7129de4b Added tooltips to App editor (#1289) 2023-03-14 22:05:34 +01:00
Ruben Fiszel 2f0acb9ffa feat(apps): state can be used as input in apps 2023-03-14 20:44:14 +01:00
Ruben Fiszel 81f989837b improve reactivity of apps 2023-03-14 18:46:31 +01:00
Faton Ramadani c71a577fea fix(frontend): fix app tabs (#1288) 2023-03-14 16:11:07 +01:00
Faton Ramadani bc870bd03e fix(frontend): fix container deletion (#1287)
* fix(frontend): fix container deletion

* fix(frontend): fix container deletion
2023-03-14 14:57:44 +01:00
Faton Ramadani 751edcf9b8 feat(frontend): app components output panel (#1283)
* feat(frontend): hierarchical output panel WIP

* feat(frontend): wip

* feat(frontend): working animations

* feat(frontend): working animations

* feat(frontend): wip

* feat(frontend): wip

* feat(frontend): improving connection

* feat(frontend): fix search

* feat(frontend): output panel v2

* feat(frontend): support table actions

* feat(frontend): support table actions

* feat(frontend): support background script

* feat(frontend): fix background scripts

* feat(frontend): simplify code

* feat(frontend): fix animation

* feat(frontend): fix wording

* feat(frontend): fix bg script click

* feat(frontend): fix bg script click

* feat(frontend): fix bg script click
2023-03-14 14:42:40 +01:00
Ruben Fiszel c2a97c53cf feat(frontend): add setTab to frontend scripts 2023-03-14 14:42:26 +01:00
Ruben Fiszel eb73f2a687 fix(backend): do not cache reference to workspace scripts 2023-03-14 14:16:07 +01:00
Ruben Fiszel cd645d0935 feat(apps): tabs can be made pages or invisible + better frontend scripts reactivity 2023-03-14 12:34:08 +01:00
Ruben Fiszel ac9bd7ef8c fix flow viewer select event 2023-03-14 00:11:37 +01:00
Ruben Fiszelandrubenfiszel 5dae6577b8 chore(main): release 1.76.0 (#1280)
* chore(main): release 1.76.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-13 23:41:53 +01:00
Ruben Fiszel 372f53b7fe improve refresh and background script behavior 2023-03-13 23:38:14 +01:00
Ruben Fiszel 5662fa0d09 fix compile 2023-03-13 20:59:27 +01:00
Ruben Fiszel c958480ce8 fix(backend): improve worker ping api 2023-03-13 20:28:11 +01:00
Ruben Fiszel f0b1b1f752 feat(frontend): add frontend (JS) scripts to apps 2023-03-13 19:29:35 +01:00
Ruben Fiszel b8e6767cca improve error messages for no workspace 2023-03-13 16:44:38 +01:00
Ruben Fiszel 75f87e7e11 feat(frontend): disabled for action buttons can now depend on row 2023-03-13 14:36:42 +01:00
Ádám Kovács 3e5a179eb8 fix(frontend): General fixes and updates (#1281)
* fix(frontend): App icon select double click issue

* fix(frontend): Update script metadata page

* fix(frontend): Set dropdown default icon to vertical dots

* fix(frontend): Clean up

* fix(frontend): Update table styles

* fix(frontend): Add spacing to secondary menu items

* fix(frontend): Scale down full path

* fix(frontend): Table loading state

* fix(frontend): Hide script kind setting by default
2023-03-13 14:17:57 +01:00
Ruben Fiszel c082c6350e fix runs reload 2023-03-13 13:00:23 +01:00
Ruben Fiszel cfd489a550 feat(frontend): improve drag-n-drop behavior 2023-03-13 12:44:39 +01:00
Ruben Fiszel 1f4ae53fb4 integrate svelte-grid in codebase 2023-03-13 10:01:02 +01:00
Ruben Fiszel 0dcbf270da small app fixes 2023-03-12 21:32:41 +01:00
Faton RamadaniandRuben Fiszel 82c139ed09 feat(frontend): Copy, Cut and Paste (#1279)
* feat(frontend): add copy, paste and cut

* feat(frontend): simplify code

* feat(frontend): add apple modifiers

---------

Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
2023-03-12 20:58:06 +01:00
Ruben Fiszelandrubenfiszel 0789bef120 chore(main): release 1.75.0 (#1278)
* chore(main): release 1.75.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>
2023-03-11 20:54:54 +01:00
Ruben Fiszel 1a7dc0a3bd more cli fixes 2023-03-11 20:53:34 +01:00
Ruben Fiszel ce323709a9 fix(cli): many small fixes 2023-03-11 20:16:56 +01:00
Ruben Fiszel 61a5e1f1ac feat(frontend): make runs filters synced with query args 2023-03-11 17:13:49 +01:00
Ruben Fiszel 3b44f9a72c feat: add filter jobs by args or result 2023-03-11 15:30:46 +01:00
Ruben Fiszel b349308ff7 handle better linked variables renaming 2023-03-11 11:25:47 +01:00
Ruben Fiszel f87b722a21 apps improvements 2023-03-10 23:27:55 +01:00
Ruben Fiszel 0e9be7f300 fix for app viewer 2023-03-10 20:26:49 +01:00
Ruben Fiszel 8681e83b57 feat(apps): add resource picker 2023-03-10 20:01:00 +01:00
348 changed files with 15209 additions and 8286 deletions
+1 -1
View File
@@ -4,7 +4,7 @@ VERSION=$1
echo "Updating versions to: $VERSION"
sed -i -e "/^version =/s/= .*/= \"$VERSION\"/" backend/Cargo.toml
sed -i -e "/^const VERSION =/s/= .*/= \"v$VERSION\";/" cli/main.ts
sed -i -e "/^export const VERSION =/s/= .*/= \"v$VERSION\";/" cli/main.ts
sed -i -e "/version: /s/: .*/: $VERSION/" backend/windmill-api/openapi.yaml
sed -i -e "/version: /s/: .*/: $VERSION/" openflow.openapi.yaml
sed -i -e "/\"version\": /s/: .*,/: \"$VERSION\",/" frontend/package.json
+32
View File
@@ -0,0 +1,32 @@
name: Deploy frontend to s3
on:
push:
branches: [main]
tags: ["*"]
concurrency:
group: ${{ github.ref }}-npmbuild
cancel-in-progress: true
jobs:
npm_build:
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 18
- name: "npm build"
timeout-minutes: 15
run:
cd frontend && npm ci && npm run generate-backend-client &&
NODE_OPTIONS="--max-old-space-size=8192" npm run build
- uses: reggionick/s3-deploy@v3
with:
folder: frontend/build
bucket: windmill-frontend
bucket-region: us-east-1
+193
View File
@@ -1,6 +1,199 @@
# Changelog
## [1.84.0](https://github.com/windmill-labs/windmill/compare/v1.83.1...v1.84.0) (2023-03-31)
### Features
* add force cancel ([fbe5c18](https://github.com/windmill-labs/windmill/commit/fbe5c18da02763371e6f32c898b31a6a29984b45))
* add the ability to edit previous versions ([2368da2](https://github.com/windmill-labs/windmill/commit/2368da214660ff1835b49b4c2c87256c9bd565cf))
* **backend:** reduce memory allocation for big forloops of flows ([c7506e4](https://github.com/windmill-labs/windmill/commit/c7506e4daec5b12bf908e6954bf6f3521a97b3ba))
* **frontend:** App component style input grouping ([#1334](https://github.com/windmill-labs/windmill/issues/1334)) ([01564f0](https://github.com/windmill-labs/windmill/commit/01564f0a1c26ee9f065bb0adeb7d5e8df0b2e5b5))
* **frontend:** Display frontend execution result in Debug Runs ([#1341](https://github.com/windmill-labs/windmill/issues/1341)) ([57f8dd9](https://github.com/windmill-labs/windmill/commit/57f8dd9570577a58fe91d93c7a9d1a9b4dc69598))
* **frontend:** improve input connection UI ([#1333](https://github.com/windmill-labs/windmill/issues/1333)) ([5ac646e](https://github.com/windmill-labs/windmill/commit/5ac646e859a07efb65542aae9365aa7791ce1097))
### Bug Fixes
* **backend:** add a refresh button to workspace script/hub ([bb61cef](https://github.com/windmill-labs/windmill/commit/bb61cef0e56bf7fa7f8a5f91dabd590afd5db791))
* **backend:** backend compatability on macos ([#1340](https://github.com/windmill-labs/windmill/issues/1340)) ([dfd2abc](https://github.com/windmill-labs/windmill/commit/dfd2abc76466cddca98f93fd82be91ba5d3076e0))
* **frontend:** Export python code as string ([#1339](https://github.com/windmill-labs/windmill/issues/1339)) ([2779891](https://github.com/windmill-labs/windmill/commit/277989141100b033b26b496b8a55d97d48cf7e81))
* **frontend:** improve app tables ([cd1f9b6](https://github.com/windmill-labs/windmill/commit/cd1f9b6baa0dadfb14fee3a586a4b6b164e5e402))
* **frontend:** improve loading of big args in job details ([71619ac](https://github.com/windmill-labs/windmill/commit/71619acdfac010822c1eac496a6f3f869e6ca6fb))
* **frontend:** improve loading of big jobs in run form ([b325493](https://github.com/windmill-labs/windmill/commit/b3254938fe58d8c00a0c4347e7ef519e3a6e4031))
## [1.83.1](https://github.com/windmill-labs/windmill/compare/v1.83.0...v1.83.1) (2023-03-28)
### Bug Fixes
* **cli:** plain secrets might be undefined ([569a55e](https://github.com/windmill-labs/windmill/commit/569a55e45b34641b0fb4569387166f3aa89ce35f))
## [1.83.0](https://github.com/windmill-labs/windmill/compare/v1.82.0...v1.83.0) (2023-03-28)
### Features
* **backend:** allow relative imports for python ([a5500ea](https://github.com/windmill-labs/windmill/commit/a5500ea40a77b2e0408e2a644190a8f65b18cd1d))
* **backend:** execute /bin/bash instead of /bin/sh for bash scripts ([021fa23](https://github.com/windmill-labs/windmill/commit/021fa23f9ffcd11548977a4589eb9bc2815243cf))
* **backend:** improve relative importsfor deno ([eaac598](https://github.com/windmill-labs/windmill/commit/eaac598af308cedea8f0f8fc7c189a4640b4366b))
* **backend:** increase timeout for premium workspace ([00b70d9](https://github.com/windmill-labs/windmill/commit/00b70d9aaac8ae979782492d7754060a3c2c9567))
* **frontend:** add pagination ([33c07d3](https://github.com/windmill-labs/windmill/commit/33c07d3e63f96673719ecb15e45f4cd9e18be80e))
* **frontend:** Add quick style settings to app editor ([#1308](https://github.com/windmill-labs/windmill/issues/1308)) ([ac24862](https://github.com/windmill-labs/windmill/commit/ac2486219cd91df3a7fe11d37894797a881cac6c))
* **frontend:** add recompute as a primitive ([449d3ae](https://github.com/windmill-labs/windmill/commit/449d3ae5ddeceef3fbcb7a815a4dba16c9639fb3))
* **frontend:** add textareacomponent + fix multiselect style + select multi components ([2b31653](https://github.com/windmill-labs/windmill/commit/2b31653a8aa06807678e8609cfa62cf0f2f55dce))
* **frontend:** multiselect components for apps ([577dec5](https://github.com/windmill-labs/windmill/commit/577dec5c5733cdf88e8586ce6c27159920c69c8a))
* **frontend:** use rich json editor for arrays of objects and for object in ArgInput ([b95afaa](https://github.com/windmill-labs/windmill/commit/b95afaa9bb41b102181657453a564f44f4511983))
### Bug Fixes
* **apps:** improve app table actionButtons behavior under many clicks ([8e3d8ac](https://github.com/windmill-labs/windmill/commit/8e3d8acc80de971ee115d6903d24864d8263f08b))
* **cli:** add --plain-secrets ([98d51e2](https://github.com/windmill-labs/windmill/commit/98d51e219df1680507114f9b57ec0b0a4a234b5c))
* **frontend:** add a modal that is always mounted to make sure compon… ([#1328](https://github.com/windmill-labs/windmill/issues/1328)) ([a527cb8](https://github.com/windmill-labs/windmill/commit/a527cb8222a2ff80dae38ebae7dc5ea0979d74c5))
* **frontend:** Disable app keyboard navigation on focused inputs ([#1326](https://github.com/windmill-labs/windmill/issues/1326)) ([da24e9a](https://github.com/windmill-labs/windmill/commit/da24e9ab0625a7503c498c179022ea4011a03170))
* **frontend:** persist description for schemas ([1a48673](https://github.com/windmill-labs/windmill/commit/1a4867302f72aaae8f422ac8f53812c116cc383d))
* **frontend:** Revert app upload input ([#1330](https://github.com/windmill-labs/windmill/issues/1330)) ([fa457bb](https://github.com/windmill-labs/windmill/commit/fa457bb7099bd31c2315eaf7f7f2c40900b2ae39))
* **frontend:** Small app fixes ([#1331](https://github.com/windmill-labs/windmill/issues/1331)) ([75306c8](https://github.com/windmill-labs/windmill/commit/75306c831616d9a01cc3a4681732aab93153f1a9))
## [1.82.0](https://github.com/windmill-labs/windmill/compare/v1.81.0...v1.82.0) (2023-03-24)
### Features
* **backend:** introduce RESTART_ZOMBIE_JOBS and ZOMBIE_JOB_TIMEOUT ([47a7f71](https://github.com/windmill-labs/windmill/commit/47a7f7163aae3fe807e766c824085b4d1b75c8c8))
### Bug Fixes
* **backend:** do not consider FlowPreview as potential zombie job ([f7c30b5](https://github.com/windmill-labs/windmill/commit/f7c30b5d2f16e15f36208e07126557fd7ed84801))
* **backend:** increase dynamic js timeout + improve client passing ([34e25f0](https://github.com/windmill-labs/windmill/commit/34e25f0f96fe637cc42f4017a064c40def5d67ef))
* **cli:** improve diff speed + fix replacing cli ([b999c98](https://github.com/windmill-labs/windmill/commit/b999c9894b4011b735f37df485fe403c22c00512))
* **frontend:** Fix AppTable error display + clear errors when removing a component + properly detect that latest component run had an error ([#1322](https://github.com/windmill-labs/windmill/issues/1322)) ([c15bc8a](https://github.com/windmill-labs/windmill/commit/c15bc8a7bfb3bef2634e6093088967137cd06239))
* **frontend:** fix refresh with manual dependencies ([#1319](https://github.com/windmill-labs/windmill/issues/1319)) ([a47031a](https://github.com/windmill-labs/windmill/commit/a47031a41e6a3392101e280dcd1aea098f898447))
* **frontend:** fix settings panel ([#1323](https://github.com/windmill-labs/windmill/issues/1323)) ([30b8e47](https://github.com/windmill-labs/windmill/commit/30b8e474df5b71b7e7b36d3fe5974a289cf0dfae))
* **frontend:** Fix transformer ([#1321](https://github.com/windmill-labs/windmill/issues/1321)) ([addabcc](https://github.com/windmill-labs/windmill/commit/addabcceb0c90782ba4a934bb3822f8cc9865069))
* **frontend:** remove unnecessary div ([#1318](https://github.com/windmill-labs/windmill/issues/1318)) ([e193a0b](https://github.com/windmill-labs/windmill/commit/e193a0bcdf6690b007594d2f1325a7ec26603129))
## [1.81.0](https://github.com/windmill-labs/windmill/compare/v1.80.1...v1.81.0) (2023-03-21)
### Features
* **apps:** add action on form/button/formbutton ([2593218](https://github.com/windmill-labs/windmill/commit/2593218cbf07c05521a270797055ddb22dc22b8d))
### Bug Fixes
* **frontend:** Remove action outline on preview mode ([#1313](https://github.com/windmill-labs/windmill/issues/1313)) ([a7c4f1a](https://github.com/windmill-labs/windmill/commit/a7c4f1a12e02e8627a5955b75d572e9cf11d8122))
## [1.80.1](https://github.com/windmill-labs/windmill/compare/v1.80.0...v1.80.1) (2023-03-21)
### Bug Fixes
* **cli:** add support for non metadataed scripts ([42f6d2e](https://github.com/windmill-labs/windmill/commit/42f6d2e0ee6294f8a1d97f5f62f2adb6edfd2fed))
## [1.80.0](https://github.com/windmill-labs/windmill/compare/v1.79.0...v1.80.0) (2023-03-20)
### Features
* **apps:** add transformers for data sources ([0abacac](https://github.com/windmill-labs/windmill/commit/0abacac06c7dd586b48c66ff47b7589fe692205b))
* **frontend:** App set tab ([#1307](https://github.com/windmill-labs/windmill/issues/1307)) ([48413a7](https://github.com/windmill-labs/windmill/commit/48413a78c5e7e0ee8208711f15135d81136b7386))
### Bug Fixes
* **frontend:** add missing optional chaining ([#1306](https://github.com/windmill-labs/windmill/issues/1306)) ([29b1cc6](https://github.com/windmill-labs/windmill/commit/29b1cc6ff0ebc5edcad24a1780113889c507075d))
* **frontend:** App button triggered by ([#1304](https://github.com/windmill-labs/windmill/issues/1304)) ([cf2d031](https://github.com/windmill-labs/windmill/commit/cf2d031e8e89faa2cd7fa58436cbe7cf4d9045f9))
## [1.79.0](https://github.com/windmill-labs/windmill/compare/v1.78.0...v1.79.0) (2023-03-17)
### Features
* **frontend:** add listeners for frontend scripts ([597e38e](https://github.com/windmill-labs/windmill/commit/597e38ef367d38fa97fc443ccb2c721e5964fece))
* **frontend:** add table actions navigation ([#1298](https://github.com/windmill-labs/windmill/issues/1298)) ([c3ba1a6](https://github.com/windmill-labs/windmill/commit/c3ba1a6ab97484a08a5a20187bb858a5af7025cb))
* **frontend:** App component triggers ([#1303](https://github.com/windmill-labs/windmill/issues/1303)) ([078cb1b](https://github.com/windmill-labs/windmill/commit/078cb1bf3e4de08cb018578f04d24392a6462f69))
* **frontend:** Component control ([#1293](https://github.com/windmill-labs/windmill/issues/1293)) ([bd927a2](https://github.com/windmill-labs/windmill/commit/bd927a27ed9581dbf67ea3694f9d989f8d71d2ed))
### Bug Fixes
* **frontend:** App panel styling ([#1284](https://github.com/windmill-labs/windmill/issues/1284)) ([c1dd35c](https://github.com/windmill-labs/windmill/commit/c1dd35c3f0fcbc1be43273f82a873c3c07863417))
* **frontend:** Display app context search on top ([#1300](https://github.com/windmill-labs/windmill/issues/1300)) ([bd3ee81](https://github.com/windmill-labs/windmill/commit/bd3ee81b14846f16ccd16461de99b46fe68be6ba))
* **frontend:** fix horizontal splitpanes ([#1301](https://github.com/windmill-labs/windmill/issues/1301)) ([ea3dab4](https://github.com/windmill-labs/windmill/commit/ea3dab411b3d5dd772e04c8831e789e2470aaf28))
* **frontend:** fix map render ([#1297](https://github.com/windmill-labs/windmill/issues/1297)) ([0092721](https://github.com/windmill-labs/windmill/commit/00927210fd68c31cb793ef4f0efea05711ebcf00))
* **frontend:** Hide archive toggle with empty list ([#1296](https://github.com/windmill-labs/windmill/issues/1296)) ([bac831b](https://github.com/windmill-labs/windmill/commit/bac831b23ce85a683ddbd4537900670a0b7d12a8))
## [1.78.0](https://github.com/windmill-labs/windmill/compare/v1.77.0...v1.78.0) (2023-03-16)
### Features
* **frontend:** app textcomponent editable + tooltip ([11567d6](https://github.com/windmill-labs/windmill/commit/11567d6280ea60f1a8c3c6607c724179775cbbe3))
### Bug Fixes
* **backend:** whitelist for include_header was ignored in some cases ([183a459](https://github.com/windmill-labs/windmill/commit/183a4591df700ab4720de6e92a83631256940089))
* **frontend:** improve rendering performance after component moving ([6f890f2](https://github.com/windmill-labs/windmill/commit/6f890f2120885f90d986fbd655096b45bf9de539))
* **frontend:** remove staticOutputs from apps ([dbdfd62](https://github.com/windmill-labs/windmill/commit/dbdfd626386398180ecba7976714f86365eeccd8))
## [1.77.0](https://github.com/windmill-labs/windmill/compare/v1.76.0...v1.77.0) (2023-03-14)
### Features
* **apps:** state can be used as input in apps ([2f0acb9](https://github.com/windmill-labs/windmill/commit/2f0acb9ffa8dace4a886527dcee49809d019b271))
* **apps:** tabs can be made pages or invisible + better frontend scripts reactivity ([cd645d0](https://github.com/windmill-labs/windmill/commit/cd645d0935f2d06e0ff71f14d2cf63accd378ff3))
* **deno:** add support for custom npm repo ([#1291](https://github.com/windmill-labs/windmill/issues/1291)) ([944795f](https://github.com/windmill-labs/windmill/commit/944795f6eeaa7d01ab1a35a80570a55c363723e6))
* **frontend:** add setTab to frontend scripts ([c2a97c5](https://github.com/windmill-labs/windmill/commit/c2a97c53cfff0fdb35dd8bc249490566eebdc1a9))
* **frontend:** app components output panel ([#1283](https://github.com/windmill-labs/windmill/issues/1283)) ([751edcf](https://github.com/windmill-labs/windmill/commit/751edcf9b8e0976a1d073603c9eff5dc6e714490))
### Bug Fixes
* **backend:** do not cache reference to workspace scripts ([eb73f2a](https://github.com/windmill-labs/windmill/commit/eb73f2a687f6faad301b9038ab8585450bec7481))
* **frontend:** fix app tabs ([#1288](https://github.com/windmill-labs/windmill/issues/1288)) ([c71a577](https://github.com/windmill-labs/windmill/commit/c71a577fead90c9cd01a736b54d859ec4f0b7807))
* **frontend:** fix container deletion ([#1287](https://github.com/windmill-labs/windmill/issues/1287)) ([bc870bd](https://github.com/windmill-labs/windmill/commit/bc870bd03eb76cb8bc0e0c861f6cd8a9c661186b))
* **frontend:** Update setting accordion ([#1285](https://github.com/windmill-labs/windmill/issues/1285)) ([dea12e8](https://github.com/windmill-labs/windmill/commit/dea12e8870ece998bb6607723cbaab9b9a958f22))
## [1.76.0](https://github.com/windmill-labs/windmill/compare/v1.75.0...v1.76.0) (2023-03-13)
### Features
* **frontend:** add frontend (JS) scripts to apps ([f0b1b1f](https://github.com/windmill-labs/windmill/commit/f0b1b1f752731ba434b960a75624118152f53c00))
* **frontend:** Copy, Cut and Paste ([#1279](https://github.com/windmill-labs/windmill/issues/1279)) ([82c139e](https://github.com/windmill-labs/windmill/commit/82c139ed0992be401e250cfb7ecc0fca61b76772))
* **frontend:** disabled for action buttons can now depend on row ([75f87e7](https://github.com/windmill-labs/windmill/commit/75f87e7e1117a9c12afcf626379e94b134a9a493))
* **frontend:** improve drag-n-drop behavior ([cfd489a](https://github.com/windmill-labs/windmill/commit/cfd489a55059e7b6843f99bab261c70b3852e6a2))
### Bug Fixes
* **backend:** improve worker ping api ([c958480](https://github.com/windmill-labs/windmill/commit/c958480ce83844a989f58dd5a70eb288582e2194))
* **frontend:** General fixes and updates ([#1281](https://github.com/windmill-labs/windmill/issues/1281)) ([3e5a179](https://github.com/windmill-labs/windmill/commit/3e5a179eb8cd8001f49c92305141dade1571e20f))
## [1.75.0](https://github.com/windmill-labs/windmill/compare/v1.74.2...v1.75.0) (2023-03-11)
### Features
* add filter jobs by args or result ([3b44f9a](https://github.com/windmill-labs/windmill/commit/3b44f9a72ca0466a44963a4b9657a0ee59b44753))
* **apps:** add resource picker ([8681e83](https://github.com/windmill-labs/windmill/commit/8681e83b574141acbf7e5a389a9e8a4f340336d1))
* **bash:** add default argument handling for bash ([1d5c194](https://github.com/windmill-labs/windmill/commit/1d5c194f09ffba963d52e418c5954843d84ae337))
* **frontend-apps:** add variable picker for static string input on apps ([bc440f8](https://github.com/windmill-labs/windmill/commit/bc440f8d4154ce464c0e027d93b7a0a3b76d782e))
* **frontend:** make runs filters synced with query args ([61a5e1f](https://github.com/windmill-labs/windmill/commit/61a5e1f1accc988628b785b3b9be04c4ea719874))
### Bug Fixes
* **backend:** add killpill for lines reading ([7c825c2](https://github.com/windmill-labs/windmill/commit/7c825c212dd0f1e8be427eabd9a9756303241d1b))
* **cli:** many small fixes ([ce32370](https://github.com/windmill-labs/windmill/commit/ce323709a94d27fb24214719180ea1aafc66d646))
## [1.74.2](https://github.com/windmill-labs/windmill/compare/v1.74.1...v1.74.2) (2023-03-09)
+1 -1
View File
@@ -11,5 +11,5 @@ http://{$BASE_URL} {
https://{$BASE_URL} {
bind {$ADDRESS}
reverse_proxy /ws/* http://localhost:3001
reverse_proxy /ws/* http://lsp:3001
}
+40 -34
View File
@@ -26,7 +26,8 @@ Open-source developer infrastructure for internal tools. Self-hostable alternati
# Windmill - Turn scripts into workflows and UIs that you can share and run at scale
Windmill is <b>fully open-sourced (AGPLv3)</b> and Windmill Labs offers dedicated instance and commercial support and licenses.
Windmill is <b>fully open-sourced (AGPLv3)</b> and Windmill Labs offers
dedicated instance and commercial support and licenses.
![Windmill Diagram](/imgs/stacks.svg)
@@ -74,7 +75,7 @@ https://user-images.githubusercontent.com/275584/218350457-bc2fdc3b-e667-4da5-a2
4. Build complex UI on top of your scripts and flows.
![Step 5](./imgs/windmill-builder.png)
Scripts and flows can also be triggered by a cron schedule '*/5 * * * *' or
Scripts and flows can also be triggered by a cron schedule '_/5 _ \* \* \*' or
through webhooks.
You can build your entire infra on top of Windmill!
@@ -82,46 +83,49 @@ You can build your entire infra on top of Windmill!
## Show me some actual script code
```typescript
import * as wmill from "https://deno.land/x/windmill@v1.62.0/mod.ts"
import * as wmill from "https://deno.land/x/windmill@v1.62.0/mod.ts";
//import any dependency from npm
import cowsay from 'npm:cowsay@1.5.0'
import cowsay from "npm:cowsay@1.5.0";
export async function main(
a: number,
// unions generate enums
b: "my" | "enum",
// default parameters prefill the field
d = "default arg",
// nested objects work c = { nested: "object" },
// permissioned and typed json
db: wmill.Resource<"postgresql">) {
a: number,
// unions generate enums
b: "my" | "enum",
// default parameters prefill the field
d = "default arg",
// nested objects work c = { nested: "object" },
// permissioned and typed json
db: wmill.Resource<"postgresql">
) {
const email = Deno.env.get("WM_EMAIL");
// variables are permissioned and by path
let variable = await wmill.getVariable("f/company-folder/my_secret");
const lastTimeRun = await wmill.getState();
// logs are printed and always inspectable
console.log(cowsay.say({ text: "hello " + email + " " + lastTimeRun }));
await wmill.setState(Date.now());
const email = Deno.env.get('WM_EMAIL')
// variables are permissioned and by path
let variable = await wmill.getVariable('f/company-folder/my_secret')
const lastTimeRun = await wmill.getState()
// logs are printed and always inspectable
console.log(cowsay.say({ text: "hello " + email + " " + lastTimeRun }))
await wmill.setState(Date.now())
// return is serialized as JSON
return { foo: d, variable };
// return is serialized as JSON
return { foo: d, variable };
}
```
## CLI
We have a powerful CLI to interact with the windmill platform and sync your
scripts from local files, github repos and to run scripts and flows on the instance from local commands. See
scripts from local files, github repos and to run scripts and flows on the
instance from local commands. See
[more details](https://github.com/windmill-labs/windmill/tree/main/cli)
![CLI Screencast](./cli/vhs/output/setup.gif)
### Running scripts locally
You can run your script locally easily, you simply need to pass the right environment variables for the `wmill` client library to fetch resource and variables from your instance if necessary. See more: <https://docs.windmill.dev/docs/advanced/local_development/>
You can run your script locally easily, you simply need to pass the right
environment variables for the `wmill` client library to fetch resource and
variables from your instance if necessary. See more:
<https://docs.windmill.dev/docs/advanced/local_development/>
## Stack
@@ -187,7 +191,6 @@ compiling from source or using without a postgres super user, see
Go to http://localhost et voilà :)
The default super-admin user is: admin@windmill.dev / changeme
From there, you can create other users (do not forget to change the password!)
@@ -199,9 +202,9 @@ We publish helm charts at:
### Postgres without superuser
If you do not want, or cannot (for instance, in AWS Aurora or Cloud sql) use a postgres superuser,
you can run `./init-db-as-superuser.sql` to init the required users for windmill.
If you do not want, or cannot (for instance, in AWS Aurora or Cloud sql) use a
postgres superuser, you can run `./init-db-as-superuser.sql` to init the
required users for windmill.
### Commercial license
@@ -275,8 +278,8 @@ You may also add your own custom OAuth2 IdP and OAuth2 Resource provider:
### Resource types
You will also want to import all the approved resource types from
[WindmillHub](https://hub.windmill.dev). A setup script will prompt
you to have it being synced automatically everyday.
[WindmillHub](https://hub.windmill.dev). A setup script will prompt you to have
it being synced automatically everyday.
## Environment Variables
@@ -285,14 +288,16 @@ you to have it being synced automatically everyday.
| DATABASE_URL | | The Postgres database url. | All |
| DISABLE_NSJAIL | true | Disable Nsjail Sandboxing | Worker |
| SERVER_BIND_ADDR | 0.0.0.0 | IP Address on which to bind listening socket | Server |
| PORT | 8000 | Exposed port | Server | |
| PORT | 8000 | Exposed port | Server |
| NUM_WORKERS | 3 | The number of worker per Worker instance (set to 1 on Eks to have 1 pod = 1 worker, set to 0 for an API only instance) | Worker |
| DISABLE_SERVER | false | Binary would operate as a worker only instance | Worker |
| METRICS_ADDR | None | The socket addr at which to expose Prometheus metrics at the /metrics path. Set to "true" to expose it on port 8001 | All |
| JSON_FMT | false | Output the logs in json format instead of logfmt | All |
| BASE_URL | http://localhost:8000 | The base url that is exposed publicly to access your instance | Server |
| BASE_INTERNAL_URL | http://localhost:8000 | The base url that is reachable by your workers to talk to the Servers. This help avoiding going through the external load balancer for VPC-internal requests. | Worker |
| TIMEOUT | 300 | The timeout in seconds for the execution of a script | Worker |
| TIMEOUT | 300 | The maximum time of execution of a script. When reached, the job is failed as having timedout. | Worker |
| ZOMBIE_JOB_TIMEOUT | 30 | The timeout after which a job is considered to be zombie if the worker did not send pings about processing the job (every server check for zombie jobs every 30s) | Server |
| RESTART_ZOMBIE_JOBS | true | If true then a zombie job is restarted (in-place with the same uuid and some logs), if false the zombie job is failed | Server |
| SLEEP_QUEUE | 50 | The number of ms to sleep in between the last check for new jobs in the DB. It is multiplied by NUM_WORKERS such that in average, for one worker instance, there is one pull every SLEEP_QUEUE ms. | Worker |
| MAX_LOG_SIZE | 500000 | The maximum number of characters a job can emit (log + result) | Worker |
| DISABLE_NUSER | false | If Nsjail is enabled, disable the nsjail's `clone_newuser` setting | Worker |
@@ -301,7 +306,7 @@ you to have it being synced automatically everyday.
| S3_CACHE_BUCKET (EE only) | None | The S3 bucket to sync the cache of the workers to | Worker |
| TAR_CACHE_RATE (EE only) | 100 | The rate at which to tar the cache of the workers. 100 means every 100th job in average (uniformly randomly distributed). | Worker |
| SLACK_SIGNING_SECRET | None | The signing secret of your Slack app. See [Slack documentation](https://api.slack.com/authentication/verifying-requests-from-slack) | Server |
| COOKIE_DOMAIN | None | The domain of the cookie. If not set, the cookie will be set by the browser based on the full origin | Server | |
| COOKIE_DOMAIN | None | The domain of the cookie. If not set, the cookie will be set by the browser based on the full origin | Server |
| DENO_PATH | /usr/bin/deno | The path to the deno binary. | Worker |
| PYTHON_PATH | /usr/local/bin/python3 | The path to the python binary. | Worker |
| GO_PATH | /usr/bin/go | The path to the go binary. | Worker |
@@ -316,6 +321,7 @@ you to have it being synced automatically everyday.
| QUEUE_LIMIT_WAIT_RESULT | None | The number of max jobs in the queue before rejecting immediately the request in 'run_wait_result' endpoint. Takes precedence on the query arg. If none is specified, there are no limit. | Worker |
| DENO_AUTH_TOKENS | None | Custom DENO_AUTH_TOKENS to pass to worker to allow the use of private modules | Worker |
| DENO_FLAGS | None | Override the flags passed to deno (default --allow-all) to tighten permissions. Minimum permissions needed are "--allow-read=args.json --allow-write=result.json" | Worker |
| NPM_CONFIG_REGISTRY | None | Registry to use for NPM dependencies, set if you have a private repository you need to use instead of the default public NPM registry | Worker |
| PIP_LOCAL_DEPENDENCIES | None | Specify dependencies that are installed locally and do not need to be solved nor installed again | |
| ADDITIONAL_PYTHON_PATHS | None | Specify python paths (separated by a :) to be appended to the PYTHONPATH of the python jobs. To be used with PIP_LOCAL_DEPENDENCIES to use python codebases within Windmill | Worker |
| INCLUDE_HEADERS | None | Whitelist of headers that are passed to jobs as args (separated by a comma) | Server |
+11
View File
@@ -8,3 +8,14 @@ rustflags = [
]
incremental = true
[target.x86_64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]
[target.aarch64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]
+409 -360
View File
File diff suppressed because it is too large Load Diff
+4 -3
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.74.2"
version = "1.84.0"
authors.workspace = true
edition.workspace = true
@@ -19,7 +19,7 @@ members = [
]
[workspace.package]
version = "1.74.2"
version = "1.84.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
@@ -122,7 +122,7 @@ itertools = "^0"
regex = "^1"
deno_core = "^0"
async-recursion = "^1"
swc_common = "^0"
swc_common = "0.29.39"
swc_ecma_parser = "0.128.2"
swc_ecma_ast = "0.98.1"
base64 = "0.21.0"
@@ -153,3 +153,4 @@ async-stripe = { version = "0.14", features = [
"checkout",
] }
async_zip = { version = "0.0.11", features = ["full"] }
once_cell = "1.17.1"
@@ -0,0 +1 @@
-- Add down migration script here
File diff suppressed because it is too large Load Diff
@@ -0,0 +1 @@
-- Add down migration script here
@@ -0,0 +1,11 @@
-- Add up migration script here
CREATE POLICY see_extra_perms_user ON app FOR ALL
USING (extra_perms ? CONCAT('u/', current_setting('session.user')))
WITH CHECK ((extra_perms ->> CONCAT('u/', current_setting('session.user')))::boolean);
CREATE POLICY see_extra_perms_groups ON app FOR ALL
USING (extra_perms ?| regexp_split_to_array(current_setting('session.pgroups'), ',')::text[])
WITH CHECK (exists(
SELECT key, value FROM jsonb_each_text(extra_perms)
WHERE SPLIT_PART(key, '/', 1) = 'g' AND key = ANY(regexp_split_to_array(current_setting('session.pgroups'), ',')::text[])
AND value::boolean));
@@ -58,6 +58,9 @@ fn parse_go_typ(typ: &parser_go_ast::Expr) -> (Option<String>, Typ) {
Some((*name).to_string()),
match *name {
"int" => Typ::Int,
"int16" => Typ::Int,
"int32" => Typ::Int,
"int64" => Typ::Int,
"string" => Typ::Str(None),
"bool" => Typ::Bool,
_ => Typ::Unknown,
+14 -3
View File
@@ -181,6 +181,7 @@ static PYTHON_IMPORTS_REPLACEMENT: phf::Map<&'static str, &'static str> = phf_ma
"git" => "GitPython",
"u" => "requests",
"f" => "requests",
"." => "requests",
"shopify" => "ShopifyAPI",
"seleniumwire" => "selenium-wire",
"openbb-terminal" => "openbb[all]",
@@ -224,13 +225,22 @@ pub fn parse_python_imports(code: &str) -> error::Result<Vec<String>> {
StmtKind::Import { names } => Some(
names
.into_iter()
.map(|x| x.node.name.split('.').next().unwrap_or("").to_string())
.map(|x| {
let name = x.node.name;
if name.starts_with('.') {
".".to_string()
} else {
name.split('.').next().unwrap_or("").to_string()
}
})
.map(replace_import)
.collect::<Vec<String>>(),
),
StmtKind::ImportFrom { level: Some(i), .. } if i > 0 => {
Some(vec!["requests".to_string()])
}
StmtKind::ImportFrom { level: _, module: Some(mod_), names: _ } => {
let imprt = mod_.split('.').next().unwrap_or("").replace("_", "-");
Some(vec![replace_import(imprt)])
}
_ => None,
@@ -442,6 +452,7 @@ import os
import wmill
from zanzibar.estonie import talin
import matplotlib.pyplot as plt
from . import tests
def main():
pass
@@ -449,7 +460,7 @@ def main():
";
let r = parse_python_imports(code)?;
// println!("{}", serde_json::to_string(&r)?);
assert_eq!(r, vec!["wmill", "zanzibar", "matplotlib"]);
assert_eq!(r, vec!["wmill", "zanzibar", "matplotlib", "requests"]);
Ok(())
}
@@ -246,7 +246,7 @@ fn tstype_to_typ(ts_type: &TsType) -> (Typ, bool) {
pub fn eval_sync(code: &str) -> Result<serde_json::Value, String> {
let mut context = JsRuntime::new(RuntimeOptions::default());
let code = format!("let x = {}; x", code);
let res = context.execute_script("<anon>", &code);
let res = context.execute_script("<anon>", code);
match res {
Ok(global) => {
let scope = &mut context.handle_scope();
+176 -135
View File
@@ -2372,29 +2372,6 @@
},
"query": "INSERT INTO workspace_key\n (workspace_id, kind, key)\n VALUES ($1, 'cloud', $2)"
},
"651361530f153de49f300e4d647903794052ddc7d083d16b66d9ebd142239794": {
"describe": {
"columns": [
{
"name": "id",
"ordinal": 0,
"type_info": "Uuid"
}
],
"nullable": [
false
],
"parameters": {
"Left": [
"Varchar",
"Text",
"Uuid",
"Text"
]
}
},
"query": "UPDATE queue SET canceled = true, canceled_by = $1, canceled_reason = $2 WHERE parent_job = $3 AND workspace_id = $4 RETURNING id"
},
"653685b39d93008762818d0518b953632040122a9af98332d3fd1d12244b1b80": {
"describe": {
"columns": [],
@@ -2501,28 +2478,7 @@
},
"query": "SELECT set_config('session.user', $1, true)"
},
"6dfd1ac7168f098a9e7c92d530a470c0a156041d5081053d43d9efb00d81c464": {
"describe": {
"columns": [
{
"name": "policy",
"ordinal": 0,
"type_info": "Jsonb"
}
],
"nullable": [
false
],
"parameters": {
"Left": [
"Text",
"Text"
]
}
},
"query": "SELECT policy from app WHERE path = $1 AND workspace_id = $2"
},
"6fc2cfae9df83eb24ea33e4c9567740100f4dd2285afc3ef474fc70041b0567b": {
"6d50a8dc9cfc040b6f37b58053daa0709671e008a10ab4189114ceff66efe603": {
"describe": {
"columns": [
{
@@ -2536,9 +2492,9 @@
"type_info": "Varchar"
},
{
"name": "ping_at",
"name": "last_ping",
"ordinal": 2,
"type_info": "Timestamptz"
"type_info": "Int4"
},
{
"name": "started_at",
@@ -2559,7 +2515,7 @@
"nullable": [
false,
false,
false,
null,
false,
false,
false
@@ -2571,7 +2527,28 @@
]
}
},
"query": "SELECT * FROM worker_ping ORDER BY ping_at desc LIMIT $1 OFFSET $2"
"query": "SELECT worker, worker_instance, EXTRACT(EPOCH FROM (now() - ping_at))::integer as last_ping, started_at, ip, jobs_executed FROM worker_ping ORDER BY ping_at desc LIMIT $1 OFFSET $2"
},
"6dfd1ac7168f098a9e7c92d530a470c0a156041d5081053d43d9efb00d81c464": {
"describe": {
"columns": [
{
"name": "policy",
"ordinal": 0,
"type_info": "Jsonb"
}
],
"nullable": [
false
],
"parameters": {
"Left": [
"Text",
"Text"
]
}
},
"query": "SELECT policy from app WHERE path = $1 AND workspace_id = $2"
},
"701f215eb14ba67a79afea15d7effc0dd394ba6c4a72c95d4560c5a377015d4e": {
"describe": {
@@ -2984,20 +2961,6 @@
},
"query": "UPDATE group_ SET summary = $1 WHERE name = $2 AND workspace_id = $3"
},
"83fc9bf32b9b40987feff61fea82e20613e74efb7bf8ba52485566667ccf2284": {
"describe": {
"columns": [],
"nullable": [],
"parameters": {
"Left": [
"Varchar",
"Text",
"Text"
]
}
},
"query": "UPDATE resource SET path = $1 WHERE path = $2 AND workspace_id = $3"
},
"8543f029d9784234e4c6a6dcd7b03e62d544b98be261334ee210594e0bb839f2": {
"describe": {
"columns": [
@@ -3168,6 +3131,21 @@
},
"query": "INSERT INTO usage (id, is_workspace, month_, usage) \n VALUES ($1, false, EXTRACT(YEAR FROM current_date) * 12 + EXTRACT(MONTH FROM current_date), 0) \n ON CONFLICT (id, is_workspace, month_) DO UPDATE SET usage = usage.usage + 1 \n RETURNING usage.usage"
},
"8c0131a9cc61f2daa258d49767242bcaab6bb34a977ff7fb0c18aa9202d11f47": {
"describe": {
"columns": [],
"nullable": [],
"parameters": {
"Left": [
"Varchar",
"Jsonb",
"Text",
"Text"
]
}
},
"query": "UPDATE resource SET path = $1, value = $2 WHERE path = $3 AND workspace_id = $4"
},
"8c11511a74a41a65f448249a00ebe6964a61d00c2f7b4875a55e64741bf1f0ca": {
"describe": {
"columns": [
@@ -4142,6 +4120,30 @@
},
"query": "\n UPDATE queue\n SET flow_status = JSONB_SET(flow_status, ARRAY['failure_module'], $1)\n WHERE id = $2\n "
},
"a676fb0538e279c054130bd6bbf9584ccb8c7ed41738617895b17f338ab7f26a": {
"describe": {
"columns": [
{
"name": "id",
"ordinal": 0,
"type_info": "Uuid"
}
],
"nullable": [
false
],
"parameters": {
"Left": [
"Varchar",
"Text",
"Bool",
"Uuid",
"Text"
]
}
},
"query": "UPDATE queue SET canceled = true, canceled_by = $1, canceled_reason = $2, scheduled_for = now(), suspend = 0, running = CASE WHEN $3 THEN false ELSE running END WHERE id = $4 AND workspace_id = $5 RETURNING id"
},
"a7abe946c8d0653169e292560eb7a5aa50a9c0f590444480ac642afcb83bbffb": {
"describe": {
"columns": [
@@ -4576,6 +4578,30 @@
},
"query": "SELECT label, concat(substring(token for 10)) as token_prefix, expiration, created_at, last_used_at FROM token WHERE email = $1\n ORDER BY created_at DESC"
},
"b658b75e7a40eeace5f81a390fb254852d535726b08169cca6966df8fdbeea51": {
"describe": {
"columns": [
{
"name": "id",
"ordinal": 0,
"type_info": "Uuid"
}
],
"nullable": [
false
],
"parameters": {
"Left": [
"Varchar",
"Text",
"Bool",
"Uuid",
"Text"
]
}
},
"query": "UPDATE queue SET canceled = true, canceled_by = $1, canceled_reason = $2, running = CASE WHEN $3 THEN false ELSE running END WHERE parent_job = $4 AND workspace_id = $5 RETURNING id"
},
"b69f747eae8b13a0a8d6914a3a7ad322554fcfe62cd28c5f0a475f18dd770d61": {
"describe": {
"columns": [
@@ -4697,6 +4723,27 @@
},
"query": "SELECT null FROM queue WHERE id = $1 FOR UPDATE"
},
"be1de9116b18a40681cb8c5ec7578fd7c5f7ce77f7af63f766c2899d85fe0bef": {
"describe": {
"columns": [
{
"name": "value",
"ordinal": 0,
"type_info": "Jsonb"
}
],
"nullable": [
true
],
"parameters": {
"Left": [
"Text",
"Text"
]
}
},
"query": "SELECT value FROM resource WHERE path = $1 AND workspace_id = $2"
},
"be7a99a5bb6858323ca61dd51077010f51ba58ae76b9a413339255024dcb524d": {
"describe": {
"columns": [],
@@ -5100,55 +5147,6 @@
},
"query": "select hash from script where path = $1 AND workspace_id = $2 AND\n created_at = (SELECT max(created_at) FROM script WHERE path = $1 AND workspace_id = $2) AND\n deleted = false"
},
"cc34f0298cfdaffe892a286e6b4390c287d33298e3342ccd1e10d48a9c3cc5c0": {
"describe": {
"columns": [
{
"name": "id",
"ordinal": 0,
"type_info": "Uuid"
},
{
"name": "workspace_id",
"ordinal": 1,
"type_info": "Varchar"
},
{
"name": "last_ping",
"ordinal": 2,
"type_info": "Timestamptz"
}
],
"nullable": [
false,
false,
false
],
"parameters": {
"Left": [
"Text",
{
"Custom": {
"kind": {
"Enum": [
"script",
"preview",
"flow",
"dependencies",
"flowpreview",
"script_hub",
"identity",
"flowdependencies"
]
},
"name": "job_kind"
}
}
]
}
},
"query": "UPDATE queue SET running = false WHERE last_ping < now() - ($1 || ' seconds')::interval AND running = true AND job_kind != $2 AND same_worker = false RETURNING id, workspace_id, last_ping"
},
"d0308abac80575038203b60bb66d3b39b586939da0421a595e47c7a759616431": {
"describe": {
"columns": [],
@@ -6093,29 +6091,6 @@
},
"query": "SELECT result FROM completed_job WHERE id = $1 AND workspace_id = $2"
},
"f1f7a27738deae46de6acc10cbff701ff0c3872cc5b387e304bd9a058bb0c8a8": {
"describe": {
"columns": [
{
"name": "id",
"ordinal": 0,
"type_info": "Uuid"
}
],
"nullable": [
false
],
"parameters": {
"Left": [
"Varchar",
"Text",
"Uuid",
"Text"
]
}
},
"query": "UPDATE queue SET canceled = true, canceled_by = $1, canceled_reason = $2, scheduled_for = now(), suspend = 0 WHERE id = $3 AND workspace_id = $4 RETURNING id"
},
"f2485c69b3ab6bec11c4e1eac2934d6e49f83b71a72fe74eba7b49abc225df7c": {
"describe": {
"columns": [
@@ -6436,6 +6411,72 @@
},
"query": "SELECT client, refresh_token FROM account WHERE workspace_id = $1 AND id = $2"
},
"fc2ec6fc4e22e46cc35b00503eff1eac35a6fdbc82a634b7eb9da4d7931f3582": {
"describe": {
"columns": [
{
"name": "id",
"ordinal": 0,
"type_info": "Uuid"
},
{
"name": "workspace_id",
"ordinal": 1,
"type_info": "Varchar"
},
{
"name": "last_ping",
"ordinal": 2,
"type_info": "Timestamptz"
}
],
"nullable": [
false,
false,
false
],
"parameters": {
"Left": [
"Text",
{
"Custom": {
"kind": {
"Enum": [
"script",
"preview",
"flow",
"dependencies",
"flowpreview",
"script_hub",
"identity",
"flowdependencies"
]
},
"name": "job_kind"
}
},
{
"Custom": {
"kind": {
"Enum": [
"script",
"preview",
"flow",
"dependencies",
"flowpreview",
"script_hub",
"identity",
"flowdependencies"
]
},
"name": "job_kind"
}
}
]
}
},
"query": "UPDATE queue SET running = false, started_at = null, logs = logs || '\nRestarted job after not receiving job''s ping for too long the ' || now() || '\n\n' WHERE last_ping < now() - ($1 || ' seconds')::interval AND running = true AND job_kind != $2 AND job_kind != $3 AND same_worker = false RETURNING id, workspace_id, last_ping"
},
"fcb6ce4fab662602827281e3e85fffaa48542f8d590100d0e7eead2ad4ad4873": {
"describe": {
"columns": [],
+6 -1
View File
@@ -42,7 +42,7 @@ async fn main() -> anyhow::Result<()> {
let server_bind_address: IpAddr = std::env::var("SERVER_BIND_ADDR")
.ok()
.and_then(|x| x.parse().ok() )
.and_then(|x| x.parse().ok())
.unwrap_or(IpAddr::from(DEFAULT_SERVER_BIND_ADDR));
let port: u16 = std::env::var("PORT")
@@ -91,6 +91,8 @@ Windmill Community Edition {GIT_VERSION}
"BASE_URL",
"BASE_INTERNAL_URL",
"TIMEOUT",
"ZOMBIE_JOB_TIMEOUT",
"RESTART_ZOMBIE_JOBS",
"SLEEP_QUEUE",
"MAX_LOG_SIZE",
"SERVER_BIND_ADDR",
@@ -112,6 +114,7 @@ Windmill Community Edition {GIT_VERSION}
"QUEUE_LIMIT_WAIT_RESULT",
"DENO_AUTH_TOKENS",
"DENO_FLAGS",
"NPM_CONFIG_REGISTRY",
"PIP_LOCAL_DEPENDENCIES",
"ADDITIONAL_PYTHON_PATHS",
"INCLUDE_HEADERS",
@@ -161,6 +164,8 @@ Windmill Community Edition {GIT_VERSION}
};
futures::try_join!(shutdown_signal, server_f, metrics_f, workers_f, monitor_f)?;
} else {
tracing::info!("Nothing to do, exiting.");
}
Ok(())
}
+99 -108
View File
@@ -154,12 +154,12 @@ mod suspend_resume {
serde_json::from_value(serde_json::json!({
"modules": [{
"id": "a",
"input_transform": {
"n": { "type": "javascript", "expr": "flow_input.n", },
"port": { "type": "javascript", "expr": "flow_input.port", },
"op": { "type": "javascript", "expr": "flow_input.op ?? 'resume'", },
},
"value": {
"input_transforms": {
"n": { "type": "javascript", "expr": "flow_input.n", },
"port": { "type": "javascript", "expr": "flow_input.port", },
"op": { "type": "javascript", "expr": "flow_input.op ?? 'resume'", },
},
"type": "rawscript",
"language": "deno",
"content": "\
@@ -193,12 +193,12 @@ mod suspend_resume {
},
}, {
"id": "b",
"input_transform": {
"n": { "type": "javascript", "expr": "results.a", },
"resume": { "type": "javascript", "expr": "resume", },
"resumes": { "type": "javascript", "expr": "resumes", },
},
"value": {
"input_transforms": {
"n": { "type": "javascript", "expr": "results.a", },
"resume": { "type": "javascript", "expr": "resume", },
"resumes": { "type": "javascript", "expr": "resumes", },
},
"type": "rawscript",
"language": "deno",
"content": "export function main(n, resume, resumes) { return { n: n + 1, resume, resumes } }"
@@ -207,12 +207,12 @@ mod suspend_resume {
"required_events": 1
},
}, {
"input_transform": {
"last": { "type": "javascript", "expr": "results.b", },
"resume": { "type": "javascript", "expr": "resume", },
"resumes": { "type": "javascript", "expr": "resumes", },
},
"value": {
"input_transforms": {
"last": { "type": "javascript", "expr": "results.b", },
"resume": { "type": "javascript", "expr": "resume", },
"resumes": { "type": "javascript", "expr": "resumes", },
},
"type": "rawscript",
"language": "deno",
"content": "export function main(last, resume, resumes) { return { last, resume, resumes } }"
@@ -477,11 +477,11 @@ def main(last, port):
"iterator": { "type": "javascript", "expr": "flow_input.items" },
"skip_failures": false,
"modules": [{
"input_transform": {
"index": { "type": "javascript", "expr": "flow_input.iter.index" },
"port": { "type": "javascript", "expr": "flow_input.port" },
},
"value": {
"input_transforms": {
"index": { "type": "javascript", "expr": "flow_input.iter.index" },
"port": { "type": "javascript", "expr": "flow_input.port" },
},
"type": "rawscript",
"language": "deno",
"content": inner_step(),
@@ -490,11 +490,11 @@ def main(last, port):
},
"retry": { "constant": { "attempts": 2, "seconds": 0 } },
}, {
"input_transform": {
"last": { "type": "javascript", "expr": "results.a" },
"port": { "type": "javascript", "expr": "flow_input.port" },
},
"value": {
"input_transforms": {
"last": { "type": "javascript", "expr": "results.a" },
"port": { "type": "javascript", "expr": "flow_input.port" },
},
"type": "rawscript",
"language": "python3",
"content": last_step(),
@@ -632,7 +632,7 @@ def main(last, port):
"modules": [{
"id": "a",
"value": {
"input_transform": { "port": { "type": "javascript", "expr": "flow_input.port" } },
"input_transforms": { "port": { "type": "javascript", "expr": "flow_input.port" } },
"type": "rawscript",
"language": "python3",
"content": r#"
@@ -645,7 +645,7 @@ def main(port):
}],
"failure_module": {
"value": {
"input_transform": { "error": { "type": "javascript", "expr": "previous_result", },
"input_transforms": { "error": { "type": "javascript", "expr": "previous_result", },
"port": { "type": "javascript", "expr": "flow_input.port" } },
"type": "rawscript",
"language": "python3",
@@ -686,7 +686,7 @@ def main(error, port):
result,
json!({
"recv": 42,
"from failure module": {"error": {"name": "IndexError", "stack": " File \"/tmp/inner.py\", line 5, in main\n return sock.recv(1)[0]\n", "message": "index out of range"}},
"from failure module": {"error": {"name": "IndexError", "stack": " File \"/tmp/tmp/main/step_0.py\", line 5, in main\n return sock.recv(1)[0]\n", "message": "index out of range"}},
})
);
}
@@ -705,13 +705,13 @@ async fn test_iteration(db: Pool<Postgres>) {
"iterator": { "type": "javascript", "expr": "result.items" },
"skip_failures": false,
"modules": [{
"input_transform": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
},
},
"value": {
"input_transforms": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
},
},
"type": "rawscript",
"language": "python3",
"content": "def main(n):\n if 1 < n:\n raise StopIteration(n)",
@@ -762,13 +762,13 @@ async fn test_iteration_parallel(db: Pool<Postgres>) {
"skip_failures": false,
"parallel": true,
"modules": [{
"input_transform": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
},
},
"value": {
"input_transforms": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
},
},
"type": "rawscript",
"language": "python3",
"content": "def main(n):\n if 1 < n:\n raise StopIteration(n)",
@@ -1001,7 +1001,6 @@ async fn test_deno_flow(db: Pool<Postgres>) {
path: None,
lock: None,
},
input_transforms: Default::default(),
stop_after_if: Default::default(),
summary: Default::default(),
suspend: Default::default(),
@@ -1029,7 +1028,6 @@ async fn test_deno_flow(db: Pool<Postgres>) {
path: None,
lock: None,
},
input_transforms: Default::default(),
stop_after_if: Default::default(),
summary: Default::default(),
suspend: Default::default(),
@@ -1037,7 +1035,6 @@ async fn test_deno_flow(db: Pool<Postgres>) {
sleep: None,
}],
},
input_transforms: Default::default(),
stop_after_if: Default::default(),
summary: Default::default(),
suspend: Default::default(),
@@ -1131,7 +1128,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) {
path: None,
lock: None,
},
input_transforms: Default::default(),
stop_after_if: Default::default(),
summary: Default::default(),
suspend: Default::default(),
@@ -1147,25 +1143,24 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) {
modules: vec![
FlowModule {
id: "d".to_string(),
input_transforms: [
(
"i".to_string(),
InputTransform::Javascript {
expr: "flow_input.iter.value".to_string(),
},
),
(
"loop".to_string(),
InputTransform::Static { value: json!(true) },
),
(
"path".to_string(),
InputTransform::Static { value: json!("inner.txt") },
),
]
.into(),
value: FlowModuleValue::RawScript {
input_transforms: [].into(),
input_transforms: [
(
"i".to_string(),
InputTransform::Javascript {
expr: "flow_input.iter.value".to_string(),
},
),
(
"loop".to_string(),
InputTransform::Static { value: json!(true) },
),
(
"path".to_string(),
InputTransform::Static { value: json!("inner.txt") },
),
]
.into(),
language: ScriptLang::Deno,
content: write_file,
path: None,
@@ -1196,7 +1191,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) {
path: None,
lock: None,
},
input_transforms: [].into(),
stop_after_if: Default::default(),
summary: Default::default(),
suspend: Default::default(),
@@ -1205,7 +1199,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) {
},
],
},
input_transforms: Default::default(),
stop_after_if: Default::default(),
summary: Default::default(),
suspend: Default::default(),
@@ -1240,7 +1233,6 @@ async fn test_deno_flow_same_worker(db: Pool<Postgres>) {
path: None,
lock: None,
},
input_transforms: [].into(),
stop_after_if: Default::default(),
summary: Default::default(),
suspend: Default::default(),
@@ -1437,7 +1429,6 @@ async fn test_python_flow(db: Pool<Postgres>) {
let doubles = "def main(n): return n * 2";
let flow: FlowValue = serde_json::from_value(serde_json::json!( {
"input_transform": {},
"modules": [
{
"value": {
@@ -1453,16 +1444,16 @@ async fn test_python_flow(db: Pool<Postgres>) {
"skip_failures": false,
"modules": [{
"value": {
"input_transforms": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
},
},
"type": "rawscript",
"language": "python3",
"content": doubles,
},
"input_transform": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
},
},
}],
},
},
@@ -1495,11 +1486,11 @@ async fn test_python_flow_2(db: Pool<Postgres>) {
"modules": [
{
"value": {
"input_transforms": {},
"type": "rawscript",
"content": "import wmill\ndef main(): return \"Hello\"",
"language": "python3"
},
"input_transform": {}
}
]
}))
@@ -1678,7 +1669,7 @@ async fn test_empty_loop(db: Pool<Postgres>) {
"modules": [
{
"value": {
"input_transform": {
"input_transforms": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
@@ -1694,7 +1685,7 @@ async fn test_empty_loop(db: Pool<Postgres>) {
},
{
"value": {
"input_transform": {
"input_transforms": {
"items": {
"type": "javascript",
"expr": "results.a",
@@ -1771,13 +1762,13 @@ async fn test_empty_loop_2(db: Pool<Postgres>) {
"iterator": { "type": "static", "value": [] },
"modules": [
{
"input_transform": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
},
},
"value": {
"input_transforms": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
},
},
"type": "rawscript",
"language": "python3",
"content": "def main(n): return n",
@@ -1813,13 +1804,13 @@ async fn test_step_after_loop(db: Pool<Postgres>) {
"iterator": { "type": "static", "value": [2,3,4] },
"modules": [
{
"input_transform": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
},
},
"value": {
"input_transforms": {
"n": {
"type": "javascript",
"expr": "flow_input.iter.value",
},
},
"type": "rawscript",
"language": "python3",
"content": "def main(n): return n",
@@ -1829,13 +1820,13 @@ async fn test_step_after_loop(db: Pool<Postgres>) {
},
},
{
"input_transform": {
"items": {
"type": "javascript",
"expr": "results.a",
},
},
"value": {
"input_transforms": {
"items": {
"type": "javascript",
"expr": "results.a",
},
},
"type": "rawscript",
"language": "python3",
"content": "def main(items): return sum(items)",
@@ -1857,17 +1848,17 @@ async fn test_step_after_loop(db: Pool<Postgres>) {
fn module_add_item_to_list(i: i32, id: &str) -> serde_json::Value {
json!({
"id": format!("id_{}", i.to_string().replace("-", "_")),
"input_transform": {
"array": {
"type": "javascript",
"expr": format!("results.{id}"),
},
"i": {
"type": "static",
"value": json!(i),
}
},
"value": {
"input_transforms": {
"array": {
"type": "javascript",
"expr": format!("results.{id}"),
},
"i": {
"type": "static",
"value": json!(i),
}
},
"type": "rawscript",
"language": "deno",
"content": "export function main(array, i){ array.push(i); return array }",
@@ -1877,8 +1868,8 @@ fn module_add_item_to_list(i: i32, id: &str) -> serde_json::Value {
fn module_failure() -> serde_json::Value {
json!({
"input_transform": {},
"value": {
"input_transforms": {},
"type": "rawscript",
"language": "deno",
"content": "export function main(){ throw Error('failure') }",
@@ -2069,7 +2060,7 @@ async fn test_branchall_skip_failure(db: Pool<Postgres>) {
assert_eq!(
result,
serde_json::json!([{"error": {"name": "Error", "stack": "Error: failure\n at main (file:///tmp/inner.ts:1:31)\n at run (file:///tmp/main.ts:9:26)\n at file:///tmp/main.ts:14:1", "message": "failure"}}, [1,3]])
serde_json::json!([{"error": {"name": "Error", "stack": "Error: failure\n at main (file:///tmp/main.ts:1:31)\n at run (file:///tmp/wrapper.ts:9:26)\n at file:///tmp/wrapper.ts:14:1", "message": "failure"}}, [1,3]])
);
let flow: FlowValue = serde_json::from_value(json!({
@@ -2103,7 +2094,7 @@ async fn test_branchall_skip_failure(db: Pool<Postgres>) {
assert_eq!(
result,
serde_json::json!([ {"error": {"name": "Error", "stack": "Error: failure\n at main (file:///tmp/inner.ts:1:31)\n at run (file:///tmp/main.ts:9:26)\n at file:///tmp/main.ts:14:1", "message": "failure"}}, [1, 2]])
serde_json::json!([ {"error": {"name": "Error", "stack": "Error: failure\n at main (file:///tmp/main.ts:1:31)\n at run (file:///tmp/wrapper.ts:9:26)\n at file:///tmp/wrapper.ts:14:1", "message": "failure"}}, [1, 2]])
);
}
@@ -2236,7 +2227,7 @@ async fn test_failure_module(db: Pool<Postgres>) {
"modules": [{
"id": "a",
"value": {
"input_transform": {
"input_transforms": {
"l": { "type": "javascript", "expr": "[]", },
"n": { "type": "javascript", "expr": "flow_input.n", },
},
@@ -2247,7 +2238,7 @@ async fn test_failure_module(db: Pool<Postgres>) {
}, {
"id": "b",
"value": {
"input_transform": {
"input_transforms": {
"l": { "type": "javascript", "expr": "results.a.l", },
"n": { "type": "javascript", "expr": "flow_input.n", },
},
@@ -2257,7 +2248,7 @@ async fn test_failure_module(db: Pool<Postgres>) {
},
}, {
"value": {
"input_transform": {
"input_transforms": {
"l": { "type": "javascript", "expr": "results.b.l", },
"n": { "type": "javascript", "expr": "flow_input.n", },
},
@@ -2267,8 +2258,8 @@ async fn test_failure_module(db: Pool<Postgres>) {
},
}],
"failure_module": {
"input_transform": { "error": { "type": "javascript", "expr": "previous_result", } },
"value": {
"input_transforms": { "error": { "type": "javascript", "expr": "previous_result", } },
"type": "rawscript",
"language": "deno",
"content": "export function main(error) { return { 'from failure module': error } }",
+1
View File
@@ -69,3 +69,4 @@ async-stripe = { workspace = true, optional = true }
lazy_static.workspace = true
prometheus.workspace = true
async_zip.workspace = true
regex.workspace = true
+132 -62
View File
@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.74.2
version: 1.84.0
title: Windmill API
contact:
@@ -117,9 +117,10 @@ paths:
responses:
"200":
description: >
Successfully authenticated.
The session ID is returned in a cookie named `token` and as plaintext response.
Preferred method of authorization is through the bearer token. The cookie is only for browser convenience.
Successfully authenticated. The session ID is returned in a cookie
named `token` and as plaintext response. Preferred method of
authorization is through the bearer token. The cookie is only for
browser convenience.
headers:
Set-Cookie:
@@ -204,7 +205,6 @@ paths:
schema:
type: string
/w/{workspace}/users/is_owner/{path}:
get:
summary: is owner of path
@@ -1258,9 +1258,10 @@ paths:
responses:
"200":
description: >
Successfully authenticated.
The session ID is returned in a cookie named `token` and as plaintext response.
Preferred method of authorization is through the bearer token. The cookie is only for browser convenience.
Successfully authenticated. The session ID is returned in a cookie
named `token` and as plaintext response. Preferred method of
authorization is through the bearer token. The cookie is only for
browser convenience.
headers:
Set-Cookie:
@@ -1549,7 +1550,7 @@ paths:
required: true
content:
application/json:
schema:
schema:
type: object
properties:
value: {}
@@ -2020,7 +2021,8 @@ paths:
schema:
type: string
- name: first_parent_hash
description: mask to filter scripts whom first direct parent has exact hash
description:
mask to filter scripts whom first direct parent has exact hash
in: query
schema:
type: string
@@ -2103,7 +2105,6 @@ paths:
items:
type: string
/w/{workspace}/scripts/create:
post:
summary: create script
@@ -2278,7 +2279,8 @@ paths:
/w/{workspace}/scripts/delete/h/{hash}:
post:
summary: delete script by hash (erase content but keep hash, require admin)
summary:
delete script by hash (erase content but keep hash, require admin)
operationId: deleteScriptByHash
tags:
- script
@@ -2433,13 +2435,15 @@ paths:
type: string
format: date-time
- name: scheduled_in_secs
description: schedule the script to execute in the number of seconds starting now
description:
schedule the script to execute in the number of seconds starting now
in: query
schema:
type: integer
- $ref: "#/components/parameters/ParentJob"
- name: invisible_to_owner
description: make the run invisible to the the script owner (default false)
description:
make the run invisible to the the script owner (default false)
in: query
schema:
type: boolean
@@ -2476,7 +2480,8 @@ paths:
type: string
format: date-time
- name: scheduled_in_secs
description: schedule the script to execute in the number of seconds starting now
description:
schedule the script to execute in the number of seconds starting now
in: query
schema:
type: integer
@@ -2515,7 +2520,8 @@ paths:
type: string
format: date-time
- name: scheduled_in_secs
description: schedule the script to execute in the number of seconds starting now
description:
schedule the script to execute in the number of seconds starting now
in: query
schema:
type: integer
@@ -2835,7 +2841,6 @@ paths:
schema:
$ref: "#/components/schemas/AppWithLastVersion"
/w/{workspace}/apps/secret_of/{path}:
get:
summary: get public secret of app
@@ -3018,14 +3023,16 @@ paths:
type: string
format: date-time
- name: scheduled_in_secs
description: schedule the script to execute in the number of seconds starting now
description:
schedule the script to execute in the number of seconds starting now
in: query
schema:
type: integer
- $ref: "#/components/parameters/ParentJob"
- $ref: "#/components/parameters/IncludeHeader"
- name: invisible_to_owner
description: make the run invisible to the the flow owner (default false)
description:
make the run invisible to the the flow owner (default false)
in: query
schema:
type: boolean
@@ -3063,14 +3070,16 @@ paths:
type: string
format: date-time
- name: scheduled_in_secs
description: schedule the script to execute in the number of seconds starting now
description:
schedule the script to execute in the number of seconds starting now
in: query
schema:
type: integer
- $ref: "#/components/parameters/ParentJob"
- $ref: "#/components/parameters/IncludeHeader"
- name: invisible_to_owner
description: make the run invisible to the the script owner (default false)
description:
make the run invisible to the the script owner (default false)
in: query
schema:
type: boolean
@@ -3101,7 +3110,8 @@ paths:
- $ref: "#/components/parameters/WorkspaceId"
- $ref: "#/components/parameters/IncludeHeader"
- name: invisible_to_owner
description: make the run invisible to the the script owner (default false)
description:
make the run invisible to the the script owner (default false)
in: query
schema:
type: boolean
@@ -3132,7 +3142,8 @@ paths:
- $ref: "#/components/parameters/WorkspaceId"
- $ref: "#/components/parameters/IncludeHeader"
- name: invisible_to_owner
description: make the run invisible to the the script owner (default false)
description:
make the run invisible to the the script owner (default false)
in: query
schema:
type: boolean
@@ -3168,12 +3179,14 @@ paths:
- $ref: "#/components/parameters/ScriptExactPath"
- $ref: "#/components/parameters/ScriptStartPath"
- $ref: "#/components/parameters/ScriptExactHash"
- $ref: "#/components/parameters/CreatedBefore"
- $ref: "#/components/parameters/CreatedAfter"
- $ref: "#/components/parameters/StartedBefore"
- $ref: "#/components/parameters/StartedAfter"
- $ref: "#/components/parameters/Success"
- $ref: "#/components/parameters/JobKinds"
- $ref: "#/components/parameters/Suspended"
- $ref: "#/components/parameters/Running"
- $ref: "#/components/parameters/ArgsFilter"
- $ref: "#/components/parameters/ResultFilter"
responses:
"200":
description: All available queued jobs
@@ -3198,10 +3211,12 @@ paths:
- $ref: "#/components/parameters/ScriptExactPath"
- $ref: "#/components/parameters/ScriptStartPath"
- $ref: "#/components/parameters/ScriptExactHash"
- $ref: "#/components/parameters/CreatedBefore"
- $ref: "#/components/parameters/CreatedAfter"
- $ref: "#/components/parameters/StartedBefore"
- $ref: "#/components/parameters/StartedAfter"
- $ref: "#/components/parameters/Success"
- $ref: "#/components/parameters/JobKinds"
- $ref: "#/components/parameters/ArgsFilter"
- $ref: "#/components/parameters/ResultFilter"
- name: is_skipped
description: is the job skipped
in: query
@@ -3235,9 +3250,11 @@ paths:
- $ref: "#/components/parameters/ScriptExactPath"
- $ref: "#/components/parameters/ScriptStartPath"
- $ref: "#/components/parameters/ScriptExactHash"
- $ref: "#/components/parameters/CreatedBefore"
- $ref: "#/components/parameters/CreatedAfter"
- $ref: "#/components/parameters/StartedBefore"
- $ref: "#/components/parameters/StartedAfter"
- $ref: "#/components/parameters/JobKinds"
- $ref: "#/components/parameters/ArgsFilter"
- $ref: "#/components/parameters/ResultFilter"
- name: is_skipped
description: is the job skipped
in: query
@@ -3382,7 +3399,7 @@ paths:
schema:
$ref: "#/components/schemas/CompletedJob"
/w/{workspace}/jobs/queue/cancel/{id}:
/w/{workspace}/jobs_u/queue/cancel/{id}:
post:
summary: cancel queued job
operationId: cancelQueuedJob
@@ -3410,6 +3427,34 @@ paths:
schema:
type: string
/w/{workspace}/jobs_u/queue/force_cancel/{id}:
post:
summary: force cancel queued job
operationId: forceCancelQueuedJob
tags:
- job
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- $ref: "#/components/parameters/JobId"
requestBody:
description: reason
required: true
content:
application/json:
schema:
type: object
properties:
reason:
type: string
responses:
"200":
description: job canceled
content:
text/plain:
schema:
type: string
/w/{workspace}/jobs/job_signature/{id}/{resume_id}:
get:
summary: create an HMac signature given a job id and a resume id
@@ -3438,7 +3483,8 @@ paths:
/w/{workspace}/jobs/resume_urls/{id}/{resume_id}:
get:
summary: get resume urls given a job_id, resume_id and a nonce to resume a flow
summary:
get resume urls given a job_id, resume_id and a nonce to resume a flow
operationId: getResumeUrls
tags:
- job
@@ -3464,7 +3510,7 @@ paths:
properties:
approvalPage:
type: string
resume:
resume:
type: string
cancel:
type: string
@@ -3701,7 +3747,8 @@ paths:
- schedule
responses:
"200":
description: the preview of the next 10 time this schedule would apply to
description:
the preview of the next 10 time this schedule would apply to
content:
application/json:
schema:
@@ -3888,7 +3935,8 @@ paths:
- $ref: "#/components/parameters/WorkspaceId"
- name: only_member_of
in: query
description: only list the groups the user is member of (default false)
description:
only list the groups the user is member of (default false)
schema:
type: boolean
responses:
@@ -4076,7 +4124,8 @@ paths:
- $ref: "#/components/parameters/WorkspaceId"
- name: only_member_of
in: query
description: only list the folders the user is member of (default false)
description:
only list the folders the user is member of (default false)
schema:
type: boolean
responses:
@@ -4109,7 +4158,7 @@ paths:
type: string
owners:
type: array
items:
items:
type: string
extra_perms:
additionalProperties:
@@ -4143,7 +4192,7 @@ paths:
properties:
owners:
type: array
items:
items:
type: string
extra_perms:
additionalProperties:
@@ -4314,7 +4363,8 @@ paths:
required: true
schema:
type: string
enum: [script, group_, resource, schedule, variable, flow, folder, app]
enum:
[script, group_, resource, schedule, variable, flow, folder, app]
responses:
"200":
description: acls
@@ -4339,7 +4389,8 @@ paths:
required: true
schema:
type: string
enum: [script, group_, resource, schedule, variable, flow, folder, app]
enum:
[script, group_, resource, schedule, variable, flow, folder, app]
requestBody:
description: acl to add
required: true
@@ -4375,7 +4426,8 @@ paths:
required: true
schema:
type: string
enum: [script, group_, resource, schedule, variable, flow, folder, app]
enum:
[script, group_, resource, schedule, variable, flow, folder, app]
requestBody:
description: acl to add
required: true
@@ -4395,19 +4447,18 @@ paths:
schema:
type: string
/w/{workspace}/capture_u/{path}:
post:
summary: update flow preview capture
operationId: updateCapture
tags:
- capture
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- $ref: "#/components/parameters/Path"
responses:
"204":
description: flow preview captured
post:
summary: update flow preview capture
operationId: updateCapture
tags:
- capture
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- $ref: "#/components/parameters/Path"
responses:
"204":
description: flow preview captured
/w/{workspace}/capture/{path}:
put:
@@ -4557,7 +4608,8 @@ components:
type: integer
PerPage:
name: per_page
description: number of items to return for a given page (default 30, max 100)
description:
number of items to return for a given page (default 30, max 100)
in: query
schema:
type: integer
@@ -4575,7 +4627,9 @@ components:
type: string
ParentJob:
name: parent_job
description: The parent job that is at the origin and responsible for the execution of this script if any
description:
The parent job that is at the origin and responsible for the execution
of this script if any
in: query
schema:
type: string
@@ -4614,15 +4668,15 @@ components:
in: query
schema:
type: string
CreatedBefore:
name: created_before
StartedBefore:
name: started_before
description: filter on created before (inclusive) timestamp
in: query
schema:
type: string
format: date-time
CreatedAfter:
name: created_after
StartedAfter:
name: started_after
description: filter on created after (exclusive) timestamp
in: query
schema:
@@ -4646,6 +4700,21 @@ components:
in: query
schema:
type: boolean
ArgsFilter:
name: args
description:
filter on jobs containing those args as a json subset (@> in postgres)
in: query
schema:
type: string
ResultFilter:
name: result
description:
filter on jobs containing those result as a json subset (@> in postgres)
in: query
schema:
type: string
After:
name: after
description: filter on created after (exclusive) timestamp
@@ -4687,7 +4756,9 @@ components:
enum: [Create, Update, Delete, Execute]
JobKinds:
name: job_kinds
description: filter on job kind (values 'preview', 'script', 'dependencies', 'flow') separated by,
description:
filter on job kind (values 'preview', 'script', 'dependencies', 'flow')
separated by,
in: query
schema:
type: string
@@ -5550,9 +5621,8 @@ components:
type: string
worker_instance:
type: string
ping_at:
type: string
format: date-time
last_ping:
type: number
started_at:
type: string
format: date-time
+4
View File
@@ -320,6 +320,10 @@ async fn create_app(
app.policy.on_behalf_of = Some(username_to_permissioned_as(&authed.username));
app.policy.on_behalf_of_email = Some(authed.email);
if &app.path == "" {
return Err(Error::BadRequest("App path cannot be empty".to_string()));
}
let id = sqlx::query_scalar!(
"INSERT INTO app
(workspace_id, path, summary, policy, versions)
-29
View File
@@ -552,7 +552,6 @@ mod tests {
modules: vec![
FlowModule {
id: "a".to_string(),
input_transforms: [].into(),
value: FlowModuleValue::Script {
path: "test".to_string(),
input_transforms: [(
@@ -570,7 +569,6 @@ mod tests {
},
FlowModule {
id: "b".to_string(),
input_transforms: HashMap::new(),
value: FlowModuleValue::RawScript {
input_transforms: HashMap::new(),
content: "test".to_string(),
@@ -589,7 +587,6 @@ mod tests {
},
FlowModule {
id: "c".to_string(),
input_transforms: HashMap::new(),
value: FlowModuleValue::ForloopFlow {
iterator: InputTransform::Static { value: serde_json::json!([1, 2, 3]) },
modules: vec![],
@@ -608,7 +605,6 @@ mod tests {
],
failure_module: Some(FlowModule {
id: "d".to_string(),
input_transforms: HashMap::new(),
value: FlowModuleValue::Script {
path: "test".to_string(),
input_transforms: HashMap::new(),
@@ -695,31 +691,6 @@ mod tests {
assert_eq!(dbg!(serde_json::json!(fv)), dbg!(expect));
}
#[test]
fn test_back_compat() {
/* renamed input_transform -> input_transforms but should deserialize old name */
let s = r#"
{
"value": {
"type": "rawscript",
"content": "def main(n): return",
"language": "python3"
},
"input_transform": {
"n": {
"expr": "flow_input.iter.value",
"type": "javascript"
}
}
}
"#;
let module: FlowModule = serde_json::from_str(s).unwrap();
assert_eq!(
module.input_transforms["n"],
InputTransform::Javascript { expr: "flow_input.iter.value".to_string() }
);
}
#[test]
fn retry_serde() {
assert_eq!(Retry::default(), serde_json::from_str(r#"{}"#).unwrap());
+29 -5
View File
@@ -16,10 +16,11 @@ use axum::{
routing::{delete, get, post},
Json, Router,
};
use itertools::Itertools;
use lazy_static::lazy_static;
use regex::Regex;
use windmill_audit::{audit_log, ActionKind};
use windmill_common::{
error::{self, Error, JsonResult, Result},
error::{self, to_anyhow, Error, JsonResult, Result},
users::username_to_permissioned_as,
utils::{not_found_if_none, paginate, Pagination},
};
@@ -137,6 +138,10 @@ async fn check_name_conflict<'c>(
return Ok(());
}
lazy_static! {
static ref VALID_FOLDER_NAME: Regex = Regex::new(r#"^[a-zA-Z_0-9]+$"#).unwrap();
}
async fn create_folder(
authed: Authed,
Extension(user_db): Extension<UserDB>,
@@ -146,6 +151,12 @@ async fn create_folder(
) -> Result<String> {
let mut tx = user_db.begin(&authed).await?;
if !VALID_FOLDER_NAME.is_match(&ng.name) {
return Err(windmill_common::error::Error::BadRequest(format!(
"Folder name can only contain alphanumeric characters, underscores"
)));
}
check_name_conflict(&mut tx, &w_id, &ng.name).await?;
let owner = username_to_permissioned_as(&authed.username);
let owners = &ng.owners.unwrap_or(vec![owner.clone()]);
@@ -262,13 +273,26 @@ async fn update_folder(
sqlb.and_where_eq("workspace_id", "?".bind(&w_id));
if let Some(display_name) = ng.display_name {
sqlb.set("display_name", display_name);
sqlb.set("display_name", "?".bind(&display_name));
}
if let Some(owners) = ng.owners {
sqlb.set_str("owners", format!("{{{}}}", owners.into_iter().join(",")));
sqlb.set(
"owners",
"?".bind(&format!(
"{{{}}}",
owners
.iter()
.map(|x| format!("\"{x}\""))
.collect::<Vec<_>>()
.join(","),
)),
);
}
if let Some(extra_perms) = ng.extra_perms {
sqlb.set_str("extra_perms", extra_perms.to_string());
sqlb.set(
"extra_perms",
"?".bind(&serde_json::to_string(&extra_perms).map_err(to_anyhow)?),
);
}
sqlb.returning("*");
+98 -31
View File
@@ -34,7 +34,7 @@ use windmill_queue::{get_queued_job, push, JobKind, JobPayload, QueuedJob, RawCo
use crate::{
db::{UserDB, DB},
users::{require_owner_of_path, Authed},
users::{require_owner_of_path, Authed, OptAuthed},
variables::get_workspace_key,
BASE_URL,
};
@@ -60,7 +60,6 @@ pub fn workspaced_service() -> Router {
.route("/run/preview_flow", post(run_preview_flow_job))
.route("/list", get(list_jobs))
.route("/queue/list", get(list_queue_jobs))
.route("/queue/cancel/:id", post(cancel_job_api))
.route("/completed/list", get(list_completed_jobs))
.route("/completed/get/:id", get(get_completed_job))
.route("/completed/get_result/:id", get(get_completed_job_result))
@@ -98,6 +97,8 @@ pub fn global_service() -> Router {
)
.route("/get/:id", get(get_job))
.route("/getupdate/:id", get(get_job_update))
.route("/queue/cancel/:id", post(cancel_job_api))
.route("/queue/force_cancel/:id", post(force_cancel))
}
async fn get_result_by_id(
@@ -109,20 +110,25 @@ async fn get_result_by_id(
}
async fn cancel_job_api(
authed: Authed,
Extension(user_db): Extension<UserDB>,
OptAuthed(opt_authed): OptAuthed,
Extension(db): Extension<DB>,
Path((w_id, id)): Path<(String, Uuid)>,
Json(CancelJob { reason }): Json<CancelJob>,
) -> error::Result<String> {
let tx = user_db.begin(&authed).await?;
let tx = db.begin().await?;
let username = match opt_authed {
Some(authed) => authed.username,
None => "anonymous".to_string(),
};
let (mut tx, job_option) =
windmill_queue::cancel_job(&authed.username, reason, id, &w_id, tx).await?;
windmill_queue::cancel_job(&username, reason, id, &w_id, tx, false).await?;
if let Some(id) = job_option {
audit_log(
&mut tx,
&authed.username,
&username,
"jobs.cancel",
ActionKind::Delete,
&w_id,
@@ -145,6 +151,48 @@ async fn cancel_job_api(
}
}
async fn force_cancel(
OptAuthed(opt_authed): OptAuthed,
Extension(db): Extension<DB>,
Path((w_id, id)): Path<(String, Uuid)>,
Json(CancelJob { reason }): Json<CancelJob>,
) -> error::Result<String> {
let tx = db.begin().await?;
let username = match opt_authed {
Some(authed) => authed.username,
None => "anonymous".to_string(),
};
let (mut tx, job_option) =
windmill_queue::cancel_job(&username, reason, id, &w_id, tx, true).await?;
if let Some(id) = job_option {
audit_log(
&mut tx,
&username,
"jobs.force_cancel",
ActionKind::Delete,
&w_id,
Some(&id.to_string()),
None,
)
.await?;
tx.commit().await?;
Ok(id.to_string())
} else {
let (job_o, tx) = get_job_by_id(tx, &w_id, id).await?;
tx.commit().await?;
let err = match job_o {
Some(Job::CompletedJob(_)) => {
return Ok(format!("queued job id {} is already completed", id))
}
_ => error::Error::NotFound(format!("queued job id {} does not exist", id)),
};
Err(err)
}
}
pub async fn get_path_for_hash<'c>(
db: &mut Transaction<'c, Postgres>,
w_id: &str,
@@ -300,9 +348,6 @@ pub fn add_include_headers(
headers: HeaderMap,
mut args: serde_json::Map<String, serde_json::Value>,
) -> serde_json::Map<String, serde_json::Value> {
if include_header.is_none() {
return args;
}
let whitelist = include_header
.as_ref()
.map(|s| s.split(",").map(|s| s.to_string()).collect::<Vec<_>>())
@@ -321,19 +366,22 @@ pub fn add_include_headers(
});
args
}
#[derive(Deserialize)]
pub struct ListQueueQuery {
pub script_path_start: Option<String>,
pub script_path_exact: Option<String>,
pub script_hash: Option<String>,
pub created_by: Option<String>,
pub created_before: Option<chrono::DateTime<chrono::Utc>>,
pub created_after: Option<chrono::DateTime<chrono::Utc>>,
pub started_before: Option<chrono::DateTime<chrono::Utc>>,
pub started_after: Option<chrono::DateTime<chrono::Utc>>,
pub running: Option<bool>,
pub parent_job: Option<String>,
pub order_desc: Option<bool>,
pub job_kinds: Option<String>,
pub suspended: Option<bool>,
// filter by matching a subset of the args using base64 encoded json subset
pub args: Option<String>,
}
fn list_queue_jobs_query(w_id: &str, lq: &ListQueueQuery, fields: &[&str]) -> SqlBuilder {
@@ -362,11 +410,11 @@ fn list_queue_jobs_query(w_id: &str, lq: &ListQueueQuery, fields: &[&str]) -> Sq
if let Some(pj) = &lq.parent_job {
sqlb.and_where_eq("parent_job", "?".bind(pj));
}
if let Some(dt) = &lq.created_before {
sqlb.and_where_lt("created_at", format!("to_timestamp({})", dt.timestamp()));
if let Some(dt) = &lq.started_before {
sqlb.and_where_le("started_at", format!("to_timestamp({})", dt.timestamp()));
}
if let Some(dt) = &lq.created_after {
sqlb.and_where_gt("created_at", format!("to_timestamp({})", dt.timestamp()));
if let Some(dt) = &lq.started_after {
sqlb.and_where_ge("started_at", format!("to_timestamp({})", dt.timestamp()));
}
if let Some(s) = &lq.suspended {
@@ -376,6 +424,7 @@ fn list_queue_jobs_query(w_id: &str, lq: &ListQueueQuery, fields: &[&str]) -> Sq
sqlb.and_where_eq("suspend", 0);
}
}
if let Some(jk) = &lq.job_kinds {
sqlb.and_where_in(
"job_kind",
@@ -383,6 +432,10 @@ fn list_queue_jobs_query(w_id: &str, lq: &ListQueueQuery, fields: &[&str]) -> Sq
);
}
if let Some(args) = &lq.args {
sqlb.and_where("args @> ?".bind(&args.replace("'", "''")));
}
sqlb
}
@@ -455,13 +508,14 @@ async fn list_jobs(
script_path_exact: lq.script_path_exact,
script_hash: lq.script_hash,
created_by: lq.created_by,
created_before: lq.created_before,
created_after: lq.created_after,
started_before: lq.started_before,
started_after: lq.started_after,
running: None,
parent_job: lq.parent_job,
order_desc: Some(true),
job_kinds: lq.job_kinds,
suspended: lq.suspended,
args: lq.args,
},
&[
"'QueuedJob' as typ",
@@ -756,6 +810,7 @@ pub async fn cancel_suspended_job(
parent_flow,
&w_id,
tx,
false,
)
.await?;
if job.is_some() {
@@ -1136,15 +1191,15 @@ where
struct InPayload {
payload: Option<String>,
}
fn decode_payload<D: DeserializeOwned, T: AsRef<[u8]>>(t: T) -> anyhow::Result<D> {
let vec = base64::engine::general_purpose::URL_SAFE
.decode(t)
.context("invalid base64")?;
serde_json::from_slice(vec.as_slice()).context("invalid json")
}
}
}
fn decode_payload<D: DeserializeOwned, T: AsRef<[u8]>>(t: T) -> anyhow::Result<D> {
let vec = base64::engine::general_purpose::URL_SAFE
.decode(t)
.context("invalid base64")?;
serde_json::from_slice(vec.as_slice()).context("invalid json")
}
pub async fn run_flow_by_path(
authed: Authed,
Extension(user_db): Extension<UserDB>,
@@ -1687,11 +1742,11 @@ fn list_completed_jobs_query(
if let Some(pj) = &lq.parent_job {
sqlb.and_where_eq("parent_job", "?".bind(pj));
}
if let Some(dt) = &lq.created_before {
sqlb.and_where_lt("created_at", format!("to_timestamp({})", dt.timestamp()));
if let Some(dt) = &lq.started_before {
sqlb.and_where_le("started_at", format!("to_timestamp({})", dt.timestamp()));
}
if let Some(dt) = &lq.created_after {
sqlb.and_where_gt("created_at", format!("to_timestamp({})", dt.timestamp()));
if let Some(dt) = &lq.started_after {
sqlb.and_where_ge("started_at", format!("to_timestamp({})", dt.timestamp()));
}
if let Some(sk) = &lq.is_skipped {
sqlb.and_where_eq("is_skipped", sk);
@@ -1706,6 +1761,14 @@ fn list_completed_jobs_query(
);
}
if let Some(args) = &lq.args {
sqlb.and_where("args @> ?".bind(&args.replace("'", "''")));
}
if let Some(result) = &lq.result {
sqlb.and_where("result @> ?".bind(&result.replace("'", "''")));
}
sqlb
}
#[derive(Deserialize, Clone)]
@@ -1714,8 +1777,8 @@ pub struct ListCompletedQuery {
pub script_path_exact: Option<String>,
pub script_hash: Option<String>,
pub created_by: Option<String>,
pub created_before: Option<chrono::DateTime<chrono::Utc>>,
pub created_after: Option<chrono::DateTime<chrono::Utc>>,
pub started_before: Option<chrono::DateTime<chrono::Utc>>,
pub started_after: Option<chrono::DateTime<chrono::Utc>>,
pub success: Option<bool>,
pub parent_job: Option<String>,
pub order_desc: Option<bool>,
@@ -1723,6 +1786,10 @@ pub struct ListCompletedQuery {
pub is_skipped: Option<bool>,
pub is_flow_step: Option<bool>,
pub suspended: Option<bool>,
// filter by matching a subset of the args using base64 encoded json subset
pub args: Option<String>,
// filter by matching a subset of the result using base64 encoded json subset
pub result: Option<String>,
}
async fn list_completed_jobs(
+27 -1
View File
@@ -25,7 +25,9 @@ use axum::{
Json, Router,
};
use hyper::{header::LOCATION, StatusCode};
use lazy_static::lazy_static;
use rand::rngs::OsRng;
use regex::Regex;
use retainer::Cache;
use serde::{Deserialize, Serialize};
use sqlx::FromRow;
@@ -35,6 +37,7 @@ use tracing::{Instrument, Span};
use windmill_audit::{audit_log, ActionKind};
use windmill_common::{
error::{self, Error, JsonResult, Result},
users::SUPERADMIN_SECRET_EMAIL,
utils::{not_found_if_none, rd_string, require_admin, Pagination, StripPath},
};
use windmill_queue::CLOUD_HOSTED;
@@ -266,7 +269,7 @@ impl AuthCache {
.unwrap_or(false)
{
Some(Authed {
email: "superadmin_secret@windmill.dev".to_string(),
email: SUPERADMIN_SECRET_EMAIL.to_string(),
username: "superadmin_secret".to_string(),
is_admin: true,
groups: Vec::new(),
@@ -997,6 +1000,10 @@ async fn decline_invite(
}
}
lazy_static! {
pub static ref VALID_USERNAME: Regex = Regex::new(r#"^[a-zA-Z_0-9]+$"#).unwrap();
}
async fn accept_invite(
Authed { email, .. }: Authed,
Extension(db): Extension<DB>,
@@ -1005,6 +1012,13 @@ async fn accept_invite(
if &nu.username == "bot" {
return Err(Error::BadRequest("bot is a reserved username".to_string()));
}
if !VALID_USERNAME.is_match(&nu.username) {
return Err(windmill_common::error::Error::BadRequest(format!(
"Usermame can only contain alphanumeric characters and underscores"
)));
}
let mut tx = db.begin().await?;
let r = sqlx::query!(
@@ -1077,6 +1091,12 @@ async fn add_user_to_workspace<'c>(
)));
}
if !VALID_USERNAME.is_match(username) {
return Err(windmill_common::error::Error::BadRequest(format!(
"Usermame can only contain alphanumeric characters and underscores"
)));
}
let already_exists_email = sqlx::query_scalar!(
"SELECT EXISTS(SELECT 1 FROM usr WHERE workspace_id = $1 AND email = $2)",
&w_id,
@@ -1282,6 +1302,12 @@ async fn create_user(
require_super_admin(&mut tx, &email).await?;
if nu.email == SUPERADMIN_SECRET_EMAIL {
return Err(Error::BadRequest(
"The superadmin email is a reserved email".into(),
));
}
sqlx::query!(
"INSERT INTO password(email, verified, password_hash, login_type, super_admin, name, \
company)
+11 -3
View File
@@ -7,16 +7,24 @@
*/
use sqlx::{Postgres, Transaction};
use windmill_common::error::{self, Error};
use windmill_common::{
error::{self, Error},
users::SUPERADMIN_SECRET_EMAIL,
};
pub async fn require_super_admin<'c>(
db: &mut Transaction<'c, Postgres>,
email: &str,
) -> error::Result<()> {
if email == SUPERADMIN_SECRET_EMAIL {
return Ok(());
}
let is_admin = sqlx::query_scalar!("SELECT super_admin FROM password WHERE email = $1", email)
.fetch_one(db)
.fetch_optional(db)
.await
.map_err(|e| Error::InternalErr(format!("fetching super admin: {e}")))?;
.map_err(|e| Error::InternalErr(format!("fetching super admin: {e}")))?
.unwrap_or(false);
if !is_admin {
Err(Error::NotAuthorized(
"This endpoint require caller to be a super admin".to_owned(),
+37 -1
View File
@@ -26,6 +26,7 @@ use axum::{
Json, Router,
};
use hyper::StatusCode;
use serde_json::Value;
use windmill_audit::{audit_log, ActionKind};
use windmill_common::{
error::{Error, JsonResult, Result},
@@ -378,9 +379,27 @@ async fn update_variable(
if !authed.is_admin {
require_owner_of_path(&w_id, &authed.username, &authed.groups, &path, &db).await?;
}
let mut v = sqlx::query_scalar!(
"SELECT value FROM resource WHERE path = $1 AND workspace_id = $2",
path,
w_id
)
.fetch_optional(&mut tx)
.await?
.flatten();
if let Some(old_v) = v {
v = Some(replace_path(
old_v,
&format!("$var:{path}"),
&format!("$var:{npath}"),
))
}
sqlx::query!(
"UPDATE resource SET path = $1 WHERE path = $2 AND workspace_id = $3",
"UPDATE resource SET path = $1, value = $2 WHERE path = $3 AND workspace_id = $4",
npath,
v,
path,
w_id
)
@@ -419,6 +438,23 @@ async fn update_variable(
Ok(format!("variable {} updated (npath: {:?})", path, npath))
}
fn replace_path(v: serde_json::Value, path: &str, npath: &str) -> Value {
match v {
Value::Object(v) => Value::Object(
v.into_iter()
.map(|(k, v)| (k, replace_path(v, path, npath)))
.collect(),
),
Value::Array(arr) => Value::Array(
arr.into_iter()
.map(|v| replace_path(v, path, npath))
.collect(),
),
Value::String(s) if s == path => Value::String(npath.to_owned()),
_ => v,
}
}
pub async fn build_crypt<'c>(
db: &mut Transaction<'c, Postgres>,
w_id: &str,
+2 -2
View File
@@ -28,7 +28,7 @@ pub fn global_service() -> Router {
struct WorkerPing {
worker: String,
worker_instance: String,
ping_at: chrono::DateTime<chrono::Utc>,
last_ping: Option<i32>,
started_at: chrono::DateTime<chrono::Utc>,
ip: String,
jobs_executed: i32,
@@ -45,7 +45,7 @@ async fn list_worker_pings(
let rows = sqlx::query_as!(
WorkerPing,
"SELECT * FROM worker_ping ORDER BY ping_at desc LIMIT $1 OFFSET $2",
"SELECT worker, worker_instance, EXTRACT(EPOCH FROM (now() - ping_at))::integer as last_ping, started_at, ip, jobs_executed FROM worker_ping ORDER BY ping_at desc LIMIT $1 OFFSET $2",
per_page as i64,
offset as i64
)
+31 -3
View File
@@ -16,8 +16,9 @@ use crate::{
db::{UserDB, DB},
folders::Folder,
resources::{Resource, ResourceType},
users::{Authed, WorkspaceInvite, NEW_USER_WEBHOOK},
users::{Authed, WorkspaceInvite, NEW_USER_WEBHOOK, VALID_USERNAME},
utils::require_super_admin,
variables::build_crypt,
HTTP_CLIENT,
};
#[cfg(feature = "enterprise")]
@@ -30,6 +31,7 @@ use axum::{
routing::{delete, get, post},
Json, Router,
};
use magic_crypt::MagicCryptTrait;
#[cfg(feature = "enterprise")]
use stripe::CustomerId;
use windmill_audit::{audit_log, ActionKind};
@@ -986,6 +988,11 @@ async fn add_user(
require_admin(is_admin, &username)?;
let mut tx = db.begin().await?;
if !VALID_USERNAME.is_match(&nu.username) {
return Err(windmill_common::error::Error::BadRequest(format!(
"Usermame can only contain alphanumeric characters and underscores"
)));
}
sqlx::query!(
"INSERT INTO usr
@@ -1000,6 +1007,16 @@ async fn add_user(
.execute(&mut tx)
.await?;
sqlx::query_as!(
Group,
"INSERT INTO usr_to_group (workspace_id, usr, group_) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING",
&w_id,
nu.username,
"all",
)
.execute(&mut tx)
.await?;
tx.commit().await?;
if let Some(new_user_webhook) = NEW_USER_WEBHOOK.clone() {
@@ -1073,6 +1090,7 @@ struct ScriptMetadata {
schema: Option<Schema>,
is_template: bool,
lock: Vec<String>,
kind: String,
}
enum ArchiveImpl {
@@ -1124,6 +1142,7 @@ impl ArchiveImpl {
#[derive(Deserialize)]
struct ArchiveQueryParams {
archive_type: Option<String>,
plain_secret: Option<bool>,
}
#[inline]
@@ -1169,7 +1188,7 @@ async fn tarball_workspace(
authed: Authed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Query(ArchiveQueryParams { archive_type }): Query<ArchiveQueryParams>,
Query(ArchiveQueryParams { archive_type, plain_secret }): Query<ArchiveQueryParams>,
) -> Result<([(headers::HeaderName, String); 2], impl IntoResponse)> {
require_admin(authed.is_admin, &authed.username)?;
@@ -1235,6 +1254,7 @@ async fn tarball_workspace(
description: script.description,
schema: script.schema,
is_template: script.is_template,
kind: script.kind.to_string(),
lock,
};
let metadata_str = serde_json::to_string_pretty(&metadata).unwrap();
@@ -1305,7 +1325,15 @@ async fn tarball_workspace(
.fetch_all(&db)
.await?;
for var in variables {
let mc = build_crypt(&mut db.begin().await?, &w_id).await?;
for mut var in variables {
if plain_secret.unwrap_or(false) && var.value.is_some() && var.is_secret {
var.value = Some(
mc.decrypt_base64_to_string(var.value.unwrap())
.map_err(|e| Error::InternalErr(e.to_string()))?,
);
}
let var_str = &to_string_without_metadata(&var, false).unwrap();
archive
.write_to_archive(&var_str, &format!("{}.variable.json", var.path))
+14 -6
View File
@@ -6,9 +6,12 @@
* LICENSE-AGPL for a copy of the license.
*/
use std::{collections::HashMap, time::Duration};
use std::{
collections::{BTreeMap, HashMap},
time::Duration,
};
use serde::{self, Deserialize, Serialize};
use serde::{self, Deserialize, Serialize, Serializer};
use crate::{
more_serde::{
@@ -149,9 +152,6 @@ pub struct Suspend {
pub struct FlowModule {
#[serde(default = "default_id")]
pub id: String,
#[serde(default)]
#[serde(alias = "input_transform")]
pub input_transforms: HashMap<String, InputTransform>,
pub value: FlowModuleValue,
#[serde(skip_serializing_if = "Option::is_none")]
pub stop_after_if: Option<StopAfterIf>,
@@ -245,7 +245,7 @@ pub enum FlowModuleValue {
},
RawScript {
#[serde(default)]
#[serde(alias = "input_transform")]
#[serde(alias = "input_transform", serialize_with = "ordered_map")]
input_transforms: HashMap<String, InputTransform>,
content: String,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -257,6 +257,14 @@ pub enum FlowModuleValue {
Identity,
}
fn ordered_map<S>(value: &HashMap<String, InputTransform>, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
let ordered: BTreeMap<_, _> = value.iter().collect();
ordered.serialize(serializer)
}
#[derive(Deserialize)]
pub struct ListFlowQuery {
pub path_start: Option<String>,
+1
View File
@@ -0,0 +1 @@
+13 -1
View File
@@ -7,7 +7,7 @@
*/
use std::{
fmt::Display,
fmt::{self, Display},
hash::{Hash, Hasher},
};
@@ -103,6 +103,18 @@ pub enum ScriptKind {
Approval,
}
impl Display for ScriptKind {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
fmt.write_str(match self {
ScriptKind::Trigger => "trigger",
ScriptKind::Failure => "failure",
ScriptKind::Script => "script",
ScriptKind::Approval => "approval",
})?;
Ok(())
}
}
#[derive(Serialize)]
#[cfg_attr(feature = "sqlx", derive(sqlx::FromRow))]
pub struct Script {
+2
View File
@@ -6,6 +6,8 @@
* LICENSE-AGPL for a copy of the license.
*/
pub const SUPERADMIN_SECRET_EMAIL: &str = "superadmin_secret@windmill.dev";
pub fn username_to_permissioned_as(user: &str) -> String {
if user.contains('@') {
user.to_string()
+10 -8
View File
@@ -58,12 +58,14 @@ pub async fn cancel_job<'c>(
id: Uuid,
w_id: &str,
mut tx: Transaction<'c, Postgres>,
force_rerun: bool,
) -> error::Result<(Transaction<'c, Postgres>, Option<Uuid>)> {
let job_option = sqlx::query_scalar!(
"UPDATE queue SET canceled = true, canceled_by = $1, canceled_reason = $2, scheduled_for = now(), suspend = 0 WHERE id = $3 \
AND workspace_id = $4 RETURNING id",
"UPDATE queue SET canceled = true, canceled_by = $1, canceled_reason = $2, scheduled_for = now(), suspend = 0, running = CASE WHEN $3 THEN false ELSE running END WHERE id = $4 \
AND workspace_id = $5 RETURNING id",
username,
reason,
force_rerun,
id,
w_id
)
@@ -73,10 +75,11 @@ pub async fn cancel_job<'c>(
while !jobs.is_empty() {
let p_job = jobs.pop();
let new_jobs = sqlx::query_scalar!(
"UPDATE queue SET canceled = true, canceled_by = $1, canceled_reason = $2 WHERE parent_job = $3 \
AND workspace_id = $4 RETURNING id",
"UPDATE queue SET canceled = true, canceled_by = $1, canceled_reason = $2, running = CASE WHEN $3 THEN false ELSE running END WHERE parent_job = $4 \
AND workspace_id = $5 RETURNING id",
username,
reason,
force_rerun,
p_job,
w_id
)
@@ -262,8 +265,8 @@ pub async fn push<'c>(
let job_id: Uuid = Ulid::new().into();
if cfg!(feature = "enterprise") {
let premium_workspace =
sqlx::query_scalar!("SELECT premium FROM workspace WHERE id = $1", workspace_id)
let premium_workspace = *CLOUD_HOSTED
&& sqlx::query_scalar!("SELECT premium FROM workspace WHERE id = $1", workspace_id)
.fetch_one(&mut tx)
.await
.map_err(|e| {
@@ -486,7 +489,6 @@ pub async fn push<'c>(
modules.push(FlowModule {
id: format!("{}-v", flow.modules[flow.modules.len() - 1].id),
value: FlowModuleValue::Identity,
input_transforms: HashMap::new(),
stop_after_if: None,
summary: Some(
"Virtual module needed for suspend/sleep when last module".to_string(),
@@ -658,7 +660,7 @@ impl QueuedJob {
self.script_path
.as_ref()
.map(String::as_str)
.unwrap_or("NO_FLOW_PATH")
.unwrap_or("tmp/main")
}
}
+1
View File
@@ -48,3 +48,4 @@ deno_core.workspace = true
const_format.workspace = true
git-version.workspace = true
dyn-iter.workspace = true
once_cell.workspace = true
@@ -61,8 +61,8 @@ mount {
mount {
src: "{JOB_DIR}/inner.ts"
dst: "/tmp/inner.ts"
src: "{JOB_DIR}/wrapper.ts"
dst: "/tmp/wrapper.ts"
is_bind: true
}
@@ -78,12 +78,6 @@ mount {
is_bind: true
}
mount {
src: "{JOB_DIR}/lock.json"
dst: "/tmp/lock.json"
is_bind: true
}
mount {
src: "{JOB_DIR}/args.json"
dst: "/tmp/args.json"
@@ -66,8 +66,8 @@ mount {
}
mount {
src: "{JOB_DIR}/inner.py"
dst: "/tmp/inner.py"
src: "{JOB_DIR}/{MAIN}.py"
dst: "/tmp/{MAIN}.py"
is_bind: true
}
@@ -79,8 +79,8 @@ mount {
}
mount {
src: "{JOB_DIR}/main.py"
dst: "/tmp/main.py"
src: "{JOB_DIR}/wrapper.py"
dst: "/tmp/wrapper.py"
is_bind: true
}
+3 -3
View File
@@ -10,7 +10,7 @@ use sqlx::{Pool, Postgres, Transaction};
use tracing::instrument;
use uuid::Uuid;
use windmill_common::{error::Error, flow_status::FlowStatusModule, schedule::Schedule};
use windmill_queue::{delete_job, schedule::get_schedule_opt, JobKind, QueuedJob};
use windmill_queue::{delete_job, schedule::get_schedule_opt, JobKind, QueuedJob, CLOUD_HOSTED};
#[instrument(level = "trace", skip_all)]
pub async fn add_completed_job_error(
@@ -175,8 +175,8 @@ pub async fn add_completed_job(
let additional_usage = duration.unwrap() as i32 / 1000;
let w_id = &queued_job.workspace_id;
let premium_workspace =
sqlx::query_scalar!("SELECT premium FROM workspace WHERE id = $1", w_id)
let premium_workspace = *CLOUD_HOSTED
&& sqlx::query_scalar!("SELECT premium FROM workspace WHERE id = $1", w_id)
.fetch_one(db)
.await
.map_err(|e| Error::InternalErr(format!("fetching if {w_id} is premium: {e}")))?;
+97 -80
View File
@@ -6,9 +6,11 @@
* LICENSE-AGPL for a copy of the license.
*/
use std::collections::HashMap;
use std::{cell::RefCell, collections::HashMap, rc::Rc};
use deno_core::{op, serde_v8, v8, v8::IsolateHandle, Extension, JsRuntime, RuntimeOptions};
use deno_core::{
op, serde_v8, v8, v8::IsolateHandle, Extension, JsRuntime, OpState, RuntimeOptions,
};
use itertools::Itertools;
use lazy_static::lazy_static;
use regex::Regex;
@@ -17,10 +19,7 @@ use tokio::{sync::oneshot, time::timeout};
use uuid::Uuid;
use windmill_common::{error::Error, flow_status::JobResult};
pub struct EvalCreds {
pub workspace: String,
pub token: String,
}
use crate::AuthedClient;
#[derive(Debug, Clone)]
pub struct IdContext {
@@ -29,22 +28,23 @@ pub struct IdContext {
pub previous_id: String,
}
pub struct OptAuthedClient(Option<AuthedClient>);
pub async fn eval_timeout(
expr: String,
env: Vec<(String, serde_json::Value)>,
creds: Option<EvalCreds>,
authed_client: Option<&AuthedClient>,
by_id: Option<IdContext>,
base_internal_url: &str,
) -> anyhow::Result<serde_json::Value> {
let expr2 = expr.clone();
let (sender, mut receiver) = oneshot::channel::<IsolateHandle>();
let base_internal_url: String = base_internal_url.to_string();
let has_client = authed_client.is_some();
let authed_client = authed_client.cloned();
timeout(
std::time::Duration::from_millis(3000),
std::time::Duration::from_millis(10000),
tokio::task::spawn_blocking(move || {
let mut ops = vec![];
if creds.is_some() {
if authed_client.is_some() {
ops.extend([
// An op for summing an array of numbers
// The op-layer automatically deserializes inputs
@@ -54,12 +54,12 @@ pub async fn eval_timeout(
])
}
if by_id.is_some() {
if by_id.is_some() && authed_client.is_some() {
ops.push(op_get_result::decl());
ops.push(op_get_id::decl());
}
let ext = Extension::builder().ops(ops).build();
let ext = Extension::builder("js_eval").ops(ops).build();
// Use our snapshot to provision our new runtime
let options = RuntimeOptions {
extensions: vec![ext],
@@ -68,6 +68,11 @@ pub async fn eval_timeout(
};
let mut js_runtime = JsRuntime::new(options);
{
let op_state = js_runtime.op_state();
let mut op_state = op_state.borrow_mut();
op_state.put(OptAuthedClient(authed_client.clone()));
}
sender
.send(js_runtime.v8_isolate().thread_safe_handle())
@@ -86,14 +91,7 @@ pub async fn eval_timeout(
let expr = replace_with_await_result(expr);
let r = runtime.block_on(eval(
&mut js_runtime,
&expr,
env,
creds,
by_id,
&base_internal_url,
))?;
let r = runtime.block_on(eval(&mut js_runtime, &expr, env, by_id, has_client))?;
Ok(r) as anyhow::Result<Value>
}),
@@ -104,7 +102,7 @@ pub async fn eval_timeout(
isolate.terminate_execution();
};
Error::ExecutionErr(format!(
"The expression of evaluation `{expr2}` took too long to execute (>3000ms)"
"The expression of evaluation `{expr2}` took too long to execute (>10000ms)"
))
})??
}
@@ -150,9 +148,8 @@ async fn eval(
context: &mut JsRuntime,
expr: &str,
env: Vec<(String, serde_json::Value)>,
creds: Option<EvalCreds>,
by_id: Option<IdContext>,
base_internal_url: &str,
has_client: bool,
) -> anyhow::Result<serde_json::Value> {
let exprs = expr
.trim()
@@ -169,7 +166,7 @@ async fn eval(
exprs.last().unwrap()
)
};
let (api_code, by_id_code) = if let Some(EvalCreds { workspace, token }) = creds {
let (api_code, by_id_code) = if has_client {
let by_id_code = if let Some(by_id) = by_id {
format!(
r#"
@@ -186,12 +183,12 @@ async function result_by_id(node_id) {{
}}
}} else {{
let flow_job_id = "{}";
return await Deno.core.opAsync("op_get_id", [workspace, flow_job_id, token, base_url, node_id]);
return await Deno.core.opAsync("op_get_id", [ flow_job_id, node_id]);
}}
}}
async function get_result(id) {{
return await Deno.core.opAsync("op_get_result", [workspace, id, token, base_url]);
return await Deno.core.opAsync("op_get_result", [id]);
}}
const results = new Proxy({{}}, {{
get: function(target, name, receiver) {{
@@ -222,17 +219,13 @@ const results = new Proxy({{}}, {{
let api_code = format!(
r#"
let workspace = "{workspace}";
let base_url = "{}";
let token = "{token}";
async function variable(path) {{
return await Deno.core.opAsync("op_variable", [workspace, path, token, base_url]);
return await Deno.core.opAsync("op_variable", [path]);
}}
async function resource(path) {{
return await Deno.core.opAsync("op_resource", [workspace, path, token, base_url]);
return await Deno.core.opAsync("op_resource", [path]);
}}
"#,
base_internal_url,
);
(api_code, by_id_code)
} else {
@@ -259,7 +252,7 @@ async function resource(path) {{
.join(""),
);
tracing::debug!("{}", code);
let global = context.execute_script("<anon>", &code)?;
let global = context.execute_script("<anon>", code)?;
let global = context.resolve_value(global).await?;
let scope = &mut context.handle_scope();
@@ -281,59 +274,83 @@ async function resource(path) {{
// TODO: Can we a) share the api configuration here somehow or b) just implement this natively in deno, via the deno client?
#[op]
async fn op_variable(args: Vec<String>) -> Result<String, anyhow::Error> {
let workspace = &args[0];
let path = &args[1];
let token = &args[2];
let base_url = &args[3];
let client = windmill_api_client::create_client(base_url, token.clone());
let result = client.get_variable(workspace, path, None).await?;
Ok(result.into_inner().value.unwrap_or_else(|| "".to_owned()))
async fn op_variable(
op_state: Rc<RefCell<OpState>>,
args: Vec<String>,
) -> Result<String, anyhow::Error> {
let path = &args[0];
let client = op_state.borrow().borrow::<OptAuthedClient>().0.clone();
if let Some(client) = client {
let result = client
.get_client()
.get_variable(&client.workspace, path, None)
.await?;
Ok(result.into_inner().value.unwrap_or_else(|| "".to_owned()))
} else {
anyhow::bail!("No client found in op state");
}
}
#[op]
async fn op_get_result(args: Vec<String>) -> Result<serde_json::Value, anyhow::Error> {
let workspace = &args[0];
let id = &args[1];
let token = &args[2];
let base_url = &args[3];
let client = windmill_api_client::create_client(base_url, token.clone());
let result = client
.get_completed_job_result(workspace, &id.parse()?)
.await?
.clone();
Ok(serde_json::json!(result))
async fn op_get_result(
op_state: Rc<RefCell<OpState>>,
args: Vec<String>,
) -> Result<serde_json::Value, anyhow::Error> {
let id = &args[0];
let client = op_state.borrow().borrow::<OptAuthedClient>().0.clone();
if let Some(client) = client {
let result = client
.get_client()
.get_completed_job_result(&client.workspace, &id.parse()?)
.await?
.clone();
Ok(serde_json::json!(result))
} else {
anyhow::bail!("No client found in op state");
}
}
#[op]
async fn op_get_id(args: Vec<String>) -> Result<Option<serde_json::Value>, anyhow::Error> {
let workspace = &args[0];
let flow_job_id = &args[1];
let token = &args[2];
let base_url = &args[3];
let node_id = &args[4];
async fn op_get_id(
op_state: Rc<RefCell<OpState>>,
args: Vec<String>,
) -> Result<Option<serde_json::Value>, anyhow::Error> {
let flow_job_id = &args[0];
let node_id = &args[1];
let client = windmill_api_client::create_client(base_url, token.clone());
let result = client
.result_by_id(workspace, flow_job_id, node_id)
.await
.map_or(None, |e| Some(e.into_inner()));
Ok(result)
let client = op_state.borrow().borrow::<OptAuthedClient>().0.clone();
if let Some(client) = client {
let result = client
.get_client()
.result_by_id(&client.workspace, flow_job_id, node_id)
.await
.map_or(None, |e| Some(e.into_inner()));
Ok(result)
} else {
anyhow::bail!("No client found in op state");
}
}
#[op]
async fn op_resource(args: Vec<String>) -> Result<serde_json::Value, anyhow::Error> {
let workspace = &args[0];
let path = &args[1];
let token = &args[2];
let base_url = &args[3];
let client = windmill_api_client::create_client(base_url, token.clone());
let result = client.get_resource(workspace, path).await?;
Ok(result
.into_inner()
.value
.unwrap_or_else(|| serde_json::json!({})))
async fn op_resource(
op_state: Rc<RefCell<OpState>>,
args: Vec<String>,
) -> Result<serde_json::Value, anyhow::Error> {
let path = &args[0];
let client = op_state.borrow().borrow::<OptAuthedClient>().0.clone();
if let Some(client) = client {
let result = client
.get_client()
.get_resource(&client.workspace, path)
.await?;
Ok(result
.into_inner()
.value
.unwrap_or_else(|| serde_json::json!({})))
} else {
anyhow::bail!("No client found in op state");
}
}
#[cfg(test)]
@@ -353,7 +370,7 @@ mod tests {
let code = "value.test + params.test";
let mut runtime = JsRuntime::new(RuntimeOptions::default());
let res = eval(&mut runtime, code, env, None, None, String::new().as_str()).await?;
let res = eval(&mut runtime, code, env, None, false).await?;
assert_eq!(res, json!(4));
Ok(())
}
@@ -366,7 +383,7 @@ mod tests {
multiline template`";
let mut runtime = JsRuntime::new(RuntimeOptions::default());
let res = eval(&mut runtime, code, env, None, None, String::new().as_str()).await?;
let res = eval(&mut runtime, code, env, None, false).await?;
assert_eq!(res, json!("my 5\nmultiline template"));
Ok(())
}
@@ -379,7 +396,7 @@ multiline template`";
];
let code = r#"params.test"#;
let res = eval_timeout(code.to_string(), env, None, None, String::new().as_str()).await?;
let res = eval_timeout(code.to_string(), env, None, None).await?;
assert_eq!(res, json!(2));
Ok(())
}
+195 -106
View File
@@ -9,8 +9,10 @@
use const_format::concatcp;
use itertools::Itertools;
use lazy_static::lazy_static;
use once_cell::sync::OnceCell;
use regex::Regex;
use sqlx::{Pool, Postgres, Transaction};
use windmill_api_client::Client;
use std::{
borrow::Borrow, collections::HashMap, io, os::unix::process::ExitStatusExt, panic,
process::Stdio, time::Duration,
@@ -23,7 +25,7 @@ use windmill_common::{
flows::{FlowModuleValue, FlowValue},
scripts::{ScriptHash, ScriptLang},
utils::rd_string,
variables, BASE_URL,
variables, BASE_URL, users::SUPERADMIN_SECRET_EMAIL,
};
use windmill_queue::{canceled_job_to_result, get_queued_job, pull, JobKind, QueuedJob, CLOUD_HOSTED};
@@ -34,7 +36,7 @@ use tokio::{
io::{AsyncBufReadExt, AsyncReadExt, AsyncWriteExt, BufReader},
process::{Child, Command},
sync::{
mpsc::{self, Sender}, watch, broadcast,
mpsc::{self, Sender}, watch, broadcast
},
time::{interval, sleep, Instant, MissedTickBehavior},
};
@@ -236,11 +238,10 @@ pub async fn create_token_for_owner<'c>(
) -> error::Result<(Transaction<'c, Postgres>, String)> {
// TODO: Bad implementation. We should not have access to this DB here.
let token: String = rd_string(30);
let is_super_admin = owner.contains('@')
&& sqlx::query_scalar!("SELECT super_admin FROM password WHERE email = $1", owner)
let is_super_admin = sqlx::query_scalar!("SELECT super_admin FROM password WHERE email = $1", email)
.fetch_optional(&mut tx)
.await?
.unwrap_or(false);
.unwrap_or(false) || email == SUPERADMIN_SECRET_EMAIL;
sqlx::query_scalar!(
"INSERT INTO token
@@ -264,7 +265,7 @@ const ROOT_CACHE_DIR: &str = "/tmp/windmill/cache/";
const PIP_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "pip");
const DENO_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "deno");
const GO_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "go");
const NUM_SECS_ENV_CHECK: u64 = 15;
const NUM_SECS_PING: u64 = 5;
const NUM_SECS_SYNC: u64 = 60 * 10;
const INCLUDE_DEPS_PY_SH_CONTENT: &str = include_str!("../nsjail/download_deps.py.sh");
@@ -331,8 +332,7 @@ lazy_static::lazy_static! {
.map(|x| format!(";{x}"))
.unwrap_or_else(|| String::new());
static ref NPM_CONFIG_REGISTRY: Option<String> = std::env::var("NPM_CONFIG_REGISTRY").ok();
static ref DENO_FLAGS: Option<Vec<String>> = std::env::var("DENO_FLAGS")
.ok()
@@ -390,10 +390,19 @@ lazy_static::lazy_static! {
.ok()
.and_then(|x| x.parse::<u16>().ok())
.unwrap_or(DEFAULT_TIMEOUT as u16);
static ref TIMEOUT_DURATION: Duration = Duration::from_secs(*TIMEOUT as u64);
static ref ZOMBIE_JOB_TIMEOUT: String = std::env::var("ZOMBIE_JOB_TIMEOUT")
.ok()
.and_then(|x| x.parse::<String>().ok())
.unwrap_or_else(|| "30".to_string());
static ref ZOMBIE_JOB_TIMEOUT: String = (*TIMEOUT as u32 * 5).to_string();
pub static ref RESTART_ZOMBIE_JOBS: bool = std::env::var("RESTART_ZOMBIE_JOBS")
.ok()
.and_then(|x| x.parse::<bool>().ok())
.unwrap_or(true);
static ref SESSION_TOKEN_EXPIRY: i32 = (*TIMEOUT as i32) * 2;
}
@@ -401,6 +410,21 @@ lazy_static::lazy_static! {
//only matter if CLOUD_HOSTED
const MAX_RESULT_SIZE: usize = 1024 * 1024 * 2; // 2MB
#[derive(Clone)]
pub struct AuthedClient {
pub base_internal_url: String,
pub workspace: String,
pub token: String,
pub client: OnceCell<Client>
}
impl AuthedClient {
pub fn get_client(&self) -> &Client {
return self.client.get_or_init(|| {
windmill_api_client::create_client(&self.base_internal_url, self.token.clone())
});
}
}
#[tracing::instrument(level = "trace")]
@@ -553,7 +577,7 @@ pub async fn run_worker(
let do_break = async {
if last_ping.elapsed().as_secs() > NUM_SECS_ENV_CHECK {
if last_ping.elapsed().as_secs() > NUM_SECS_PING {
sqlx::query!(
"UPDATE worker_ping SET ping_at = now(), jobs_executed = $1 WHERE worker = $2",
jobs_executed,
@@ -627,7 +651,8 @@ pub async fn run_worker(
.with_label_values(label_values.as_slice()),
};
tracing::info!(worker = %worker_name, id = %job.id, "fetched job {}", job.id);
let job_root = job.root_job.map(|x| x.to_string()).unwrap_or_else(|| "none".to_string());
tracing::info!(worker = %worker_name, id = %job.id, root_id = %job_root, "fetched job {}, root job: {}", job.id, job_root);
let job_dir = format!("{worker_dir}/{}", job.id);
@@ -670,13 +695,13 @@ pub async fn run_worker(
)
.await.expect("could not create job token");
tx.commit().await.expect("could not commit job token");
let job_client = windmill_api_client::create_client(base_internal_url, token.clone());
let authed_client = AuthedClient { base_internal_url: base_internal_url.to_string(), token: token.clone(), workspace: job.workspace_id.to_string(), client: OnceCell::new() };
let is_flow = job.job_kind == JobKind::Flow || job.job_kind == JobKind::FlowPreview || job.job_kind == JobKind::FlowDependencies;
if let Some(err) = handle_queued_job(
job.clone(),
db,
&job_client,
&authed_client,
token,
&worker_name,
&worker_dir,
@@ -690,7 +715,7 @@ pub async fn run_worker(
{
handle_job_error(
db,
&job_client,
&authed_client,
job,
err,
Some(metrics),
@@ -734,7 +759,7 @@ pub async fn run_worker(
async fn handle_job_error(
db: &Pool<Postgres>,
client: &windmill_api_client::Client,
client: &AuthedClient,
job: QueuedJob,
err: Error,
metrics: Option<Metrics>,
@@ -833,7 +858,7 @@ fn extract_error_value(log_lines: &str) -> serde_json::Value {
async fn handle_queued_job(
job: QueuedJob,
db: &sqlx::Pool<sqlx::Postgres>,
client: &windmill_api_client::Client,
client: &AuthedClient,
token: String,
worker_name: &str,
worker_dir: &str,
@@ -865,6 +890,9 @@ async fn handle_queued_job(
}
_ => {
let mut logs = "".to_string();
if let Some(log_str) = &job.logs {
logs.push_str(&log_str);
}
if job.is_flow_step {
update_flow_status_in_progress(
@@ -1013,14 +1041,14 @@ async fn write_file(dir: &str, path: &str, content: &str) -> error::Result<File>
#[async_recursion]
async fn transform_json_value(
name: &str,
client: &windmill_api_client::Client,
client: &AuthedClient,
workspace: &str,
v: Value,
) -> error::Result<Value> {
match v {
Value::String(y) if y.starts_with("$var:") => {
let path = y.strip_prefix("$var:").unwrap();
let v = client
let v = client.get_client()
.get_variable(workspace, path, Some(true))
.await
.map_err(|_| Error::NotFound(format!("Variable {path} not found for `{name}`")))
@@ -1036,7 +1064,7 @@ async fn transform_json_value(
"Argument `{name}` is an invalid resource path: {path}",
)));
}
let v = client
let v = client.get_client()
.get_resource_value(workspace, path)
.await
.map_err(|_| Error::NotFound(format!("Resource {path} not found for `{name}`")))?
@@ -1060,7 +1088,7 @@ async fn transform_json_value(
async fn handle_code_execution_job(
job: &QueuedJob,
db: &sqlx::Pool<sqlx::Postgres>,
client: &windmill_api_client::Client,
client: &AuthedClient,
token: String,
job_dir: &str,
worker_dir: &str,
@@ -1152,7 +1180,6 @@ mount {{
job_dir,
&inner_content,
&shared_mount,
requirements_o,
base_internal_url,
worker_name
)
@@ -1206,7 +1233,7 @@ async fn handle_go_job(
logs: &mut String,
job: &QueuedJob,
db: &sqlx::Pool<sqlx::Postgres>,
client: &windmill_api_client::Client,
client: &AuthedClient,
token: String,
inner_content: &str,
job_dir: &str,
@@ -1242,7 +1269,8 @@ async fn handle_go_job(
db,
true,
skip_go_mod,
worker_name
worker_name,
&job.workspace_id,
)
.await?;
@@ -1360,7 +1388,7 @@ func Run(req Req) (interface{{}}, error){{
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()?;
handle_child(&job.id, db, logs, build_go, false, worker_name).await?;
handle_child(&job.id, db, logs, build_go, false, worker_name, &job.workspace_id).await?;
Command::new(NSJAIL_PATH.as_str())
.current_dir(job_dir)
@@ -1386,7 +1414,7 @@ func Run(req Req) (interface{{}}, error){{
.stderr(Stdio::piped())
.spawn()?
};
handle_child(&job.id, db, logs, child, !*DISABLE_NSJAIL, worker_name).await?;
handle_child(&job.id, db, logs, child, !*DISABLE_NSJAIL, worker_name, &job.workspace_id).await?;
read_result(job_dir).await
}
@@ -1437,7 +1465,7 @@ async fn handle_bash_job(
.replace("{SHARED_MOUNT}", shared_mount),
)
.await?;
let mut cmd_args = vec!["--config", "run.config.proto", "--", "/bin/sh", "main.sh"];
let mut cmd_args = vec!["--config", "run.config.proto", "--", "/bin/bash", "main.sh"];
cmd_args.extend(args);
Command::new(NSJAIL_PATH.as_str())
.current_dir(job_dir)
@@ -1452,7 +1480,7 @@ async fn handle_bash_job(
} else {
let mut cmd_args = vec!["main.sh"];
cmd_args.extend(&args);
Command::new("/bin/sh")
Command::new("/bin/bash")
.current_dir(job_dir)
.env_clear()
.envs(reserved_variables)
@@ -1464,7 +1492,7 @@ async fn handle_bash_job(
.stderr(Stdio::piped())
.spawn()?
};
handle_child(&job.id, db, logs, child, !*DISABLE_NSJAIL, worker_name).await?;
handle_child(&job.id, db, logs, child, !*DISABLE_NSJAIL, worker_name, &job.workspace_id).await?;
//for now bash jobs have an empty result object
Ok(serde_json::json!(logs
.lines()
@@ -1481,40 +1509,53 @@ fn capitalize(s: &str) -> String {
}
}
fn get_common_deno_proc_envs(token: &str, base_internal_url: &str) -> HashMap<String, String> {
let hostname_base = BASE_URL.split("://").last().unwrap_or("localhost");
let hostname_internal = base_internal_url.split("://").last().unwrap_or("localhost");
let deno_auth_tokens_base = DENO_AUTH_TOKENS.as_str();
let deno_auth_tokens =
format!("{token}@{hostname_base};{token}@{hostname_internal}{deno_auth_tokens_base}",);
let mut deno_envs: HashMap<String, String> = HashMap::from([
(String::from("PATH"), PATH_ENV.clone()),
(String::from("DENO_AUTH_TOKENS"), deno_auth_tokens),
(String::from("BASE_INTERNAL_URL"), base_internal_url.to_string()),
(String::from("NO_COLOR"), String::from("true")),
]);
if let Some(ref s) = *NPM_CONFIG_REGISTRY {
deno_envs.insert(String::from("NPM_CONFIG_REGISTRY"), s.clone());
}
return deno_envs;
}
#[tracing::instrument(level = "trace", skip_all)]
async fn handle_deno_job(
logs: &mut String,
job: &QueuedJob,
db: &sqlx::Pool<sqlx::Postgres>,
client: &windmill_api_client::Client,
client: &AuthedClient,
token: String,
job_dir: &str,
inner_content: &String,
shared_mount: &str,
lockfile: Option<String>,
base_internal_url: &str,
worker_name: &str
) -> error::Result<serde_json::Value> {
logs.push_str("\n\n--- DENO CODE EXECUTION ---\n");
set_logs(logs, &job.id, db).await;
let lockfile = lockfile.and_then(|e| if e.starts_with("{") { Some(e) } else { None });
let _ = write_file(
job_dir,
"lock.json",
&lockfile.clone().unwrap_or("".to_string()),
)
.await?;
// TODO: Separately cache dependencies here using `deno cache --reload --lock=lock.json src/deps.ts` (https://deno.land/manual@v1.27.0/linking_to_external_code/integrity_checking)
// Then require caching below using --cached-only. This makes it so we require zero network interaction when running the process below
let _ = write_file(job_dir, "inner.ts", inner_content).await?;
let _ = write_file(job_dir, "main.ts", inner_content).await?;
let sig = trace_span!("parse_deno_signature")
.in_scope(|| windmill_parser_ts::parse_deno_signature(inner_content))?;
create_args_and_out_file(client, job, job_dir).await?;
let spread = sig.args.into_iter().map(|x| x.name).join(",");
let wrapper_content: String = format!(
r#"
import {{ main }} from "./inner.ts";
import {{ main }} from "./main.ts";
const args = await Deno.readTextFile("args.json")
.then(JSON.parse)
@@ -1532,25 +1573,35 @@ run().catch(async (e) => {{
}});
"#,
);
write_file(job_dir, "main.ts", &wrapper_content).await?;
let w_id = job.workspace_id.clone();
let script_path_split = job.script_path().split("/");
let script_path_parts_len = script_path_split.clone().count();
let mut relative_mounts = "".to_string();
for c in 0..script_path_parts_len {
relative_mounts += ",\n ";
relative_mounts += &format!("\"./{}\": \"{base_internal_url}/api/w/{w_id}/scripts/raw/p/{}{}\"",
(0..c).map(|_| "../").join(""),
&script_path_split.clone().take(script_path_parts_len - c - 1).join("/"),
if c == script_path_parts_len - 1 { "" } else { "/" },
);
}
write_file(job_dir, "wrapper.ts", &wrapper_content).await?;
let import_map = format!(
r#"{{
"imports": {{
"/": "{base_internal_url}/api/w/{w_id}/scripts/raw/p/",
"./": "./"
"./wrapper.ts": "./wrapper.ts",
"./main.ts": "./main.ts"{relative_mounts}
}}
}}"#
}}"#,
);
write_file(job_dir, "import_map.json", &import_map).await?;
let mut reserved_variables = get_reserved_variables(job, &token, db).await?;
reserved_variables.insert("RUST_LOG".to_string(), "info".to_string());
let hostname_base = BASE_URL.split("://").last().unwrap_or("localhost");
let hostname_internal = base_internal_url.split("://").last().unwrap_or("localhost");
let deno_auth_tokens_base = DENO_AUTH_TOKENS.as_str();
let deno_auth_tokens =
format!("{token}@{hostname_base};{token}@{hostname_internal}{deno_auth_tokens_base}",);
let common_deno_proc_envs = get_common_deno_proc_envs(&token, base_internal_url);
//do not cache local dependencies
let reload = format!("--reload={base_internal_url}");
let child = async {
Ok(if !*DISABLE_NSJAIL {
let _ = write_file(
@@ -1569,11 +1620,10 @@ run().catch(async (e) => {{
args.push("--");
args.push(DENO_PATH.as_str());
args.push("run");
if lockfile.is_some() {
args.push("--lock=/tmp/lock.json");
}
args.push("--import-map");
args.push("/tmp/import_map.json");
args.push(&reload);
args.push("--unstable");
if let Some(deno_flags) = DENO_FLAGS.as_ref() {
for flag in deno_flags {
@@ -1582,26 +1632,25 @@ run().catch(async (e) => {{
} else {
args.push("-A");
}
args.push("/tmp/main.ts");
args.push("/tmp/wrapper.ts");
Command::new(NSJAIL_PATH.as_str())
.current_dir(job_dir)
.env_clear()
.envs(reserved_variables)
.env("PATH", PATH_ENV.as_str())
.env("DENO_AUTH_TOKENS", deno_auth_tokens)
.env("BASE_INTERNAL_URL", base_internal_url)
.envs(common_deno_proc_envs)
.args(args)
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()?
} else {
let mut args = Vec::new();
let script_path = format!("{job_dir}/main.ts");
let script_path = format!("{job_dir}/wrapper.ts");
let import_map_path = format!("{job_dir}/import_map.json");
args.push("run");
args.push("--import-map");
args.push(&import_map_path);
args.push(&reload);
args.push("--unstable");
if let Some(deno_flags) = DENO_FLAGS.as_ref() {
for flag in deno_flags {
@@ -1615,11 +1664,8 @@ run().catch(async (e) => {{
.current_dir(job_dir)
.env_clear()
.envs(reserved_variables)
.env("PATH", PATH_ENV.as_str())
.envs(common_deno_proc_envs)
.env("DENO_DIR", DENO_CACHE_DIR)
.env("DENO_AUTH_TOKENS", deno_auth_tokens)
.env("BASE_INTERNAL_URL", base_internal_url)
.env("NO_COLOR", "true")
.args(args)
.stdout(Stdio::piped())
.stderr(Stdio::piped())
@@ -1628,13 +1674,13 @@ run().catch(async (e) => {{
}
.instrument(trace_span!("create_deno_jail"))
.await?;
handle_child(&job.id, db, logs, child, !*DISABLE_NSJAIL, worker_name).await?;
handle_child(&job.id, db, logs, child, !*DISABLE_NSJAIL, worker_name, &job.workspace_id).await?;
read_result(job_dir).await
}
#[tracing::instrument(level = "trace", skip_all)]
async fn create_args_and_out_file(
client: &windmill_api_client::Client,
client: &AuthedClient,
job: &QueuedJob,
job_dir: &str,
) -> Result<(), Error> {
@@ -1650,7 +1696,7 @@ async fn create_args_and_out_file(
}
lazy_static! {
static ref RELATIVE_IMPORT_REGEX: Regex = Regex::new(r#"(import|from)\s(u|f)\."#).unwrap();
static ref RELATIVE_IMPORT_REGEX: Regex = Regex::new(r#"(import|from)\s(((u|f)\.)|\.)"#).unwrap();
}
#[tracing::instrument(level = "trace", skip_all)]
@@ -1662,11 +1708,11 @@ async fn handle_python_job(
job: &QueuedJob,
logs: &mut String,
db: &sqlx::Pool<sqlx::Postgres>,
client: &windmill_api_client::Client,
client: &AuthedClient,
token: String,
inner_content: &String,
shared_mount: &str,
base_internal_url: &str,
base_internal_url: &str
) -> error::Result<serde_json::Value> {
create_dependencies_dir(job_dir).await;
@@ -1680,7 +1726,7 @@ async fn handle_python_job(
if requirements.is_empty() {
"".to_string()
} else {
pip_compile(&job.id, &requirements, logs, job_dir, db, worker_name)
pip_compile(&job.id, &requirements, logs, job_dir, db, worker_name, &job.workspace_id)
.await
.map_err(|e| {
Error::ExecutionErr(format!("pip compile failed: {}", e.to_string()))
@@ -1721,9 +1767,14 @@ async fn handle_python_job(
let relative_imports = RELATIVE_IMPORT_REGEX.is_match(&inner_content);
let _ = write_file(job_dir, "inner.py", inner_content).await?;
let script_path_splitted = &job.script_path().split("/");
let dirs = script_path_splitted.clone().take(script_path_splitted.clone().count() - 1).join("/").replace("-", "_");
let last = script_path_splitted.clone().last().unwrap().replace("-", "_").replace(" ", "_").to_lowercase();
let module_dir = format!("{}/{}", job_dir, dirs );
tokio::fs::create_dir_all(format!("{module_dir}/")).await?;
let _ = write_file(&module_dir, &format!("{last}.py"), inner_content).await?;
if relative_imports {
let _ = write_file(job_dir, "loader.py", RELATIVE_PYTHON_LOADER).await?;
let _ = write_file(&job_dir, "loader.py", RELATIVE_PYTHON_LOADER).await?;
}
let sig = windmill_parser_py::parse_python_signature(inner_content)?;
@@ -1784,6 +1835,7 @@ async fn handle_python_job(
.join("\n")
};
let module_dir_dot = dirs.replace("/", ".").replace("-", "_");
let wrapper_content: String = format!(
r#"
import json
@@ -1792,8 +1844,8 @@ import json
{import_datetime}
import traceback
import sys
from {module_dir_dot} import {last} as inner_script
inner_script = __import__("inner")
with open("args.json") as f:
kwargs = json.load(f, strict=False)
@@ -1820,7 +1872,7 @@ except Exception as e:
sys.exit(1)
"#,
);
write_file(job_dir, "main.py", &wrapper_content).await?;
write_file(job_dir, "wrapper.py", &wrapper_content).await?;
let mut reserved_variables = get_reserved_variables(job, &token, db).await?;
let additional_python_paths_folders = additional_python_paths.iter().join(":");
@@ -1848,6 +1900,7 @@ mount {{
.replace("{CLONE_NEWUSER}", &(!*DISABLE_NUSER).to_string())
.replace("{SHARED_MOUNT}", shared_mount)
.replace("{SHARED_DEPENDENCIES}", shared_deps.as_str())
.replace("{MAIN}", format!("{dirs}/{last}").as_str())
.replace(
"{ADDITIONAL_PYTHON_PATHS}",
additional_python_paths_folders.as_str(),
@@ -1879,7 +1932,8 @@ mount {{
"--",
PYTHON_PATH.as_str(),
"-u",
"/tmp/main.py",
"-m",
"wrapper",
])
.stdout(Stdio::piped())
.stderr(Stdio::piped())
@@ -1891,13 +1945,13 @@ mount {{
.envs(reserved_variables)
.env("PATH", PATH_ENV.as_str())
.env("BASE_INTERNAL_URL", base_internal_url)
.args(vec!["-u", "main.py"])
.args(vec!["-u", "-m", "wrapper"])
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()?
};
handle_child(&job.id, db, logs, child, !*DISABLE_NSJAIL, worker_name).await?;
handle_child(&job.id, db, logs, child, !*DISABLE_NSJAIL, worker_name, &job.workspace_id).await?;
read_result(job_dir).await
}
@@ -1943,7 +1997,8 @@ async fn handle_dependency_job(
logs,
job_dir,
db,
worker_name
worker_name,
&job.workspace_id,
)
.await;
match content {
@@ -2009,7 +2064,8 @@ async fn handle_flow_dependency_job(
logs,
job_dir,
db,
worker_name
worker_name,
&job.workspace_id,
)
.await;
match new_lock {
@@ -2125,12 +2181,13 @@ async fn capture_dependency_job(
logs: &mut String,
job_dir: &str,
db: &sqlx::Pool<sqlx::Postgres>,
worker_name: &str
worker_name: &str,
w_id: &str,
) -> error::Result<String> {
match job_language {
ScriptLang::Python3 => {
create_dependencies_dir(job_dir).await;
pip_compile(job_id, job_raw_code, logs, job_dir, db, worker_name).await
pip_compile(job_id, job_raw_code, logs, job_dir, db, worker_name, w_id).await
}
ScriptLang::Go => {
install_go_dependencies(
@@ -2141,7 +2198,8 @@ async fn capture_dependency_job(
db,
false,
false,
worker_name
worker_name,
w_id
)
.await
}
@@ -2159,7 +2217,8 @@ async fn pip_compile(
logs: &mut String,
job_dir: &str,
db: &Pool<Postgres>,
worker_name: &str
worker_name: &str,
w_id: &str,
) -> error::Result<String> {
logs.push_str(&format!("\nresolving dependencies..."));
set_logs(logs, job_id, db).await;
@@ -2192,7 +2251,7 @@ async fn pip_compile(
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()?;
handle_child(job_id, db, logs, child, false, worker_name)
handle_child(job_id, db, logs, child, false, worker_name, &w_id)
.await
.map_err(|e| Error::ExecutionErr(format!("Lock file generation failed: {e:?}")))?;
let path_lock = format!("{job_dir}/requirements.txt");
@@ -2215,7 +2274,8 @@ async fn install_go_dependencies(
db: &sqlx::Pool<sqlx::Postgres>,
preview: bool,
skip_go_mod: bool,
worker_name: &str
worker_name: &str,
w_id: &str
) -> error::Result<String> {
if !skip_go_mod {
gen_go_mymod(code, job_dir).await?;
@@ -2226,7 +2286,7 @@ async fn install_go_dependencies(
.stderr(Stdio::piped())
.spawn()?;
handle_child(job_id, db, logs, child, false, worker_name).await?;
handle_child(job_id, db, logs, child, false, worker_name, w_id).await?;
}
let child = Command::new(GO_PATH.as_str())
.current_dir(job_dir)
@@ -2234,7 +2294,7 @@ async fn install_go_dependencies(
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()?;
handle_child(job_id, db, logs, child, false, worker_name)
handle_child(job_id, db, logs, child, false, worker_name, &w_id)
.await
.map_err(|e| Error::ExecutionErr(format!("Lock file generation failed: {e:?}")))?;
@@ -2341,11 +2401,13 @@ async fn handle_child(
mut child: Child,
nsjail: bool,
worker_name: &str,
w_id: &str,
) -> error::Result<()> {
let update_job_interval = Duration::from_millis(500);
let write_logs_delay = Duration::from_millis(500);
let pid = child.id();
#[cfg(target_os = "linux")]
if let Some(pid) = pid {
//set the highest oom priority
let mut file = File::create(format!("/proc/{pid}/oom_score_adj")).await?;
@@ -2416,11 +2478,29 @@ async fn handle_child(
let wait_on_child = async {
let db = db.clone();
#[cfg(not(feature = "enterprise"))]
let timeout_duration = *TIMEOUT_DURATION;
#[cfg(feature = "enterprise")]
let premium_workspace = *CLOUD_HOSTED && sqlx::query_scalar!("SELECT premium FROM workspace WHERE id = $1", w_id)
.fetch_one(&db)
.await
.map_err(|e| {
tracing::error!(%e, "error getting premium workspace for job {job_id}: {e}");
}).unwrap_or(false);
#[cfg(feature = "enterprise")]
let timeout_duration = if premium_workspace {
*TIMEOUT_DURATION*6 //30mins
} else {
*TIMEOUT_DURATION
};
let kill_reason = tokio::select! {
biased;
result = child.wait() => return result.map(Ok),
Ok(()) = too_many_logs.changed() => KillReason::TooManyLogs,
_ = sleep(*TIMEOUT_DURATION) => KillReason::Timeout,
_ = sleep(timeout_duration) => KillReason::Timeout,
_ = update_job => KillReason::Cancelled,
};
tx.send(()).expect("rx should never be dropped");
@@ -2679,7 +2759,7 @@ pub async fn handle_zombie_jobs_periodically(
handle_zombie_jobs(db, base_internal_url).await;
tokio::select! {
_ = tokio::time::sleep(Duration::from_secs(60)) => (),
_ = tokio::time::sleep(Duration::from_secs(30)) => (),
_ = rx.recv() => {
println!("received killpill for monitor job");
break;
@@ -2689,28 +2769,35 @@ pub async fn handle_zombie_jobs_periodically(
}
async fn handle_zombie_jobs(db: &Pool<Postgres>, base_internal_url: &str) {
let restarted = sqlx::query!(
"UPDATE queue SET running = false WHERE last_ping < now() - ($1 || ' seconds')::interval AND running = true AND job_kind != $2 AND same_worker = false RETURNING id, workspace_id, last_ping",
*ZOMBIE_JOB_TIMEOUT,
JobKind::Flow: JobKind,
)
.fetch_all(db)
.await
.ok()
.unwrap_or_else(|| vec![]);
if *RESTART_ZOMBIE_JOBS {
let restarted = sqlx::query!(
"UPDATE queue SET running = false, started_at = null, logs = logs || '\nRestarted job after not receiving job''s ping for too long the ' || now() || '\n\n' WHERE last_ping < now() - ($1 || ' seconds')::interval AND running = true AND job_kind != $2 AND job_kind != $3 AND same_worker = false RETURNING id, workspace_id, last_ping",
*ZOMBIE_JOB_TIMEOUT,
JobKind::Flow: JobKind,
JobKind::FlowPreview: JobKind,
)
.fetch_all(db)
.await
.ok()
.unwrap_or_else(|| vec![]);
QUEUE_ZOMBIE_RESTART_COUNT.inc_by(restarted.len() as _);
for r in restarted {
tracing::info!(
"restarted zombie job {} {} {}",
r.id,
r.workspace_id,
r.last_ping
);
QUEUE_ZOMBIE_RESTART_COUNT.inc_by(restarted.len() as _);
for r in restarted {
tracing::info!(
"restarted zombie job {} {} {}",
r.id,
r.workspace_id,
r.last_ping
);
}
}
let mut timeout_query = "SELECT * FROM queue WHERE last_ping < now() - ($1 || ' seconds')::interval AND running = true AND job_kind != $2".to_string();
if *RESTART_ZOMBIE_JOBS {
timeout_query.push_str(" AND same_worker = true");
};
let timeouts = sqlx::query_as::<_, QueuedJob>(
"SELECT * FROM queue WHERE last_ping < now() - ($1 || ' seconds')::interval AND running = true AND job_kind != $2 AND same_worker = true",
&timeout_query
)
.bind(ZOMBIE_JOB_TIMEOUT.as_str())
.bind(JobKind::Flow)
@@ -2722,7 +2809,7 @@ async fn handle_zombie_jobs(db: &Pool<Postgres>, base_internal_url: &str) {
QUEUE_ZOMBIE_DELETE_COUNT.inc_by(timeouts.len() as _);
for job in timeouts {
tracing::info!(
"timedouts zombie same_worker job {} {}",
"timedout zombie job {} {}",
job.id,
job.workspace_id,
);
@@ -2742,13 +2829,15 @@ async fn handle_zombie_jobs(db: &Pool<Postgres>, base_internal_url: &str) {
.await
.expect("could not create job token");
tx.commit().await.expect("could not commit job token");
let client = windmill_api_client::create_client(base_internal_url, token.clone());
let client = AuthedClient { base_internal_url: base_internal_url.to_string(), token: token.clone(), workspace: job.workspace_id.to_string(), client: OnceCell::new() };
let last_ping = job.last_ping.clone();
let _ = handle_job_error(
db,
&client,
job,
error::Error::ExecutionErr("Same worker job timed out".to_string()),
error::Error::ExecutionErr(format!("Job timed out after no ping from job since {} (ZOMBIE_JOB_TIMEOUT: {})",
last_ping.map(|x| x.to_string()).unwrap_or_else(|| "no ping".to_string()), *ZOMBIE_JOB_TIMEOUT)),
None,
true,
same_worker_tx_never_used,
@@ -2851,7 +2940,7 @@ async fn handle_python_reqs(
.spawn()?
};
let child = handle_child(&job.id, db, logs, child, false, worker_name).await;
let child = handle_child(&job.id, db, logs, child, false, worker_name, &job.workspace_id).await;
tracing::info!(
worker_name = %worker_name,
job_id = %job.id,
+159 -183
View File
@@ -7,15 +7,13 @@
*/
use std::collections::HashMap;
use std::iter;
use std::time::Duration;
use crate::jobs::{add_completed_job, add_completed_job_error, schedule_again_if_scheduled};
use crate::js_eval::{eval_timeout, EvalCreds, IdContext};
use crate::{worker, KEEP_JOB_DIR};
use crate::js_eval::{eval_timeout, IdContext};
use crate::{worker, AuthedClient, KEEP_JOB_DIR};
use anyhow::Context;
use async_recursion::async_recursion;
use dyn_iter::DynIter;
use serde_json::{json, Map, Value};
use tokio::sync::mpsc::Sender;
use tracing::instrument;
@@ -40,7 +38,7 @@ use windmill_queue::{
// #[instrument(level = "trace", skip_all)]
pub async fn update_flow_status_after_job_completion(
db: &DB,
client: &windmill_api_client::Client,
client: &AuthedClient,
flow: uuid::Uuid,
job_id_for_status: &Uuid,
w_id: &str,
@@ -120,8 +118,7 @@ pub async fn update_flow_status_after_job_completion(
let stop_early = success
&& if let Some(expr) = r.stop_early_expr.clone() {
compute_bool_from_expr(expr, &r.args, result.clone(), None, None, base_internal_url)
.await?
compute_bool_from_expr(expr, &r.args, result.clone(), None, Some(client)).await?
} else {
false
};
@@ -396,6 +393,12 @@ pub async fn update_flow_status_after_job_completion(
.await?
.ok_or_else(|| Error::InternalErr(format!("requiring flow to be in the queue")))?;
let job_root = flow_job
.root_job
.map(|x| x.to_string())
.unwrap_or_else(|| "none".to_string());
tracing::info!(id = %flow_job.id, root_id = %job_root, "update flow status");
let raw_flow = flow_job.parse_raw_flow();
let module = raw_flow.as_ref().and_then(|module| {
module_index.and_then(|i| module.modules.get(i).or(module.failure_module.as_ref()))
@@ -598,8 +601,7 @@ async fn compute_bool_from_expr(
flow_args: &Option<serde_json::Value>,
result: serde_json::Value,
by_id: Option<IdContext>,
creds: Option<EvalCreds>,
base_internal_url: &str,
client: Option<&AuthedClient>,
) -> error::Result<bool> {
let flow_input = flow_args.clone().unwrap_or_else(|| json!({}));
match eval_timeout(
@@ -610,9 +612,8 @@ async fn compute_bool_from_expr(
("previous_result".to_string(), result),
]
.into(),
creds,
client,
by_id,
base_internal_url,
)
.await?
{
@@ -687,12 +688,10 @@ async fn transform_input(
flow_args: &Option<serde_json::Value>,
last_result: serde_json::Value,
input_transforms: &HashMap<String, InputTransform>,
workspace: &str,
token: &str,
resumes: &[Value],
approvers: Vec<String>,
by_id: &IdContext,
base_internal_url: &str,
client: &AuthedClient,
) -> windmill_common::error::Result<Map<String, serde_json::Value>> {
let mut mapped = serde_json::Map::new();
@@ -731,19 +730,13 @@ async fn transform_input(
context.push(("error".to_string(), error.unwrap().clone()));
}
let v = eval_timeout(
expr.to_string(),
context,
Some(EvalCreds { workspace: workspace.to_string(), token: token.to_string() }),
Some(by_id.clone()),
base_internal_url,
)
.await
.map_err(|e| {
Error::ExecutionErr(format!(
"Error during isolated evaluation of expression `{expr}`:\n{e}"
))
})?;
let v = eval_timeout(expr.to_string(), context, Some(client), Some(by_id.clone()))
.await
.map_err(|e| {
Error::ExecutionErr(format!(
"Error during isolated evaluation of expression `{expr}`:\n{e}"
))
})?;
mapped.insert(key.to_string(), v);
()
}
@@ -757,7 +750,7 @@ async fn transform_input(
pub async fn handle_flow(
flow_job: &QueuedJob,
db: &sqlx::Pool<sqlx::Postgres>,
client: &windmill_api_client::Client,
client: &AuthedClient,
last_result: serde_json::Value,
same_worker_tx: Sender<Uuid>,
worker_dir: &str,
@@ -797,12 +790,18 @@ async fn push_next_flow_job(
mut status: FlowStatus,
flow: FlowValue,
db: &sqlx::Pool<sqlx::Postgres>,
client: &windmill_api_client::Client,
client: &AuthedClient,
mut last_result: serde_json::Value,
same_worker_tx: Sender<Uuid>,
worker_dir: &str,
base_internal_url: &str,
) -> error::Result<()> {
let job_root = flow_job
.root_job
.map(|x| x.to_string())
.unwrap_or_else(|| "none".to_string());
tracing::info!(id = %flow_job.id, root_id = %job_root, "pushing next flow job");
let mut i = usize::try_from(status.step)
.with_context(|| format!("invalid module index {}", status.step))?;
@@ -871,7 +870,6 @@ async fn push_next_flow_job(
.into(),
None,
None,
base_internal_url,
)
.await
.map_err(|e| {
@@ -1114,29 +1112,23 @@ async fn push_next_flow_job(
_ => (),
}
let mut transform_context: Option<TransformContext> = None;
let mut transform_context: Option<IdContext> = None;
let args: windmill_common::error::Result<_> = match &module.value {
FlowModuleValue::Script { input_transforms, .. }
| FlowModuleValue::RawScript { input_transforms, .. }
| FlowModuleValue::Flow { input_transforms, .. } => {
let ctx = get_transform_context(db, &flow_job, previous_id.clone(), &status).await?;
let ctx = get_transform_context(&flow_job, previous_id.clone(), &status).await?;
transform_context = Some(ctx);
let (token, by_id) = transform_context.as_ref().unwrap();
let by_id = transform_context.as_ref().unwrap();
transform_input(
&flow_job.args,
last_result.clone(),
if !input_transforms.is_empty() {
input_transforms
} else {
&module.input_transforms
},
&flow_job.workspace_id,
&token,
input_transforms,
resume_messages.as_slice(),
approvers,
by_id,
base_internal_url,
client,
)
.await
}
@@ -1169,17 +1161,17 @@ async fn push_next_flow_job(
flow_job,
&flow,
transform_context,
db,
tx,
&module,
&status,
&status_module,
last_result.clone(),
previous_id,
base_internal_url,
client,
)
.await?;
tx.commit().await?;
tracing::info!(id = %flow_job.id, root_id = %job_root, "next flow transform computed");
let (job_payloads, next_status) = match next_flow_transform {
NextFlowTransform::Continue(job_payload, next_state) => (job_payload, next_state),
@@ -1209,45 +1201,62 @@ async fn push_next_flow_job(
let continue_on_same_worker =
flow.same_worker && module.suspend.is_none() && module.sleep.is_none();
let zipped = {
let all_args = match &next_status {
NextStatus::NextLoopIteration(NextIteration { new_args, .. }) => {
let args = args.as_ref().map(|args| {
let mut args = args.clone();
args.extend(new_args.clone());
args
});
DynIter::new(iter::once(args))
}
NextStatus::AllFlowJobs {
branchall: Some(BranchAllStatus { len, .. }),
iterator: None,
..
} => DynIter::new((0..*len).map(|_| args.as_ref().map(|args| args.clone()))),
NextStatus::AllFlowJobs {
branchall: None,
iterator: Some(Iterator { itered, .. }),
..
} => DynIter::new(itered.into_iter().enumerate().map(|(i, v)| {
args.as_ref().map(|args| {
let mut new_args = args.clone();
new_args.insert("iter".to_string(), json!({ "index": i, "value": v }));
new_args
})
})),
_ => DynIter::new(iter::once(args.as_ref().map(|m| m.clone()))),
};
job_payloads.into_iter().zip(all_args).collect::<Vec<_>>()
};
/* Finally, push the job into the queue */
let mut tx = db.begin().await?;
let mut uuids = vec![];
for (payload, args) in zipped {
let len = match &job_payloads {
ContinuePayload::SingleJob(_) => 1,
ContinuePayload::BranchAllJobs(payloads) => payloads.len(),
ContinuePayload::ForloopJobs { n, .. } => *n,
};
for i in (0..len).into_iter() {
let payload = match &job_payloads {
ContinuePayload::SingleJob(payload) => payload.clone(),
ContinuePayload::BranchAllJobs(payloads) => payloads[i].clone(),
ContinuePayload::ForloopJobs { modules, .. } => {
let mut fm = flow.failure_module.clone();
if let Some(mut failure_module) = flow.failure_module.clone() {
failure_module.id_append(&format!("{}/{}", status.step, i));
fm = Some(failure_module);
}
JobPayload::RawFlow {
value: FlowValue {
modules: (*modules).clone(),
failure_module: fm.clone(),
same_worker: flow.same_worker,
},
path: Some(format!("{}/loop-{}", flow_job.script_path(), i)),
}
}
};
let args = match &next_status {
NextStatus::AllFlowJobs {
branchall: Some(BranchAllStatus { .. }),
iterator: None,
..
} => args.as_ref().map(|args| args.clone()),
NextStatus::NextLoopIteration(NextIteration { new_args, .. }) => {
args.as_ref().map(|args| {
let mut args = args.clone();
args.extend(new_args.clone());
args
})
}
NextStatus::AllFlowJobs {
branchall: None,
iterator: Some(Iterator { itered, .. }),
..
} => args.as_ref().map(|args| {
let mut new_args = args.clone();
new_args.insert(
"iter".to_string(),
json!({ "index": i, "value": itered[i] }),
);
new_args
}),
_ => args.as_ref().map(|args| args.clone()),
};
let (ok, err) = match args {
Ok(v) => (Some(v), None),
Err(e) => (None, Some(e)),
@@ -1376,6 +1385,7 @@ async fn push_next_flow_job(
};
tx.commit().await?;
tracing::info!(id = %flow_job.id, root_id = %job_root, "all next flow jobs pushed");
if continue_on_same_worker {
if !is_one_uuid {
@@ -1486,9 +1496,15 @@ enum NextStatus {
},
}
enum ContinuePayload {
SingleJob(JobPayload),
ForloopJobs { n: usize, modules: Vec<FlowModule> },
BranchAllJobs(Vec<JobPayload>),
}
enum NextFlowTransform {
EmptyInnerFlows,
Continue(Vec<JobPayload>, NextStatus),
Continue(ContinuePayload, NextStatus),
}
// a similar function exists on the backend
@@ -1507,31 +1523,34 @@ async fn script_path_to_payload<'c>(
Ok(job_payload)
}
type TransformContext = (String, IdContext);
async fn compute_next_flow_transform<'c>(
flow_job: &QueuedJob,
flow: &FlowValue,
transform_context: Option<TransformContext>,
db: &DB,
by_id: Option<IdContext>,
mut tx: sqlx::Transaction<'c, sqlx::Postgres>,
module: &FlowModule,
status: &FlowStatus,
status_module: &FlowStatusModule,
last_result: serde_json::Value,
previous_id: String,
base_internal_url: &str,
client: &AuthedClient,
) -> error::Result<(sqlx::Transaction<'c, sqlx::Postgres>, NextFlowTransform)> {
match &module.value {
FlowModuleValue::Identity => Ok((
tx,
NextFlowTransform::Continue(vec![JobPayload::Identity], NextStatus::NextStep),
NextFlowTransform::Continue(
ContinuePayload::SingleJob(JobPayload::Identity),
NextStatus::NextStep,
),
)),
FlowModuleValue::Flow { path, .. } => {
let payload = JobPayload::Flow(path.to_string());
Ok((
tx,
NextFlowTransform::Continue(vec![payload], NextStatus::NextStep),
NextFlowTransform::Continue(
ContinuePayload::SingleJob(payload),
NextStatus::NextStep,
),
))
}
FlowModuleValue::Script { path: script_path, hash: script_hash, .. } => {
@@ -1545,22 +1564,25 @@ async fn compute_next_flow_transform<'c>(
};
Ok((
tx,
NextFlowTransform::Continue(vec![payload], NextStatus::NextStep),
NextFlowTransform::Continue(
ContinuePayload::SingleJob(payload),
NextStatus::NextStep,
),
))
}
FlowModuleValue::RawScript { path, content, language, lock, .. } => {
let path = path
.clone()
.or_else(|| Some(format!("{}/{}", flow_job.script_path(), status.step)));
.or_else(|| Some(format!("{}/step-{}", flow_job.script_path(), status.step)));
Ok((
tx,
NextFlowTransform::Continue(
vec![JobPayload::Code(RawCode {
ContinuePayload::SingleJob(JobPayload::Code(RawCode {
path,
content: content.clone(),
language: language.clone(),
lock: lock.clone(),
})],
})),
NextStatus::NextStep,
),
))
@@ -1573,10 +1595,10 @@ async fn compute_next_flow_transform<'c>(
FlowStatusModule::WaitingForPriorSteps { .. }
| FlowStatusModule::WaitingForEvents { .. }
| FlowStatusModule::WaitingForExecutor { .. } => {
let (token, by_id) = if let Some(x) = transform_context {
let by_id = if let Some(x) = by_id {
x
} else {
get_transform_context(db, &flow_job, previous_id, &status).await?
get_transform_context(&flow_job, previous_id, &status).await?
};
let flow_input = flow_job.args.clone().unwrap_or_else(|| json!({}));
/* Iterator is an InputTransform, evaluate it into an array. */
@@ -1589,10 +1611,8 @@ async fn compute_next_flow_transform<'c>(
("previous_result".to_string(), last_result.clone()),
]
},
token,
flow_job.workspace_id.clone(),
Some(client),
Some(by_id),
base_internal_url,
)
.await?
.into_array()
@@ -1660,14 +1680,14 @@ async fn compute_next_flow_transform<'c>(
Ok((
tx,
NextFlowTransform::Continue(
vec![JobPayload::RawFlow {
ContinuePayload::SingleJob(JobPayload::RawFlow {
value: FlowValue {
modules: (*modules).clone(),
failure_module: fm,
same_worker: flow.same_worker,
},
path: Some(format!("{}/loop-{}", flow_job.script_path(), ns.index)),
}],
}),
NextStatus::NextLoopIteration(ns),
),
))
@@ -1675,23 +1695,10 @@ async fn compute_next_flow_transform<'c>(
LoopStatus::ParallelIteration { itered } => Ok((
tx,
NextFlowTransform::Continue(
(0..itered.len())
.map(|i| {
let mut fm = flow.failure_module.clone();
if let Some(mut failure_module) = flow.failure_module.clone() {
failure_module.id_append(&format!("{}/{}", status.step, i));
fm = Some(failure_module);
}
JobPayload::RawFlow {
value: FlowValue {
modules: (*modules).clone(),
failure_module: fm.clone(),
same_worker: flow.same_worker,
},
path: Some(format!("{}/loop-{}", flow_job.script_path(), i)),
}
})
.collect(),
ContinuePayload::ForloopJobs {
n: itered.len(),
modules: (*modules).clone(),
},
NextStatus::AllFlowJobs {
branchall: None,
iterator: Some(windmill_common::flow_status::Iterator {
@@ -1709,19 +1716,14 @@ async fn compute_next_flow_transform<'c>(
| FlowStatusModule::WaitingForEvents { .. }
| FlowStatusModule::WaitingForExecutor { .. } => {
let mut branch_chosen = BranchChosen::Default;
let (token, idcontext) =
get_transform_context(db, &flow_job, previous_id, &status).await?;
let idcontext = get_transform_context(&flow_job, previous_id, &status).await?;
for (i, b) in branches.iter().enumerate() {
let pred = compute_bool_from_expr(
b.expr.to_string(),
&flow_job.args,
last_result.clone(),
Some(idcontext.clone()),
Some(EvalCreds {
workspace: flow_job.workspace_id.clone(),
token: token.to_string(),
}),
base_internal_url,
Some(client),
)
.await?;
@@ -1757,7 +1759,7 @@ async fn compute_next_flow_transform<'c>(
Ok((
tx,
NextFlowTransform::Continue(
vec![JobPayload::RawFlow {
ContinuePayload::SingleJob(JobPayload::RawFlow {
value: FlowValue {
modules,
failure_module: fm,
@@ -1768,7 +1770,7 @@ async fn compute_next_flow_transform<'c>(
flow_job.script_path(),
status.step
)),
}],
}),
NextStatus::BranchChosen(branch),
),
))
@@ -1784,32 +1786,34 @@ async fn compute_next_flow_transform<'c>(
return Ok((
tx,
NextFlowTransform::Continue(
branches
.iter()
.enumerate()
.map(|(i, b)| {
let mut fm = flow.failure_module.clone();
if let Some(mut failure_module) =
flow.failure_module.clone()
{
failure_module
.id_append(&format!("{}/{i}", status.step));
fm = Some(failure_module);
}
JobPayload::RawFlow {
value: FlowValue {
modules: b.modules.clone(),
failure_module: fm.clone(),
same_worker: flow.same_worker,
},
path: Some(format!(
"{}/branchall-{}",
flow_job.script_path(),
i
)),
}
})
.collect(),
ContinuePayload::BranchAllJobs(
branches
.iter()
.enumerate()
.map(|(i, b)| {
let mut fm = flow.failure_module.clone();
if let Some(mut failure_module) =
flow.failure_module.clone()
{
failure_module
.id_append(&format!("{}/{i}", status.step));
fm = Some(failure_module);
}
JobPayload::RawFlow {
value: FlowValue {
modules: b.modules.clone(),
failure_module: fm.clone(),
same_worker: flow.same_worker,
},
path: Some(format!(
"{}/branchall-{}",
flow_job.script_path(),
i
)),
}
})
.collect(),
),
NextStatus::AllFlowJobs {
branchall: Some(BranchAllStatus {
branch: 0,
@@ -1866,7 +1870,7 @@ async fn compute_next_flow_transform<'c>(
Ok((
tx,
NextFlowTransform::Continue(
vec![JobPayload::RawFlow {
ContinuePayload::SingleJob(JobPayload::RawFlow {
value: FlowValue {
modules,
failure_module: fm.clone(),
@@ -1877,7 +1881,7 @@ async fn compute_next_flow_transform<'c>(
flow_job.script_path(),
branch_status.branch
)),
}],
}),
NextStatus::NextBranchStep(NextBranch { status: branch_status, flow_jobs }),
),
))
@@ -1886,59 +1890,31 @@ async fn compute_next_flow_transform<'c>(
}
async fn get_transform_context(
db: &DB,
flow_job: &QueuedJob,
previous_id: String,
status: &FlowStatus,
) -> error::Result<TransformContext> {
let tx = db.begin().await?;
let (tx, new_token) = crate::create_token_for_owner(
tx,
&flow_job.workspace_id,
&flow_job.permissioned_as,
"transform-input",
10,
&flow_job.email,
)
.await?;
//we need to commit asap otherwise the token won't be valid for auth to check outside of this transaction
//which will happen with client http calls
tx.commit().await?;
) -> error::Result<IdContext> {
let steps_results: HashMap<String, JobResult> = status
.modules
.iter()
.filter_map(|x| x.job_result().map(|y| (x.id(), y)))
.collect();
Ok((
new_token,
IdContext { flow_job: flow_job.id, steps_results, previous_id },
))
Ok(IdContext { flow_job: flow_job.id, steps_results, previous_id })
}
async fn evaluate_with<F>(
transform: InputTransform,
vars: F,
token: String,
workspace: String,
client: Option<&AuthedClient>,
by_id: Option<IdContext>,
base_internal_url: &str,
) -> anyhow::Result<serde_json::Value>
where
F: FnOnce() -> Vec<(String, serde_json::Value)>,
{
match transform {
InputTransform::Static { value } => Ok(value),
InputTransform::Javascript { expr } => {
eval_timeout(
expr,
vars(),
Some(EvalCreds { workspace, token }),
by_id,
base_internal_url,
)
.await
}
InputTransform::Javascript { expr } => eval_timeout(expr, vars(), client, by_id).await,
}
}
trait IntoArray: Sized {
+102 -28
View File
@@ -1,12 +1,22 @@
import { Any, model, property } from "./decoverto.ts";
import { requireLogin, resolveWorkspace, validatePath } from "./context.ts";
import { Any, decoverto, model, property } from "./decoverto.ts";
import {
AppService,
AppWithLastVersion,
colors,
Command,
ListableApp,
microdiff,
Policy,
Table,
} from "./deps.ts";
import { Difference, PushDiffs, Resource, setValueByPath } from "./types.ts";
import {
Difference,
GlobalOptions,
PushDiffs,
Resource,
setValueByPath,
} from "./types.ts";
@model()
export class AppFile implements Resource, PushDiffs {
@@ -17,7 +27,6 @@ export class AppFile implements Resource, PushDiffs {
@property(Any)
policy: Policy;
constructor(value: string, summary: string, policy: Policy) {
this.value = value;
this.summary = summary;
@@ -26,16 +35,20 @@ export class AppFile implements Resource, PushDiffs {
async pushDiffs(
workspace: string,
remotePath: string,
diffs: Difference[],
diffs: Difference[]
): Promise<void> {
if (await AppService.existsApp({ workspace, path: remotePath })) {
let app: AppWithLastVersion | undefined = undefined;
try {
app = await AppService.getAppByPath({ workspace, path: remotePath });
} catch (e) {}
if (app) {
console.log(
colors.bold.yellow(
`Applying ${diffs.length} diffs to existing app...`,
),
`Applying ${diffs.length} diffs to existing app... ${remotePath}`
)
);
const changeset: {
path?: string | undefined;
summary?: string | undefined;
value?: any;
policy?: Policy | undefined;
@@ -43,14 +56,10 @@ export class AppFile implements Resource, PushDiffs {
for (const diff of diffs) {
if (
diff.type !== "REMOVE" &&
(
diff.path[0] !== "value" && diff.path[0] !== "policy" && (
diff.path.length !== 1 ||
!["path", "summary"].includes(
diff.path[0] as string,
)
)
)
diff.path[0] !== "value" &&
diff.path[0] !== "policy" &&
(diff.path.length !== 1 ||
!["summary"].includes(diff.path[0] as string))
) {
throw new Error("Invalid app diff with path " + diff.path);
}
@@ -61,8 +70,21 @@ export class AppFile implements Resource, PushDiffs {
}
}
const hasChanges = Object.values(changeset).some((v) =>
v !== null && typeof v !== "undefined"
if (
(!changeset?.policy ||
JSON.stringify(changeset?.policy) == JSON.stringify(app.policy)) &&
(!changeset?.value ||
JSON.stringify(changeset?.value) == JSON.stringify(app.value)) &&
(!changeset?.summary || changeset.summary == app.summary)
) {
console.log(
colors.yellow(`No changes to push for app ${remotePath}, skipping`)
);
return;
}
const hasChanges = Object.values(changeset).some(
(v) => v !== null && typeof v !== "undefined"
);
if (!hasChanges) {
return;
@@ -75,6 +97,7 @@ export class AppFile implements Resource, PushDiffs {
});
} else {
console.log(colors.yellow.bold("Creating new app..."));
await AppService.createApp({
workspace,
requestBody: {
@@ -87,19 +110,70 @@ export class AppFile implements Resource, PushDiffs {
}
}
async push(workspace: string, remotePath: string): Promise<void> {
let existing: AppWithLastVersion | undefined;
try {
existing = await AppService.getAppByPath({
workspace: workspace,
path: remotePath,
});
} catch {
existing = undefined;
}
await this.pushDiffs(
workspace,
remotePath,
microdiff(existing ?? {}, this, { cyclesFix: false }),
microdiff({}, this, { cyclesFix: false })
);
}
}
async function list(opts: GlobalOptions) {
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
let page = 0;
const perPage = 10;
const total: ListableApp[] = [];
while (true) {
const res = await AppService.listApps({
workspace: workspace.workspaceId,
page,
perPage,
});
page += 1;
total.push(...res);
if (res.length < perPage) {
break;
}
}
new Table()
.header(["path", "summary"])
.padding(2)
.border(true)
.body(total.map((x) => [x.path, x.summary]))
.render();
}
async function push(opts: GlobalOptions, filePath: string) {
const remotePath = filePath.split(".")[0];
if (!validatePath(remotePath)) {
return;
}
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
await pushApp(filePath, workspace.workspaceId, remotePath);
console.log(colors.bold.underline.green("App pushed"));
}
export async function pushApp(
filePath: string,
workspace: string,
remotePath: string
) {
const data = decoverto
.type(AppFile)
.rawToInstance(await Deno.readTextFile(filePath));
await data.push(workspace, remotePath);
}
const command = new Command()
.description("app related commands")
.action(list as any)
.command("push", "push a local app ")
.arguments("<file_path:file>")
.action(push as any);
export default command;
+15 -29
View File
@@ -18,7 +18,7 @@ export type Context = {
};
async function tryResolveWorkspace(
opts: GlobalOptions,
opts: GlobalOptions
): Promise<
{ isError: false; value: Workspace } | { isError: true; error: string }
> {
@@ -49,17 +49,20 @@ async function tryResolveWorkspace(
}
export async function resolveWorkspace(
opts: GlobalOptions,
opts: GlobalOptions
): Promise<Workspace> {
const res = await tryResolveWorkspace(opts);
if (res.isError) {
console.log(colors.red.bold(res.error));
return Deno.exit(-1);
} else {
return res.value;
}
}
export async function requireLogin(opts: GlobalOptions): Promise<GlobalUserInfo> {
export async function requireLogin(
opts: GlobalOptions
): Promise<GlobalUserInfo> {
const workspace = await resolveWorkspace(opts);
let token = await tryGetLoginInfo(opts);
@@ -73,7 +76,7 @@ export async function requireLogin(opts: GlobalOptions): Promise<GlobalUserInfo>
return await UserService.globalWhoami();
} catch {
console.log(
"! Could not reach API given existing credentials. Attempting to reauth...",
"! Could not reach API given existing credentials. Attempting to reauth..."
);
const newToken = await loginInteractive(workspace.remote);
if (!newToken) {
@@ -85,14 +88,14 @@ export async function requireLogin(opts: GlobalOptions): Promise<GlobalUserInfo>
setClient(
token,
workspace.remote.substring(0, workspace.remote.length - 1),
workspace.remote.substring(0, workspace.remote.length - 1)
);
return await UserService.globalWhoami();
}
}
export async function tryResolveVersion(
opts: GlobalOptions,
opts: GlobalOptions
): Promise<number | undefined> {
if ((opts as any).__cache_version) {
return (opts as any).__cache_version;
@@ -102,41 +105,24 @@ export async function tryResolveVersion(
if (workspaceRes.isError) return undefined;
const response = await fetch(
new URL(new URL(workspaceRes.value.remote).origin + "/api/version"),
new URL(new URL(workspaceRes.value.remote).origin + "/api/version")
);
const version = await response.text();
try {
return Number.parseInt(
version.split("-", 1)[0].replaceAll(".", "").replace("v", ""),
version.split("-", 1)[0].replaceAll(".", "").replace("v", "")
);
} catch {
return undefined;
}
}
export async function validatePath(
opts: GlobalOptions,
path: string,
): Promise<boolean> {
const backendVersion = await tryResolveVersion(opts);
if (path.startsWith("f")) {
if (!backendVersion || backendVersion >= 1550) {
return true;
}
console.log(
`Attempting to use folders, but the current remote does not have support. Remote version is ${backendVersion} but folders are supported from 1560.`,
);
return false;
}
if (
!(path.startsWith("g") ||
path.startsWith("u"))
) {
export function validatePath(path: string): boolean {
if (!(path.startsWith("g") || path.startsWith("u") || path.startsWith("f"))) {
console.log(
colors.red(
"Given remote path looks invalid. Remote paths are typically of the form <u|g|f>/<username|group|folder>/...",
),
"Given remote path looks invalid. Remote paths are typically of the form <u|g|f>/<username|group|folder>/..."
)
);
return false;
}
+1 -3
View File
@@ -33,9 +33,7 @@ export { passwordGenerator } from "https://deno.land/x/password_generator@latest
export { nanoid } from "https://deno.land/x/nanoid@v3.0.0/mod.ts";
export * as cbor from "https://deno.land/x/cbor@v1.4.1/index.js";
export { default as Murmurhash3 } from "https://deno.land/x/murmurhash@v1.0.0/mod.ts";
export {
default as microdiff,
} from "https://deno.land/x/microdiff@v1.3.1/index.ts";
export { default as microdiff } from "https://deno.land/x/microdiff@v1.3.1/index.ts";
export { default as objectHash } from "https://deno.land/x/object_hash@2.0.3.1/mod.ts";
export { default as gitignore_parser } from "npm:gitignore-parser";
export { default as JSZip } from "npm:jszip@3.7.1";
+25 -45
View File
@@ -20,7 +20,6 @@ import { requireLogin, resolveWorkspace, validatePath } from "./context.ts";
import { resolve, track_job } from "./script.ts";
import { Any, decoverto, model, property } from "./decoverto.ts";
// this is effectively "OpenFlow" but a copy as it is accepted by the CLI
@model()
export class FlowFile implements Resource, PushDiffs {
@@ -40,7 +39,7 @@ export class FlowFile implements Resource, PushDiffs {
async pushDiffs(
workspace: string,
remotePath: string,
diffs: Difference[],
diffs: Difference[]
): Promise<void> {
if (
await FlowService.existsFlowByPath({
@@ -50,8 +49,8 @@ export class FlowFile implements Resource, PushDiffs {
) {
console.log(
colors.bold.yellow(
`Applying ${diffs.length} diffs to existing flow... ${remotePath}`,
),
`Applying ${diffs.length} diffs to existing flow... ${remotePath}`
)
);
// TODO: Make these optional in backend (not path ofc)
@@ -66,14 +65,11 @@ export class FlowFile implements Resource, PushDiffs {
for (const diff of diffs) {
if (
diff.type !== "REMOVE" &&
(
diff.path[0] !== "value" && (
diff.path.length !== 1 ||
!["summary", "description", "schema"].includes(
diff.path[0] as string,
)
)
)
diff.path[0] !== "value" &&
(diff.path.length !== 1 ||
!["summary", "description", "schema"].includes(
diff.path[0] as string
))
) {
throw new Error("Invalid flow diff with path " + diff.path);
}
@@ -83,8 +79,8 @@ export class FlowFile implements Resource, PushDiffs {
setValueByPath(changeset, diff.path, null);
}
}
const hasChanges = Object.values(changeset).some((v) =>
v !== null && typeof v !== "undefined"
const hasChanges = Object.values(changeset).some(
(v) => v !== null && typeof v !== "undefined"
);
if (!hasChanges) {
return;
@@ -93,7 +89,8 @@ export class FlowFile implements Resource, PushDiffs {
const update = {
...changeset,
...base_changeset,
}
};
await FlowService.updateFlow({
workspace: workspace,
path: remotePath,
@@ -114,20 +111,10 @@ export class FlowFile implements Resource, PushDiffs {
}
}
async push(workspace: string, remotePath: string): Promise<void> {
let remote: Flow | undefined;
try {
remote = await FlowService.getFlowByPath({
workspace,
path: remotePath,
});
} catch {
remote = undefined;
}
await this.pushDiffs(
workspace,
remotePath,
microdiff(remote ?? {}, this, { cyclesFix: false }),
microdiff({}, this, { cyclesFix: false })
);
}
}
@@ -135,24 +122,24 @@ export class FlowFile implements Resource, PushDiffs {
type Options = GlobalOptions;
async function push(opts: Options, filePath: string, remotePath: string) {
if (!await validatePath(opts, remotePath)) {
if (!validatePath(remotePath)) {
return;
}
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
await pushFlow(filePath, workspace.remote, remotePath);
await pushFlow(filePath, workspace.workspaceId, remotePath);
console.log(colors.bold.underline.green("Flow pushed"));
}
export async function pushFlow(
filePath: string,
workspace: string,
remotePath: string,
remotePath: string
) {
const data = decoverto.type(FlowFile).rawToInstance(
await Deno.readTextFile(filePath),
);
const data = decoverto
.type(FlowFile)
.rawToInstance(await Deno.readTextFile(filePath));
await data.push(workspace, remotePath);
}
@@ -181,13 +168,7 @@ async function list(opts: GlobalOptions & { showArchived?: boolean }) {
.header(["path", "summary", "edited by"])
.padding(2)
.border(true)
.body(
total.map((x) => [
x.path,
x.summary,
x.edited_by,
]),
)
.body(total.map((x) => [x.path, x.summary, x.edited_by]))
.render();
}
async function run(
@@ -195,14 +176,13 @@ async function run(
data?: string;
silent: boolean;
},
path: string,
path: string
) {
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
const input = opts.data ? await resolve(opts.data) : {};
const id = await JobService.runFlowByPath({
workspace: workspace.workspaceId,
path,
@@ -237,7 +217,7 @@ async function run(
if (!opts.silent) {
console.log(colors.green.underline.bold("Flow ran to completion"));
console.log()
console.log();
}
const jobInfo = await JobService.getCompletedJob({
workspace: workspace.workspaceId,
@@ -252,7 +232,7 @@ const command = new Command()
.action(list as any)
.command(
"push",
"push a local flow spec. This overrides any remote versions.",
"push a local flow spec. This overrides any remote versions."
)
.arguments("<file_path:string> <remote_path:string>")
.action(push as any)
@@ -260,11 +240,11 @@ const command = new Command()
.arguments("<path:string>")
.option(
"-d --data <data:string>",
"Inputs specified as a JSON string or a file using @<filename> or stdin using @-.",
"Inputs specified as a JSON string or a file using @<filename> or stdin using @-."
)
.option(
"-s --silent",
"Do not ouput anything other then the final output. Useful for scripting.",
"Do not ouput anything other then the final output. Useful for scripting."
)
.action(run as any);
+47 -31
View File
@@ -1,4 +1,4 @@
import { colors, Command, Folder, FolderService, microdiff } from "./deps.ts";
import { colors, Command, FolderService, microdiff } from "./deps.ts";
import { requireLogin, resolveWorkspace, validatePath } from "./context.ts";
import {
Difference,
@@ -20,8 +20,16 @@ import {
export class FolderFile implements Resource, PushDiffs {
@property(array(() => String))
owners: Array<string> | undefined;
@property(map(() => String, () => Boolean, { shape: MapShape.Object }))
@property(
map(
() => String,
() => Boolean,
{ shape: MapShape.Object }
)
)
extra_perms: Map<string, boolean> | undefined;
@property(() => String)
display_name: string | undefined;
async push(workspace: string, remotePath: string): Promise<void> {
if (remotePath.startsWith("/")) {
@@ -31,23 +39,17 @@ export class FolderFile implements Resource, PushDiffs {
remotePath = remotePath.substring(2);
}
let existing: Folder | undefined;
try {
existing = await FolderService.getFolder({ workspace, name: remotePath });
} catch {
existing = undefined;
}
await this.pushDiffs(
workspace,
remotePath,
microdiff(existing ?? {}, this, { cyclesFix: false }),
microdiff({}, this, { cyclesFix: false })
);
}
async pushDiffs(
workspace: string,
remotePath: string,
diffs: Difference[],
diffs: Difference[]
): Promise<void> {
if (remotePath.startsWith("/")) {
remotePath = remotePath.substring(1);
@@ -59,29 +61,34 @@ export class FolderFile implements Resource, PushDiffs {
// TODO: Support this in backend
let exists: boolean;
try {
exists = !!await FolderService.getFolder({ workspace, name: remotePath });
exists = !!(await FolderService.getFolder({
workspace,
name: remotePath,
}));
} catch {
exists = false;
}
if (exists) {
console.log(
colors.bold.yellow(
`Applying ${diffs.length} diffs to existing folder...`,
),
`Applying ${diffs.length} diffs to existing folder... ${remotePath}`
)
);
const changeset: {
owners?: string[] | undefined;
extra_perms?: any;
display_name?: string | undefined;
} = {};
for (const diff of diffs) {
if (
diff.type !== "REMOVE" &&
(
diff.path.length !== 1 ||
!["owners", "extra_perms"].includes(diff.path[0] as string)
)
(diff.path.length !== 1 ||
!["owners", "extra_perms", "display_name"].includes(
diff.path[0] as string
))
) {
console.log(diff.path);
throw new Error("Invalid folder diff with path " + diff.path);
}
if (diff.type === "CREATE" || diff.type === "CHANGE") {
@@ -91,18 +98,27 @@ export class FolderFile implements Resource, PushDiffs {
}
}
const hasChanges = Object.values(changeset).some((v) =>
v !== null && typeof v !== "undefined"
const hasChanges = Object.values(changeset).some(
(v) => v !== null && typeof v !== "undefined"
);
if (!hasChanges) {
return;
}
await FolderService.updateFolder({
workspace: workspace,
name: remotePath,
requestBody: changeset,
});
try {
await FolderService.updateFolder({
workspace: workspace,
name: remotePath,
requestBody: {
...changeset,
extra_perms: changeset.extra_perms
? Object.fromEntries(this.extra_perms?.entries() ?? [])
: undefined,
},
});
} catch (e) {
console.error(colors.red.bold(e.body));
throw e;
}
} else {
console.log(colors.bold.yellow("Creating new folder: " + remotePath));
await FolderService.createFolder({
@@ -121,7 +137,7 @@ async function push(opts: GlobalOptions, filePath: string, remotePath: string) {
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
if (!await validatePath(opts, remotePath)) {
if (!validatePath(remotePath)) {
return;
}
@@ -139,11 +155,11 @@ async function push(opts: GlobalOptions, filePath: string, remotePath: string) {
export async function pushFolder(
workspace: string,
filePath: string,
remotePath: string,
remotePath: string
) {
const data = decoverto.type(FolderFile).rawToInstance(
await Deno.readTextFile(filePath),
);
const data = decoverto
.type(FolderFile)
.rawToInstance(await Deno.readTextFile(filePath));
data.push(workspace, remotePath);
}
@@ -151,7 +167,7 @@ const command = new Command()
.description("resource related commands")
.command(
"push",
"push a local folder spec. This overrides any remote versions.",
"push a local folder spec. This overrides any remote versions."
)
.arguments("<file_path:string> <remote_path:string>")
.action(push as any);
+13 -9
View File
@@ -1,5 +1,11 @@
import { Command, CompletionsCommand, DenoLandProvider, UpgradeCommand } from "./deps.ts";
import {
Command,
CompletionsCommand,
DenoLandProvider,
UpgradeCommand,
} from "./deps.ts";
import flow from "./flow.ts";
import app from "./apps.ts";
import script from "./script.ts";
import workspace from "./workspace.ts";
import resource from "./resource.ts";
@@ -13,21 +19,21 @@ import sync from "./sync.ts";
import { tryResolveVersion } from "./context.ts";
import { GlobalOptions } from "./types.ts";
const VERSION = "v1.74.2";
export const VERSION = "v1.84.0";
let command: any = new Command()
.name("wmill")
.description("A simple CLI tool for windmill.")
.action(() => command.showHelp())
.globalOption(
"--workspace <workspace:string>",
"Specify the target workspace. This overrides the default workspace.",
"Specify the target workspace. This overrides the default workspace."
)
.globalOption(
"--token <token:string>",
"Specify an API token. This will override any stored token.",
"Specify an API token. This will override any stored token."
)
.version(VERSION)
.command("app", app)
.command("flow", flow)
.command("script", script)
.command("workspace", workspace)
@@ -59,14 +65,12 @@ let command: any = new Command()
"--unstable",
],
provider: new DenoLandProvider({ name: "wmill" }),
}),
})
)
.command("completions", new CompletionsCommand());
if (Number.parseInt(VERSION.replace("v", "").replace(".", "")) > 1700) {
command = command
.command("push", push)
.command("pull", pull);
command = command.command("push", push).command("pull", pull);
}
try {
+12 -10
View File
@@ -5,25 +5,27 @@ import { Workspace } from "./workspace.ts";
export async function downloadZip(
workspace: Workspace,
plainSecrets: boolean | undefined
): Promise<JSZip | undefined> {
const requestHeaders: HeadersInit = new Headers();
requestHeaders.set("Authorization", "Bearer " + workspace.token);
requestHeaders.set("Content-Type", "application/octet-stream");
const zipResponse = await fetch(
workspace.remote + "api/w/" + workspace.workspaceId +
"/workspaces/tarball?archive_type=zip",
workspace.remote +
"api/w/" +
workspace.workspaceId +
"/workspaces/tarball?archive_type=zip&plain_secret=" +
(plainSecrets ?? false),
{
headers: requestHeaders,
method: "GET",
},
}
);
if (!zipResponse.ok) {
console.log(
colors.red(
"Failed to request tarball from API " + zipResponse.statusText,
),
colors.red("Failed to request tarball from API " + zipResponse.statusText)
);
throw new Error(await zipResponse.text());
}
@@ -33,18 +35,18 @@ export async function downloadZip(
async function stub(
_opts: GlobalOptions & { override: boolean },
_dir: string,
_dir: string
) {
console.log(
colors.red.underline(
'Pull is deprecated. Use "sync pull --raw" instead. See <TODO_LINK_HERE> for more information.',
),
'Pull is deprecated. Use "sync pull --raw" instead. See <TODO_LINK_HERE> for more information.'
)
);
}
const command = new Command()
.description(
"Pull all definitions in the current workspace from the API and write them to disk.",
"Pull all definitions in the current workspace from the API and write them to disk."
)
.arguments("<dir:string>")
.action(stub as any);
+4 -14
View File
@@ -13,7 +13,6 @@ import {
EditResourceType,
microdiff,
ResourceService,
ResourceType,
Table,
} from "./deps.ts";
import { Any, decoverto, model, property } from "./decoverto.ts";
@@ -26,19 +25,10 @@ export class ResourceTypeFile implements ResourceI, PushDiffs {
description?: string;
async push(workspace: string, remotePath: string): Promise<void> {
let existing: ResourceType | undefined;
try {
existing = await ResourceService.getResourceType({
workspace,
path: remotePath,
});
} catch {
existing = undefined;
}
this.pushDiffs(
await this.pushDiffs(
workspace,
remotePath,
microdiff(existing ?? {}, this, { cyclesFix: false }),
microdiff({}, this, { cyclesFix: false }),
);
}
@@ -65,8 +55,8 @@ export class ResourceTypeFile implements ResourceI, PushDiffs {
return;
}
console.log(
colors.yellow(
`Applying ${diffs.length} diffs to existing resource type...`,
colors.yellow.bold(
`Applying ${diffs.length} diffs to existing resource type... ${remotePath}`,
),
);
const changeset: EditResourceType = {};
+22 -33
View File
@@ -34,7 +34,7 @@ export class ResourceFile implements Resource2, PushDiffs {
async pushDiffs(
workspace: string,
remotePath: string,
diffs: Difference[],
diffs: Difference[]
): Promise<void> {
if (
await ResourceService.existsResource({
@@ -43,7 +43,9 @@ export class ResourceFile implements Resource2, PushDiffs {
})
) {
console.log(
colors.yellow(`Applying ${diffs.length} diffs to existing resource...`),
colors.yellow.bold(
`Applying ${diffs.length} diffs to existing resource... ${remotePath}`
)
);
const changeset: EditResource = {
@@ -51,22 +53,18 @@ export class ResourceFile implements Resource2, PushDiffs {
};
for (const diff of diffs) {
if (diff.path[0] === "is_oauth") {
console.log(
colors.yellow(
"! is_oauth has been removed in newer versions. Ignoring.",
),
);
//is_oauth is not updatable
continue;
}
if (
diff.type !== "REMOVE" &&
(
diff.path[0] !== "value" && (
diff.path.length !== 1 ||
diff.path[0] !== "description"
)
)
diff.path[0] !== "value" &&
(diff.path.length !== 1 || diff.path[0] !== "description") &&
diff.path[0] !== "resource_type"
) {
console.log(
colors.red("Invalid variable diff with path " + diff.path)
);
throw new Error("Invalid folder diff with path " + diff.path);
}
if (diff.type === "CREATE" || diff.type === "CHANGE") {
@@ -76,8 +74,8 @@ export class ResourceFile implements Resource2, PushDiffs {
}
}
const hasChanges = Object.values(changeset).some((v) =>
v !== null && typeof v !== "undefined"
const hasChanges = Object.values(changeset).some(
(v) => v !== null && typeof v !== "undefined"
);
if (!hasChanges) {
return;
@@ -92,8 +90,8 @@ export class ResourceFile implements Resource2, PushDiffs {
if (typeof this.is_oauth !== "undefined") {
console.log(
colors.yellow(
"! is_oauth has been removed in newer versions. Ignoring.",
),
"! is_oauth has been removed in newer versions. Ignoring."
)
);
}
@@ -110,19 +108,10 @@ export class ResourceFile implements Resource2, PushDiffs {
}
}
async push(workspace: string, remotePath: string): Promise<void> {
let existing: Resource | undefined;
try {
existing = await ResourceService.getResource({
workspace,
path: remotePath,
});
} catch {
existing = undefined;
}
await this.pushDiffs(
workspace,
remotePath,
microdiff(existing ?? {}, this, { cyclesFix: false }),
microdiff({}, this, { cyclesFix: false })
);
}
}
@@ -130,11 +119,11 @@ export class ResourceFile implements Resource2, PushDiffs {
export async function pushResource(
workspace: string,
filePath: string,
remotePath: string,
remotePath: string
) {
const data = decoverto.type(ResourceFile).rawToInstance(
await Deno.readTextFile(filePath),
);
const data = decoverto
.type(ResourceFile)
.rawToInstance(await Deno.readTextFile(filePath));
await data.push(workspace, remotePath);
}
@@ -143,7 +132,7 @@ async function push(opts: PushOptions, filePath: string, remotePath: string) {
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
if (!await validatePath(opts, remotePath)) {
if (!validatePath(remotePath)) {
return;
}
@@ -190,7 +179,7 @@ const command = new Command()
.action(list as any)
.command(
"push",
"push a local resource spec. This overrides any remote versions.",
"push a local resource spec. This overrides any remote versions."
)
.arguments("<file_path:string> <remote_path:string>")
.action(push as any);
+139 -95
View File
@@ -32,8 +32,11 @@ export class ScriptFile {
if (data == null) return data;
if (
data === "script" || data === "failure" || data === "trigger" ||
data === "command" || data === "approvial"
data === "script" ||
data === "failure" ||
data === "trigger" ||
data === "command" ||
data === "approvial"
) {
return data;
}
@@ -42,6 +45,7 @@ export class ScriptFile {
},
toPlain: (data) => data,
})
@property(() => String)
kind?: "script" | "failure" | "trigger" | "command" | "approval";
constructor(summary: string, description: string) {
@@ -51,14 +55,11 @@ export class ScriptFile {
}
type PushOptions = GlobalOptions;
async function push(
opts: PushOptions,
filePath: string,
remotePath: string,
contentPath?: string,
) {
async function push(opts: PushOptions, filePath: string) {
const workspace = await resolveWorkspace(opts);
if (!await validatePath(opts, remotePath)) {
const remotePath = filePath.split(".")[0];
if (!validatePath(remotePath)) {
return;
}
@@ -66,89 +67,136 @@ async function push(
if (!fstat.isFile) {
throw new Error("file path must refer to a file.");
}
if (!contentPath) {
let contentPath: string;
let metaPath: string | undefined;
if (filePath.endsWith(".script.json")) {
metaPath = filePath;
contentPath = await findContentFile(filePath);
} else {
const fstat = await Deno.stat(filePath);
if (!fstat.isFile) {
throw new Error("content path must refer to a file.");
}
contentPath = filePath;
metaPath = undefined;
}
await requireLogin(opts);
await pushScript(filePath, contentPath, workspace.workspaceId, remotePath);
await pushScript(metaPath, contentPath, workspace.workspaceId, remotePath);
console.log(colors.bold.underline.green(`Script ${remotePath} pushed`));
}
export async function handleScriptMetadata(path: string, workspace: string, alreadySynced: string[]): Promise<boolean> {
export async function handleScriptMetadata(
path: string,
workspace: string,
alreadySynced: string[]
): Promise<boolean> {
if (path.endsWith(".script.json")) {
const contentPath = await findContentFile(path)
return handleFile(contentPath, await Deno.readTextFile(contentPath), workspace, alreadySynced)
const contentPath = await findContentFile(path);
return handleFile(
contentPath,
await Deno.readTextFile(contentPath),
workspace,
alreadySynced
);
} else {
return false
return false;
}
}
export async function handleFile(path: string, content: string, workspace: string, alreadySynced: string[]): Promise<boolean> {
if (path.endsWith(".ts") || path.endsWith(".py") || path.endsWith(".go") || path.endsWith(".sh")) {
export async function handleFile(
path: string,
content: string,
workspace: string,
alreadySynced: string[]
): Promise<boolean> {
if (
path.endsWith(".ts") ||
path.endsWith(".py") ||
path.endsWith(".go") ||
path.endsWith(".sh")
) {
if (alreadySynced.includes(path)) {
return true
return true;
}
alreadySynced.push(path)
alreadySynced.push(path);
const remotePath = path.substring(0, path.length - 3);
const metaPath = remotePath + ".script.json";
let typed = undefined
let typed = undefined;
try {
await Deno.stat(metaPath)
typed = JSON.parse(await Deno.readTextFile(metaPath))
await Deno.stat(metaPath);
typed = JSON.parse(await Deno.readTextFile(metaPath));
typed = decoverto.type(ScriptFile).plainToInstance(typed);
} catch { }
} catch {}
const language = inferContentTypeFromFilePath(path);
let remote = undefined;
try {
const remote = await ScriptService.getScriptByPath({
remote = await ScriptService.getScriptByPath({
workspace,
path: remotePath,
});
} catch {}
if (remote) {
if (content === remote.content) {
if (
typed == undefined ||
(typed.description === remote.description &&
typed.summary === remote.summary &&
typed.is_template === remote.is_template &&
typed.kind == remote.kind &&
remote?.lock == typed.lock?.join("\n") &&
JSON.stringify(typed.schema) == JSON.stringify(remote.schema))
) {
console.log(
colors.yellow(
`No change to push for script ${remotePath}, skipping`
)
);
return true;
}
}
await ScriptService.createScript({
workspace,
requestBody: {
content,
description: typed.description,
description: typed?.description ?? "",
language,
path: remotePath,
summary: typed.summary,
is_template: typed.is_template,
kind: typed.kind,
lock: typed.lock,
summary: typed?.summary ?? "",
is_template: typed?.is_template,
kind: typed?.kind,
lock: typed?.lock,
parent_hash: remote.hash,
schema: typed.schema,
schema: typed?.schema,
},
});
console.log(colors.yellow.bold(`Creating script with a parent ${remotePath}`))
} catch {
console.log(
colors.yellow.bold(`Creating script with a parent ${remotePath}`)
);
} else {
// no parent hash
await ScriptService.createScript({
workspace: workspace,
requestBody: {
content,
description: typed.description,
description: typed?.description ?? "",
language,
path: remotePath,
summary: typed.summary,
is_template: typed.is_template,
kind: typed.kind,
lock: typed.lock,
summary: typed?.summary ?? "",
is_template: typed?.is_template,
kind: typed?.kind,
lock: typed?.lock,
parent_hash: undefined,
schema: typed.schema,
schema: typed?.schema,
},
});
console.log(colors.yellow.bold(`Creating script without parent ${remotePath}`))
console.log(
colors.yellow.bold(`Creating script without parent ${remotePath}`)
);
}
return true
return true;
}
return false
return false;
}
export async function findContentFile(filePath: string) {
@@ -167,7 +215,7 @@ export async function findContentFile(filePath: string) {
.then((e) => {
return { path: x, file: e };
});
}),
})
)
)
.filter((x) => x.file)
@@ -175,7 +223,7 @@ export async function findContentFile(filePath: string) {
if (validCandidates.length > 1) {
throw new Error(
"No content path given and more then one candidate found: " +
validCandidates.join(", "),
validCandidates.join(", ")
);
}
if (validCandidates.length < 1) {
@@ -185,7 +233,7 @@ export async function findContentFile(filePath: string) {
}
export function inferContentTypeFromFilePath(
contentPath: string,
contentPath: string
): "python3" | "deno" | "go" | "bash" {
let language = contentPath.substring(contentPath.lastIndexOf("."));
if (language == ".ts") language = "deno";
@@ -193,7 +241,9 @@ export function inferContentTypeFromFilePath(
if (language == ".sh") language = "bash";
if (language == ".go") language = "go";
if (
language != "python3" && language != "deno" && language != "go" &&
language != "python3" &&
language != "deno" &&
language != "go" &&
language != "bash"
) {
throw new Error("Invalid language: " + language);
@@ -202,18 +252,20 @@ export function inferContentTypeFromFilePath(
}
export async function pushScript(
filePath: string,
filePath: string | undefined,
contentPath: string,
workspace: string,
remotePath: string,
remotePath: string
) {
const data = decoverto.type(ScriptFile).rawToInstance(
await Deno.readTextFile(filePath),
);
const data = filePath
? decoverto
.type(ScriptFile)
.rawToInstance(await Deno.readTextFile(filePath))
: undefined;
const content = await Deno.readTextFile(contentPath);
const language = inferContentTypeFromFilePath(contentPath);
let parent_hash = data.parent_hash;
let parent_hash = data?.parent_hash;
if (!parent_hash) {
try {
parent_hash = (
@@ -232,15 +284,15 @@ export async function pushScript(
workspace: workspace,
requestBody: {
path: remotePath,
summary: data.summary,
summary: data?.summary ?? "",
content: content,
description: data.description,
description: data?.description ?? "",
language: language,
is_template: data.is_template,
kind: data.kind,
lock: data.lock,
is_template: data?.is_template,
kind: data?.kind,
lock: data?.lock,
parent_hash: parent_hash,
schema: data.schema,
schema: data?.schema,
},
});
}
@@ -270,14 +322,7 @@ async function list(opts: GlobalOptions & { showArchived?: boolean }) {
.header(["path", "summary", "language", "created by"])
.padding(2)
.border(true)
.body(
total.map((x) => [
x.path,
x.summary,
x.language,
x.created_by,
]),
)
.body(total.map((x) => [x.path, x.summary, x.language, x.created_by]))
.render();
}
@@ -288,14 +333,15 @@ export async function resolve(input: string): Promise<Record<string, any>> {
if (input == "@-") {
input = new TextDecoder().decode(await readAll(Deno.stdin));
} if (input[0] == "@") {
}
if (input[0] == "@") {
input = await Deno.readTextFile(input.substring(1));
}
try {
return JSON.parse(input);
} catch (e) {
console.error("Impossible to parse input as JSON", input)
throw e
console.error("Impossible to parse input as JSON", input);
throw e;
}
}
@@ -304,12 +350,11 @@ async function run(
data?: string;
silent: boolean;
},
path: string,
path: string
) {
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
const input = opts.data ? await resolve(opts.data) : {};
const id = await JobService.runScriptByPath({
workspace: workspace.workspaceId,
@@ -323,12 +368,13 @@ async function run(
while (true) {
try {
const result = (
await JobService.getCompletedJob({
workspace: workspace.workspaceId,
id,
})
).result ?? {};
const result =
(
await JobService.getCompletedJob({
workspace: workspace.workspaceId,
id,
})
).result ?? {};
console.log(result);
break;
@@ -343,9 +389,9 @@ export async function track_job(workspace: string, id: string) {
const result = await JobService.getCompletedJob({ workspace, id });
console.log(result.logs);
console.log()
console.log();
console.log(colors.bold.underline.green("Job Completed"));
console.log()
console.log();
return;
} catch {
/* ignore */
@@ -396,7 +442,7 @@ export async function track_job(workspace: string, id: string) {
if (running && updates.running === false) {
running = false;
console.log(
colors.yellow("Job suspended. Waiting for it to continue..."),
colors.yellow("Job suspended. Waiting for it to continue...")
);
}
}
@@ -407,15 +453,13 @@ export async function track_job(workspace: string, id: string) {
if ((final_job.logs?.length ?? -1) > logOffset) {
console.log(final_job.logs!.substring(logOffset));
}
console.log("\n")
console.log("\n");
if (final_job.success) {
console.log(colors.bold.underline.green("Job Completed"));
} else {
console.log(colors.bold.underline.red("Job Completed"));
}
console.log()
console.log();
} catch {
console.log("Job appears to have completed, but no data can be retrieved");
}
@@ -440,22 +484,22 @@ const command = new Command()
.action(list as any)
.command(
"push",
"push a local script spec. This overrides any remote versions.",
"push a local script spec. This overrides any remote versions. Can use a script file (.ts, .js, .py, .sh) or a script spec file (.json). "
)
.arguments("<file_path:string> <remote_path:string> [content_path:string]")
.arguments("<file_path:file>")
.action(push as any)
.command("show", "show a scripts content")
.arguments("<path:string>")
.arguments("<path:file>")
.action(show as any)
.command("run", "run a script by path")
.arguments("<path:string>")
.arguments("<path:file>")
.option(
"-d --data <data:string>",
"Inputs specified as a JSON string or a file using @<filename> or stdin using @-.",
"-d --data <data:file>",
"Inputs specified as a JSON string or a file using @<filename> or stdin using @-."
)
.option(
"-s --silent",
"Do not ouput anything other then the final output. Useful for scripting.",
"Do not ouput anything other then the final output. Useful for scripting."
)
.action(run as any);
+348 -219
View File
@@ -13,7 +13,7 @@ import {
ResourceService,
VariableService,
AppService,
FlowService
FlowService,
} from "./deps.ts";
import {
Difference,
@@ -25,16 +25,14 @@ import {
import { downloadZip } from "./pull.ts";
import { FolderFile } from "./folder.ts";
import { ResourceTypeFile } from "./resource-type.ts";
import {
handleScriptMetadata,
ScriptFile,
} from "./script.ts";
import { handleScriptMetadata, ScriptFile } from "./script.ts";
import { ResourceFile } from "./resource.ts";
import { FlowFile } from "./flow.ts";
import { VariableFile } from "./variable.ts";
import { handleFile } from "./script.ts";
import { equal } from "https://deno.land/x/equal/mod.ts";
import { diffCharacters } from "https://deno.land/x/diff/mod.ts";
import { equal } from "https://deno.land/x/equal@v1.5.0/mod.ts";
import * as Diff from "npm:diff";
type DynFSElement = {
isDirectory: boolean;
path: string;
@@ -44,10 +42,7 @@ type DynFSElement = {
};
async function FSFSElement(p: string): Promise<DynFSElement> {
function _internal_element(
localP: string,
isDir: boolean,
): DynFSElement {
function _internal_element(localP: string, isDir: boolean): DynFSElement {
return {
isDirectory: isDir,
path: localP.substring(p.length + 1),
@@ -113,16 +108,14 @@ function ZipFSElement(zip: JSZip): DynFSElement {
async function* readDirRecursiveWithIgnore(
ignore: (path: string, isDirectory: boolean) => boolean,
root: DynFSElement,
): AsyncGenerator<
{
path: string;
ignored: boolean;
isDirectory: boolean;
getContentBytes(): Promise<Uint8Array>;
getContentText(): Promise<string>;
}
> {
root: DynFSElement
): AsyncGenerator<{
path: string;
ignored: boolean;
isDirectory: boolean;
getContentBytes(): Promise<Uint8Array>;
getContentText(): Promise<string>;
}> {
const stack: {
path: string;
isDirectory: boolean;
@@ -130,18 +123,20 @@ async function* readDirRecursiveWithIgnore(
c(): AsyncIterable<DynFSElement>;
getContentBytes(): Promise<Uint8Array>;
getContentText(): Promise<string>;
}[] = [{
path: root.path,
ignored: ignore(root.path, root.isDirectory),
isDirectory: root.isDirectory,
c: root.getChildren,
getContentBytes(): Promise<Uint8Array> {
throw undefined;
}[] = [
{
path: root.path,
ignored: ignore(root.path, root.isDirectory),
isDirectory: root.isDirectory,
c: root.getChildren,
getContentBytes(): Promise<Uint8Array> {
throw undefined;
},
getContentText(): Promise<string> {
throw undefined;
},
},
getContentText(): Promise<string> {
throw undefined;
},
}];
];
while (stack.length > 0) {
const e = stack.pop()!;
@@ -161,17 +156,17 @@ async function* readDirRecursiveWithIgnore(
}
type Added = { name: "added"; path: string; content: string };
type Deleted = { name: "deleted"; path: string; };
type Edit = { name: "edited"; path: string; before: string; after: string; };
type Deleted = { name: "deleted"; path: string };
type Edit = { name: "edited"; path: string; before: string; after: string };
type Change = Added | Deleted | Edit;
async function elementsToMap(els: DynFSElement, ignore: (path: string, isDirectory: boolean) => boolean): Promise<{ [key: string]: string }> {
async function elementsToMap(
els: DynFSElement,
ignore: (path: string, isDirectory: boolean) => boolean
): Promise<{ [key: string]: string }> {
const map: { [key: string]: string } = {};
for await (const entry of readDirRecursiveWithIgnore(
ignore,
els,
)) {
for await (const entry of readDirRecursiveWithIgnore(ignore, els)) {
if (entry.isDirectory || entry.ignored) continue;
const content = await entry.getContentText();
map[entry.path] = content;
@@ -179,20 +174,26 @@ async function elementsToMap(els: DynFSElement, ignore: (path: string, isDirecto
return map;
}
async function compareDynFSElement(
els1: DynFSElement, els2: DynFSElement,
ignore: (path: string, isDirectory: boolean) => boolean,
raw: boolean
els1: DynFSElement,
els2: DynFSElement | undefined,
ignore: (path: string, isDirectory: boolean) => boolean
): Promise<Change[]> {
const [m1, m2] = raw ? [await elementsToMap(els1, ignore), {}] :
await Promise.all([elementsToMap(els1, ignore), elementsToMap(els2, ignore)]);
const [m1, m2] = els2
? await Promise.all([
elementsToMap(els1, ignore),
elementsToMap(els2, ignore),
])
: [await elementsToMap(els1, ignore), {}];
const changes: Change[] = [];
for (const [k, v] of Object.entries(m1)) {
if (m2[k] === undefined) {
changes.push({ name: "added", path: k, content: v });
} else if (m2[k] != v && (!k.endsWith(".json") || !equal(JSON.parse(v), JSON.parse(m2[k])))) {
} else if (
m2[k] != v &&
(!k.endsWith(".json") || !equal(JSON.parse(v), JSON.parse(m2[k])))
) {
// await Deno.writeTextFile("/tmp/k", m2[k])
// await Deno.writeTextFile("/tmp/v", v)
// console.log(k)
@@ -208,54 +209,59 @@ async function compareDynFSElement(
}
}
return changes
return changes;
}
const isNotWmillFile = (p: string, isDirectory: boolean) => {
if (p.endsWith("/")) {
return false
return false;
}
if (isDirectory) {
return !p.startsWith("u/") && !p.startsWith("f/") && !p.startsWith("g/")
return !p.startsWith("u/") && !p.startsWith("f/") && !p.startsWith("g/");
}
try {
const typ = getTypeStrFromPath(p)
if (typ == 'resource-type') {
return p.includes('/')
const typ = getTypeStrFromPath(p);
if (typ == "resource-type") {
return p.includes("/");
} else {
return !p.startsWith("u/") && !p.startsWith("f/") && !p.startsWith("g/")
return !p.startsWith("u/") && !p.startsWith("f/") && !p.startsWith("g/");
}
} catch {
return true
return true;
}
}
};
const isWhitelisted = (p: string) => {
return p == "./" || p == "" || p == "u" || p == "f" || p == "g"
}
return p == "./" || p == "" || p == "u" || p == "f" || p == "g";
};
async function ignoreF() {
try {
const ignore: {
accepts(file: string): boolean;
denies(file: string): boolean;
} = gitignore_parser.compile(
await Deno.readTextFile(".wmillignore"),
);
} = gitignore_parser.compile(await Deno.readTextFile(".wmillignore"));
return (p: string, isDirectory: boolean) => {
return !isWhitelisted(p) && (isNotWmillFile(p, isDirectory) || ignore.denies(p));
}
return (
!isWhitelisted(p) &&
(isNotWmillFile(p, isDirectory) || ignore.denies(p))
);
};
} catch (e) {
return (p: string, isDirectory: boolean) => !isWhitelisted(p) && isNotWmillFile(p, isDirectory)
return (p: string, isDirectory: boolean) =>
!isWhitelisted(p) && isNotWmillFile(p, isDirectory);
}
}
async function pull(
opts: GlobalOptions & { raw: boolean; yes: boolean, failConflicts: boolean },
opts: GlobalOptions & {
raw: boolean;
yes: boolean;
failConflicts: boolean;
plainSecrets?: boolean;
}
) {
if (!opts.raw) {
await ensureDir(path.join(Deno.cwd(), ".wmill"));
}
@@ -263,74 +269,98 @@ async function pull(
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
console.log(colors.gray("Computing the files to update locally to match remote (taking .wmillignore into account)"));
const remote = ZipFSElement((await downloadZip(workspace))!)
const local = await FSFSElement(path.join(Deno.cwd(), opts.raw ? "" : ".wmill"))
const changes = await compareDynFSElement(remote, local, await ignoreF(), opts.raw)
console.log(
colors.gray(
"Computing the files to update locally to match remote (taking .wmillignore into account)"
)
);
const remote = ZipFSElement(
(await downloadZip(workspace, opts.plainSecrets))!
);
const local = opts.raw
? undefined
: await FSFSElement(path.join(Deno.cwd(), opts.raw ? "" : ".wmill"));
const changes = await compareDynFSElement(remote, local, await ignoreF());
console.log(`remote -> local: ${changes.length} changes to apply`);
if (changes.length > 0) {
prettyChanges(changes)
prettyChanges(changes);
if (
!opts.yes && !opts.raw && !(await Confirm.prompt({ message: `Do you want to apply these ${changes.length} changes?`, default: true }))
!opts.yes &&
!opts.raw &&
!(await Confirm.prompt({
message: `Do you want to apply these ${changes.length} changes?`,
default: true,
}))
) {
return
return;
}
const conflicts = []
const conflicts = [];
console.log(colors.gray(`Applying changes to files ...`));
for await (const change of changes) {
const target = path.join(Deno.cwd(), change.path);
const stateTarget = path.join(Deno.cwd(), ".wmill", change.path)
const stateTarget = path.join(Deno.cwd(), ".wmill", change.path);
if (change.name === "edited") {
try {
const currentLocal = await Deno.readTextFile(target)
if (currentLocal !== change.before) {
console.log(colors.red(`Conflict detected on ${change.path}\nBoth local and remote have been modified.`))
const currentLocal = await Deno.readTextFile(target);
if (currentLocal !== change.before && currentLocal !== change.after) {
console.log(
colors.red(
`Conflict detected on ${change.path}\nBoth local and remote have been modified.`
)
);
if (opts.failConflicts) {
conflicts.push({ local: currentLocal, change, path: change.path })
conflicts.push({
local: currentLocal,
change,
path: change.path,
});
continue;
} else if (opts.yes) {
console.log(colors.red(`Override local version with remote since --yes was passed and no --fail-conflicts.`))
}
else {
showConflict(change.path, currentLocal, change.after)
if (await Confirm.prompt("Preserve local (push to change remote and avoid seeing this again)?")) {
console.log(
colors.red(
`Override local version with remote since --yes was passed and no --fail-conflicts.`
)
);
} else {
showConflict(change.path, currentLocal, change.after);
if (
await Confirm.prompt(
"Preserve local (push to change remote and avoid seeing this again)?"
)
) {
continue;
}
}
}
} catch { }
} catch {}
if (change.path.endsWith(".json")) {
const diffs =
microdiff(
JSON.parse(change.before),
JSON.parse(change.after),
{ cyclesFix: false },
)
console.log(`Editing ${getTypeStrFromPath(change.path)} json ${change.path}`)
await applyDiff(
diffs,
target,
const diffs = microdiff(
JSON.parse(change.before),
JSON.parse(change.after),
{ cyclesFix: false }
);
console.log(
`Editing ${getTypeStrFromPath(change.path)} json ${change.path}`
);
await applyDiff(diffs, target);
} else {
console.log(`Editing script ${change.path}`)
console.log(`Editing script ${change.path}`);
await Deno.writeTextFile(target, change.after);
}
if (!opts.raw) {
await ensureDir(path.dirname(stateTarget))
await ensureDir(path.dirname(stateTarget));
await Deno.copyFile(target, stateTarget);
}
} else if (change.name === "added") {
await ensureDir(path.dirname(target))
await ensureDir(path.dirname(target));
if (!opts.raw) {
await ensureDir(path.dirname(stateTarget))
console.log(`Adding ${getTypeStrFromPath(change.path)} ${change.path}`)
await ensureDir(path.dirname(stateTarget));
console.log(
`Adding ${getTypeStrFromPath(change.path)} ${change.path}`
);
}
await Deno.writeTextFile(target, change.content);
if (!opts.raw) {
@@ -338,8 +368,10 @@ async function pull(
}
} else if (change.name === "deleted") {
try {
console.log(`Deleting ${getTypeStrFromPath(change.path)} ${change.path}`)
await Deno.remove(target)
console.log(
`Deleting ${getTypeStrFromPath(change.path)} ${change.path}`
);
await Deno.remove(target);
if (!opts.raw) {
await Deno.remove(stateTarget);
}
@@ -352,41 +384,46 @@ async function pull(
}
if (opts.failConflicts) {
if (conflicts.length > 0) {
console.error(colors.red(`Conflicts were found`))
console.log("Conflicts:")
console.error(colors.red(`Conflicts were found`));
console.log("Conflicts:");
for (const conflict of conflicts) {
showConflict(conflict.path, conflict.local, conflict.change.after)
showConflict(conflict.path, conflict.local, conflict.change.after);
}
console.log(colors.red(`Please resolve theses conflicts manually by either:
console.log(
colors.red(`Please resolve theses conflicts manually by either:
- reverting the content back to its remote (\`wmill pull\` and refuse to preserve local when prompted)
- pushing the changes with \`wmill push --skip-pull\` to override wmill with all your local changes
`))
Deno.exit(1)
`)
);
Deno.exit(1);
}
}
console.log(colors.green.underline(`Done! All ${changes.length} changes applied locally.`));
console.log(
colors.green.underline(
`Done! All ${changes.length} changes applied locally.`
)
);
}
function showConflict(path: string, local: string, remote: string) {
console.log(colors.yellow(`- ${path}`))
console.log(colors.yellow(`- ${path}`));
let finalString = "";
for (const character of diffCharacters(local, remote)) {
if (character.wasRemoved) {
for (const part of Diff.diffLines(local, remote)) {
if (part.removed) {
// print red if removed without newline
finalString += `\x1b[31m${character.character}\x1b[0m`;
} else if (character.wasAdded) {
finalString += `\x1b[31m${part.value}\x1b[0m`;
} else if (part.added) {
// print green if added
finalString += `\x1b[32m${character.character}\x1b[0m`;
finalString += `\x1b[32m${part.value}\x1b[0m`;
} else {
// print white if unchanged
finalString += `\x1b[37m${character.character}\x1b[0m`;
finalString += `\x1b[37m${part.value}\x1b[0m`;
}
}
console.log(finalString);
console.log("\x1b[31mlocal\x1b[31m - \x1b[32mremote\x1b[32m")
console.log()
console.log("\x1b[31mlocal\x1b[31m - \x1b[32mremote\x1b[32m");
console.log();
}
async function applyDiff(diffs: Difference[], file: string) {
ensureDir(path.dirname(file));
@@ -415,175 +452,258 @@ async function pull(
}
}
function prettyChanges(changes: Change[]) {
for (const change of changes) {
if (change.name === "added") {
console.log(colors.green(`+ ${getTypeStrFromPath(change.path)} ` + change.path));
console.log(
colors.green(`+ ${getTypeStrFromPath(change.path)} ` + change.path)
);
} else if (change.name === "deleted") {
console.log(colors.red(`- ${getTypeStrFromPath(change.path)} ` + change.path));
console.log(
colors.red(`- ${getTypeStrFromPath(change.path)} ` + change.path)
);
} else if (change.name === "edited") {
console.log(colors.yellow(`~ ${getTypeStrFromPath(change.path)} ` + change.path));
}
}
}
function prettyDiff(diffs: Difference[]) {
for (const diff of diffs) {
let pathString = "";
for (const pathSegment of diff.path) {
if (typeof pathSegment === "string") {
pathString += ".";
pathString += pathSegment;
} else {
pathString += "[";
pathString += pathSegment;
pathString += "]";
}
}
if (diff.type === "REMOVE" || diff.type === "CHANGE") {
console.log(colors.red("- " + pathString + " = " + diff.oldValue));
}
if (diff.type === "CREATE" || diff.type === "CHANGE") {
console.log(colors.green("+ " + pathString + " = " + diff.value));
console.log(
colors.yellow(`~ ${getTypeStrFromPath(change.path)} ` + change.path)
);
}
}
}
// function prettyDiff(diffs: Difference[]) {
// for (const diff of diffs) {
// let pathString = "";
// for (const pathSegment of diff.path) {
// if (typeof pathSegment === "string") {
// pathString += ".";
// pathString += pathSegment;
// } else {
// pathString += "[";
// pathString += pathSegment;
// pathString += "]";
// }
// }
// if (diff.type === "REMOVE" || diff.type === "CHANGE") {
// console.log(colors.red("- " + pathString + " = " + diff.oldValue));
// }
// if (diff.type === "CREATE" || diff.type === "CHANGE") {
// console.log(colors.green("+ " + pathString + " = " + diff.value));
// }
// }
// }
function removeSuffix(str: string, suffix: string) {
return str.slice(0, str.length - suffix.length);
}
async function push(opts: GlobalOptions & { raw: boolean, yes: boolean, skipPull: boolean, failConflicts: boolean }) {
async function push(
opts: GlobalOptions & {
raw: boolean;
yes: boolean;
skipPull: boolean;
failConflicts: boolean;
plainSecrets?: boolean;
}
) {
if (!opts.raw) {
if (!opts.skipPull) {
console.log(colors.gray("You need to be up-to-date before pushing, pulling first."))
await pull(opts)
console.log(colors.green("Pull done, now pushing."))
console.log()
console.log(
colors.gray("You need to be up-to-date before pushing, pulling first.")
);
await pull(opts);
console.log(colors.green("Pull done, now pushing."));
console.log();
}
}
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
console.log(colors.gray("Computing the files to update on the remote to match local (taking .wmillignore into account)"));
const remote = ZipFSElement((await downloadZip(workspace))!)
const local = await FSFSElement(path.join(Deno.cwd(), ""))
const changes = await compareDynFSElement(local, remote, await ignoreF(), opts.raw)
console.log(
colors.gray(
"Computing the files to update on the remote to match local (taking .wmillignore into account)"
)
);
const remote = opts.raw
? undefined
: ZipFSElement((await downloadZip(workspace, opts.plainSecrets))!);
const local = await FSFSElement(path.join(Deno.cwd(), ""));
const changes = await compareDynFSElement(local, remote, await ignoreF());
console.log(`remote <- local: ${changes.length} changes to apply`);
if (changes.length > 0) {
prettyChanges(changes)
prettyChanges(changes);
if (
!opts.yes && !(await Confirm.prompt({ message: `Do you want to apply these ${changes.length} changes?`, default: true }))
!opts.yes &&
!(await Confirm.prompt({
message: `Do you want to apply these ${changes.length} changes?`,
default: true,
}))
) {
return
return;
}
console.log(colors.gray(`Applying changes to files ...`));
const alreadySynced: string[] = []
const alreadySynced: string[] = [];
for await (const change of changes) {
const stateTarget = path.join(Deno.cwd(), ".wmill", change.path)
const stateTarget = path.join(Deno.cwd(), ".wmill", change.path);
let stateExists = true;
try {
await Deno.stat(stateTarget);
} catch {
stateExists = false;
}
if (change.name === "edited") {
if (await handleScriptMetadata(change.path, workspace.workspaceId, alreadySynced)) {
if (!opts.raw) {
if (
await handleScriptMetadata(
change.path,
workspace.workspaceId,
alreadySynced
)
) {
if (!opts.raw && stateExists) {
await Deno.writeTextFile(stateTarget, change.after);
}
continue
} else if (await handleFile(change.path, change.after, workspace.workspaceId, alreadySynced)) {
if (!opts.raw) {
continue;
} else if (
await handleFile(
change.path,
change.after,
workspace.workspaceId,
alreadySynced
)
) {
if (!opts.raw && stateExists) {
await Deno.writeTextFile(stateTarget, change.after);
}
continue
continue;
}
if (!opts.raw) {
await ensureDir(path.dirname(stateTarget))
console.log(`Editing ${getTypeStrFromPath(change.path)} ${change.path}`)
await ensureDir(path.dirname(stateTarget));
console.log(
`Editing ${getTypeStrFromPath(change.path)} ${change.path}`
);
}
const obj = inferTypeFromPath(change.path, JSON.parse(change.after))
const obj = inferTypeFromPath(change.path, JSON.parse(change.after));
const diff = microdiff(inferTypeFromPath(change.path, JSON.parse(change.before)), obj, { cyclesFix: false });
const diff = microdiff(
inferTypeFromPath(change.path, JSON.parse(change.before)),
obj,
{ cyclesFix: false }
);
await applyDiff(
workspace.workspaceId,
change.path.split(".")[0],
obj,
diff,
opts.plainSecrets
);
if (!opts.raw) {
if (!opts.raw && stateExists) {
await Deno.writeTextFile(stateTarget, change.after);
}
} else if (change.name === "added") {
if (change.path.endsWith(".script.json")) {
continue
} else if (await handleFile(change.path, change.content, workspace.workspaceId, alreadySynced)) {
continue
continue;
} else if (
await handleFile(
change.path,
change.content,
workspace.workspaceId,
alreadySynced
)
) {
continue;
}
if (!opts.raw) {
await ensureDir(path.dirname(stateTarget))
console.log(`Adding ${getTypeStrFromPath(change.path)} ${change.path}`)
if (!opts.raw && stateExists) {
await ensureDir(path.dirname(stateTarget));
console.log(
`Adding ${getTypeStrFromPath(change.path)} ${change.path}`
);
}
const obj = inferTypeFromPath(change.path, JSON.parse(change.content))
const obj = inferTypeFromPath(change.path, JSON.parse(change.content));
const diff = microdiff({}, obj, { cyclesFix: false });
await applyDiff(
workspace.workspaceId,
change.path.split(".")[0],
obj,
diff,
opts.plainSecrets
);
if (!opts.raw) {
if (!opts.raw && stateExists) {
await Deno.writeTextFile(stateTarget, change.content);
}
} else if (change.name === "deleted") {
if (!change.path.includes(".json")) {
continue
continue;
}
console.log(`Deleting ${getTypeStrFromPath(change.path)} ${change.path}`)
const typ = getTypeStrFromPath(change.path)
console.log(
`Deleting ${getTypeStrFromPath(change.path)} ${change.path}`
);
const typ = getTypeStrFromPath(change.path);
const workspaceId = workspace.workspaceId;
switch (typ) {
case "script": {
const script = await ScriptService.getScriptByPath({ workspace: workspaceId, path: removeSuffix(change.path, ".script.json") })
await ScriptService.deleteScriptByHash({ workspace: workspaceId, hash: script.hash })
const script = await ScriptService.getScriptByPath({
workspace: workspaceId,
path: removeSuffix(change.path, ".script.json"),
});
await ScriptService.deleteScriptByHash({
workspace: workspaceId,
hash: script.hash,
});
break;
}
case "folder":
await FolderService.deleteFolder({ workspace: workspaceId, name: change.path.split('/')[1] })
await FolderService.deleteFolder({
workspace: workspaceId,
name: change.path.split("/")[1],
});
break;
case "resource":
await ResourceService.deleteResource({ workspace: workspaceId, path: removeSuffix(change.path, ".resource.json") })
await ResourceService.deleteResource({
workspace: workspaceId,
path: removeSuffix(change.path, ".resource.json"),
});
break;
case "resource-type":
await ResourceService.deleteResourceType({ workspace: workspaceId, path: removeSuffix(change.path, ".resource-type.json") })
break
await ResourceService.deleteResourceType({
workspace: workspaceId,
path: removeSuffix(change.path, ".resource-type.json"),
});
break;
case "flow":
await FlowService.deleteFlowByPath({ workspace: workspaceId, path: removeSuffix(change.path, ".flow.json") })
break
await FlowService.deleteFlowByPath({
workspace: workspaceId,
path: removeSuffix(change.path, ".flow.json"),
});
break;
case "app":
await AppService.deleteApp({ workspace: workspaceId, path: removeSuffix(change.path, ".app.json") })
break
await AppService.deleteApp({
workspace: workspaceId,
path: removeSuffix(change.path, ".app.json"),
});
break;
case "variable":
await VariableService.deleteVariable({ workspace: workspaceId, path: removeSuffix(change.path, ".variable.json") })
break
await VariableService.deleteVariable({
workspace: workspaceId,
path: removeSuffix(change.path, ".variable.json"),
});
break;
default:
break;
}
try {
Deno.remove(stateTarget)
} catch { }
await Deno.remove(stateTarget);
} catch {}
}
}
console.log(colors.green.underline(`Done! All ${changes.length} changes pushed to the remote workspace.`));
console.log(
colors.green.underline(
`Done! All ${changes.length} changes pushed to the remote workspace.`
)
);
}
async function applyDiff(
workspace: string,
remotePath: string,
@@ -595,10 +715,11 @@ async function push(opts: GlobalOptions & { raw: boolean, yes: boolean, skipPull
| ResourceTypeFile
| FolderFile,
diffs: Difference[],
plainSecrets?: boolean
) {
if (file instanceof ScriptFile) {
throw new Error(
"This code path should be unreachable - we should never generate diffs for scripts",
"This code path should be unreachable - we should never generate diffs for scripts"
);
} else if (file instanceof FolderFile) {
const parts = remotePath.split("/");
@@ -609,14 +730,14 @@ async function push(opts: GlobalOptions & { raw: boolean, yes: boolean, skipPull
}
}
if (diffs.length === 0) {
console.log("No diffs to apply to " + remotePath)
console.log("No diffs to apply to " + remotePath);
return;
}
try {
await file.pushDiffs(workspace, remotePath, diffs);
await file.pushDiffs(workspace, remotePath, diffs, plainSecrets);
} catch (e) {
console.error("Failing to apply diffs to " + remotePath)
console.error(e.body)
console.error("Failing to apply diffs to " + remotePath);
console.error(JSON.stringify(e));
}
}
}
@@ -624,20 +745,28 @@ async function push(opts: GlobalOptions & { raw: boolean, yes: boolean, skipPull
const command = new Command()
.command("pull")
.description(
"Pull any remote changes and apply them locally. Use --raw for usage without local state tracking.",
"Pull any remote changes and apply them locally. Use --raw for usage without local state tracking."
)
.option(
"--fail-conflicts",
"Error on conflicts (both remote and local have changes on the same item)"
)
.option("--fail-conflicts", "Error on conflicts (both remote and local have changes on the same item)")
.option("--yes", "Pull without needing confirmation")
.option("--raw", "Pull without using state, just overwrite.")
.option("--plain-secrets", "Pull secrets as plain text")
.action(pull as any)
.command("push")
.description(
"Push any local changes and apply them remotely. Use --raw for usage without local state tracking.",
"Push any local changes and apply them remotely. Use --raw for usage without local state tracking."
)
.option("--fail-conflicts", "Error on conflicts (both remote and local have changes on the same item)")
.option("--skip-pull", "Push without pulling first")
.option(
"--fail-conflicts",
"Error on conflicts (both remote and local have changes on the same item)"
)
.option("--skip-pull", "Push without pulling first (you have pulled prior)")
.option("--yes", "Push without needing confirmation")
.option("--raw", "Push without using state, just overwrite.")
.option("--plain-secrets", "Push secrets as plain text")
.action(push as any);
export default command;
+22 -9
View File
@@ -11,7 +11,11 @@ import { AppFile } from "./apps.ts";
// TODO: Remove this & replace with a "pull" that lets the object either pull the remote version or return undefined.
// Then combine those with diffing, which then gives the new push impl
export interface Resource {
push(workspace: string, remotePath: string): Promise<void>;
push(
workspace: string,
remotePath: string,
plainSecrets?: boolean
): Promise<void>;
}
export interface PushDiffs {
@@ -19,6 +23,7 @@ export interface PushDiffs {
workspace: string,
remotePath: string,
diffs: Difference[],
plainSecrets?: boolean
): Promise<void>;
}
@@ -46,7 +51,7 @@ export type Difference = DifferenceCreate | DifferenceRemove | DifferenceChange;
export function setValueByPath(
obj: any,
path: (string | number)[],
value: any,
value: any
) {
let i;
let lastObj = undefined;
@@ -84,7 +89,7 @@ export type GlobalOptions = {
export function inferTypeFromPath(
p: string,
obj: any,
obj: any
):
| ScriptFile
| VariableFile
@@ -115,7 +120,7 @@ export function inferTypeFromPath(
}
export function getTypeStrFromPath(
p: string,
p: string
):
| "script"
| "variable"
@@ -125,8 +130,13 @@ export function getTypeStrFromPath(
| "folder"
| "app" {
const parsed = path.parse(p);
if (parsed.ext == ".go" || parsed.ext == ".ts" || parsed.ext == ".sh" || parsed.ext == ".py") {
return 'script'
if (
parsed.ext == ".go" ||
parsed.ext == ".ts" ||
parsed.ext == ".sh" ||
parsed.ext == ".py"
) {
return "script";
}
if (parsed.name === "folder.meta") {
@@ -135,9 +145,12 @@ export function getTypeStrFromPath(
const typeEnding = parsed.name.split(".").at(-1);
if (
typeEnding === "script" || typeEnding === "variable" ||
typeEnding === "flow" || typeEnding === "resource" ||
typeEnding === "resource-type" || typeEnding === "app"
typeEnding === "script" ||
typeEnding === "variable" ||
typeEnding === "flow" ||
typeEnding === "resource" ||
typeEnding === "resource-type" ||
typeEnding === "app"
) {
return typeEnding;
} else {
+47 -35
View File
@@ -11,7 +11,6 @@ import {
colors,
Command,
EditVariable,
ListableVariable,
microdiff,
Table,
VariableService,
@@ -36,7 +35,7 @@ async function list(opts: GlobalOptions) {
x.is_secret ? "true" : "false",
x.account ?? "-",
x.value ?? "-",
]),
])
)
.render();
}
@@ -63,24 +62,31 @@ export class VariableFile implements Resource, PushDiffs {
workspace: string,
remotePath: string,
diffs: Difference[],
plainSecrets?: boolean
): Promise<void> {
if (await VariableService.existsVariable({ workspace, path: remotePath })) {
console.log(
colors.bold.yellow(
`Applying ${diffs.length} diffs to existing variable...`,
),
`Applying ${diffs.length} diffs to existing variable... ${remotePath}`
)
);
const changeset: EditVariable = {};
for (const diff of diffs) {
if (
diff.type !== "REMOVE" &&
(
diff.path.length !== 1 ||
!["path", "value", "is_secret", "description"].includes(
diff.path[0] as string,
)
)
(diff.path.length !== 1 ||
![
"path",
"value",
"is_secret",
"description",
"account",
"is_oauth",
].includes(diff.path[0] as string))
) {
console.log(
colors.red("Invalid variable diff with path " + diff.path)
);
throw new Error("Invalid variable diff with path " + diff.path);
}
if (diff.type === "CREATE" || diff.type === "CHANGE") {
@@ -90,25 +96,24 @@ export class VariableFile implements Resource, PushDiffs {
}
}
const hasChanges = Object.values(changeset).some((v) =>
v !== null && typeof v !== "undefined"
const hasChanges = Object.values(changeset).some(
(v) => v !== null && typeof v !== "undefined"
);
if (!hasChanges) {
return;
}
await VariableService.updateVariable({
workspace,
path: remotePath,
alreadyEncrypted: true,
alreadyEncrypted: !plainSecrets,
requestBody: changeset,
});
console.log(changeset);
} else {
console.log(colors.yellow.bold("Creating new variable..."));
await VariableService.createVariable({
workspace,
alreadyEncrypted: true,
alreadyEncrypted: !plainSecrets,
requestBody: {
path: remotePath,
description: this.description,
@@ -120,29 +125,29 @@ export class VariableFile implements Resource, PushDiffs {
});
}
}
async push(workspace: string, remotePath: string): Promise<void> {
let existing: ListableVariable | undefined;
try {
existing = await VariableService.getVariable({
workspace: workspace,
path: remotePath,
});
} catch {
existing = undefined;
}
async push(
workspace: string,
remotePath: string,
plainSecrets?: boolean
): Promise<void> {
await this.pushDiffs(
workspace,
remotePath,
microdiff(existing ?? {}, this, { cyclesFix: false }),
microdiff({}, this, { cyclesFix: false }),
plainSecrets
);
}
}
async function push(opts: GlobalOptions, filePath: string, remotePath: string) {
async function push(
opts: GlobalOptions & { plainSecrets: boolean },
filePath: string,
remotePath: string
) {
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
if (!await validatePath(opts, remotePath)) {
if (!validatePath(remotePath)) {
return;
}
@@ -153,7 +158,12 @@ async function push(opts: GlobalOptions, filePath: string, remotePath: string) {
console.log(colors.bold.yellow("Pushing variable..."));
await pushVariable(workspace.workspaceId, filePath, remotePath);
await pushVariable(
workspace.workspaceId,
filePath,
remotePath,
opts.plainSecrets
);
console.log(colors.bold.underline.green(`Variable ${remotePath} pushed`));
}
@@ -161,11 +171,12 @@ export async function pushVariable(
workspace: string,
filePath: string,
remotePath: string,
plainSecrets: boolean
) {
const data = decoverto.type(VariableFile).rawToInstance(
await Deno.readTextFile(filePath),
);
await data.push(workspace, remotePath);
const data = decoverto
.type(VariableFile)
.rawToInstance(await Deno.readTextFile(filePath));
await data.push(workspace, remotePath, plainSecrets);
}
const command = new Command()
@@ -173,9 +184,10 @@ const command = new Command()
.action(list as any)
.command(
"push",
"Push a local variable spec. This overrides any remote versions.",
"Push a local variable spec. This overrides any remote versions."
)
.arguments("<file_path:string> <remote_path:string>")
.option("--plain-secrets", "Push secrets as plain text")
.action(push as any);
export default command;
+90 -54
View File
@@ -30,7 +30,7 @@ export class Workspace {
remote: string,
workspaceId: string,
name: string,
token: string,
token: string
) {
this.remote = remote;
this.workspaceId = workspaceId;
@@ -40,7 +40,7 @@ export class Workspace {
}
function makeWorkspaceStream(
readable: ReadableStream<Uint8Array>,
readable: ReadableStream<Uint8Array>
): ReadableStream<Workspace> {
return readable
.pipeThrough(new DelimiterStream(new TextEncoder().encode("\n")))
@@ -59,7 +59,7 @@ function makeWorkspaceStream(
/* ignore */
}
},
}),
})
);
}
@@ -86,7 +86,7 @@ export async function allWorkspaces(): Promise<Workspace[]> {
}
async function getActiveWorkspaceName(
opts: GlobalOptions,
opts: GlobalOptions
): Promise<string | undefined> {
if (opts.workspace) {
return opts.workspace;
@@ -99,7 +99,7 @@ async function getActiveWorkspaceName(
}
export async function getActiveWorkspace(
opts: GlobalOptions,
opts: GlobalOptions
): Promise<Workspace | undefined> {
const name = await getActiveWorkspaceName(opts);
if (!name) {
@@ -109,7 +109,7 @@ export async function getActiveWorkspace(
}
export async function getWorkspaceByName(
workspaceName: string,
workspaceName: string
): Promise<Workspace | undefined> {
const workspaceStream = await getWorkspaceStream();
for await (const workspace of workspaceStream) {
@@ -136,36 +136,40 @@ async function list(opts: GlobalOptions) {
} else {
return a;
}
}),
})
)
.render();
console.log('Active: ' + colors.green.bold(activeName || 'none'))
console.log("Active: " + colors.green.bold(activeName || "none"));
}
async function switchC(opts: GlobalOptions, workspaceName: string) {
if (opts.workspace) {
console.log(
colors.red.bold(
"! Workspace needs to be specified as positional argument, not as option.",
),
"! Workspace needs to be specified as positional argument, not as option."
)
);
return;
}
const all = await allWorkspaces();
if (all.findIndex((x) => x.name === workspaceName) === -1) {
console.log(colors.red.bold(`! This workspace profile ${workspaceName} does not exist locally.`));
console.log("available workspace profiles:")
console.log(
colors.red.bold(
`! This workspace profile ${workspaceName} does not exist locally.`
)
);
console.log("available workspace profiles:");
for (const w of all) {
console.log(' - ' + w.name)
console.log(" - " + w.name);
}
return;
}
return await Deno.writeTextFile(
(await getRootStore()) + "/activeWorkspace",
workspaceName,
workspaceName
);
}
@@ -177,13 +181,13 @@ export async function add(
},
workspaceName: string | undefined,
workspaceId: string | undefined,
remote: string | undefined,
remote: string | undefined
) {
if (opts.workspace) {
console.log(
colors.red.bold(
"! Workspace needs to be specified as positional argument, not as option.",
),
"! Workspace needs to be specified as positional argument, not as option."
)
);
return;
}
@@ -193,7 +197,11 @@ export async function add(
}
if (!workspaceId) {
workspaceId = await Input.prompt({ message: "Enter the ID of this workspace", default: workspaceName, suggestions: [workspaceName] });
workspaceId = await Input.prompt({
message: "Enter the ID of this workspace",
default: workspaceName,
suggestions: [workspaceName],
});
}
if (!remote) {
@@ -209,8 +217,8 @@ export async function add(
await Input.prompt({
message: "Enter the Remote URL",
suggestions: ["https://app.windmill.dev/"],
default: "https://app.windmill.dev/"
}),
default: "https://app.windmill.dev/",
})
).toString();
}
}
@@ -223,22 +231,26 @@ export async function add(
setClient(
token,
remote.endsWith("/") ? remote.substring(0, remote.length - 1) : remote,
remote.endsWith("/") ? remote.substring(0, remote.length - 1) : remote
);
let alreadyExists = false
let alreadyExists = false;
try {
alreadyExists = await WorkspaceService.existsWorkspace({
requestBody: { id: workspaceId },
})
});
} catch (e) {
console.log(colors.red.bold("! Credentials or instance is invalid. Aborting."));
throw e
console.log(
colors.red.bold("! Credentials or instance is invalid. Aborting.")
);
throw e;
}
if (opts.create) {
if (
!alreadyExists
) {
console.log(colors.yellow(`Workspace at id ${workspaceId} on ${remote} does not exist. Creating...`));
if (!alreadyExists) {
console.log(
colors.yellow(
`Workspace at id ${workspaceId} on ${remote} does not exist. Creating...`
)
);
await WorkspaceService.createWorkspace({
requestBody: {
id: workspaceId,
@@ -248,63 +260,87 @@ export async function add(
});
}
} else if (!alreadyExists) {
console.log(colors.red.bold(`! Workspace at id ${workspaceId} on ${remote} does not exist. Re-run with --create to create it. Aborting.`));
console.log("On that instance and with those credentials, the workspaces that you can access are:")
const workspaces = await WorkspaceService.listWorkspaces()
console.log(
colors.red.bold(
`! Workspace at id ${workspaceId} on ${remote} does not exist. Re-run with --create to create it. Aborting.`
)
);
console.log(
"On that instance and with those credentials, the workspaces that you can access are:"
);
const workspaces = await WorkspaceService.listWorkspaces();
for (const workspace of workspaces) {
console.log(`- ${workspace.id} (name: ${workspace.name})`)
console.log(`- ${workspace.id} (name: ${workspace.name})`);
}
Deno.exit(1);
}
await addWorkspace({
name: workspaceName,
remote: remote,
workspaceId: workspaceId,
token: token,
}, opts);
await addWorkspace(
{
name: workspaceName,
remote: remote,
workspaceId: workspaceId,
token: token,
},
opts
);
await Deno.writeTextFile(
(await getRootStore()) + "/activeWorkspace",
workspaceName,
workspaceName
);
console.log(
colors.green.underline(
`Added workspace ${workspaceName} for ${workspaceId} on ${remote}!`
)
);
console.log(colors.green.underline(`Added workspace ${workspaceName} for ${workspaceId} on ${remote}!`));
}
export async function addWorkspace(workspace: Workspace, opts: any) {
workspace.remote = new URL(workspace.remote).toString(); // add trailing slash in all cases!
await removeWorkspace(workspace.name, true, opts);
const file = await Deno.open((await getRootStore()) + "remotes.ndjson", {
append: true,
write: true,
read: true,
create: true,
});
await removeWorkspace(workspace.name, true, opts);
await file.write(new TextEncoder().encode(JSON.stringify(workspace) + "\n"));
file.close();
}
export async function removeWorkspace(name: string, silent: boolean, opts: any) {
export async function removeWorkspace(
name: string,
silent: boolean,
opts: any
) {
const orgWorkspaces = await allWorkspaces();
if (orgWorkspaces.findIndex((x) => x.name === name) === -1) {
if (!silent) {
console.log(colors.red.bold(`! Workspace profile ${name} does not exist locally`));
console.log("available workspace profiles:")
await list(opts)
console.log(
colors.red.bold(`! Workspace profile ${name} does not exist locally`)
);
console.log("available workspace profiles:");
await list(opts);
}
return;
}
if (silent) {
console.log(colors.yellow(`Replacing existing workspace ${name}`))
console.log(colors.yellow(`Replacing existing workspace ${name}`));
}
await Deno.writeTextFile(
(await getRootStore()) + "remotes.ndjson",
orgWorkspaces
.filter((x) => x.name !== name)
.map((x) => JSON.stringify(x))
.join("\n"),
.join("\n") + "\n"
);
if (!silent) {
console.log(colors.green.underline(`Succesfully removed workspace ${name}!`));
console.log(
colors.green.underline(`Succesfully removed workspace ${name}!`)
);
}
}
@@ -313,10 +349,10 @@ async function remove(_opts: GlobalOptions, name: string) {
}
async function whoami(_opts: GlobalOptions) {
await requireLogin(_opts)
console.log(await UserService.globalWhoami())
await requireLogin(_opts);
console.log(await UserService.globalWhoami());
const activeName = await getActiveWorkspaceName(_opts);
console.log('Active: ' + colors.green.bold(activeName || 'none'))
console.log("Active: " + colors.green.bold(activeName || "none"));
}
const command = new Command()
@@ -333,14 +369,14 @@ const command = new Command()
.option("-c --create", "Create the workspace if it does not exist")
.option(
"--create-workspace-name <workspace_name:string>",
"Specify the workspace name. Ignored if --create is not specified or the workspace already exists. Will default to the workspace id.",
"Specify the workspace name. Ignored if --create is not specified or the workspace already exists. Will default to the workspace id."
)
.option(
"--create-username <username:string>",
"Specify your own username in the newly created workspace. Ignored if --create is not specified or the workspace already exists.",
{
default: "admin",
},
}
)
.action(add as any)
.command("remove")
+173 -126
View File
@@ -1,41 +1,59 @@
import { JobService, ResourceService, VariableService } from './windmill-api/index.ts'
import { OpenAPI } from './windmill-api/index.ts'
import {
JobService,
ResourceService,
VariableService,
} from "./windmill-api/index.ts";
import { OpenAPI } from "./windmill-api/index.ts";
export {
AdminService, AuditService, FlowService, GranularAclService, GroupService,
JobService, ResourceService, VariableService, ScriptService, ScheduleService, SettingsService,
UserService, WorkspaceService
} from './windmill-api/index.ts'
AdminService,
AuditService,
FlowService,
GranularAclService,
GroupService,
JobService,
ResourceService,
VariableService,
ScriptService,
ScheduleService,
SettingsService,
UserService,
WorkspaceService,
} from "./windmill-api/index.ts";
// @ts-ignore: Otherwise BigInt is not supported for export
BigInt.prototype.toJSON = function () {
return this.toString();
return this.toString();
};
export { pgSql, pgClient } from './pg.ts'
export { pgSql, pgClient } from "./pg.ts";
export type Sql = string
export type Email = string
export type Base64 = string
export type Resource<S extends string> = any
export type Sql = string;
export type Email = string;
export type Base64 = string;
export type Resource<S extends string> = any;
export const SHARED_FOLDER = '/shared'
export const SHARED_FOLDER = "/shared";
export function setClient(token: string, baseUrl: string) {
OpenAPI.WITH_CREDENTIALS = true
OpenAPI.TOKEN = token
OpenAPI.BASE = baseUrl + '/api'
OpenAPI.WITH_CREDENTIALS = true;
OpenAPI.TOKEN = token;
OpenAPI.BASE = baseUrl + "/api";
}
setClient(Deno.env.get("WM_TOKEN") ?? 'no_token', Deno.env.get("BASE_INTERNAL_URL") ?? Deno.env.get("BASE_URL") ?? 'http://localhost:8000')
setClient(
Deno.env.get("WM_TOKEN") ?? "no_token",
Deno.env.get("BASE_INTERNAL_URL") ??
Deno.env.get("BASE_URL") ??
"http://localhost:8000"
);
/**
* Create a client configuration from env variables
* @returns client configuration
*/
export function getWorkspace(): string {
return Deno.env.get("WM_WORKSPACE") ?? 'no_workspace'
return Deno.env.get("WM_WORKSPACE") ?? "no_workspace";
}
/**
@@ -44,27 +62,30 @@ export function getWorkspace(): string {
* @param undefinedIfEmpty if the resource does not exist, return undefined instead of throwing an error
* @returns resource value
*/
export async function getResource(path?: string, undefinedIfEmpty?: boolean): Promise<any> {
const workspace = getWorkspace()
path = path ?? getStatePath()
try {
const resource = await ResourceService.getResource({ workspace, path })
return await _transformLeaf(resource.value)
} catch (e: any) {
if (undefinedIfEmpty && e.status === 404) {
return undefined
} else {
throw Error(`Resource not found at ${path} or not visible to you`)
}
export async function getResource(
path?: string,
undefinedIfEmpty?: boolean
): Promise<any> {
const workspace = getWorkspace();
path = path ?? getStatePath();
try {
const resource = await ResourceService.getResource({ workspace, path });
return await _transformLeaf(resource.value);
} catch (e: any) {
if (undefinedIfEmpty && e.status === 404) {
return undefined;
} else {
throw Error(`Resource not found at ${path} or not visible to you`);
}
}
}
export function getStatePath(): string {
const state_path = Deno.env.get("WM_STATE_PATH")
if (state_path === undefined) {
throw Error("State path not set")
}
return state_path
const state_path = Deno.env.get("WM_STATE_PATH");
if (state_path === undefined) {
throw Error("State path not set");
}
return state_path;
}
/**
@@ -73,16 +94,29 @@ export function getStatePath(): string {
* @param value new value of the resource to set
* @param initializeToTypeIfNotExist if the resource does not exist, initialize it with this type
*/
export async function setResource(value: any, path?: string, initializeToTypeIfNotExist?: string): Promise<void> {
path = path ?? getStatePath()
const workspace = getWorkspace()
if (await ResourceService.existsResource({ workspace, path })) {
await ResourceService.updateResourceValue({ workspace, path, requestBody: { value } })
} else if (initializeToTypeIfNotExist) {
await ResourceService.createResource({ workspace, requestBody: { path, value, resource_type: initializeToTypeIfNotExist } })
} else {
throw Error(`Resource at path ${path} does not exist and no type was provided to initialize it`)
}
export async function setResource(
value: any,
path?: string,
initializeToTypeIfNotExist?: string
): Promise<void> {
path = path ?? getStatePath();
const workspace = getWorkspace();
if (await ResourceService.existsResource({ workspace, path })) {
await ResourceService.updateResourceValue({
workspace,
path,
requestBody: { value },
});
} else if (initializeToTypeIfNotExist) {
await ResourceService.createResource({
workspace,
requestBody: { path, value, resource_type: initializeToTypeIfNotExist },
});
} else {
throw Error(
`Resource at path ${path} does not exist and no type was provided to initialize it`
);
}
}
/**
@@ -91,7 +125,7 @@ export async function setResource(value: any, path?: string, initializeToTypeIfN
* @deprecated use setState instead
*/
export async function setInternalState(state: any): Promise<void> {
await setResource(state, undefined, 'state')
await setResource(state, undefined, "state");
}
/**
@@ -99,39 +133,41 @@ export async function setInternalState(state: any): Promise<void> {
* @param state state to set
*/
export async function setState(state: any): Promise<void> {
await setResource(state, undefined, 'state')
await setResource(state, undefined, "state");
}
/**
* Set the shared state
* @param state state to set
*/
export async function setSharedState(state: any, path = 'state.json'): Promise<void> {
await Deno.writeTextFile(SHARED_FOLDER + '/' + path, JSON.stringify(state))
export async function setSharedState(
state: any,
path = "state.json"
): Promise<void> {
await Deno.writeTextFile(SHARED_FOLDER + "/" + path, JSON.stringify(state));
}
/**
* Get the shared state
* @param state state to set
*/
export async function getSharedState(path = 'state.json'): Promise<any> {
return JSON.parse(await Deno.readTextFile(SHARED_FOLDER + '/' + path))
export async function getSharedState(path = "state.json"): Promise<any> {
return JSON.parse(await Deno.readTextFile(SHARED_FOLDER + "/" + path));
}
/**
* Get the internal state
* @deprecated use getState instead
*/
export async function getInternalState(): Promise<any> {
return await getResource(getStatePath(), true)
return await getResource(getStatePath(), true);
}
/**
* Get the state shared across executions
*/
export async function getState(): Promise<any> {
return await getResource(getStatePath(), true)
return await getResource(getStatePath(), true);
}
/**
@@ -140,38 +176,43 @@ export async function getState(): Promise<any> {
* @returns variable value
*/
export async function getVariable(path: string): Promise<string | undefined> {
const workspace = getWorkspace()
try {
const variable = await VariableService.getVariable({ workspace, path })
return variable.value
} catch (e: any) {
throw Error(`Variable not found at ${path} or not visible to you`)
}
const workspace = getWorkspace();
try {
const variable = await VariableService.getVariable({ workspace, path });
return variable.value;
} catch (e: any) {
throw Error(`Variable not found at ${path} or not visible to you`);
}
}
async function transformLeaves(d: { [key: string]: any }): Promise<{ [key: string]: any }> {
for (const k in d) {
d[k] = await _transformLeaf(d[k])
}
return d
async function transformLeaves(d: {
[key: string]: any;
}): Promise<{ [key: string]: any }> {
for (const k in d) {
d[k] = await _transformLeaf(d[k]);
}
return d;
}
const VAR_RESOURCE_PREFIX = "$var:"
const VAR_RESOURCE_PREFIX = "$var:";
const RES_RESOURCE_PREFIX = "$res:";
async function _transformLeaf(v: any): Promise<any> {
if (typeof v === 'object') {
return transformLeaves(v)
}
else if (typeof v === 'string' && v.startsWith(VAR_RESOURCE_PREFIX)) {
const varName = v.substring(VAR_RESOURCE_PREFIX.length)
return await getVariable(varName)
} else {
return v
}
if (typeof v === "object") {
return transformLeaves(v);
} else if (typeof v === "string" && v.startsWith(VAR_RESOURCE_PREFIX)) {
const varName = v.substring(VAR_RESOURCE_PREFIX.length);
return await getVariable(varName);
} else if (typeof v === "string" && v.startsWith(RES_RESOURCE_PREFIX)) {
const resName = v.substring(RES_RESOURCE_PREFIX.length);
return await getResource(resName);
} else {
return v;
}
}
export async function databaseUrlFromResource(path: string): Promise<string> {
const resource = await getResource(path)
return `postgresql://${resource.user}:${resource.password}@${resource.host}:${resource.port}/${resource.dbname}?sslmode=${resource.sslmode}`
const resource = await getResource(path);
return `postgresql://${resource.user}:${resource.password}@${resource.host}:${resource.port}/${resource.dbname}?sslmode=${resource.sslmode}`;
}
/**
@@ -180,78 +221,84 @@ export async function databaseUrlFromResource(path: string): Promise<string> {
* @returns approval page UI URL, resume and cancel API URLs for resumeing the flow
*/
export async function getResumeUrls(approver?: string): Promise<{
approvalPage: string;
resume: string;
cancel: string;
approvalPage: string;
resume: string;
cancel: string;
}> {
const nonce = Math.floor(Math.random() * 4294967295);
const workspace = getWorkspace()
return await JobService.getResumeUrls({ workspace, resumeId: nonce, approver, id: Deno.env.get("WM_JOB_ID") ?? 'NO_JOB_ID' })
const nonce = Math.floor(Math.random() * 4294967295);
const workspace = getWorkspace();
return await JobService.getResumeUrls({
workspace,
resumeId: nonce,
approver,
id: Deno.env.get("WM_JOB_ID") ?? "NO_JOB_ID",
});
}
/**
* @deprecated use getResumeUrls instead
*/
export function getResumeEndpoints(approver?: string): Promise<{
approvalPage: string;
resume: string;
cancel: string;
approvalPage: string;
resume: string;
cancel: string;
}> {
return getResumeUrls(approver)
return getResumeUrls(approver);
}
export function base64ToUint8Array(data: string): Uint8Array {
return Uint8Array.from(atob(data), c => c.charCodeAt(0))
return Uint8Array.from(atob(data), (c) => c.charCodeAt(0));
}
export function uint8ArrayToBase64(arrayBuffer: Uint8Array): string {
let base64 = ''
const encodings = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
let base64 = "";
const encodings =
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
const bytes = new Uint8Array(arrayBuffer)
const byteLength = bytes.byteLength
const byteRemainder = byteLength % 3
const mainLength = byteLength - byteRemainder
const bytes = new Uint8Array(arrayBuffer);
const byteLength = bytes.byteLength;
const byteRemainder = byteLength % 3;
const mainLength = byteLength - byteRemainder;
let a, b, c, d
let chunk
let a, b, c, d;
let chunk;
// Main loop deals with bytes in chunks of 3
for (let i = 0; i < mainLength; i = i + 3) {
// Combine the three bytes into a single integer
chunk = (bytes[i] << 16) | (bytes[i + 1] << 8) | bytes[i + 2]
// Main loop deals with bytes in chunks of 3
for (let i = 0; i < mainLength; i = i + 3) {
// Combine the three bytes into a single integer
chunk = (bytes[i] << 16) | (bytes[i + 1] << 8) | bytes[i + 2];
// Use bitmasks to extract 6-bit segments from the triplet
a = (chunk & 16515072) >> 18 // 16515072 = (2^6 - 1) << 18
b = (chunk & 258048) >> 12 // 258048 = (2^6 - 1) << 12
c = (chunk & 4032) >> 6 // 4032 = (2^6 - 1) << 6
d = chunk & 63 // 63 = 2^6 - 1
// Use bitmasks to extract 6-bit segments from the triplet
a = (chunk & 16515072) >> 18; // 16515072 = (2^6 - 1) << 18
b = (chunk & 258048) >> 12; // 258048 = (2^6 - 1) << 12
c = (chunk & 4032) >> 6; // 4032 = (2^6 - 1) << 6
d = chunk & 63; // 63 = 2^6 - 1
// Convert the raw binary segments to the appropriate ASCII encoding
base64 += encodings[a] + encodings[b] + encodings[c] + encodings[d]
}
// Convert the raw binary segments to the appropriate ASCII encoding
base64 += encodings[a] + encodings[b] + encodings[c] + encodings[d];
}
// Deal with the remaining bytes and padding
if (byteRemainder == 1) {
chunk = bytes[mainLength]
// Deal with the remaining bytes and padding
if (byteRemainder == 1) {
chunk = bytes[mainLength];
a = (chunk & 252) >> 2 // 252 = (2^6 - 1) << 2
a = (chunk & 252) >> 2; // 252 = (2^6 - 1) << 2
// Set the 4 least significant bits to zero
b = (chunk & 3) << 4 // 3 = 2^2 - 1
// Set the 4 least significant bits to zero
b = (chunk & 3) << 4; // 3 = 2^2 - 1
base64 += encodings[a] + encodings[b] + '=='
} else if (byteRemainder == 2) {
chunk = (bytes[mainLength] << 8) | bytes[mainLength + 1]
base64 += encodings[a] + encodings[b] + "==";
} else if (byteRemainder == 2) {
chunk = (bytes[mainLength] << 8) | bytes[mainLength + 1];
a = (chunk & 64512) >> 10 // 64512 = (2^6 - 1) << 10
b = (chunk & 1008) >> 4 // 1008 = (2^6 - 1) << 4
a = (chunk & 64512) >> 10; // 64512 = (2^6 - 1) << 10
b = (chunk & 1008) >> 4; // 1008 = (2^6 - 1) << 4
// Set the 2 least significant bits to zero
c = (chunk & 15) << 2 // 15 = 2^4 - 1
// Set the 2 least significant bits to zero
c = (chunk & 15) << 2; // 15 = 2^4 - 1
base64 += encodings[a] + encodings[b] + encodings[c] + '='
}
base64 += encodings[a] + encodings[b] + encodings[c] + "=";
}
return base64
return base64;
}
+1
View File
@@ -0,0 +1 @@
VITE_DEFAULT_WEBHOOK_TYPE=async
+1
View File
@@ -7,3 +7,4 @@ node_modules
CaddyfileRemoteRuben
tests-out/
storageState.json
.env.production
+23 -1
View File
@@ -17,7 +17,7 @@ In the root folder:
```bash
docker build . -t windmill
docker-compose up db windmill_server windmill_worker
docker compose up db windmill_server windmill_worker
```
### 2. Backend is run by cargo
@@ -145,3 +145,25 @@ Recommended config for VS Code:
```
- turn _format on save_ on
## Building
The project is built with [SvelteKit](https://kit.svelte.dev/) and uses as output static files.
There are others adapters for sveltekit, but we use the static adapter.
To build the frontend as static assets, use:
```
npm run build
```
The output is in the `build` folder.
The default build assume you serve every non static files as the 200.html file which is catchall. If you prefer a normal layout, you can use:
```
NOTCATCHALL=true npm run build
```
which will generate an index.html and allow you to serve the frontend with any static server.
Env variables used for build are set in .env file. See [https://vitejs.dev/guide/env-and-mode.html#env-files](https://vitejs.dev/guide/env-and-mode.html#env-files) for more details.
+65 -23
View File
@@ -1,12 +1,12 @@
{
"name": "windmill",
"version": "1.74.2",
"version": "1.84.0",
"lockfileVersion": 2,
"requires": true,
"packages": {
"": {
"name": "windmill",
"version": "1.74.2",
"version": "1.84.0",
"dependencies": {
"@fortawesome/free-brands-svg-icons": "^6.2.1",
"@fortawesome/free-solid-svg-icons": "^6.2.1",
@@ -23,6 +23,7 @@
"date-fns": "^2.29.3",
"fast-equals": "^4.0.3",
"highlight.js": "^11.7.0",
"lodash": "^4.17.21",
"lucide-svelte": "^0.115.0",
"monaco-editor": "0.33.0",
"monaco-editor-workers": "0.33.0",
@@ -44,11 +45,11 @@
"@tailwindcss/typography": "^0.5.8",
"@types/d3": "^7.4.0",
"@types/d3-zoom": "^3.0.2",
"@types/lodash": "^4.14.191",
"@types/node": "^18.11.18",
"@types/vscode": "~1.74.0",
"@typescript-eslint/eslint-plugin": "^5.49.0",
"@typescript-eslint/parser": "^5.48.0",
"@windmill-labs/svelte-grid": "^5.1.6",
"@zerodevx/svelte-toast": "^0.8.1",
"autoprefixer": "^10.4.13",
"cssnano": "^5.1.14",
@@ -65,6 +66,7 @@
"prettier": "^2.8.3",
"prettier-plugin-svelte": "^2.9.0",
"simple-svelte-autocomplete": "^2.5.1",
"style-to-object": "^0.4.1",
"stylelint-config-recommended": "^9.0.0",
"svelte": "^3.55.1",
"svelte-awesome": "^3.0.0",
@@ -76,7 +78,7 @@
"svelte-preprocess": "^5.0.1",
"svelte-range-slider-pips": "^2.1.1",
"svelte-splitpanes": "^0.7.3",
"svelte2tsx": "^0.6.1",
"svelte2tsx": "^0.6.10",
"tailwindcss": "^3.2.2",
"tslib": "^2.4.1",
"typescript": "^4.9.3",
@@ -1286,6 +1288,12 @@
"resolved": "https://registry.npmjs.org/@types/json-schema/-/json-schema-7.0.11.tgz",
"integrity": "sha512-wOuvG1SN4Us4rez+tylwwwCV1psiNVOkJeM3AUWUNWg/jDQY2+HE/444y5gc+jBmRqASOm2Oeh5c1axHobwRKQ=="
},
"node_modules/@types/lodash": {
"version": "4.14.191",
"resolved": "https://registry.npmjs.org/@types/lodash/-/lodash-4.14.191.tgz",
"integrity": "sha512-BdZ5BCCvho3EIXw6wUCXHe7rS53AIDPLE+JzwgT+OsJk53oBfbSmZZ7CX4VaRoN78N+TJpFi9QPlfIVNmJYWxQ==",
"dev": true
},
"node_modules/@types/minimist": {
"version": "1.2.2",
"resolved": "https://registry.npmjs.org/@types/minimist/-/minimist-1.2.2.tgz",
@@ -1601,12 +1609,6 @@
"url": "https://opencollective.com/typescript-eslint"
}
},
"node_modules/@windmill-labs/svelte-grid": {
"version": "5.1.6",
"resolved": "https://registry.npmjs.org/@windmill-labs/svelte-grid/-/svelte-grid-5.1.6.tgz",
"integrity": "sha512-wXHIG7XeMxwxnfaHajhjvz9c0u5uGgiQtTk/QxAVyViGdshsclfIfIWZF113QQBKCKp2jY/zP1cj64BrqioRoQ==",
"dev": true
},
"node_modules/@zeit/schemas": {
"version": "2.21.0",
"resolved": "https://registry.npmjs.org/@zeit/schemas/-/schemas-2.21.0.tgz",
@@ -4078,6 +4080,12 @@
"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
"dev": true
},
"node_modules/inline-style-parser": {
"version": "0.1.1",
"resolved": "https://registry.npmjs.org/inline-style-parser/-/inline-style-parser-0.1.1.tgz",
"integrity": "sha512-7NXolsK4CAS5+xvdj5OMMbI962hU/wvwoxk+LWR9Ek9bVtyuuYScDN6eS0rUm6TxApFpw7CX1o4uJzcd4AyD3Q==",
"dev": true
},
"node_modules/internmap": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/internmap/-/internmap-2.0.3.tgz",
@@ -4408,6 +4416,11 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/lodash": {
"version": "4.17.21",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz",
"integrity": "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg=="
},
"node_modules/lodash.castarray": {
"version": "4.4.0",
"resolved": "https://registry.npmjs.org/lodash.castarray/-/lodash.castarray-4.4.0.tgz",
@@ -6953,6 +6966,15 @@
"dev": true,
"peer": true
},
"node_modules/style-to-object": {
"version": "0.4.1",
"resolved": "https://registry.npmjs.org/style-to-object/-/style-to-object-0.4.1.tgz",
"integrity": "sha512-HFpbb5gr2ypci7Qw+IOhnP2zOU7e77b+rzM+wTzXzfi1PrtBCX0E7Pk4wL4iTLnhzZ+JgEGAhX81ebTg/aYjQw==",
"dev": true,
"dependencies": {
"inline-style-parser": "0.1.1"
}
},
"node_modules/stylehacks": {
"version": "5.1.1",
"resolved": "https://registry.npmjs.org/stylehacks/-/stylehacks-5.1.1.tgz",
@@ -7351,9 +7373,9 @@
}
},
"node_modules/svelte2tsx": {
"version": "0.6.1",
"resolved": "https://registry.npmjs.org/svelte2tsx/-/svelte2tsx-0.6.1.tgz",
"integrity": "sha512-O/1+5UyChfmhp1/GUv8b8iveTrn6eZwHxEXc+rw7LMKRidr9KHk5w/EiliLjDUwHa2VA6CoEty+CQylROVU4Sw==",
"version": "0.6.10",
"resolved": "https://registry.npmjs.org/svelte2tsx/-/svelte2tsx-0.6.10.tgz",
"integrity": "sha512-7CtUexhSHppSpLwVX7yI9OxJp7+esCNTuOF/CaVxWTgWp88QF/2/dCbpND/cTZmBxht5cFFQBwoLXXqFLdYe3Q==",
"dev": true,
"dependencies": {
"dedent-js": "^1.0.1",
@@ -7361,7 +7383,7 @@
},
"peerDependencies": {
"svelte": "^3.55",
"typescript": "^4.9.4"
"typescript": "^4.9.4 || ^5.0.0"
}
},
"node_modules/svg-tags": {
@@ -9089,6 +9111,12 @@
"resolved": "https://registry.npmjs.org/@types/json-schema/-/json-schema-7.0.11.tgz",
"integrity": "sha512-wOuvG1SN4Us4rez+tylwwwCV1psiNVOkJeM3AUWUNWg/jDQY2+HE/444y5gc+jBmRqASOm2Oeh5c1axHobwRKQ=="
},
"@types/lodash": {
"version": "4.14.191",
"resolved": "https://registry.npmjs.org/@types/lodash/-/lodash-4.14.191.tgz",
"integrity": "sha512-BdZ5BCCvho3EIXw6wUCXHe7rS53AIDPLE+JzwgT+OsJk53oBfbSmZZ7CX4VaRoN78N+TJpFi9QPlfIVNmJYWxQ==",
"dev": true
},
"@types/minimist": {
"version": "1.2.2",
"resolved": "https://registry.npmjs.org/@types/minimist/-/minimist-1.2.2.tgz",
@@ -9284,12 +9312,6 @@
"eslint-visitor-keys": "^3.3.0"
}
},
"@windmill-labs/svelte-grid": {
"version": "5.1.6",
"resolved": "https://registry.npmjs.org/@windmill-labs/svelte-grid/-/svelte-grid-5.1.6.tgz",
"integrity": "sha512-wXHIG7XeMxwxnfaHajhjvz9c0u5uGgiQtTk/QxAVyViGdshsclfIfIWZF113QQBKCKp2jY/zP1cj64BrqioRoQ==",
"dev": true
},
"@zeit/schemas": {
"version": "2.21.0",
"resolved": "https://registry.npmjs.org/@zeit/schemas/-/schemas-2.21.0.tgz",
@@ -11121,6 +11143,12 @@
"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
"dev": true
},
"inline-style-parser": {
"version": "0.1.1",
"resolved": "https://registry.npmjs.org/inline-style-parser/-/inline-style-parser-0.1.1.tgz",
"integrity": "sha512-7NXolsK4CAS5+xvdj5OMMbI962hU/wvwoxk+LWR9Ek9bVtyuuYScDN6eS0rUm6TxApFpw7CX1o4uJzcd4AyD3Q==",
"dev": true
},
"internmap": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/internmap/-/internmap-2.0.3.tgz",
@@ -11371,6 +11399,11 @@
"p-locate": "^5.0.0"
}
},
"lodash": {
"version": "4.17.21",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz",
"integrity": "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg=="
},
"lodash.castarray": {
"version": "4.4.0",
"resolved": "https://registry.npmjs.org/lodash.castarray/-/lodash.castarray-4.4.0.tgz",
@@ -13211,6 +13244,15 @@
"dev": true,
"peer": true
},
"style-to-object": {
"version": "0.4.1",
"resolved": "https://registry.npmjs.org/style-to-object/-/style-to-object-0.4.1.tgz",
"integrity": "sha512-HFpbb5gr2ypci7Qw+IOhnP2zOU7e77b+rzM+wTzXzfi1PrtBCX0E7Pk4wL4iTLnhzZ+JgEGAhX81ebTg/aYjQw==",
"dev": true,
"requires": {
"inline-style-parser": "0.1.1"
}
},
"stylehacks": {
"version": "5.1.1",
"resolved": "https://registry.npmjs.org/stylehacks/-/stylehacks-5.1.1.tgz",
@@ -13491,9 +13533,9 @@
}
},
"svelte2tsx": {
"version": "0.6.1",
"resolved": "https://registry.npmjs.org/svelte2tsx/-/svelte2tsx-0.6.1.tgz",
"integrity": "sha512-O/1+5UyChfmhp1/GUv8b8iveTrn6eZwHxEXc+rw7LMKRidr9KHk5w/EiliLjDUwHa2VA6CoEty+CQylROVU4Sw==",
"version": "0.6.10",
"resolved": "https://registry.npmjs.org/svelte2tsx/-/svelte2tsx-0.6.10.tgz",
"integrity": "sha512-7CtUexhSHppSpLwVX7yI9OxJp7+esCNTuOF/CaVxWTgWp88QF/2/dCbpND/cTZmBxht5cFFQBwoLXXqFLdYe3Q==",
"dev": true,
"requires": {
"dedent-js": "^1.0.1",
+5 -3
View File
@@ -1,6 +1,6 @@
{
"name": "windmill",
"version": "1.74.2",
"version": "1.84.0",
"scripts": {
"dev": "vite dev",
"build": "vite build",
@@ -23,11 +23,11 @@
"@tailwindcss/typography": "^0.5.8",
"@types/d3": "^7.4.0",
"@types/d3-zoom": "^3.0.2",
"@types/lodash": "^4.14.191",
"@types/node": "^18.11.18",
"@types/vscode": "~1.74.0",
"@typescript-eslint/eslint-plugin": "^5.49.0",
"@typescript-eslint/parser": "^5.48.0",
"@windmill-labs/svelte-grid": "^5.1.6",
"@zerodevx/svelte-toast": "^0.8.1",
"autoprefixer": "^10.4.13",
"cssnano": "^5.1.14",
@@ -44,6 +44,7 @@
"prettier": "^2.8.3",
"prettier-plugin-svelte": "^2.9.0",
"simple-svelte-autocomplete": "^2.5.1",
"style-to-object": "^0.4.1",
"stylelint-config-recommended": "^9.0.0",
"svelte": "^3.55.1",
"svelte-awesome": "^3.0.0",
@@ -55,7 +56,7 @@
"svelte-preprocess": "^5.0.1",
"svelte-range-slider-pips": "^2.1.1",
"svelte-splitpanes": "^0.7.3",
"svelte2tsx": "^0.6.1",
"svelte2tsx": "^0.6.10",
"tailwindcss": "^3.2.2",
"tslib": "^2.4.1",
"typescript": "^4.9.3",
@@ -79,6 +80,7 @@
"date-fns": "^2.29.3",
"fast-equals": "^4.0.3",
"highlight.js": "^11.7.0",
"lodash": "^4.17.21",
"lucide-svelte": "^0.115.0",
"monaco-editor": "0.33.0",
"monaco-editor-workers": "0.33.0",
+9 -13
View File
@@ -1,17 +1,13 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<link rel="icon" href="/logo.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
%sveltekit.head%
</head>
<head>
<meta charset="utf-8" />
<link rel="icon" href="/logo.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
%sveltekit.head%
</head>
<body data-sveltekit-preload-code="viewport" class="outline-none focus:outline-none">
<div style="display: contents">
%sveltekit.body%
</div>
</body>
<body data-sveltekit-preload-code="viewport" class="outline-none focus:outline-none">
<div style="display: contents"> %sveltekit.body% </div>
</body>
</html>
-77
View File
@@ -1,79 +1,2 @@
/// <reference types="@sveltejs/kit" />
declare type Item = import('svelte-dnd-action').Item
declare type DndEvent<ItemType = Item> = import('svelte-dnd-action').DndEvent<ItemType>
declare namespace svelte.JSX {
interface HTMLAttributes<T> {
onconsider?: (event: CustomEvent<DndEvent<ItemType>> & { target: EventTarget & T }) => void
onfinalize?: (event: CustomEvent<DndEvent<ItemType>> & { target: EventTarget & T }) => void
}
}
declare module '@windmill-labs/svelte-grid' {
import type { SvelteComponentTyped } from 'svelte'
export interface Size {
w: number
h: number
}
export interface Positon {
x: number
y: number
}
interface ItemLayout extends Size, Positon {
fixed?: boolean
resizable?: boolean
draggable?: boolean
customDragger?: boolean
customResizer?: boolean
min?: Size
max?: Size
}
export type Item<T> = T & { [width: number]: ItemLayout; data: any }
export type FilledItem<T> = T & { [width: number]: Required<ItemLayout>; data: any }
export interface Props<T> {
fillSpace?: boolean
items: FilledItem<T>[]
rowHeight: number
cols: [number, number][]
gap?: [number, number]
fastStart?: boolean
throttleUpdate?: number
throttleResize?: number
onTopId?: string
scroller?: undefined
sensor?: number
parentWidth?: number
}
export interface Slots<T> {
default: { item: ItemLayout; dataItem: Item<T> }
}
export default class Grid<T = {}> extends SvelteComponentTyped<
Props<T>,
{
pointerup: CustomEvent<{ id: string }>
mount: CustomEvent<>
},
Slots<T>
> { }
}
declare module '@windmill-labs/svelte-grid/build/helper/index.mjs' {
import { ItemLayout } from '@windmill-labs/svelte-grid'
const x: {
normalize(items: any[], col: any): unknown[]
adjust(items: any[], col: any): unknown[]
findSpace(item: any, items: any, cols: any): unknown
item<T>(obj: ItemLayout): Required<ItemLayout>
}
export default x
}
+8 -5
View File
@@ -1,4 +1,4 @@
import type { Script } from "./gen"
import type { Script } from './gen'
export type OwnerKind = 'group' | 'user' | 'folder'
@@ -14,7 +14,7 @@ export interface SchemaProperty {
enum?: string[]
contentEncoding?: 'base64' | 'binary'
format?: string
items?: { type?: 'string' | 'number' | 'bytes'; contentEncoding?: 'base64' }
items?: { type?: 'string' | 'number' | 'bytes' | 'object'; contentEncoding?: 'base64' }
properties?: { [name: string]: SchemaProperty }
}
@@ -28,11 +28,14 @@ export type Schema = {
export type Meta = { ownerKind: OwnerKind; owner: string; name: string }
type Enumerate<N extends number, Acc extends number[] = []> = Acc['length'] extends N
? Acc[number]
: Enumerate<N, [...Acc, Acc['length']]>
? Acc[number]
: Enumerate<N, [...Acc, Acc['length']]>
/** An inclusive range of integer numbers */
export type IntRange<F extends number, T extends number> = F | Exclude<Enumerate<T>, Enumerate<F>> | T
export type IntRange<F extends number, T extends number> =
| F
| Exclude<Enumerate<T>, Enumerate<F>>
| T
export function pathToMeta(path: string): Meta {
const splitted = path.split('/')
@@ -81,7 +81,7 @@
$: key =
apiTokenApps[resource_type]?.key ??
(args != undefined
? Object.keys(args).filter((x) => ['token', 'password', 'api_key'].includes(x))[0]
? Object.keys(args).filter((x) => ['token', 'password', 'api_key', 'key'].includes(x))[0]
: undefined)
let scopes: string[] = []
@@ -259,6 +259,7 @@
args['token'] == '' &&
args['password'] == '' &&
args['api_key'] == '' &&
args['key'] == '' &&
key != undefined) ||
(step == 3 && pathError != '') ||
!isValid
@@ -396,7 +397,7 @@
bind:error={pathError}
bind:path
initialPath=""
namePlaceholder="{resource_type}"
namePlaceholder={resource_type}
kind="resource"
/>
@@ -433,7 +434,7 @@
{:else}
<Path
initialPath=""
namePlaceholder="{resource_type}"
namePlaceholder={resource_type}
bind:error={pathError}
bind:path
kind="resource"
+64 -83
View File
@@ -7,7 +7,7 @@
faPlus
} from '@fortawesome/free-solid-svg-icons'
import { setInputCat as computeInputCat, type InputCat } from '$lib/utils'
import { setInputCat as computeInputCat } from '$lib/utils'
import { Badge, Button } from './common'
import { createEventDispatcher } from 'svelte'
import Icon from 'svelte-awesome'
@@ -26,6 +26,9 @@
import type ItemPicker from './ItemPicker.svelte'
import NumberTypeNarrowing from './NumberTypeNarrowing.svelte'
import Range from './Range.svelte'
import JsonEditor from './apps/editor/settingsPanel/inputEditor/JsonEditor.svelte'
import { fade } from 'svelte/transition'
import { X } from 'lucide-svelte'
export let label: string = ''
export let value: any
@@ -44,7 +47,7 @@
export let disabled = false
export let editableSchema = false
export let itemsType:
| { type?: 'string' | 'number' | 'bytes'; contentEncoding?: 'base64' }
| { type?: 'string' | 'number' | 'bytes' | 'object'; contentEncoding?: 'base64' }
| undefined = undefined
export let displayHeader = true
export let properties: { [name: string]: SchemaProperty } | undefined = undefined
@@ -93,7 +96,7 @@
export function evalValueToRaw() {
if (value) {
rawValue = JSON.stringify(value, null, 4)
rawValue = JSON.stringify(value, null, 2)
}
}
@@ -152,18 +155,23 @@
value = enum_?.[0]
} else if (inputCat == 'boolean') {
value = false
} else if (inputCat == 'list') {
value = []
}
}
}
}
$: inputCat = computeInputCat(type, format, itemsType?.type, enum_, contentEncoding)
let redraw = 0
let itemsLimit = 50
</script>
<div class="flex flex-col w-full min-w-[250px]">
<div>
{#if displayHeader}
<FieldHeader {label} {required} {type} {contentEncoding} {format} {itemsType} />
<FieldHeader {label} {required} {type} {contentEncoding} {format} />
{/if}
{#if editableSchema}
<div class="p-2 my-1 text-xs border-solid border border-gray-400">
@@ -186,6 +194,7 @@
use:autosize
rows="1"
bind:value={description}
on:keydown|stopPropagation
placeholder="Field description"
/>
{#if type == 'string' && format != 'date-time'}
@@ -198,6 +207,7 @@
<select bind:value={itemsType}>
<option value={undefined}>No specific item type</option>
<option value={{ type: 'string' }}> Items are strings</option>
<option value={{ type: 'object' }}> Items are objects (JSON)</option>
<option value={{ type: 'number' }}>Items are numbers</option>
<option value={{ type: 'string', contentEncoding: 'base64' }}
>Items are bytes</option
@@ -232,7 +242,6 @@
<input
{autofocus}
on:focus={(e) => {
window.dispatchEvent(new Event('pointerup'))
dispatch('focus')
}}
{disabled}
@@ -244,14 +253,12 @@
bind:value
min={extra['min']}
max={extra['max']}
on:input={() => dispatch('input', { value, isRaw: true })}
/>
{/if}
{:else if inputCat == 'boolean'}
<Toggle
on:pointerdown={(e) => {
e?.stopPropagation()
window.dispatchEvent(new Event('pointerup'))
}}
{disabled}
class={valid
@@ -263,44 +270,53 @@
<span>&nbsp; Not set</span>
{/if}
{:else if inputCat == 'list'}
<div>
<div>
{#each value ?? [] as v, i}
<div class="flex flex-row max-w-md mt-1">
{#if itemsType?.type == 'number'}
<input autofocus={autofocus && i == 0} type="number" bind:value={v} />
{:else if itemsType?.type == 'string' && itemsType?.contentEncoding == 'base64'}
<input
autofocus={autofocus && i == 0}
type="file"
class="my-6"
on:change={(x) => fileChanged(x, (val) => (value[i] = val))}
multiple={false}
/>
{:else}
<input autofocus={autofocus && i == 0} type="text" bind:value={v} />
<div class="w-full">
<div class="w-full">
{#key redraw}
{#if Array.isArray(value)}
{#each value ?? [] as v, i}
{#if i < itemsLimit}
<div class="flex max-w-md mt-1 w-full items-center">
{#if itemsType?.type == 'number'}
<input type="number" bind:value={v} />
{:else if itemsType?.type == 'string' && itemsType?.contentEncoding == 'base64'}
<input
type="file"
class="my-6"
on:change={(x) => fileChanged(x, (val) => (value[i] = val))}
multiple={false}
/>
{:else if itemsType?.type == 'object'}
<JsonEditor code={JSON.stringify(v, null, 2)} bind:value={v} />
{:else}
<input type="text" bind:value={v} />
{/if}
<button
transition:fade|local={{ duration: 100 }}
class="rounded-full p-1 bg-white/60 duration-200 hover:bg-gray-200"
aria-label="Clear"
on:click={() => {
value.splice(i, 1)
redraw += 1
}}
>
<X size={14} />
</button>
</div>
{/if}
{/each}
{#if value.length > itemsLimit}
<button on:click={() => (itemsLimit += 50)} class="text-xs py-2 text-blue-600"
>{itemsLimit}/{value.length}: Load 50 more...</button
>
{/if}
<Button
variant="border"
color="red"
size="sm"
btnClasses="mx-6"
on:click={() => {
value = value.filter((el) => el != v)
if (value.length == 0) {
value = undefined
}
}}
>
<Icon data={faMinus} />
</Button>
</div>
{/each}
{/if}
{/key}
</div>
<Button
variant="border"
color="blue"
size="sm"
color="dark"
size="xs"
btnClasses="mt-1"
on:click={() => {
if (value == undefined || !Array.isArray(value)) {
@@ -312,9 +328,6 @@
<Icon data={faPlus} class="mr-2" />
Add item
</Button>
<span class="ml-2">
{(value ?? []).length} item{(value ?? []).length > 1 ? 's' : ''}
</span>
</div>
{:else if inputCat == 'resource-object'}
<ObjectResourceInput {format} bind:value />
@@ -327,31 +340,21 @@
bind:args={value}
/>
</div>
{:else if disabled}
<textarea disabled />
{:else}
<textarea
bind:this={el}
<JsonEditor
bind:editor
on:focus={(e) => {
window.dispatchEvent(new Event('pointerup'))
dispatch('focus')
}}
{autofocus}
{disabled}
use:autosize
style="max-height: {maxHeight}"
on:input={() => {
dispatch('input', { rawValue: value, isRaw: false })
}}
class="col-span-10 {valid
? ''
: 'border border-red-700 border-opacity-30 focus:border-red-700 focus:border-opacity-30 bg-red-100'}"
placeholder={defaultValue ? JSON.stringify(defaultValue, null, 4) : ''}
bind:value={rawValue}
code={rawValue}
bind:value
/>
{/if}
{:else if inputCat == 'enum'}
<select
on:focus={(e) => {
window.dispatchEvent(new Event('pointerup'))
dispatch('focus')
}}
{disabled}
@@ -368,16 +371,11 @@
<div class="border my-1 mb-4 w-full border-gray-400">
<SimpleEditor
on:focus={(e) => {
window.dispatchEvent(new Event('pointerup'))
dispatch('focus')
}}
on:blur={() => dispatch('blur')}
bind:this={editor}
lang={inputCat}
bind:code={value}
on:change={async () => {
dispatch('input', { rawValue: value, isRaw: false })
}}
autoHeight
/>
</div>
@@ -407,11 +405,10 @@
rows="1"
bind:this={el}
on:focus={(e) => {
window.dispatchEvent(new Event('pointerup'))
dispatch('focus')
}}
on:blur={() => dispatch('blur')}
use:autosize
on:keydown|stopPropagation
type="text"
{disabled}
class="col-span-10 {valid
@@ -419,9 +416,6 @@
: 'border border-red-700 border-opacity-30 focus:border-red-700 focus:border-opacity-30 bg-red-100'}"
placeholder={defaultValue ?? ''}
bind:value
on:input={() => {
dispatch('input', { rawValue: value, isRaw: false })
}}
/>
{#if itemPicker}
<div class="ml-1 relative">
@@ -450,19 +444,6 @@
{/if}
</div>
{/if}
{#if !required && inputCat != 'resource-object'}
<!-- <Tooltip placement="bottom" content="Reset to default value">
<Button
on:click={() => (value = undefined)}
{disabled}
color="alternative"
size="sm"
class="h-8"
>
<Icon data={faArrowRotateLeft} />
</Button>
</Tooltip> -->
{/if}
<slot name="actions" />
</div>
{#if !compact || (error && error != '')}
@@ -6,9 +6,11 @@
import { Button, Drawer, DrawerContent } from './common'
import autosize from 'svelte-autosize'
import { ClipboardCopy } from 'lucide-svelte'
import Portal from 'svelte-portal'
export let result: any
export let requireHtmlApproval = false
export let filename: string | undefined = undefined
let resultKind:
| 'json'
@@ -91,28 +93,13 @@
let jsonViewer: Drawer
</script>
<Drawer bind:this={jsonViewer} size="900px">
<DrawerContent title="Expanded Result" on:close={jsonViewer.closeDrawer}>
<svelte:fragment slot="actions">
<Button
on:click={() => copyToClipboard(JSON.stringify(result, null, 4))}
color="light"
size="xs"
>
<div class="flex gap-2 items-center">Copy to clipboard <ClipboardCopy /> </div>
</Button>
</svelte:fragment>
<Highlight language={json} code={JSON.stringify(result, null, 4).replace(/\\n/g, '\n')} />
</DrawerContent>
</Drawer>
<div class="inline-highlight">
{#if result != undefined}
{#if resultKind && resultKind != 'json'}
<div class="mb-2 text-gray-500 text-sm bg-gray-50/20">
as JSON&nbsp;<input class="windmillapp" type="checkbox" bind:checked={forceJson} /></div
>{/if}{#if typeof result == 'object' && Object.keys(result).length > 0}<div
class="mb-2 min-w-[400px] text-sm text-gray-700 relative"
class="mb-2 min-w-[360px] text-sm text-gray-700 relative"
>The result keys are: <b>{truncate(Object.keys(result).join(', '), 50)}</b>
<div class="text-gray-500 text-sm absolute top-0 right-2">
<button on:click={jsonViewer.openDrawer}>Expand JSON</button>
@@ -245,9 +232,34 @@
>
</div>
{:else}
<Highlight language={json} code={JSON.stringify(result, null, 4).replace(/\\n/g, '\n')} />
{@const jsonStr = JSON.stringify(result, null, 4).replace(/\\n/g, '\n')}
{#if jsonStr.length > 10000}
JSON too large. <a
download="{filename ?? 'result'}.json"
href="data:text/json;charset=utf-8,{encodeURIComponent(jsonStr)}">Download</a
>
{:else}
<Highlight language={json} code={jsonStr} />
{/if}
{/if}
{:else}
<div class="text-gray-500 text-sm">No result</div>
<div class="text-gray-500 text-sm">No result: {JSON.stringify(result)}</div>
{/if}
</div>
<Portal>
<Drawer bind:this={jsonViewer} size="900px">
<DrawerContent title="Expanded Result" on:close={jsonViewer.closeDrawer}>
<svelte:fragment slot="actions">
<Button
on:click={() => copyToClipboard(JSON.stringify(result, null, 4))}
color="light"
size="xs"
>
<div class="flex gap-2 items-center">Copy to clipboard <ClipboardCopy /> </div>
</Button>
</svelte:fragment>
<Highlight language={json} code={JSON.stringify(result, null, 4).replace(/\\n/g, '\n')} />
</DrawerContent>
</Drawer>
</Portal>
+14 -4
View File
@@ -2,9 +2,10 @@
import type { DropdownItem } from '$lib/utils'
import { createEventDispatcher } from 'svelte'
import Icon from 'svelte-awesome'
import { MoreHorizontal } from 'lucide-svelte'
import { MoreVertical } from 'lucide-svelte'
import { Button, Menu } from './common'
import { goto } from '$app/navigation'
import { twMerge } from 'tailwind-merge'
type Alignment = 'start' | 'end'
type Side = 'top' | 'bottom'
@@ -13,15 +14,24 @@
export let dropdownItems: DropdownItem[]
export let name: string | undefined = undefined
export let placement: Placement = 'bottom-start'
export let btnClasses = '!text-blue-500 bg-transparent'
export let btnClasses = ''
$: buttonClass = twMerge('!border-0 bg-transparent !p-[6px]', btnClasses)
const dispatch = createEventDispatcher()
</script>
<Menu {placement} let:close>
<Button nonCaptureEvent color="light" size="xs" {btnClasses} {...$$restProps} slot="trigger">
<Button
nonCaptureEvent
color="dark"
variant="border"
size="xs"
btnClasses={buttonClass}
{...$$restProps}
slot="trigger"
>
{#if !$$slots.default}
<MoreHorizontal size={20} />
<MoreVertical size={20} />
{:else}
<slot />
{/if}
@@ -461,10 +461,6 @@
}
})
editor.onDidBlurEditorText(() => {
dispatch('blur')
})
reloadWebsocket()
return () => {
@@ -6,9 +6,6 @@
export let contentEncoding = ''
export let type: string | undefined = undefined
export let required = false
export let itemsType:
| { type?: 'string' | 'number' | 'bytes'; contentEncoding?: 'base64' }
| undefined = undefined
</script>
<div class="inline-flex flex-row items-center truncated">
@@ -118,6 +118,7 @@
if (leave) {
goto(`/flows/get/${$flowStore.path}?workspace_id=${$workspaceStore}`)
} else if (initialPath !== $flowStore.path) {
initialPath = $flowStore.path
goto(`/flows/edit/${$flowStore.path}?workspace_id=${$workspaceStore}`)
}
}
@@ -194,6 +195,14 @@
loadHubScripts()
function onKeyDown(event: KeyboardEvent) {
let classes = event.target?.['className']
if (
(typeof classes === 'string' && classes.includes('inputarea')) ||
['INPUT', 'TEXTAREA'].includes(document.activeElement?.tagName!)
) {
return
}
switch (event.key) {
case 'Z':
if (event.ctrlKey) {
@@ -42,11 +42,7 @@
</div>
<div class="text-2xs mb-4">
<h3 class="mb-2">Step Inputs</h3>
<InputTransformsViewer
inputTransforms={stepDetail?.value?.input_transforms ??
stepDetail?.input_transforms ??
{}}
/>
<InputTransformsViewer inputTransforms={stepDetail?.value?.input_transforms ?? {}} />
</div>
{#if stepDetail.value.path.startsWith('hub/')}
<div class="mt-6">
@@ -62,11 +58,7 @@
{:else if stepDetail.value.type == 'rawscript'}
<div class="text-2xs mb-4">
<h3 class="mb-2">Step Inputs</h3>
<InputTransformsViewer
inputTransforms={stepDetail?.value?.input_transforms ??
stepDetail?.input_transforms ??
{}}
/>
<InputTransformsViewer inputTransforms={stepDetail?.value?.input_transforms ?? {}} />
</div>
<h3 class="mb-2">Code</h3>
@@ -88,7 +80,7 @@
minHeight={400}
modules={flow?.value?.modules}
failureModule={flow?.value?.failure_module}
on:click={(e) => (stepDetail = e.detail)}
on:select={(e) => (stepDetail = e.detail)}
/>
</div>
{#if !noSide}
@@ -115,11 +107,7 @@
{:else if stepDetail.value.type == 'rawscript'}
<div class="text-2xs mb-4">
<h3 class="mb-2">Step Inputs</h3>
<InputTransformsViewer
inputTransforms={stepDetail?.value?.input_transforms ??
stepDetail?.input_transforms ??
{}}
/>
<InputTransformsViewer inputTransforms={stepDetail?.value?.input_transforms ?? {}} />
</div>
<h3 class="mb-2"
@@ -143,11 +131,7 @@
</div>
<div class="text-2xs mb-4">
<h3 class="mb-2">Step Inputs</h3>
<InputTransformsViewer
inputTransforms={stepDetail?.value?.input_transforms ??
stepDetail?.input_transforms ??
{}}
/>
<InputTransformsViewer inputTransforms={stepDetail?.value?.input_transforms ?? {}} />
</div>
{#if stepDetail.value.path.startsWith('hub/')}
<div class="mt-6">
@@ -8,6 +8,7 @@
export let col: boolean = false
export let noBorder = false
export let loading
export let filename: string | undefined = undefined
</script>
<div
@@ -17,7 +18,7 @@
<div class="bg-white {col ? '' : 'max-h-80'} h-full p-1 overflow-auto relative">
<span class="text-gray-500">Result</span>
{#if result}
<DisplayResult {result} />
<DisplayResult {filename} {result} />
{:else if loading}
<Loader2 class="animate-spin" />
{:else}
@@ -22,7 +22,7 @@
const dispatch = createEventDispatcher()
export let jobId: string
export let workspaceId: string | undefined = undefined
export let flowState: FlowState | undefined = undefined
export let flowJobIds:
| {
@@ -102,7 +102,7 @@
localFlowModuleStates[mod.id ?? '']?.scheduled_for == undefined
) {
JobService.getJob({
workspace: $workspaceStore ?? '',
workspace: workspaceId ?? $workspaceStore ?? '',
id: mod.job ?? ''
}).then((job) => {
localFlowModuleStates[mod.id ?? ''] = {
@@ -122,7 +122,7 @@
if (jobId != '00000000-0000-0000-0000-000000000000') {
try {
const newJob = await JobService.getJob({
workspace: $workspaceStore ?? '',
workspace: workspaceId ?? $workspaceStore ?? '',
id: jobId ?? ''
})
if (JSON.stringify(newJob) !== JSON.stringify(job)) {
@@ -159,7 +159,7 @@
})
async function loadOwner(path: string) {
is_owner = await isOwner(path, $userStore!, $workspaceStore!)
is_owner = await isOwner(path, $userStore!, workspaceId ?? $workspaceStore!)
}
let selected: 'graph' | 'sequence' = 'graph'
@@ -207,7 +207,7 @@
variant="border"
on:click={async () =>
await JobService.resumeSuspendedFlowAsOwner({
workspace: $workspaceStore ?? '',
workspace: workspaceId ?? $workspaceStore ?? '',
id: job?.id ?? '',
requestBody: JSON.parse(payload)
})}
@@ -305,6 +305,7 @@
</Button>
<div class="border p-6" class:hidden={forloop_selected != loopJobId}>
<svelte:self
{workspaceId}
bind:suspend_status
bind:retry_status
bind:flowState
@@ -379,6 +380,7 @@
<li class="w-full border border-gray-600 p-6 space-y-2 bg-blue-50/50">
{#if [FlowStatusModule.type.IN_PROGRESS, FlowStatusModule.type.SUCCESS, FlowStatusModule.type.FAILURE].includes(mod.type)}
<svelte:self
{workspaceId}
bind:suspend_status
bind:retry_status
bind:flowState
@@ -474,6 +476,7 @@
{@const node = localFlowModuleStates[selectedNode]}
{#if selectedNode == 'end'}
<FlowJobResult
filename={job.id}
loading={job['running']}
noBorder
col
@@ -64,7 +64,6 @@
type={toAny(v)?.type}
contentEncoding={toAny(v)?.contentEncoding}
format={toAny(v)?.format}
itemsType={toAny(v)?.itemsType}
/><span class="ml-4 mt-2 text-xs"
>{toAny(v)?.default != undefined
? 'default: ' + JSON.stringify(toAny(v)?.default)
@@ -3,6 +3,8 @@
import { workspaceStore } from '$lib/stores'
export let name: string
export let tabular = false
export let order = ['scripts', 'flows', 'apps', 'schedules', 'variables', 'resources']
$: $workspaceStore && loadUsage()
@@ -13,10 +15,18 @@
}
</script>
<div class="flex flex-col text-xs text-gray-600">
{#each Object.entries(usage) as [k, v]}
<div>
{k}: {v}
</div>
{#if tabular}
{#each order as key}
<td class="text-center">
{usage[key] ?? ''}
</td>
{/each}
</div>
{:else}
<div class="flex flex-col text-xs text-gray-600">
{#each Object.entries(usage) as [k, v]}
<div>
{k}: {v}
</div>
{/each}
</div>
{/if}
@@ -234,6 +234,14 @@
name,
requestBody: { username: member_name }
})
await GranularAclService.removeGranularAcls({
workspace: $workspaceStore ?? '',
path: name,
kind: 'group_',
requestBody: {
owner: 'u/' + member_name
}
})
loadGroup()
}}>remove</button
>
@@ -133,7 +133,6 @@
contentEncoding={schema.properties[argName].contentEncoding}
required={schema.required.includes(argName)}
type={schema.properties[argName].type}
itemsType={schema.properties[argName].items}
/>
{#if isStaticTemplate(inputCat)}
@@ -0,0 +1,318 @@
<script lang="ts">
import { faMinus, faPlus } from '@fortawesome/free-solid-svg-icons'
import { setInputCat as computeInputCat } from '$lib/utils'
import { Badge, Button } from './common'
import { createEventDispatcher } from 'svelte'
import Icon from 'svelte-awesome'
import FieldHeader from './FieldHeader.svelte'
import ObjectResourceInput from './ObjectResourceInput.svelte'
import ResourcePicker from './ResourcePicker.svelte'
import type { SchemaProperty } from '$lib/common'
import autosize from 'svelte-autosize'
import Toggle from './Toggle.svelte'
import Range from './Range.svelte'
import LightweightSchemaForm from './LightweightSchemaForm.svelte'
export let label: string = ''
export let value: any
export let defaultValue: any = undefined
export let description: string = ''
export let format: string = ''
export let contentEncoding: 'base64' | 'binary' | undefined = undefined
export let type: string | undefined = undefined
export let required = false
export let pattern: undefined | string = undefined
export let valid = required ? false : true
export let maxRows = 10
export let enum_: string[] | undefined = undefined
export let itemsType:
| { type?: 'string' | 'number' | 'bytes' | 'object'; contentEncoding?: 'base64' }
| undefined = undefined
export let displayHeader = true
export let properties: { [name: string]: SchemaProperty } | undefined = undefined
export let extra: Record<string, any> = {}
const dispatch = createEventDispatcher()
$: maxHeight = maxRows ? `${1 + maxRows * 1.2}em` : `auto`
$: validateInput(pattern, value)
let error: string = ''
let el: HTMLTextAreaElement | undefined = undefined
let rawValue: string | undefined = undefined
$: {
if (rawValue) {
try {
value = JSON.parse(rawValue)
error = ''
} catch (err) {
error = err.toString()
}
}
}
$: {
error = ''
if (inputCat === 'object') {
evalValueToRaw()
validateInput(pattern, value)
}
}
export function evalValueToRaw() {
if (value) {
rawValue = JSON.stringify(value, null, 4)
}
}
function fileChanged(e: any, cb: (v: string | undefined) => void) {
let t = e.target
if (t && 'files' in t && t.files.length > 0) {
let reader = new FileReader()
reader.onload = (e: any) => {
cb(e.target.result.split('base64,')[1])
}
reader.readAsDataURL(t.files[0])
} else {
cb(undefined)
}
}
export function focus() {
el?.focus()
if (el) {
el.style.height = '5px'
el.style.height = el.scrollHeight + 50 + 'px'
}
}
function validateInput(pattern: string | undefined, v: any): void {
if (required && (v == undefined || v == null || v === '')) {
error = 'This field is required'
valid = false
} else {
if (pattern && !testRegex(pattern, v)) {
error = `Should match ${pattern}`
valid = false
} else {
error = ''
valid = true
}
}
}
function testRegex(pattern: string, value: any): boolean {
try {
const regex = new RegExp(pattern)
return regex.test(value)
} catch (err) {
return false
}
}
$: {
if (value == undefined || value == null) {
value = defaultValue
if (defaultValue === undefined || defaultValue === null) {
if (inputCat === 'string') {
value = ''
} else if (inputCat == 'enum') {
value = enum_?.[0]
} else if (inputCat == 'boolean') {
value = false
}
}
}
}
$: inputCat = computeInputCat(type, format, itemsType?.type, enum_, contentEncoding)
</script>
<div class="flex flex-col w-full min-w-[250px]">
<div>
{#if displayHeader}
<FieldHeader {label} {required} {type} {contentEncoding} {format} />
{/if}
{#if description}
<div class="text-sm italic pb-1">
{description}
</div>
{/if}
<div class="flex space-x-1">
{#if inputCat == 'number'}
{#if extra['min'] != undefined && extra['max'] != undefined}
<div class="flex w-full gap-1">
<span>{extra['min']}</span>
<div class="grow">
<Range bind:value min={extra['min']} max={extra['max']} />
</div>
<span>{extra['max']}</span>
<span class="mx-2"><Badge large color="blue">{value}</Badge></span>
</div>
{:else}
<input
on:focus={(e) => {
dispatch('focus')
}}
type="number"
class={valid
? ''
: 'border border-red-700 border-opacity-30 focus:border-red-700 focus:border-opacity-30 bg-red-100'}
placeholder={defaultValue ?? ''}
bind:value
min={extra['min']}
max={extra['max']}
/>
{/if}
{:else if inputCat == 'boolean'}
<Toggle
on:pointerdown={(e) => {
e?.stopPropagation()
}}
class={valid
? ''
: 'border border-red-700 border-opacity-30 focus:border-red-700 focus:border-opacity-30 bg-red-100'}
bind:checked={value}
/>
{#if type == 'boolean' && value == undefined}
<span>&nbsp; Not set</span>
{/if}
{:else if inputCat == 'list'}
<div>
<div>
{#each value ?? [] as v, i}
<div class="flex flex-row max-w-md mt-1">
{#if itemsType?.type == 'number'}
<input type="number" bind:value={v} />
{:else if itemsType?.type == 'string' && itemsType?.contentEncoding == 'base64'}
<input
type="file"
class="my-6"
on:change={(x) => fileChanged(x, (val) => (value[i] = val))}
multiple={false}
/>
{:else}
<input type="text" bind:value={v} />
{/if}
<Button
variant="border"
color="red"
size="sm"
btnClasses="mx-6"
on:click={() => {
value = value.filter((el) => el != v)
if (value.length == 0) {
value = undefined
}
}}
>
<Icon data={faMinus} />
</Button>
</div>
{/each}
</div>
<Button
variant="border"
color="blue"
size="sm"
btnClasses="mt-1"
on:click={() => {
if (value == undefined || !Array.isArray(value)) {
value = []
}
value = value.concat('')
}}
>
<Icon data={faPlus} class="mr-2" />
Add item
</Button>
<span class="ml-2">
{(value ?? []).length} item{(value ?? []).length > 1 ? 's' : ''}
</span>
</div>
{:else if inputCat == 'resource-object'}
<ObjectResourceInput {format} bind:value />
{:else if inputCat == 'object'}
{#if properties && Object.keys(properties).length > 0}
<div class="p-4 pl-8 border rounded w-full">
<LightweightSchemaForm
schema={{ properties, $schema: '', required: [], type: 'object' }}
bind:args={value}
/>
</div>
{:else}
<textarea
bind:this={el}
on:focus={(e) => {
dispatch('focus')
}}
use:autosize
style="max-height: {maxHeight}"
class="col-span-10 {valid
? ''
: 'border border-red-700 border-opacity-30 focus:border-red-700 focus:border-opacity-30 bg-red-100'}"
placeholder={defaultValue ? JSON.stringify(defaultValue, null, 4) : ''}
bind:value={rawValue}
/>
{/if}
{:else if inputCat == 'enum'}
<select
on:focus={(e) => {
dispatch('focus')
}}
class="px-6"
bind:value
>
{#each enum_ ?? [] as e}
<option>{e}</option>
{/each}
</select>
{:else if inputCat == 'date'}
<input class="inline-block" type="datetime-local" bind:value />
{:else if inputCat == 'base64'}
<input
type="file"
class="my-6"
on:change={(x) => fileChanged(x, (val) => (value = val))}
multiple={false}
/>
{:else if inputCat == 'resource-string'}
<ResourcePicker
bind:value
resourceType={format.split('-').length > 1
? format.substring('resource-'.length)
: undefined}
/>
{:else if inputCat == 'string'}
<div class="flex flex-col w-full">
<div class="flex flex-row w-full items- justify-between">
<textarea
rows="1"
bind:this={el}
on:focus={(e) => {
dispatch('focus')
}}
use:autosize
type="text"
class="col-span-10 {valid
? ''
: 'border border-red-700 border-opacity-30 focus:border-red-700 focus:border-opacity-30 bg-red-100'}"
placeholder={defaultValue ?? ''}
bind:value
/>
</div>
</div>
{/if}
<slot name="actions" />
</div>
</div>
</div>
@@ -0,0 +1,35 @@
<script lang="ts">
import type { Schema } from '$lib/common'
import LightweightArgInput from './LightweightArgInput.svelte'
export let schema: Schema
export let args: Record<string, any> | undefined = undefined
$: if (args === undefined) {
args = {}
}
</script>
<div class="w-full">
{#each Object.keys(schema.properties ?? {}) as argName, i (argName)}
<div>
{#if typeof args == 'object' && schema?.properties[argName] && args}
<LightweightArgInput
label={argName}
description={schema.properties[argName].description}
bind:value={args[argName]}
type={schema.properties[argName].type}
required={schema.required.includes(argName)}
pattern={schema.properties[argName].pattern}
defaultValue={schema.properties[argName].default}
enum_={schema.properties[argName].enum}
format={schema.properties[argName].format}
contentEncoding={schema.properties[argName].contentEncoding}
properties={schema.properties[argName].properties}
itemsType={schema.properties[argName].items}
extra={schema.properties[argName]}
/>
{/if}
</div>
{/each}
</div>
@@ -60,6 +60,7 @@
if (testJob && !testJob.canceled && testJob.type == 'CompletedJob' && `result` in testJob) {
if ($flowStateStore[mod.id]?.previewResult) {
$flowStateStore[mod.id].previewResult = testJob.result
$flowStateStore = $flowStateStore
}
}
}
@@ -11,7 +11,7 @@
<span class="flex items-center space-x-2 mb-2">
<h1>{title}</h1>
{#if tooltip}
<Tooltip>{tooltip}</Tooltip>
<Tooltip scale={0.9}>{tooltip}</Tooltip>
{/if}
</span>
{:else}
+26 -3
View File
@@ -23,6 +23,7 @@
import FolderEditor from './FolderEditor.svelte'
import GroupEditor from './GroupEditor.svelte'
import { random_adj } from './random_positive_adjetive'
import Badge from './common/badge/Badge.svelte'
type PathKind = 'resource' | 'script' | 'variable' | 'flow' | 'schedule' | 'app'
let meta: Meta | undefined = undefined
@@ -199,6 +200,10 @@
}
function initPath() {
if (path != undefined && path != '') {
meta = pathToMeta(path)
return
}
if (initialPath == undefined || initialPath == '') {
reset()
} else {
@@ -306,7 +311,7 @@
{#if meta != undefined}
<div class="flex gap-4 shrink">
<label class="block">
<span class="text-gray-700 text-sm whitespace-nowrap">&nbsp;</span>
<span class="text-gray-700 text-sm whitespace-nowrap">Owner</span>
<ToggleButtonGroup
class="mt-0.5"
@@ -409,8 +414,26 @@
{/if}
</div>
<div class="flex-row flex justify-between w-full">
<div><span class="font-mono text-sm break-all">{path}</span></div>
<div class="flex flex-col w-full mt-4">
<div class="flex justify-start w-full">
<Badge
color="gray"
class="center-center !bg-gray-300 !text-gray-600 !w-[70px] !h-[24px] rounded-r-none"
>
Full path
</Badge>
<input
type="text"
readonly
value={path}
size={path?.length || 50}
class="font-mono !text-xs max-w-[calc(100%-70px)] !w-auto !h-[24px] !py-0 !border-l-0 !rounded-l-none"
on:focus={({ currentTarget }) => {
currentTarget.select()
}}
/>
<!-- <span class="font-mono text-sm break-all">{path}</span> -->
</div>
<div class="text-red-600 text-2xs">{error}</div>
</div>
</div>
@@ -88,7 +88,7 @@
}}
items={collection}
class="text-clip grow min-w-0"
placeholder="{resourceType} resource"
placeholder="{resourceType ?? 'any'} resource"
inputStyles={SELECT_INPUT_DEFAULT_STYLE.inputStyles}
containerStyles={SELECT_INPUT_DEFAULT_STYLE.containerStyles}
/>
+28 -5
View File
@@ -45,7 +45,7 @@
label: 'Failed',
data:
failed?.map((job) => ({
x: job.created_at as any,
x: job.started_at as any,
y: job.duration_ms,
id: job.id,
path: job.script_path
@@ -57,7 +57,7 @@
label: 'Successful',
data:
success?.map((job) => ({
x: job.created_at as any,
x: job.started_at as any,
y: job.duration_ms,
id: job.id,
path: job.script_path
@@ -71,7 +71,10 @@
enabled: true,
modifierKey: 'ctrl' as 'ctrl',
onPanComplete: ({ chart }) => {
dispatch('zoom', { min: new Date(chart.scales.x.min), max: new Date(chart.scales.x.max) })
dispatch('zoom', {
min: addSeconds(new Date(chart.scales.x.min), -1),
max: addSeconds(new Date(chart.scales.x.max), 1)
})
}
},
zoom: {
@@ -80,7 +83,10 @@
},
mode: 'x' as 'x',
onZoom: ({ chart }) => {
dispatch('zoom', { min: new Date(chart.scales.x.min), max: new Date(chart.scales.x.max) })
dispatch('zoom', {
min: addSeconds(new Date(chart.scales.x.min), -1),
max: addSeconds(new Date(chart.scales.x.max), 1)
})
}
}
}
@@ -88,6 +94,22 @@
function getPath(x: any): string {
return x.path
}
$: minTime = getMinTime(jobs)
function addSeconds(date: Date, seconds: number): Date {
date.setTime(date.getTime() + seconds * 1000)
return date
}
function getMinTime(jobs: CompletedJob[] | undefined): Date {
return addSeconds(new Date(jobs?.[jobs?.length - 1]?.started_at ?? new Date().toString()), -15)
}
$: maxTime = getMaxTime(jobs)
function getMaxTime(jobs: CompletedJob[] | undefined): Date {
return addSeconds(new Date(jobs?.[0]?.started_at ?? new Date().toString()), 15)
}
</script>
<Scatter
@@ -115,7 +137,8 @@
display: false
},
type: 'time',
min: jobs?.[jobs?.length - 1]?.created_at ?? new Date().toString()
min: minTime,
max: maxTime
},
y: {
grid: {
+7 -2
View File
@@ -16,7 +16,7 @@
import { Badge, Button } from './common'
import SharedBadge from './SharedBadge.svelte'
import Toggle from './Toggle.svelte'
import { userStore } from '$lib/stores'
import { runFormStore, userStore } from '$lib/stores'
import Tooltip from './Tooltip.svelte'
import CliHelpBox from './CliHelpBox.svelte'
import InlineCodeCopy from './InlineCodeCopy.svelte'
@@ -51,7 +51,12 @@
export let viewCliRun = false
export let isFlow: boolean
export let args: Record<string, any> = decodeArgs($page.url.searchParams.get('args') ?? undefined)
export let args: Record<string, any> = {}
if ($runFormStore) {
args = $runFormStore
$runFormStore = undefined
}
export function run() {
runAction(scheduledForStr, args, invisible_to_owner)
+15 -10
View File
@@ -12,6 +12,7 @@
import Toggle from './Toggle.svelte'
import Tooltip from './Tooltip.svelte'
import { flip } from 'svelte/animate'
import Portal from 'svelte-portal'
const dispatch = createEventDispatcher()
@@ -178,8 +179,10 @@
/>
<div class="ml-2">
<Tooltip>
Arguments can be edited either using the wizard, or by editing their JSON Schema,
<a href="https://docs.windmill.dev/docs/reference/#script-parameters-to-json-schema">see docs</a>
Arguments can be edited either using the wizard, or by editing their JSON Schema,
<a href="https://docs.windmill.dev/docs/reference/#script-parameters-to-json-schema"
>see docs</a
>
</Tooltip>
</div>
</div>
@@ -281,11 +284,13 @@
</div>
</div>
<SchemaModal
bind:this={schemaModal}
bind:property={modalProperty}
bind:error={argError}
on:save={handleAddOrEditArgument}
bind:editing
bind:oldArgName
/>
<Portal>
<SchemaModal
bind:this={schemaModal}
bind:property={modalProperty}
bind:error={argError}
on:save={handleAddOrEditArgument}
bind:editing
bind:oldArgName
/>
</Portal>
@@ -1,6 +1,6 @@
<script lang="ts">
import type { Schema } from '$lib/common'
import { VariableService, type InputTransform } from '$lib/gen'
import { VariableService } from '$lib/gen'
import { workspaceStore } from '$lib/stores'
import { allTrue } from '$lib/utils'
import { faPlus } from '@fortawesome/free-solid-svg-icons'
@@ -10,7 +10,7 @@
import VariableEditor from './VariableEditor.svelte'
export let schema: Schema
export let args: Record<string, InputTransform | any> = {}
export let args: Record<string, any> = {}
export let disabledArgs: string[] = []
export let disabled = false
+132 -133
View File
@@ -1,29 +1,30 @@
<script lang="ts">
import { Script, ScriptService } from '$lib/gen'
import { goto } from '$app/navigation'
import { page } from '$app/stores'
import { inferArgs } from '$lib/infer'
import { initialCode, isInitialCode } from '$lib/script_helpers'
import { initialCode } from '$lib/script_helpers'
import { userStore, workspaceStore } from '$lib/stores'
import { emptySchema, encodeState, sendUserToast, setQueryWithoutLoad } from '$lib/utils'
import Path from './Path.svelte'
import RadioButton from './RadioButton.svelte'
import ScriptEditor from './ScriptEditor.svelte'
import ScriptSchema from './ScriptSchema.svelte'
import CenteredPage from './CenteredPage.svelte'
import { dirtyStore } from './common/confirmationModal/dirtyStore'
import { Button, ButtonPopup, ButtonPopupItem, Kbd } from './common'
import { faChevronDown, faChevronUp, faPen, faSave } from '@fortawesome/free-solid-svg-icons'
import { faPen, faSave } from '@fortawesome/free-solid-svg-icons'
import Breadcrumb from './common/breadcrumb/Breadcrumb.svelte'
import LanguageIcon from './common/languageIcons/LanguageIcon.svelte'
import type { SupportedLanguage } from '$lib/common'
import Tooltip from './Tooltip.svelte'
import SettingSection from './SettingSection.svelte'
export let script: Script
export let initialPath: string = ''
export let template: 'pgsql' | 'mysql' | 'script' = 'script'
export let initialArgs: Record<string, any> = {}
export let lockedLanguage = false
export let topHash: string | undefined = undefined
const langs: [string, SupportedLanguage][] = [
['Typescript', Script.language.DENO],
@@ -31,14 +32,34 @@
['Go', Script.language.GO],
['Bash', Script.language.BASH]
]
let viewScriptKind = script.kind !== Script.kind.SCRIPT
const scriptKindOptions: { value: Script.kind; title: string; desc?: string }[] = [
{
value: Script.kind.SCRIPT,
title: 'Action'
},
{
value: Script.kind.TRIGGER,
title: 'Trigger',
desc: 'First module of flows to trigger them based on external changes. These kind of scripts are usually running on a schedule to periodically look for changes.'
},
{
value: Script.kind.APPROVAL,
title: 'Approval',
desc: 'Send notifications externally to ask for approval to continue a flow.'
},
{
value: Script.kind.FAILURE,
title: 'Error Handler',
desc: 'Handle errors in flows after all retry attempts have been exhausted.'
}
]
let pathError = ''
let summaryC: HTMLInputElement | undefined = undefined
let pathC: Path | undefined = undefined
let loadingSave = false
$: setQueryWithoutLoad($page.url, 'state', encodeState(script))
$: setQueryWithoutLoad($page.url, [{ key: 'state', value: encodeState(script) }])
$: step = Number($page.url.searchParams.get('step')) || 1
if (script.content == '') {
@@ -53,7 +74,6 @@
script.content = initialCode(language, kind, template)
}
let loadingSave = false
async function editScript(leave: boolean): Promise<void> {
loadingSave = true
try {
@@ -65,8 +85,7 @@
await inferArgs(script.language, script.content, script.schema)
} catch (error) {
sendUserToast(
`Impossible to infer the schema. Assuming this is a script without main function`,
true
`The main signature was not parsable. This script is considered to be without main function`
)
}
@@ -77,7 +96,7 @@
summary: script.summary,
description: script.description ?? '',
content: script.content,
parent_hash: script.hash != '' ? script.hash : undefined,
parent_hash: script.hash != '' ? topHash ?? script.hash : undefined,
schema: script.schema,
is_template: script.is_template,
language: script.language,
@@ -90,9 +109,10 @@
} else {
await goto(`/scripts/edit/${newHash}?step=2`)
script.hash = newHash
topHash = undefined
}
} catch (error) {
sendUserToast(`Impossible to save the script: ${error.body || error.message}`, true)
sendUserToast(`Error while saving the script: ${error.body || error.message}`, true)
}
loadingSave = false
}
@@ -104,7 +124,7 @@
await inferArgs(script.language, script.content, script.schema)
} catch (error) {
console.info(
'Impossible to infer the schema. Assuming this is a script without main function'
'The main signature was not parsable. This script is considered to be without main function'
)
}
}
@@ -211,132 +231,111 @@
<!-- metadata -->
{#if step === 1}
<CenteredPage>
<h2 class="border-b pb-1 mt-8 mb-2">Path</h2>
<Path
bind:this={pathC}
bind:error={pathError}
bind:path={script.path}
{initialPath}
on:enter={() => changeStep(2)}
namePlaceholder="script"
kind="script"
/>
<h2 class="border-b pb-1 mt-12 mb-4">Summary</h2>
<input
type="text"
bind:this={summaryC}
bind:value={script.summary}
placeholder="Short summary to be displayed when listed"
/>
<h2 class="border-b pb-1 mt-12 mb-6">Language</h2>
{#if lockedLanguage}
<div class="text-sm text-gray-600 italic mb-2">
As a forked script, the language '{script.language}' cannot be modified.
</div>
{/if}
<div class="flex flex-row gap-2 flex-wrap">
{#each langs as [label, lang]}
{@const isPicked = script.language == lang && template == 'script'}
<SettingSection title="Path" element="h2">
<Path
bind:this={pathC}
bind:error={pathError}
bind:path={script.path}
{initialPath}
on:enter={() => changeStep(2)}
namePlaceholder="script"
kind="script"
/>
</SettingSection>
<SettingSection title="Summary" element="h2">
<input
type="text"
bind:this={summaryC}
bind:value={script.summary}
placeholder="Short summary to be displayed when listed"
/>
</SettingSection>
<SettingSection title="Language" element="h2">
{#if lockedLanguage}
<div class="text-sm text-gray-600 italic mb-2">
As a forked script, the language '{script.language}' cannot be modified.
</div>
{/if}
<div class="flex flex-row gap-2 flex-wrap">
{#each langs as [label, lang]}
{@const isPicked = script.language == lang && template == 'script'}
<Button
size="sm"
variant="border"
color={isPicked ? 'blue' : 'dark'}
btnClasses={isPicked ? '!border-2 !bg-blue-50/75' : 'm-[1px]'}
on:click={() => {
script.language = lang
template = 'script'
initContent(lang, script.kind, template)
}}
disabled={lockedLanguage}
>
<LanguageIcon {lang} />
<span class="ml-2 py-2">{label}</span>
</Button>
{/each}
<Button
size="sm"
variant="border"
color={isPicked ? 'blue' : 'dark'}
btnClasses={isPicked ? '!border-2 !bg-blue-50/75' : 'm-[1px]'}
on:click={() => {
script.language = lang
template = 'script'
initContent(lang, script.kind, template)
}}
color={template == 'pgsql' ? 'blue' : 'dark'}
btnClasses={template == 'pgsql' ? '!border-2 !bg-blue-50/75' : 'm-[1px]'}
disabled={lockedLanguage}
>
<LanguageIcon {lang} /><span class="ml-2 py-4">{label}</span>
</Button>
{/each}
<Button
size="sm"
variant="border"
color={template == 'pgsql' ? 'blue' : 'dark'}
btnClasses={template == 'pgsql' ? '!border-2 !bg-blue-50/75' : 'm-[1px]'}
disabled={lockedLanguage}
on:click={() => {
script.language = Script.language.DENO
template = 'pgsql'
initContent(script.language, script.kind, template)
}}
>
<LanguageIcon lang="pgsql" /><span class="ml-2 py-4">PostgreSQL</span>
</Button>
<!-- <Button
size="sm"
variant="border"
color={template == 'mysql' ? 'blue' : 'dark'}
btnClasses={template == 'mysql' ? '!border-2 !bg-blue-50/75' : 'm-[1px]'}
on:click={() => {
script.language = Script.language.DENO
template = 'mysql'
initContent(script.language, script.kind, template)
}}
>
<LanguageIcon lang="mysql" /><span class="ml-2 py-4">MySQL</span>
</Button> -->
</div>
<div class="mt-16 mb-4">
<Button
color="light"
size="sm"
endIcon={{ icon: viewScriptKind ? faChevronUp : faChevronDown }}
on:click={() => (viewScriptKind = !viewScriptKind)}
>
Tag this script as having a specific purpose inside flows
</Button>
</div>
{#if viewScriptKind}
<div class="max-w-lg">
<RadioButton
label="Script Type"
options={[
['Action', Script.kind.SCRIPT],
[
{
title: 'Trigger',
desc: `First module of flows to trigger them based on watching changes external periodically using an internal state`
},
Script.kind.TRIGGER
],
[
{
title: 'Approval',
desc: `Send notification externally to ask for approval to continue a flow`
},
Script.kind.APPROVAL
],
[
{
title: 'Error Handler',
desc: `Handle errors for flows after all retries attempts have been exhausted`
},
Script.kind.FAILURE
]
// ['Command Handler', Script.kind.COMMAND]
]}
on:change={(e) => {
if (isInitialCode(script.content)) {
template = 'script'
initContent(script.language, e.detail, template)
}
on:click={() => {
script.language = Script.language.DENO
template = 'pgsql'
initContent(script.language, script.kind, template)
}}
bind:value={script.kind}
/>
>
<LanguageIcon lang="pgsql" /><span class="ml-2 py-2">PostgreSQL</span>
</Button>
<!-- <Button
size="sm"
variant="border"
color={template == 'mysql' ? 'blue' : 'dark'}
btnClasses={template == 'mysql' ? '!border-2 !bg-blue-50/75' : 'm-[1px]'}
on:click={() => {
script.language = Script.language.DENO
template = 'mysql'
initContent(script.language, script.kind, template)
}}
>
<LanguageIcon lang="mysql" /><span class="ml-2 py-2">MySQL</span>
</Button> -->
</div>
{/if}
<!-- <div class="ml-3">
<Toggle
bind:checked={script.is_template}
options={{ right: 'Save as a workspace template' }}
/>
</div> -->
</SettingSection>
<SettingSection
title="Script kind"
element="h3"
tooltip="Tag this script as having a specific purpose inside flows. If it won't be used in flows,
you don't have to worry about this."
accordion
>
<div class="flex flex-wrap gap-2">
{#each scriptKindOptions as { value, title, desc }}
{@const isPicked = script.kind === value}
<Button
size="sm"
variant="border"
color={isPicked ? 'blue' : 'dark'}
btnClasses="font-medium {isPicked ? '!bg-blue-50/75' : ''}"
on:click={() => {
template = 'script'
script.kind = value
initContent(script.language, value, template)
}}
disabled={lockedLanguage}
>
{title}
{#if desc}
<Tooltip class="mb-0.5 ml-1">
{desc}
</Tooltip>
{/if}
</Button>
{/each}
</div>
</SettingSection>
</CenteredPage>
{:else if step === 2}
<ScriptEditor
@@ -0,0 +1,47 @@
<script lang="ts">
import { ChevronDown } from 'lucide-svelte'
import { slide } from 'svelte/transition'
import type { IntRange } from '../common'
import Tooltip from './Tooltip.svelte'
export let title: string
export let tooltip: string = ''
export let element: `h${IntRange<1, 6>}` = 'h2'
export let accordion = false
let showContent = !accordion
</script>
<div class="border-b [&:has(button:hover)]:border-gray-400 duration-200 pb-1 mt-8 mb-2">
{#if accordion}
<button
class="flex w-full justify-start items-center"
on:click={() => (showContent = !showContent)}
>
<span class="rounded-full hover:bg-gray-100 focus:bg-gray-100 p-1 mr-1">
<ChevronDown size={22} class="rotate-0 duration-300 {showContent ? '!-rotate-180' : ''}" />
</span>
<svelte:element this={element}>
{title}
{#if tooltip}
<Tooltip scale={0.9} class="mb-0.5">
{tooltip}
</Tooltip>
{/if}
</svelte:element>
</button>
{:else}
<svelte:element this={element}>
{title}
{#if tooltip}
<Tooltip scale={0.9} class="mb-0.5">
{tooltip}
</Tooltip>
{/if}
</svelte:element>
{/if}
</div>
{#if showContent}
<div transition:slide|local={{ duration: 300 }}>
<slot />
</div>
{/if}
@@ -99,7 +99,7 @@
}
}
function format() {
export function format() {
if (editor) {
code = getCode()
editor.getAction('editor.action.formatDocument').run()
@@ -176,7 +176,6 @@
editor.onDidBlurEditorText(() => {
code = getCode()
dispatch('blur')
})
if (lang == 'javascript') {
@@ -2,13 +2,18 @@
import { ChevronDown, ChevronUp } from 'lucide-svelte'
import { slide } from 'svelte/transition'
import Button from './common/button/Button.svelte'
import Tooltip from './Tooltip.svelte'
export let text: string
export let tooltip: string | undefined = undefined
export let view = false
</script>
<Button color="light" on:click={() => (view = !view)} variant="border"
>{text}
{#if tooltip}
<Tooltip wrapperClass="mx-1">{tooltip}</Tooltip>
{/if}
{#if !view}<ChevronDown />{:else}<ChevronUp />{/if}</Button
>
{#if view}
@@ -573,10 +573,13 @@
editor.onDidBlurEditorText(() => {
code = getCode()
dispatch('blur')
})
}
export function focus() {
editor?.focus()
}
onMount(() => {
if (browser) {
loadMonaco()
+6 -3
View File
@@ -10,6 +10,7 @@
export let disabled = false
export let textClass = ''
export let textStyle = ''
export let color: 'blue' | 'red' = 'blue'
export let size: 'sm' | 'xs' = 'sm'
const id = (Math.random() + 1).toString(36).substring(10)
@@ -26,7 +27,7 @@
{#if Boolean(options?.left)}
<span
class={twMerge(
'ml-2 font-medium duration-200',
'mr-2 font-medium duration-200',
disabled ? 'text-gray-500' : 'text-gray-900',
size === 'xs' ? 'text-xs' : 'text-sm',
textClass
@@ -37,7 +38,7 @@
</span>
{/if}
<!-- svelte-ignore a11y-click-events-have-key-events -->
<div class="relative" on:pointerdown on:click|stopPropagation>
<div class="relative" on:click|stopPropagation>
<input
on:focus
on:click
@@ -55,7 +56,9 @@
class="w-11 h-6 bg-gray-200 rounded-full peer peer-focus:ring-4 peer-focus:ring-blue-300
peer-checked:after:translate-x-full peer-checked:after:border-white after:content-['']
after:absolute after:top-0.5 after:left-[2px] after:bg-white after:border-gray-300
after:border after:rounded-full after:h-5 after:w-5 after:transition-all peer-checked:bg-blue-600"
after:border after:rounded-full after:h-5 after:w-5 after:transition-all {color == 'red'
? 'peer-checked:bg-red-600'
: 'peer-checked:bg-blue-600'}"
/>
</div>
{#if Boolean(options?.right)}
+3 -1
View File
@@ -18,5 +18,7 @@
data={faInfoCircle}
{scale}
/>
<span slot="text"> <slot /> </span>
<svelte:fragment slot="text">
<slot />
</svelte:fragment>
</Popover>
@@ -57,7 +57,8 @@
decryptSecret: false
})
can_write =
getV.workspace_id == $workspaceStore && canWrite(path, getV.extra_perms ?? {}, $userStore)
getV.workspace_id == $workspaceStore &&
canWrite(edit_path, getV.extra_perms ?? {}, $userStore)
variable = {
value: getV.value ?? '',
@@ -9,9 +9,7 @@
...new Set(
workers
.filter((worker) => {
const date = new Date().getTime() - 300 * 60
const ping_at = new Date(worker.ping_at).getTime()
return worker.ip != 'unretrievable IP' && ping_at > date
return worker.ip != 'unretrievable IP' && worker.last_ping && worker.last_ping < 300
})
.map((worker) => worker.ip)
)
@@ -1,176 +1,149 @@
<script lang="ts">
import { Button, type ButtonType } from '$lib/components/common'
import { Button } from '$lib/components/common'
import { getContext } from 'svelte'
import type { AppInput } from '../../inputType'
import type { Output } from '../../rx'
import type { AppViewerContext, ComponentCustomCSS } from '../../types'
import type { AppViewerContext, ComponentCustomCSS, RichConfigurations } from '../../types'
import AlignWrapper from '../helpers/AlignWrapper.svelte'
import InputValue from '../helpers/InputValue.svelte'
import type RunnableComponent from '../helpers/RunnableComponent.svelte'
import RunnableWrapper from '../helpers/RunnableWrapper.svelte'
import { loadIcon } from '../icon'
import { twMerge } from 'tailwind-merge'
import { goto } from '$app/navigation'
import { initConfig, initOutput } from '../../editor/appUtils'
import { components } from '../../editor/component'
import ResolveConfig from '../helpers/ResolveConfig.svelte'
export let id: string
export let componentInput: AppInput | undefined
export let configuration: Record<string, AppInput>
export let configuration: RichConfigurations
export let recomputeIds: string[] | undefined = undefined
export let extraQueryParams: Record<string, any> = {}
export let horizontalAlignment: 'left' | 'center' | 'right' | undefined = undefined
export let verticalAlignment: 'top' | 'center' | 'bottom' | undefined = undefined
export let noWFull = false
export let preclickAction: (() => Promise<void>) | undefined = undefined
export let customCss: ComponentCustomCSS<'button'> | undefined = undefined
export let customCss: ComponentCustomCSS<'buttoncomponent'> | undefined = undefined
export let render: boolean
export let initializing: boolean | undefined = false
export let extraKey: string | undefined = undefined
export const staticOutputs: string[] = ['loading', 'result']
export let controls: { left: () => boolean; right: () => boolean | string } | undefined =
undefined
const { runnableComponents, worldStore, app } = getContext<AppViewerContext>('AppViewerContext')
const { worldStore, app, componentControl, selectedComponent } =
getContext<AppViewerContext>('AppViewerContext')
let resolvedConfig = initConfig(
components['buttoncomponent'].initialData.configuration,
configuration
)
$: initializing = resolvedConfig?.label == undefined
let outputs = initOutput($worldStore, id, {
result: undefined,
loading: false
})
if (controls) {
$componentControl[id] = controls
}
let labelValue: string
let color: ButtonType.Color
let size: ButtonType.Size
let runnableComponent: RunnableComponent
let disabled: boolean | undefined = undefined
let fillContainer: boolean | undefined = undefined
let gotoUrl: string | undefined = undefined
let gotoNewTab: boolean | undefined = undefined
let isLoading: boolean = false
let ownClick: boolean = false
let triggerOnAppLoad = false
let beforeIcon: undefined | string = undefined
let afterIcon: undefined | string = undefined
let beforeIconComponent: any
let afterIconComponent: any
$: beforeIcon && handleBeforeIcon()
$: afterIcon && handleAfterIcon()
$: resolvedConfig.beforeIcon && handleBeforeIcon()
$: resolvedConfig.afterIcon && handleAfterIcon()
async function handleBeforeIcon() {
if (beforeIcon) {
beforeIconComponent = await loadIcon(beforeIcon)
if (resolvedConfig.beforeIcon) {
beforeIconComponent = await loadIcon(resolvedConfig.beforeIcon)
}
}
async function handleAfterIcon() {
if (afterIcon) {
afterIconComponent = await loadIcon(afterIcon)
if (resolvedConfig.afterIcon) {
afterIconComponent = await loadIcon(resolvedConfig.afterIcon)
}
}
$: outputs = $worldStore?.outputsById[id] as {
result: Output<Array<any>>
loading: Output<boolean>
}
$: resolvedConfig?.triggerOnAppLoad && runnableComponent?.runComponent()
$: triggerOnAppLoad && runnableComponent?.runComponent()
$: if (outputs?.loading != undefined) {
outputs.loading.set(false, true)
}
$: outputs?.loading.subscribe({
next: (value) => {
isLoading = value
if (ownClick && !value) {
ownClick = false
}
}
})
$: loading = isLoading && ownClick
let errors: Record<string, string> = {}
$: errorsMessage = Object.values(errors)
.filter((x) => x != '')
.join('\n')
let runnableWrapper: RunnableWrapper
async function handleClick(event: CustomEvent) {
event?.stopPropagation()
event?.preventDefault()
$selectedComponent = [id]
if (preclickAction) {
await preclickAction()
}
ownClick = true
if (!runnableComponent) {
if (gotoUrl) {
if (gotoNewTab) {
window.open(gotoUrl, '_blank')
} else {
goto(gotoUrl)
}
}
runnableWrapper.onSuccess()
} else {
await runnableComponent?.runComponent()
}
if (recomputeIds) {
await Promise.all(recomputeIds.map((id) => $runnableComponents?.[id]?.()))
}
}
let loading = false
</script>
<InputValue {id} input={configuration.label} bind:value={labelValue} />
<InputValue {id} input={configuration.goto} bind:value={gotoUrl} />
<InputValue {id} input={configuration.color} bind:value={color} />
<InputValue {id} input={configuration.size} bind:value={size} />
<InputValue {id} input={configuration.beforeIcon} bind:value={beforeIcon} />
<InputValue {id} input={configuration.afterIcon} bind:value={afterIcon} />
<InputValue {id} input={configuration.triggerOnAppLoad} bind:value={triggerOnAppLoad} />
{#each Object.keys(components['buttoncomponent'].initialData.configuration) as key (key)}
<ResolveConfig
{id}
{extraKey}
{key}
bind:resolvedConfig={resolvedConfig[key]}
configuration={configuration[key]}
/>
{/each}
<InputValue
{id}
input={configuration.disabled}
bind:value={disabled}
bind:error={errors.disabled}
/>
<InputValue {id} input={configuration.fillContainer} bind:value={fillContainer} />
<InputValue {id} input={configuration.gotoNewTab} bind:value={gotoNewTab} />
<!-- gotoNewTab={resolvedConfig.onSuccess.selected == 'goto'} -->
<RunnableWrapper
flexWrap
bind:this={runnableWrapper}
{recomputeIds}
bind:runnableComponent
bind:loading
{componentInput}
doOnSuccess={resolvedConfig.onSuccess}
{id}
{extraQueryParams}
autoRefresh={false}
goto={gotoUrl}
{gotoNewTab}
{render}
{outputs}
{extraKey}
>
<AlignWrapper {noWFull} {horizontalAlignment} {verticalAlignment}>
{#if errorsMessage}
<div class="text-red-500 text-xs">{errorsMessage}</div>
{/if}
<Button
on:pointerdown={(e) => e.stopPropagation()}
btnClasses={twMerge(
$app.css?.['buttoncomponent']?.['button']?.class,
customCss?.button.class,
fillContainer ? 'w-full h-full' : ''
customCss?.button?.class,
resolvedConfig.fillContainer ? 'w-full h-full' : ''
)}
style={[$app.css?.['buttoncomponent']?.['button']?.style, customCss?.button.style].join(';')}
{disabled}
on:pointerdown={(e) => {
e?.stopPropagation()
window.dispatchEvent(new Event('pointerup'))
}}
style={[$app.css?.['buttoncomponent']?.['button']?.style, customCss?.button?.style].join(';')}
disabled={resolvedConfig.disabled}
on:click={handleClick}
{size}
{color}
size={resolvedConfig.size}
color={resolvedConfig.color}
{loading}
>
<span class="truncate inline-flex gap-2 items-center">
{#if beforeIcon && beforeIconComponent}
{#if resolvedConfig.beforeIcon && beforeIconComponent}
<svelte:component this={beforeIconComponent} size={14} />
{/if}
<div>{labelValue}</div>
{#if afterIcon && afterIconComponent}
<div>{resolvedConfig.label}</div>
{#if resolvedConfig.afterIcon && afterIconComponent}
<svelte:component this={afterIconComponent} size={14} />
{/if}
</span>
@@ -3,34 +3,39 @@
import { faUser } from '@fortawesome/free-solid-svg-icons'
import { getContext } from 'svelte'
import { Icon } from 'svelte-awesome'
import { initConfig, initOutput } from '../../editor/appUtils'
import { components } from '../../editor/component'
import type { AppInput } from '../../inputType'
import type { Output } from '../../rx'
import type { AppViewerContext, ComponentCustomCSS } from '../../types'
import type { AppViewerContext, ComponentCustomCSS, RichConfigurations } from '../../types'
import { concatCustomCss } from '../../utils'
import AlignWrapper from '../helpers/AlignWrapper.svelte'
import InputValue from '../helpers/InputValue.svelte'
import ResolveConfig from '../helpers/ResolveConfig.svelte'
import type RunnableComponent from '../helpers/RunnableComponent.svelte'
import RunnableWrapper from '../helpers/RunnableWrapper.svelte'
export let id: string
export let componentInput: AppInput | undefined
export let configuration: Record<string, AppInput>
export let configuration: RichConfigurations
export let recomputeIds: string[] | undefined = undefined
export let extraQueryParams: Record<string, any> = {}
export let horizontalAlignment: 'left' | 'center' | 'right' | undefined = undefined
export let customCss: ComponentCustomCSS<'container' | 'button'> | undefined = undefined
export let customCss: ComponentCustomCSS<'formcomponent'> | undefined = undefined
export let render: boolean
export const staticOutputs: string[] = ['loading', 'result']
const { app, runnableComponents, worldStore, stateId } =
getContext<AppViewerContext>('AppViewerContext')
const { app, worldStore, stateId } = getContext<AppViewerContext>('AppViewerContext')
let labelValue: string = 'Default label'
let color: ButtonType.Color
let size: ButtonType.Size
let outputs = initOutput($worldStore, id, {
result: undefined,
loading: false
})
let resolvedConfig = initConfig(
components['formcomponent'].initialData.configuration,
configuration
)
let runnableComponent: RunnableComponent
let goto: string | undefined = undefined
let isLoading: boolean = false
@@ -38,40 +43,34 @@
$stateId != undefined &&
(componentInput?.type != 'runnable' || Object.keys(componentInput?.fields ?? {}).length == 0)
$: outputs = $worldStore?.outputsById[id] as {
result: Output<Array<any>>
loading: Output<boolean>
}
$: if (outputs?.loading != undefined) {
outputs.loading.set(false, true)
}
$: outputs?.loading.subscribe({
next: (value) => {
isLoading = value
}
})
$: css = concatCustomCss($app.css?.formcomponent, customCss)
let loading = false
</script>
<InputValue {id} input={configuration.goto} bind:value={goto} />
<InputValue {id} input={configuration.label} bind:value={labelValue} />
<InputValue {id} input={configuration.color} bind:value={color} />
<InputValue {id} input={configuration.size} bind:value={size} />
{#each Object.keys(components['formcomponent'].initialData.configuration) as key (key)}
<ResolveConfig
{id}
{key}
bind:resolvedConfig={resolvedConfig[key]}
configuration={configuration[key]}
/>
{/each}
<RunnableWrapper
{recomputeIds}
{render}
bind:runnableComponent
bind:loading
{componentInput}
{id}
{goto}
doOnSuccess={resolvedConfig.onSuccess}
{extraQueryParams}
autoRefresh={false}
forceSchemaDisplay={true}
runnableClass="!block"
runnableStyle={css?.container.style}
runnableStyle={css?.container?.style}
{outputs}
>
<AlignWrapper {horizontalAlignment}>
<div
@@ -99,21 +98,14 @@
style={css?.button?.style ?? ''}
on:pointerdown={(e) => {
e?.stopPropagation()
window.dispatchEvent(new Event('pointerup'))
}}
on:click={() => {
runnableComponent?.runComponent()
if (recomputeIds) {
recomputeIds.forEach((id) => {
$runnableComponents[id]?.()
})
}
}}
{size}
{color}
size={resolvedConfig.size}
color={resolvedConfig.color}
>
{labelValue}
{resolvedConfig.label}
</Button>
{/if}
</div>
@@ -1,44 +1,46 @@
<script lang="ts">
import { Button, type ButtonType } from '$lib/components/common'
import { Button } from '$lib/components/common'
import { faUser } from '@fortawesome/free-solid-svg-icons'
import { getContext } from 'svelte'
import { Icon } from 'svelte-awesome'
import type { AppInput } from '../../inputType'
import type { Output } from '../../rx'
import type { AppViewerContext, ComponentCustomCSS } from '../../types'
import type { AppViewerContext, ComponentCustomCSS, RichConfigurations } from '../../types'
import AlignWrapper from '../helpers/AlignWrapper.svelte'
import InputValue from '../helpers/InputValue.svelte'
import type RunnableComponent from '../helpers/RunnableComponent.svelte'
import RunnableWrapper from '../helpers/RunnableWrapper.svelte'
import Portal from 'svelte-portal'
import Modal from '$lib/components/common/modal/Modal.svelte'
import { concatCustomCss } from '../../utils'
import { initConfig, initOutput } from '../../editor/appUtils'
import { components } from '../../editor/component'
import ResolveConfig from '../helpers/ResolveConfig.svelte'
import AlwaysMountedModal, {
getModal
} from '$lib/components/common/modal/AlwaysMountedModal.svelte'
export let id: string
export let componentInput: AppInput | undefined
export let configuration: Record<string, AppInput>
export let configuration: RichConfigurations
export let recomputeIds: string[] | undefined = undefined
export let extraQueryParams: Record<string, any> = {}
export let horizontalAlignment: 'left' | 'center' | 'right' | undefined = undefined
export let verticalAlignment: 'top' | 'center' | 'bottom' | undefined = undefined
export let customCss: ComponentCustomCSS<'button' | 'popup'> | undefined = undefined
export let customCss: ComponentCustomCSS<'formbuttoncomponent'> | undefined = undefined
export let render: boolean
export const staticOutputs: string[] = ['loading', 'result']
const { app, worldStore } = getContext<AppViewerContext>('AppViewerContext')
const { app, runnableComponents, worldStore } = getContext<AppViewerContext>('AppViewerContext')
let outputs = initOutput($worldStore, id, {
result: undefined,
loading: false
})
let labelValue: string = 'Default label'
let color: ButtonType.Color
let size: ButtonType.Size
let resolvedConfig = initConfig(
components['formbuttoncomponent'].initialData.configuration,
configuration
)
let runnableComponent: RunnableComponent
let disabled: boolean | undefined = undefined
let isLoading: boolean = false
let ownClick: boolean = false
let errors: Record<string, string> = {}
let open: boolean = false
$: errorsMessage = Object.values(errors)
.filter((x) => x != '')
@@ -47,120 +49,96 @@
$: noInputs =
componentInput?.type != 'runnable' || Object.keys(componentInput?.fields ?? {}).length == 0
$: outputs = $worldStore?.outputsById[id] as {
result: Output<Array<any>>
loading: Output<boolean>
}
$: if (outputs?.loading != undefined) {
outputs.loading.set(false, true)
}
$: outputs?.loading.subscribe({
next: (value) => {
isLoading = value
if (ownClick && !value) {
ownClick = false
}
}
})
$: loading = isLoading && ownClick
$: css = concatCustomCss($app?.css?.formbuttoncomponent, customCss)
let runnableWrapper: RunnableWrapper
let loading = false
</script>
<InputValue {id} input={configuration.label} bind:value={labelValue} />
<InputValue {id} input={configuration.color} bind:value={color} />
<InputValue {id} input={configuration.size} bind:value={size} />
<InputValue
{id}
input={configuration.disabled}
bind:value={disabled}
bind:error={errors.disabled}
/>
{#each Object.keys(components['formbuttoncomponent'].initialData.configuration) as key (key)}
<ResolveConfig
{id}
{key}
bind:resolvedConfig={resolvedConfig[key]}
configuration={configuration[key]}
/>
{/each}
<Portal>
<Modal
{open}
title={labelValue}
class={css?.popup.class}
style={css?.popup.style}
on:canceled={() => {
open = false
}}
on:confirmed={() => {
open = false
}}
<AlwaysMountedModal
title={resolvedConfig.label ?? ''}
class={css?.popup?.class}
style={css?.popup?.style}
>
<RunnableWrapper
bind:this={runnableWrapper}
{recomputeIds}
{render}
bind:runnableComponent
{componentInput}
{id}
{extraQueryParams}
autoRefresh={false}
forceSchemaDisplay={true}
runnableClass="!block"
{outputs}
doOnSuccess={resolvedConfig.onSuccess}
>
<RunnableWrapper
{render}
bind:runnableComponent
{componentInput}
{id}
{extraQueryParams}
autoRefresh={false}
forceSchemaDisplay={true}
runnableClass="!block"
>
<div class="flex flex-col gap-2 px-4 w-full">
<div>
{#if noInputs}
<div class="text-gray-600 italic text-sm my-4">
Run forms are associated with a runnable that has user inputs.
<br />
Once a script or flow is chosen, set some <strong>Runnable Inputs</strong> to
<strong>
User Input
<Icon data={faUser} scale={1.3} class="rounded-sm bg-gray-200 p-1 ml-0.5" />
</strong>
</div>
{/if}
</div>
<div class="flex justify-end">
<Button
{loading}
btnClasses="my-1"
on:pointerdown={(e) => {
e?.stopPropagation()
window.dispatchEvent(new Event('pointerup'))
}}
on:click={async () => {
await runnableComponent?.runComponent()
if (recomputeIds) {
recomputeIds.forEach((id) => {
$runnableComponents[id]?.()
})
}
open = false
}}
size="xs"
color="dark"
>
Submit
</Button>
</div>
<div class="flex flex-col gap-2 px-4 w-full">
<div>
{#if noInputs}
<div class="text-gray-600 italic text-sm my-4">
Run forms are associated with a runnable that has user inputs.
<br />
Once a script or flow is chosen, set some <strong>Runnable Inputs</strong> to
<strong>
User Input
<Icon data={faUser} scale={1.3} class="rounded-sm bg-gray-200 p-1 ml-0.5" />
</strong>
</div>
{/if}
</div>
</RunnableWrapper>
</Modal>
</Portal>
<div class="flex justify-end">
<Button
{loading}
btnClasses="my-1"
on:pointerdown={(e) => {
e?.stopPropagation()
}}
on:click={async () => {
if (!runnableComponent) {
runnableWrapper.onSuccess()
} else {
await runnableComponent?.runComponent()
}
getModal().close()
}}
size="xs"
color="dark"
>
Submit
</Button>
</div>
</div>
</RunnableWrapper>
</AlwaysMountedModal>
<AlignWrapper {horizontalAlignment} {verticalAlignment}>
{#if errorsMessage}
<div class="text-red-500 text-xs">{errorsMessage}</div>
{/if}
<Button
{disabled}
{size}
{color}
disabled={resolvedConfig.disabled ?? false}
size={resolvedConfig.size ?? 'md'}
color={resolvedConfig.color}
btnClasses={css?.button?.class ?? ''}
style={css?.button?.style ?? ''}
on:click={(e) => {
open = true
getModal().open()
}}
>
{labelValue}
{resolvedConfig.label}
</Button>
</AlignWrapper>

Some files were not shown because too many files have changed in this diff Show More