chore(main): release 1.339.0 (#3843)

* chore(main): release 1.339.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>

.github/workflows/docker-image.yml

@@ -74,7 +74,7 @@ jobs:
           platforms: linux/amd64,linux/arm64
           push: true
           build-args: |
-            features=embedding,parquet,openidconnect
+            features=embedding,parquet,openidconnect,jemalloc
           tags: |
             ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:dev
             ${{ steps.meta-public.outputs.tags }}
@@ -199,7 +199,7 @@ jobs:
           platforms: linux/amd64
           push: true
           build-args: |
-            features=enterprise,enterprise_saml,stripe,embedding,parquet,prometheus,openidconnect,cloud
+            features=enterprise,enterprise_saml,stripe,embedding,parquet,prometheus,openidconnect,cloud,jemalloc
             PYTHON_IMAGE=python:3.12.2-slim-bookworm
           tags: |
             ${{ steps.meta-ee-public-py312.outputs.tags }}

CHANGELOG.md

@@ -1,5 +1,93 @@
 # Changelog
 
+## [1.339.0](https://github.com/windmill-labs/windmill/compare/v1.338.3...v1.339.0) (2024-05-29)
+
+
+### Features
+
+* display last key renewal attempt ([#3839](https://github.com/windmill-labs/windmill/issues/3839)) ([dc2d7e1](https://github.com/windmill-labs/windmill/commit/dc2d7e1c8f5f7c50be04cfea6986ce03a3e33c7b))
+
+## [1.338.3](https://github.com/windmill-labs/windmill/compare/v1.338.2...v1.338.3) (2024-05-29)
+
+
+### Bug Fixes
+
+* fix resource type search ([b01e335](https://github.com/windmill-labs/windmill/commit/b01e33523da72050bc08b47bd84e0d8fc88f6dda))
+
+## [1.338.2](https://github.com/windmill-labs/windmill/compare/v1.338.1...v1.338.2) (2024-05-28)
+
+
+### Bug Fixes
+
+* worker metrics ([#3837](https://github.com/windmill-labs/windmill/issues/3837)) ([5d2b244](https://github.com/windmill-labs/windmill/commit/5d2b244869fb9a6a7d006cfc95b22d9e83d3b9bb))
+
+## [1.338.1](https://github.com/windmill-labs/windmill/compare/v1.338.0...v1.338.1) (2024-05-28)
+
+
+### Bug Fixes
+
+* fix retry not working on single step flow due to opt ([6166c4d](https://github.com/windmill-labs/windmill/commit/6166c4d7aae9eddcee92f950959b9d2290bc917a))
+* remove quotes around interpolated arg values of tags ([8b66636](https://github.com/windmill-labs/windmill/commit/8b66636a8908c0043d2be63c6462ce28acc43fd9))
+
+## [1.338.0](https://github.com/windmill-labs/windmill/compare/v1.337.0...v1.338.0) (2024-05-27)
+
+
+### Features
+
+* allow user resources in apps with a toggle ([#3821](https://github.com/windmill-labs/windmill/issues/3821)) ([5877727](https://github.com/windmill-labs/windmill/commit/5877727c167bf9b62bedaa7eeafad3c913b1089d))
+* **frontend:** add open/close state of the modal in the outputs ([#3822](https://github.com/windmill-labs/windmill/issues/3822)) ([c246bb0](https://github.com/windmill-labs/windmill/commit/c246bb07052e65e9b336529ad638bcb9cc9686cd))
+* **frontend:** upgrade to gpt-4o + add AI support for php ([#3820](https://github.com/windmill-labs/windmill/issues/3820)) ([2561853](https://github.com/windmill-labs/windmill/commit/256185308e2c0f1b5395a76bc7de5d3857f867d4))
+* public apps can require login ([#3825](https://github.com/windmill-labs/windmill/issues/3825)) ([e2a1219](https://github.com/windmill-labs/windmill/commit/e2a1219373cace1bf4130004c27b9bf3595f7dd9))
+* worker vcpus/memory limits + mem usage ([#3828](https://github.com/windmill-labs/windmill/issues/3828)) ([621464b](https://github.com/windmill-labs/windmill/commit/621464b55025d601f747f611cbb3485d363a480a))
+
+
+### Bug Fixes
+
+* filtering not always working with concurrency keys ([#3823](https://github.com/windmill-labs/windmill/issues/3823)) ([47df9f3](https://github.com/windmill-labs/windmill/commit/47df9f33cce5a20231b6b38d4eb1ff370699fc38))
+* improve concurrency re-scheduling at scale ([f2d9c3c](https://github.com/windmill-labs/windmill/commit/f2d9c3c77729fe32366a1c0628d5c0af4d603dff))
+* improve concurrency re-scheduling at scale ([e187fa6](https://github.com/windmill-labs/windmill/commit/e187fa6263100cb26a24726123bcc3bbe641a6e8))
+* make all error strings more verbose ([7a36f30](https://github.com/windmill-labs/windmill/commit/7a36f30fe3138089df5c3a4ce87744c4a459044d))
+
+## [1.337.0](https://github.com/windmill-labs/windmill/compare/v1.336.1...v1.337.0) (2024-05-25)
+
+
+### Features
+
+* add GetResumeUrls to go client ([#3810](https://github.com/windmill-labs/windmill/issues/3810)) ([59f980f](https://github.com/windmill-labs/windmill/commit/59f980f75a9a507f22349f822579a5542f69b988))
+* automatic key renewal ([#3815](https://github.com/windmill-labs/windmill/issues/3815)) ([831cf89](https://github.com/windmill-labs/windmill/commit/831cf89b6367009d0d861975cb8f78b616b0868e))
+
+
+### Bug Fixes
+
+* rename email_from_username to username_to_email ([#3813](https://github.com/windmill-labs/windmill/issues/3813)) ([7584c2b](https://github.com/windmill-labs/windmill/commit/7584c2b0bbc216896daa6a366028cfc3ab5af00c))
+* use hub script language as tag instead of 'hub' ([#3816](https://github.com/windmill-labs/windmill/issues/3816)) ([47bb5d2](https://github.com/windmill-labs/windmill/commit/47bb5d2492856c5cecd77d386c802958f0aa7211))
+
+## [1.336.1](https://github.com/windmill-labs/windmill/compare/v1.336.0...v1.336.1) (2024-05-24)
+
+
+### Bug Fixes
+
+* revert bun to 1.1.8 ([ba9cdb1](https://github.com/windmill-labs/windmill/commit/ba9cdb1ae2aca13fd06876b48e0acdb65f2a1b00))
+
+## [1.336.0](https://github.com/windmill-labs/windmill/compare/v1.335.0...v1.336.0) (2024-05-23)
+
+
+### Features
+
+* **frontend:** improve graph edges + fix depedencies detection ([#3802](https://github.com/windmill-labs/windmill/issues/3802)) ([d11e350](https://github.com/windmill-labs/windmill/commit/d11e350de395811daad8a49c0eef7b3b19460ead))
+* improve parsers when no main func ([#3805](https://github.com/windmill-labs/windmill/issues/3805)) ([001278c](https://github.com/windmill-labs/windmill/commit/001278cddd6e91f767f090e358db4299611f0451))
+* improve permissioned audit logs ([#3799](https://github.com/windmill-labs/windmill/issues/3799)) ([62f3180](https://github.com/windmill-labs/windmill/commit/62f318077cd77d8332bbe02ac279a5bca7351e51))
+* improve premissioned audit logs ([#3793](https://github.com/windmill-labs/windmill/issues/3793)) ([21a077e](https://github.com/windmill-labs/windmill/commit/21a077ecfaa2dfabe6a46e425214182845b5259e))
+* store failed_retries per module state and display failed retries in flow status viewer ([62e8816](https://github.com/windmill-labs/windmill/commit/62e88169ea47e6357931bafd564ab45df40a62ff))
+* store failed_retries per module state and display failed retries in flow status viewer ([7e0be89](https://github.com/windmill-labs/windmill/commit/7e0be8914bcb33caf680358f083e80edfb83d00f))
+
+
+### Bug Fixes
+
+* **frontend:** Add missing loading state for non-runnable components … ([#3797](https://github.com/windmill-labs/windmill/issues/3797)) ([a0acdc1](https://github.com/windmill-labs/windmill/commit/a0acdc1217967562bb349a2b718c82af74b8bc45))
+* show code option in tree view ([#3803](https://github.com/windmill-labs/windmill/issues/3803)) ([1745a9d](https://github.com/windmill-labs/windmill/commit/1745a9dd1617de4686480e0cfc9ba2ef2da0c625))
+* update bun to 1.1.9 ([bfd2d27](https://github.com/windmill-labs/windmill/commit/bfd2d27415a906f9133777b69653ddacf659de41))
+
 ## [1.335.0](https://github.com/windmill-labs/windmill/compare/v1.334.0...v1.335.0) (2024-05-22)
 
 

Dockerfile

@@ -171,7 +171,7 @@ COPY --from=builder /windmill/target/release/windmill ${APP}/windmill
 
 COPY --from=downloader --chmod=755 /deno /usr/bin/deno
 
-COPY --from=oven/bun:1.1.7 /usr/local/bin/bun /usr/bin/bun
+COPY --from=oven/bun:1.1.8 /usr/local/bin/bun /usr/bin/bun
 
 COPY --from=php:8.3.7-cli /usr/local/bin/php /usr/bin/php
 COPY --from=composer:2.7.6 /usr/bin/composer /usr/bin/composer

backend/.sqlx/query-04be51a152d7c9644f11173da2cc386a71e178685364e7da4b910d1648ea55ba.json

@@ -1,48 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT running, substr(concat(coalesce(queue.logs, ''), job_logs.logs), greatest($1 - job_logs.log_offset, 0)) as logs, mem_peak, \n        CASE WHEN is_flow_step is true then NULL else flow_status END as flow_status,\n        job_logs.log_offset + char_length(job_logs.logs) + 1 as log_offset\n        FROM queue\n        LEFT JOIN job_logs ON job_logs.job_id =  queue.id \n        WHERE queue.workspace_id = $2 AND queue.id = $3",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "running",
-        "type_info": "Bool"
-      },
-      {
-        "ordinal": 1,
-        "name": "logs",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 2,
-        "name": "mem_peak",
-        "type_info": "Int4"
-      },
-      {
-        "ordinal": 3,
-        "name": "flow_status",
-        "type_info": "Jsonb"
-      },
-      {
-        "ordinal": 4,
-        "name": "log_offset",
-        "type_info": "Int4"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Int4",
-        "Text",
-        "Uuid"
-      ]
-    },
-    "nullable": [
-      false,
-      null,
-      true,
-      null,
-      null
-    ]
-  },
-  "hash": "04be51a152d7c9644f11173da2cc386a71e178685364e7da4b910d1648ea55ba"
-}

backend/.sqlx/query-0a7212dd507ed8f7a311724185e39ecc1809abb208a681ad711614c27baadd83.json

@@ -1,23 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT flow_status FROM queue WHERE id = $1 AND workspace_id = $2",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "flow_status",
-        "type_info": "Jsonb"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Uuid",
-        "Text"
-      ]
-    },
-    "nullable": [
-      true
-    ]
-  },
-  "hash": "0a7212dd507ed8f7a311724185e39ecc1809abb208a681ad711614c27baadd83"
-}

backend/.sqlx/query-126be8832776644e0d2c5d004acb26f6d4820a6a1fc8e028c3550c438248a82b.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE queue\n                SET running = false\n                , started_at = null\n                , scheduled_for = $1\n                , last_ping = null\n                WHERE id = $2",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Timestamptz",
+        "Uuid"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "126be8832776644e0d2c5d004acb26f6d4820a6a1fc8e028c3550c438248a82b"
+}

backend/.sqlx/query-293054a4d6a2b00228a5029a81c026e8891576c7e87cdab41d7fcc3afc32d583.json

@@ -0,0 +1,24 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT COUNT(*) FROM queue LEFT JOIN concurrency_key ON concurrency_key.job_id = queue.id\n                 WHERE key = $1 AND running = false AND canceled = false AND scheduled_for >= $2 AND scheduled_for < $3",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "count",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Timestamptz",
+        "Timestamptz"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "293054a4d6a2b00228a5029a81c026e8891576c7e87cdab41d7fcc3afc32d583"
+}

backend/.sqlx/query-360d13c0a776e794063c99acad935a424f79017f3769e1150faa87cbb91367a8.json

@@ -0,0 +1,28 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT value, created_at FROM metrics WHERE id = $1 ORDER BY created_at DESC LIMIT 1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "value",
+        "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 1,
+        "name": "created_at",
+        "type_info": "Timestamptz"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      false
+    ]
+  },
+  "hash": "360d13c0a776e794063c99acad935a424f79017f3769e1150faa87cbb91367a8"
+}

backend/.sqlx/query-3da14e6c89a33871e4bd40524e2fd9222aa707ee76de6ce348337c3e403eb955.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "DELETE FROM usr WHERE email = $1 AND workspace_id = $2",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "3da14e6c89a33871e4bd40524e2fd9222aa707ee76de6ce348337c3e403eb955"
+}

backend/.sqlx/query-4398bc6f701dcba647a259ec2d091a70938a3ecd7ff8a89277cc35d119e2bad4.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2, occupancy_rate = $3, current_job_id = NULL, current_job_workspace_id = NULL WHERE worker = $4",
+  "query": "UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2, occupancy_rate = $3, memory_usage = $4, wm_memory_usage = $5, current_job_id = NULL, current_job_workspace_id = NULL WHERE worker = $6",
   "describe": {
     "columns": [],
     "parameters": {
@@ -8,10 +8,12 @@
         "Int4",
         "TextArray",
         "Float4",
+        "Int8",
+        "Int8",
         "Text"
       ]
     },
     "nullable": []
   },
-  "hash": "54fef88cc6b9e8db7c07fccbaa845edfefa339153a32e468faad9f063008863d"
+  "hash": "4398bc6f701dcba647a259ec2d091a70938a3ecd7ff8a89277cc35d119e2bad4"
 }

backend/.sqlx/query-55d6c6f887f1cceb311d9ab034d3485b8ccb2d07d723f184cef2696760d3ccf9.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT worker, worker_instance,  EXTRACT(EPOCH FROM (now() - ping_at))::integer as last_ping, started_at, ip, jobs_executed, CASE WHEN $4 IS TRUE THEN current_job_id ELSE NULL END as current_job_id, CASE WHEN $4 IS TRUE THEN current_job_workspace_id ELSE NULL END as current_job_workspace_id, custom_tags, worker_group, wm_version, occupancy_rate\n        FROM worker_ping\n        WHERE ($1::integer IS NULL AND ping_at > now() - interval '5 minute') OR (ping_at > now() - ($1 || ' seconds')::interval)\n        ORDER BY ping_at desc LIMIT $2 OFFSET $3",
+  "query": "SELECT worker, worker_instance,  EXTRACT(EPOCH FROM (now() - ping_at))::integer as last_ping, started_at, ip, jobs_executed, CASE WHEN $4 IS TRUE THEN current_job_id ELSE NULL END as current_job_id, CASE WHEN $4 IS TRUE THEN current_job_workspace_id ELSE NULL END as current_job_workspace_id, custom_tags, worker_group, wm_version, occupancy_rate, memory, vcpus, memory_usage, wm_memory_usage\n        FROM worker_ping\n        WHERE ($1::integer IS NULL AND ping_at > now() - interval '5 minute') OR (ping_at > now() - ($1 || ' seconds')::interval)\n        ORDER BY ping_at desc LIMIT $2 OFFSET $3",
   "describe": {
     "columns": [
       {
@@ -62,6 +62,26 @@
         "ordinal": 11,
         "name": "occupancy_rate",
         "type_info": "Float4"
+      },
+      {
+        "ordinal": 12,
+        "name": "memory",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 13,
+        "name": "vcpus",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 14,
+        "name": "memory_usage",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 15,
+        "name": "wm_memory_usage",
+        "type_info": "Int8"
       }
     ],
     "parameters": {
@@ -84,8 +104,12 @@
       true,
       false,
       false,
+      true,
+      true,
+      true,
+      true,
       true
     ]
   },
-  "hash": "e00171cc3fc8f32922562d4fc4d3db56955fbc6ee9872d4b2e99157ebed131e0"
+  "hash": "55d6c6f887f1cceb311d9ab034d3485b8ccb2d07d723f184cef2696760d3ccf9"
 }

backend/.sqlx/query-62cd69e3dc0f8757db3481d078921f4722bbc93d52e05656b82cb42ba9803ef6.json

@@ -1,23 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM usr WHERE email = $1 AND workspace_id = $2 RETURNING username",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "username",
-        "type_info": "Varchar"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": [
-      false
-    ]
-  },
-  "hash": "62cd69e3dc0f8757db3481d078921f4722bbc93d52e05656b82cb42ba9803ef6"
-}

backend/.sqlx/query-95cb1fe8658f98fb736d899fa21cd7378b0c9d3b5f3d6bd6cafcba273f8277d4.json

@@ -1,16 +1,18 @@
 {
   "db_name": "PostgreSQL",
-  "query": "UPDATE worker_ping SET ping_at = now(), current_job_id = $1, current_job_workspace_id = $2 WHERE worker = $3",
+  "query": "UPDATE worker_ping SET ping_at = now(), current_job_id = $1, current_job_workspace_id = $2, memory_usage = $3, wm_memory_usage = $4 WHERE worker = $5",
   "describe": {
     "columns": [],
     "parameters": {
       "Left": [
         "Uuid",
         "Varchar",
+        "Int8",
+        "Int8",
         "Text"
       ]
     },
     "nullable": []
   },
-  "hash": "8c62e3bb264c7336b3b3a68677993ba3b729c7358cc3da14513d8b447ce65aaf"
+  "hash": "95cb1fe8658f98fb736d899fa21cd7378b0c9d3b5f3d6bd6cafcba273f8277d4"
 }

backend/.sqlx/query-98c11a7d870a997c74e533a75bb48f39b0ec43816f05ac1d1eb39ea1903138f9.json

@@ -0,0 +1,23 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT EXISTS(SELECT 1 FROM app WHERE id = $1 AND workspace_id = $2)",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "exists",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Text"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "98c11a7d870a997c74e533a75bb48f39b0ec43816f05ac1d1eb39ea1903138f9"
+}

backend/.sqlx/query-9ed77e78e6295c62745ac3ac3b7e5f544f654d7cddc13acb5d3f4fdc12a8875f.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT created_at FROM metrics WHERE id = 'license_key_renewal' ORDER BY created_at DESC LIMIT 1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "created_at",
+        "type_info": "Timestamptz"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "9ed77e78e6295c62745ac3ac3b7e5f544f654d7cddc13acb5d3f4fdc12a8875f"
+}

backend/.sqlx/query-a0ce703def7e976947513029874fb571893c75730e0a8feeb609852423404cf0.json

@@ -1,22 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT CAST(ROUND(AVG(duration_ms) / 1000, 0) AS BIGINT) AS avg_duration_s FROM\n                (SELECT duration_ms FROM completed_job WHERE script_path = $1\n                ORDER BY started_at\n                DESC LIMIT 10) AS t",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "avg_duration_s",
-        "type_info": "Int8"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text"
-      ]
-    },
-    "nullable": [
-      null
-    ]
-  },
-  "hash": "a0ce703def7e976947513029874fb571893c75730e0a8feeb609852423404cf0"
-}

backend/.sqlx/query-a73b57229602d68cc25a8d963753271619e74df9fd7fc6cb05fc0614b10e6001.json

@@ -1,42 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT substr(concat(coalesce(completed_job.logs, ''), job_logs.logs), greatest($1 - job_logs.log_offset, 0))  as logs, mem_peak, \n            CASE WHEN is_flow_step is true then NULL else flow_status END as flow_status,\n            job_logs.log_offset + char_length(job_logs.logs) + 1 as log_offset\n            FROM completed_job \n            LEFT JOIN job_logs ON job_logs.job_id = completed_job.id \n            WHERE completed_job.workspace_id = $2 AND id = $3",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "logs",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 1,
-        "name": "mem_peak",
-        "type_info": "Int4"
-      },
-      {
-        "ordinal": 2,
-        "name": "flow_status",
-        "type_info": "Jsonb"
-      },
-      {
-        "ordinal": 3,
-        "name": "log_offset",
-        "type_info": "Int4"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Int4",
-        "Text",
-        "Uuid"
-      ]
-    },
-    "nullable": [
-      null,
-      true,
-      null,
-      null
-    ]
-  },
-  "hash": "a73b57229602d68cc25a8d963753271619e74df9fd7fc6cb05fc0614b10e6001"
-}

backend/.sqlx/query-c845c06aa46b52a1e3672fd379dbcec167f4044e45e932aeeefd3d9237e5042c.json

@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT CAST(ROUND(AVG(duration_ms), 0) AS BIGINT) AS avg_duration_s FROM\n                (SELECT duration_ms FROM concurrency_key LEFT JOIN completed_job ON completed_job.id = concurrency_key.job_id WHERE key = $1 AND ended_at IS NOT NULL\n                ORDER BY ended_at\n                DESC LIMIT 10) AS t",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "avg_duration_s",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "c845c06aa46b52a1e3672fd379dbcec167f4044e45e932aeeefd3d9237e5042c"
+}

backend/.sqlx/query-d113bc301f2f094c3a2a4c6ed385da65edde2fbfee0fac5103ba7bd18c64563d.json

@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO metrics (id, value) VALUES ('license_key_renewal', $1)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Jsonb"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "d113bc301f2f094c3a2a4c6ed385da65edde2fbfee0fac5103ba7bd18c64563d"
+}

backend/.sqlx/query-da5da57ea48ddc6ab271b6c18baa5f7360008e082e2fb8d58faff4461e18c83a.json

@@ -1,23 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT username FROM usr WHERE workspace_id = $1 AND email = $2",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "username",
-        "type_info": "Varchar"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": [
-      false
-    ]
-  },
-  "hash": "da5da57ea48ddc6ab271b6c18baa5f7360008e082e2fb8d58faff4461e18c83a"
-}

backend/.sqlx/query-f5a0346034f3bef54260962f7783b3fde7b1aa1f37a72058e292eebbcae08d74.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO concurrency_key (job_id, key) SELECT id, $1 FROM unnest($2::uuid[]) as id",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "UuidArray"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "f5a0346034f3bef54260962f7783b3fde7b1aa1f37a72058e292eebbcae08d74"
+}

backend/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "windmill"
-version = "1.335.0"
+version = "1.339.0"
 authors.workspace = true
 edition.workspace = true
 
@@ -24,7 +24,7 @@ members = [
 ]
 
 [workspace.package]
-version = "1.335.0"
+version = "1.339.0"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]
 edition = "2021"
 
@@ -51,7 +51,7 @@ prometheus = ["windmill-common/prometheus", "windmill-api/prometheus", "windmill
 flow_testing = ["windmill-worker/flow_testing"]
 openidconnect = ["windmill-api/openidconnect"]
 cloud = ["windmill-queue/cloud"]
-jemalloc = ["dep:tikv-jemallocator", "dep:tikv-jemalloc-sys", "dep:tikv-jemalloc-ctl"]
+jemalloc = ["windmill-common/jemalloc", "dep:tikv-jemallocator", "dep:tikv-jemalloc-sys", "dep:tikv-jemalloc-ctl"]
 
 [dependencies]
 anyhow.workspace = true
@@ -84,9 +84,9 @@ deno_core.workspace = true
 pg-embed = {git = "https://github.com/faokunega/pg-embed", optional = true, default-features = false, features = ['rt_tokio']} 
 
 [target.'cfg(not(target_env = "msvc"))'.dependencies]
-tikv-jemallocator = { optional = true, version = "0.5", features = ["profiling"] }
-tikv-jemalloc-sys  = { optional = true, version = "^0.5", features = ["profiling"] }
-tikv-jemalloc-ctl = { optional = true, version = "^0.5" }
+tikv-jemallocator = { optional = true, workspace = true }
+tikv-jemalloc-sys = { optional = true, workspace = true }
+tikv-jemalloc-ctl = { optional = true, workspace = true }
 
 [dev-dependencies]
 serde_json.workspace = true
@@ -248,3 +248,12 @@ crc = "^3"
 tar = "^0"
 http = "^1"
 async-stream = "^0"
+
+sysinfo = "0.30.12"
+
+tikv-jemallocator = { version = "0.5" }
+tikv-jemalloc-sys  = { version = "^0.5" }
+tikv-jemalloc-ctl = { version = "^0.5" }
+
+# 0.1.12 broken (nested dependency of swc_common) 
+triomphe = "<0.1.12"

backend/ee-repo-ref.txt

@@ -1 +1 @@
-229b7acc60691f7e6577c44f782676d6871defdd
+982774ea62ab95914bd0502aac8c5c9b5a91b5c2

backend/migrations/20240523095814_increase_token_label_size.down.sql

@@ -0,0 +1 @@
+-- Add down migration script here

backend/migrations/20240523095814_increase_token_label_size.up.sql

@@ -0,0 +1,2 @@
+-- Add up migration script here
+ALTER TABLE token ALTER COLUMN label TYPE VARCHAR(1000)

backend/migrations/20240527151853_add_worker_memory_usage.down.sql

@@ -0,0 +1 @@
+-- Add down migration script here

backend/migrations/20240527151853_add_worker_memory_usage.up.sql

@@ -0,0 +1,2 @@
+-- Add up migration script here
+ALTER TABLE worker_ping ADD COLUMN memory_usage BIGINT;

backend/migrations/20240528135719_add_windmill_memory_usage.down.sql

@@ -0,0 +1 @@
+-- Add down migration script here

backend/migrations/20240528135719_add_windmill_memory_usage.up.sql

@@ -0,0 +1,2 @@
+-- Add up migration script here
+ALTER TABLE worker_ping ADD COLUMN wm_memory_usage BIGINT;

backend/parsers/windmill-parser-bash/src/lib.rs

@@ -11,7 +11,7 @@ pub fn parse_bash_sig(code: &str) -> anyhow::Result<MainArgSignature> {
     let parsed = parse_bash_file(&code)?;
     if let Some(x) = parsed {
         let args = x;
-        Ok(MainArgSignature { star_args: false, star_kwargs: false, args })
+        Ok(MainArgSignature { star_args: false, star_kwargs: false, args, no_main_func: None })
     } else {
         Err(anyhow!("Error parsing bash script".to_string()))
     }
@@ -21,7 +21,7 @@ pub fn parse_powershell_sig(code: &str) -> anyhow::Result<MainArgSignature> {
     let parsed = parse_powershell_file(&code)?;
     if let Some(x) = parsed {
         let args = x;
-        Ok(MainArgSignature { star_args: false, star_kwargs: false, args })
+        Ok(MainArgSignature { star_args: false, star_kwargs: false, args, no_main_func: None })
     } else {
         Err(anyhow!("Error parsing powershell script".to_string()))
     }
@@ -160,7 +160,8 @@ non_required="${5:-}"
                         default: Some(json!("")),
                         has_default: true
                     }
-                ]
+                ],
+                no_main_func: None
             }
         );
 

backend/parsers/windmill-parser-go/src/lib.rs

@@ -30,9 +30,19 @@ pub fn parse_go_sig(code: &str) -> anyhow::Result<MainArgSignature> {
                 Arg { name: get_name(param), otyp, typ, default: None, has_default: false }
             })
             .collect_vec();
-        Ok(MainArgSignature { star_args: false, star_kwargs: false, args })
+        Ok(MainArgSignature {
+            star_args: false,
+            star_kwargs: false,
+            args,
+            no_main_func: Some(false),
+        })
     } else {
-        Err(anyhow::anyhow!("no main function found".to_string(),))
+        Ok(MainArgSignature {
+            star_args: false,
+            star_kwargs: false,
+            args: vec![],
+            no_main_func: Some(true),
+        })
     }
 }
 
@@ -216,7 +226,8 @@ func main(x int, y string, z bool, l []string, o struct { Name string `json:"nam
                         default: None,
                         has_default: false
                     },
-                ]
+                ],
+                no_main_func: Some(false)
             }
         );
 

backend/parsers/windmill-parser-graphql/src/lib.rs

@@ -10,7 +10,7 @@ pub fn parse_graphql_sig(code: &str) -> anyhow::Result<MainArgSignature> {
     let parsed = parse_graphql_file(&code)?;
     if let Some(x) = parsed {
         let args = x;
-        Ok(MainArgSignature { star_args: false, star_kwargs: false, args })
+        Ok(MainArgSignature { star_args: false, star_kwargs: false, args, no_main_func: None })
     } else {
         Err(anyhow!("Error parsing sql".to_string()))
     }
@@ -101,7 +101,8 @@ query($s: String, $arr: [String]) {
                         default: None,
                         has_default: false
                     },
-                ]
+                ],
+                no_main_func: None
             }
         );
 

backend/parsers/windmill-parser-php/src/lib.rs

@@ -89,11 +89,19 @@ pub fn parse_php_signature(
             })
             .collect();
 
-        Ok(MainArgSignature { star_args: false, star_kwargs: false, args })
+        Ok(MainArgSignature {
+            star_args: false,
+            star_kwargs: false,
+            args,
+            no_main_func: Some(false),
+        })
     } else {
-        Err(anyhow::anyhow!(
-            "main function was not findable".to_string(),
-        ))
+        Ok(MainArgSignature {
+            star_args: false,
+            star_kwargs: false,
+            args: vec![],
+            no_main_func: Some(true),
+        })
     }
 }
 
@@ -157,7 +165,8 @@ function main(string $input1 = \"hey\", bool $input2 = false, int $input3 = 3, f
                         has_default: false,
                         default: None
                     }
-                ]
+                ],
+                no_main_func: Some(false)
             }
         );
 

backend/parsers/windmill-parser-py-imports/src/lib.rs

@@ -53,6 +53,9 @@ static PYTHON_IMPORTS_REPLACEMENT: phf::Map<&'static str, &'static str> = phf_ma
     "pymsql" => "PyMySQL",
     "haystack" => "haystack-ai",
     "github" => "PyGithub",
+    "ldap" => "python-ldap",
+    "opensearchpy" => "opensearch-py",
+    
 };
 
 fn replace_import(x: String) -> String {

backend/parsers/windmill-parser-py/src/lib.rs

@@ -117,11 +117,15 @@ pub fn parse_python_signature(
                     }
                 })
                 .collect(),
+            no_main_func: Some(false),
         })
     } else {
-        Err(anyhow::anyhow!(
-            "main function was not findable".to_string(),
-        ))
+        Ok(MainArgSignature {
+            star_args: false,
+            star_kwargs: false,
+            args: vec![],
+            no_main_func: Some(true),
+        })
     }
 }
 
@@ -308,7 +312,8 @@ def main(test1: str, name: datetime.datetime = datetime.now(), byte: bytes = byt
                         default: Some(json!(true)),
                         has_default: true
                     },
-                ]
+                ],
+                no_main_func: Some(false),
             }
         );
 
@@ -367,7 +372,8 @@ def main(test1: str,
                         default: Some(json!("$res:g/all/resource")),
                         has_default: true
                     }
-                ]
+                ],
+                no_main_func: Some(false),
             }
         );
 
@@ -421,7 +427,8 @@ def main(test1: str,
                         default: Some(json!("<function call>")),
                         has_default: true
                     }
-                ]
+                ],
+                no_main_func: Some(false),
             }
         );
 
@@ -461,7 +468,8 @@ def main(test1: Literal["foo", "bar"], test2: List[Literal["foo", "bar"]]): retu
                         default: None,
                         has_default: false
                     }
-                ]
+                ],
+                no_main_func: Some(false),
             }
         );
 

backend/parsers/windmill-parser-sql/src/lib.rs

@@ -11,7 +11,7 @@ pub fn parse_mysql_sig(code: &str) -> anyhow::Result<MainArgSignature> {
     let parsed = parse_mysql_file(&code)?;
     if let Some(x) = parsed {
         let args = x;
-        Ok(MainArgSignature { star_args: false, star_kwargs: false, args })
+        Ok(MainArgSignature { star_args: false, star_kwargs: false, args, no_main_func: None })
     } else {
         Err(anyhow!("Error parsing sql".to_string()))
     }
@@ -21,7 +21,7 @@ pub fn parse_pgsql_sig(code: &str) -> anyhow::Result<MainArgSignature> {
     let parsed = parse_pg_file(&code)?;
     if let Some(x) = parsed {
         let args = x;
-        Ok(MainArgSignature { star_args: false, star_kwargs: false, args })
+        Ok(MainArgSignature { star_args: false, star_kwargs: false, args, no_main_func: None })
     } else {
         Err(anyhow!("Error parsing sql".to_string()))
     }
@@ -31,7 +31,7 @@ pub fn parse_bigquery_sig(code: &str) -> anyhow::Result<MainArgSignature> {
     let parsed = parse_bigquery_file(&code)?;
     if let Some(x) = parsed {
         let args = x;
-        Ok(MainArgSignature { star_args: false, star_kwargs: false, args })
+        Ok(MainArgSignature { star_args: false, star_kwargs: false, args, no_main_func: None })
     } else {
         Err(anyhow!("Error parsing sql".to_string()))
     }
@@ -41,7 +41,7 @@ pub fn parse_snowflake_sig(code: &str) -> anyhow::Result<MainArgSignature> {
     let parsed = parse_snowflake_file(&code)?;
     if let Some(x) = parsed {
         let args = x;
-        Ok(MainArgSignature { star_args: false, star_kwargs: false, args })
+        Ok(MainArgSignature { star_args: false, star_kwargs: false, args, no_main_func: None })
     } else {
         Err(anyhow!("Error parsing sql".to_string()))
     }
@@ -51,7 +51,7 @@ pub fn parse_mssql_sig(code: &str) -> anyhow::Result<MainArgSignature> {
     let parsed = parse_mssql_file(&code)?;
     if let Some(x) = parsed {
         let args = x;
-        Ok(MainArgSignature { star_args: false, star_kwargs: false, args })
+        Ok(MainArgSignature { star_args: false, star_kwargs: false, args, no_main_func: None })
     } else {
         Err(anyhow!("Error parsing sql".to_string()))
     }
@@ -396,7 +396,8 @@ SELECT * FROM table WHERE token=$1::TEXT AND image=$2::BIGINT
                         default: None,
                         has_default: false
                     },
-                ]
+                ],
+                no_main_func: None
             }
         );
 
@@ -431,7 +432,8 @@ SELECT * FROM table WHERE token=@token AND image=@image
                         default: None,
                         has_default: false
                     },
-                ]
+                ],
+                no_main_func: None
             }
         );
 

backend/parsers/windmill-parser-ts/Cargo.toml

@@ -16,6 +16,7 @@ serde-wasm-bindgen.workspace = true
 [dependencies]
 windmill-parser.workspace = true
 swc_common.workspace = true
+triomphe.workspace = true
 swc_ecma_parser.workspace = true
 swc_ecma_ast.workspace = true
 swc_ecma_visit.workspace = true

backend/parsers/windmill-parser-ts/src/lib.rs

@@ -176,13 +176,16 @@ pub fn parse_deno_signature(
                 .into_iter()
                 .map(|x| parse_param(x, &cm, skip_dflt, &mut c))
                 .collect::<anyhow::Result<Vec<Arg>>>()?,
+            no_main_func: Some(false),
         };
         Ok(r)
     } else {
-        Err(anyhow::anyhow!(
-            "main function was not findable (expected to find 'export function main(...)'"
-                .to_string(),
-        ))
+        Ok(MainArgSignature {
+            star_args: false,
+            star_kwargs: false,
+            args: vec![],
+            no_main_func: Some(true),
+        })
     }
 }
 

backend/parsers/windmill-parser-wasm/pkg/package.json

@@ -3,7 +3,7 @@
   "collaborators": [
     "Ruben Fiszel <ruben@windmill.dev>"
   ],
-  "version": "1.327.0",
+  "version": "1.335.0",
   "files": [
     "windmill_parser_wasm_bg.wasm",
     "windmill_parser_wasm.js",

backend/parsers/windmill-parser-wasm/pkg/windmill_parser_wasm.js

@@ -587,10 +587,6 @@ function __wbg_get_imports() {
     imports.wbg.__wbindgen_object_drop_ref = function(arg0) {
         takeObject(arg0);
     };
-    imports.wbg.__wbg_eval_f2b8ae7add53626d = function(arg0, arg1) {
-        const ret = eval(getStringFromWasm0(arg0, arg1));
-        return addHeapObject(ret);
-    };
     imports.wbg.__wbindgen_string_get = function(arg0, arg1) {
         const obj = getObject(arg1);
         const ret = typeof(obj) === 'string' ? obj : undefined;
@@ -639,6 +635,10 @@ function __wbg_get_imports() {
         const ret = getObject(arg0) in getObject(arg1);
         return ret;
     };
+    imports.wbg.__wbg_eval_4ecb84501b3abc94 = function(arg0, arg1) {
+        const ret = eval(getStringFromWasm0(arg0, arg1));
+        return addHeapObject(ret);
+    };
     imports.wbg.__wbindgen_jsval_loose_eq = function(arg0, arg1) {
         const ret = getObject(arg0) == getObject(arg1);
         return ret;

backend/parsers/windmill-parser-wasm/tests/wasm.rs

@@ -116,7 +116,8 @@ export function main(test1?: string, test2: string = \"burkina\",
                     default: None,
                     has_default: true
                 }
-            ]
+            ],
+            no_main_func: Some(false)
         }
     );
 
@@ -185,7 +186,8 @@ export function main(test2 = \"burkina\",
                     default: Some(json!({"a": "test", "b": 42})),
                     has_default: true
                 }
-            ]
+            ],
+            no_main_func: Some(false)
         }
     );
 
@@ -230,7 +232,8 @@ export function main(foo: FooBar, {a, b}: FooBar, {c, d}: FooBar = {a: \"foo\",
                     default: Some(json!({"a": "foo", "b": 42})),
                     has_default: true
                 }
-            ]
+            ],
+            no_main_func: Some(false)
         }
     );
 
@@ -258,7 +261,8 @@ export function main(foo: (\"foo\" | \"bar\")[]) {
                 ])))),
                 default: None,
                 has_default: false
-            }]
+            }],
+            no_main_func: Some(false)
         }
     );
 
@@ -388,7 +392,8 @@ Write-Output 'Testing...'
                     default: None,
                     has_default: false
                 }
-            ]
+            ],
+            no_main_func: None,
         }
     );
 

backend/parsers/windmill-parser/src/lib.rs

@@ -14,6 +14,7 @@ pub struct MainArgSignature {
     pub star_args: bool,
     pub star_kwargs: bool,
     pub args: Vec<Arg>,
+    pub no_main_func: Option<bool>,
 }
 
 #[derive(Serialize, Clone, Debug, PartialEq)]

backend/src/main.rs

@@ -16,6 +16,10 @@ use std::{
 };
 use tokio::fs::DirBuilder;
 use windmill_api::HTTP_CLIENT;
+
+#[cfg(feature = "enterprise")]
+use windmill_common::ee::schedule_key_renewal;
+
 use windmill_common::{
     global_settings::{
         BASE_URL_SETTING, BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ERROR_CHANNELS_SETTING,
@@ -325,7 +329,9 @@ Windmill Community Edition {GIT_VERSION}
         monitor_pool(&db).await;
 
         #[cfg(all(not(target_env = "msvc"), feature = "jemalloc"))]
-        monitor_mem().await;
+        if !worker_mode {
+            monitor_mem().await;
+        }
 
         let addr = SocketAddr::from((server_bind_address, port));
 
@@ -353,7 +359,7 @@ Windmill Community Edition {GIT_VERSION}
                 base_internal_tx
                     .send(base_internal_url.clone())
                     .map_err(|e| {
-                        anyhow::anyhow!("Could not send base_internal_url to agent: {e}")
+                        anyhow::anyhow!("Could not send base_internal_url to agent: {e:#}")
                     })?;
             }
             Ok(()) as anyhow::Result<()>
@@ -453,7 +459,7 @@ Windmill Community Edition {GIT_VERSION}
                                                 },
                                                 DEFAULT_TAGS_PER_WORKSPACE_SETTING => {
                                                     if let Err(e) = load_tag_per_workspace_enabled(&db).await {
-                                                        tracing::error!("Error loading default tag per workpsace: {e}");
+                                                        tracing::error!("Error loading default tag per workpsace: {e:#}");
                                                     }
                                                 },
                                                 RETENTION_PERIOD_SECS_SETTING => {
@@ -557,7 +563,7 @@ Windmill Community Edition {GIT_VERSION}
             });
 
             if let Err(e) = h.await {
-                tracing::error!("Error waiting for monitor handle:{e}")
+                tracing::error!("Error waiting for monitor handle:{e:#}")
             }
             Ok(()) as anyhow::Result<()>
         };
@@ -579,6 +585,11 @@ Windmill Community Edition {GIT_VERSION}
             schedule_stats(instance_name, &db, &HTTP_CLIENT).await;
         }
 
+        #[cfg(feature = "enterprise")]
+        if mode == Mode::Server || mode == Mode::Standalone {
+            schedule_key_renewal(&HTTP_CLIENT, &db).await;
+        }
+
         futures::try_join!(shutdown_signal, workers_f, monitor_f, server_f, metrics_f)?;
     } else {
         tracing::info!("Nothing to do, exiting.");

backend/src/monitor.rs

@@ -30,7 +30,7 @@ use windmill_common::{
         PIP_INDEX_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING,
         REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RETENTION_PERIOD_SECS_SETTING,
         SAML_METADATA_SETTING, SCIM_TOKEN_SETTING,
-    }, jobs::QueuedJob, oauth2::REQUIRE_PREEXISTING_USER_FOR_OAUTH, server::load_server_config, stats_ee::get_user_usage, users::truncate_token, worker::{
+    }, jobs::QueuedJob, oauth2::REQUIRE_PREEXISTING_USER_FOR_OAUTH, server::load_server_config, stats_ee::get_user_usage, users::truncate_token, utils::now_from_db, worker::{
         load_worker_config, reload_custom_tags_setting, DEFAULT_TAGS_PER_WORKSPACE, SERVER_CONFIG,
         WORKER_CONFIG,
     }, BASE_URL, CRITICAL_ERROR_CHANNELS, DB, DEFAULT_HUB_BASE_URL, HUB_BASE_URL, METRICS_DEBUG_ENABLED, METRICS_ENABLED
@@ -104,15 +104,15 @@ pub async fn initial_load(
     _is_agent: bool,
 ) {
     if let Err(e) = load_metrics_enabled(db).await {
-        tracing::error!("Error loading expose metrics: {e}");
+        tracing::error!("Error loading expose metrics: {e:#}");
     }
 
     if let Err(e) = load_metrics_debug_enabled(db).await {
-        tracing::error!("Error loading expose debug metrics: {e}");
+        tracing::error!("Error loading expose debug metrics: {e:#}");
     }
 
     if let Err(e) = load_tag_per_workspace_enabled(db).await {
-        tracing::error!("Error loading default tag per workpsace: {e}");
+        tracing::error!("Error loading default tag per workpsace: {e:#}");
     }
 
     if server_mode {
@@ -250,20 +250,44 @@ pub async fn monitor_mem() {
 
         // Obtain a MIB for the `epoch`, `stats.allocated`, and
         // `atats.resident` keys:
-        let e = epoch::mib().unwrap();
-        let allocated = stats::allocated::mib().unwrap();
-        let resident = stats::resident::mib().unwrap();
+        let e = match epoch::mib() {
+            Ok(mib) => mib,
+            Err(e) => {
+                tracing::error!("Error getting jemalloc epoch mib: {:?}", e);
+                return;
+            }
+        };
+        let allocated = match stats::allocated::mib() {
+            Ok(mib) => mib,
+            Err(e) => {
+                tracing::error!("Error getting jemalloc allocated mib: {:?}", e);
+                return;
+            }
+        };
+        let resident = match stats::resident::mib() {
+            Ok(mib) => mib,
+            Err(e) => {
+                tracing::error!("Error getting jemalloc resident mib: {:?}", e);
+                return;
+            }
+        }; 
+        
         
         loop {
             // Many statistics are cached and only updated 
             // when the epoch is advanced:
-            e.advance().unwrap();
-            
-            // Read statistics using MIB key:
-            let allocated = allocated.read().unwrap();
-            let resident = resident.read().unwrap();
-            tracing::info!("{} mb allocated/{} mb resident", bytes_to_mb(allocated as u64), bytes_to_mb(resident as u64));
-            tokio::time::sleep(Duration::from_secs(10)).await;
+            match e.advance() {
+                Ok(_) => {
+                    // Read statistics using MIB key:
+                    let allocated = allocated.read().unwrap_or_default();
+                    let resident = resident.read().unwrap_or_default();
+                    tracing::info!("{} mb allocated/{} mb resident", bytes_to_mb(allocated as u64), bytes_to_mb(resident as u64));
+                },
+                Err(e) => {
+                    tracing::error!("Error advancing jemalloc epoch: {:?}", e);
+                }
+            }
+            tokio::time::sleep(Duration::from_secs(30)).await;
         }
     });
 }
@@ -273,7 +297,7 @@ pub async fn load_keep_job_dir(db: &DB) {
     match value {
         Ok(Some(serde_json::Value::Bool(t))) => KEEP_JOB_DIR.store(t, Ordering::Relaxed),
         Err(e) => {
-            tracing::error!("Error loading keep job dir metrics: {e}");
+            tracing::error!("Error loading keep job dir metrics: {e:#}");
         }
         _ => (),
     };
@@ -287,7 +311,7 @@ pub async fn load_require_preexisting_user(db: &DB) {
             REQUIRE_PREEXISTING_USER_FOR_OAUTH.store(t, Ordering::Relaxed)
         }
         Err(e) => {
-            tracing::error!("Error loading keep job dir metrics: {e}");
+            tracing::error!("Error loading keep job dir metrics: {e:#}");
         }
         _ => (),
     };
@@ -950,7 +974,7 @@ pub async fn reload_worker_config(
                     tracing::info!("Waiting 5 seconds to allow others workers to start potential jobs that depend on a potential shared cache volume");
                     tokio::time::sleep(Duration::from_secs(5)).await;
                     if let Err(e) = windmill_worker::common::clean_cache().await {
-                        tracing::error!("Error cleaning the cache: {e}");
+                        tracing::error!("Error cleaning the cache: {e:#}");
                     }
                 }
             }
@@ -1079,11 +1103,16 @@ async fn handle_zombie_jobs<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
             mpsc::channel::<SameWorkerPayload>(1);
         let (send_result_never_used, _send_result_rx_never_used) = mpsc::channel::<SendResult>(1);
 
+        let label = if job.permissioned_as != format!("u/{}", job.created_by) && job.permissioned_as != job.created_by {
+            format!("ephemeral-script-end-user-{}", job.created_by)
+        } else {
+            "ephemeral-script".to_string()
+        };
         let token = create_token_for_owner(
             &db,
             &job.workspace_id,
             &job.permissioned_as,
-            "ephemeral-script",
+            &label,
             *SCRIPT_TOKEN_EXPIRY,
             &job.email,
             &job.id,
@@ -1159,11 +1188,13 @@ async fn handle_zombie_flows(
             .await?;
         } else {
             let id = flow.id.clone();
+            let last_ping = flow.last_ping.clone();
+            let now = now_from_db(db).await?;
             cancel_zombie_flow_job(
                 db,
                 flow,
                 &rsmq,
-                format!("Flow {} cancelled as it was hanging in between 2 steps", id),
+                format!("Flow {} cancelled as it was hanging in between 2 steps. Last ping: {last_ping:?} (now: {now})", id),
             )
             .await?;
         }

backend/windmill-api/Cargo.toml

@@ -89,4 +89,5 @@ openidconnect = { workspace = true, optional = true}
 pin-project.workspace = true
 crc.workspace = true
 http.workspace = true
-async-stream.workspace = true
+async-stream.workspace = true
+ulid.workspace = true

backend/windmill-api/openapi.yaml

@@ -1,7 +1,7 @@
 openapi: "3.0.3"
 
 info:
-  version: 1.335.0
+  version: 1.339.0
   title: Windmill API
 
   contact:
@@ -766,6 +766,45 @@ paths:
               schema:
                 type: string
 
+  
+  /settings/latest_key_renewal_attempt:
+    get:
+      summary: get latest key renewal attempt
+      operationId: getLatestKeyRenewalAttempt
+      tags:
+        - setting
+      responses:
+        "200":
+          description: status
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  result:
+                    type: string
+                  attempted_at:
+                    type: string
+                    format: date-time
+                required:
+                  - result
+                  - attempted_at
+                nullable: true
+
+  /settings/renew_license_key:
+    post:
+      summary: renew license key
+      operationId: renewLicenseKey
+      tags:
+        - setting
+      responses:
+        "200":
+          description: status
+          content:
+            text/plain:
+              schema:
+                type: string
+
   /saml/test_metadata:
     post:
       summary: test metadata
@@ -2025,6 +2064,27 @@ paths:
                 items:
                   type: string
 
+  /w/{workspace}/users/username_to_email/{username}:
+    get:
+      summary: get email from username
+      operationId: usernameToEmail
+      tags:
+        - user
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+        - name: username
+          in: path
+          required: true
+          schema:
+            type: string
+      responses:
+        "200":
+          description: email
+          content:
+            text/plain:
+              schema:
+                type: string
+
   /users/tokens/create:
     post:
       summary: create token
@@ -2557,16 +2617,41 @@ paths:
           content:
             application/json:
               schema:
-                additionalProperties:
-                  type: object
-                  properties:
-                    extra_params:
-                      additionalProperties:
-                        type: string
-                    scopes:
-                      type: array
-                      items:
-                        type: string
+                type: array
+                items:
+                  type: string
+
+  /oauth/get_connect/{client}:
+    get:
+      summary: get oauth connect
+      operationId: getOAuthConnect
+      tags:
+        - oauth
+      parameters:
+        - name: client
+          description: client name
+          in: path
+          required: true
+          schema:
+            type: string
+      responses:
+        "200":
+          description: get 
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  extra_params:
+                    additionalProperties:
+                      type: string
+                  scopes:
+                    type: array
+                    items:
+                      type: string
+
+
+
 
   /w/{workspace}/resources/create:
     post:
@@ -4900,6 +4985,8 @@ paths:
                       type: string
                     path:
                       type: string
+                    lock:
+                      type: string
                     cache_ttl:
                       type: integer
                   required:
@@ -4909,6 +4996,10 @@ paths:
                   type: object
                 force_viewer_one_of_fields:
                   type: object
+                force_viewer_allow_user_resources:
+                  type: array
+                  items:
+                    type: string
               required:
                 - args
                 - component
@@ -9416,12 +9507,16 @@ components:
             required:
               - name
               - typ
+        no_main_func:
+          type: boolean
+          nullable: true
       required:
         - star_args
         - start_kwargs
         - args
         - type
         - error
+        - no_main_func
 
     Preview:
       type: object
@@ -9851,6 +9946,14 @@ components:
           type: string
         occupancy_rate:
           type: number
+        memory:
+          type: number
+        vcpus:
+          type: number
+        memory_usage:
+          type: number
+        wm_memory_usage:
+          type: number
       required:
         - worker
         - worker_instance

backend/windmill-api/src/apps.rs

@@ -9,7 +9,9 @@ use std::collections::HashMap;
  */
 use crate::{
     db::{ApiAuthed, DB},
+    resources::get_resource_value_interpolated_internal,
     users::{require_owner_of_path, OptAuthed},
+    variables::encrypt,
     webhook_util::{WebhookMessage, WebhookShared},
     HTTP_CLIENT,
 };
@@ -38,7 +40,7 @@ use windmill_common::{
     utils::{
         http_get_from_hub, not_found_if_none, paginate, query_elems_from_hub, Pagination, StripPath,
     },
-    variables::build_crypt,
+    variables::{build_crypt, build_crypt_with_key_suffix},
     worker::to_raw_value,
     HUB_BASE_URL,
 };
@@ -145,6 +147,7 @@ pub struct AppHistoryUpdate {
 
 pub type StaticFields = HashMap<String, Box<RawValue>>;
 pub type OneOfFields = HashMap<String, Vec<Box<RawValue>>>;
+pub type AllowUserResources = Vec<String>;
 
 #[derive(Serialize, Deserialize, Debug, PartialEq, Clone)]
 #[serde(rename_all = "lowercase")]
@@ -158,6 +161,7 @@ pub enum ExecutionMode {
 pub struct PolicyTriggerableInputs {
     static_inputs: StaticFields,
     one_of_inputs: OneOfFields,
+    allow_user_resources: AllowUserResources,
 }
 
 #[derive(Serialize, Deserialize, Debug, Clone)]
@@ -421,6 +425,8 @@ async fn get_app_by_id(
 }
 
 async fn get_public_app_by_secret(
+    OptAuthed(opt_authed): OptAuthed,
+    Extension(user_db): Extension<UserDB>,
     Extension(db): Extension<DB>,
     Path((w_id, secret)): Path<(String, String)>,
 ) -> JsonResult<AppWithLastVersion> {
@@ -450,10 +456,32 @@ async fn get_public_app_by_secret(
 
     let policy = serde_json::from_str::<Policy>(app.policy.0.get()).map_err(to_anyhow)?;
 
-    if !matches!(policy.execution_mode, ExecutionMode::Anonymous) {
-        return Err(Error::NotAuthorized(
-            "App visibility does not allow public access".to_string(),
-        ));
+    if matches!(policy.execution_mode, ExecutionMode::Anonymous) {
+        return Ok(Json(app));
+    }
+
+    if opt_authed.is_none() {
+        {
+            return Err(Error::NotAuthorized(
+                "App visibility does not allow public access and you are not logged in".to_string(),
+            ));
+        }
+    } else {
+        let authed = opt_authed.unwrap();
+        let mut tx = user_db.begin(&authed).await?;
+        let is_visible = sqlx::query_scalar!(
+            "SELECT EXISTS(SELECT 1 FROM app WHERE id = $1 AND workspace_id = $2)",
+            id,
+            &w_id
+        )
+        .fetch_one(&mut *tx)
+        .await?;
+        tx.commit().await?;
+        if !is_visible.unwrap_or(false) {
+            return Err(Error::NotAuthorized(
+                "App visibility does not allow public access and you are logged in but you have no read-access to that app".to_string(),
+            ));
+        }
     }
 
     Ok(Json(app))
@@ -585,7 +613,7 @@ async fn create_app(
 
     audit_log(
         &mut tx,
-        &authed.username,
+        &authed,
         "apps.create",
         ActionKind::Create,
         &w_id,
@@ -701,7 +729,7 @@ async fn delete_app(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "apps.delete",
         ActionKind::Delete,
         &w_id,
@@ -736,7 +764,7 @@ async fn delete_app(
     .await
     .map_err(|e| {
         Error::InternalErr(format!(
-            "error deleting deployment metadata for script with path {path} in workspace {w_id}: {e}"
+            "error deleting deployment metadata for script with path {path} in workspace {w_id}: {e:#}"
         ))
     })?;
 
@@ -873,7 +901,7 @@ async fn update_app(
 
     audit_log(
         &mut tx,
-        &authed.username,
+        &authed,
         "apps.update",
         ActionKind::Update,
         &w_id,
@@ -940,6 +968,7 @@ pub struct ExecuteApp {
     // if set, the app is executed as viewer with the given static fields
     pub force_viewer_static_fields: Option<StaticFields>,
     pub force_viewer_one_of_fields: Option<OneOfFields>,
+    pub force_viewer_allow_user_resources: Option<AllowUserResources>,
 }
 
 fn digest(code: &str) -> String {
@@ -952,6 +981,7 @@ fn digest(code: &str) -> String {
 async fn execute_component(
     OptAuthed(opt_authed): OptAuthed,
     Extension(db): Extension<DB>,
+    Extension(user_db): Extension<UserDB>,
     Extension(rsmq): Extension<Option<rsmq_async::MultiplexedRsmq>>,
     Path((w_id, path)): Path<(String, StripPath)>,
     Json(payload): Json<ExecuteApp>,
@@ -976,6 +1006,7 @@ async fn execute_component(
         ExecuteApp {
             force_viewer_static_fields: Some(static_fields),
             force_viewer_one_of_fields: Some(one_of_fields),
+            force_viewer_allow_user_resources: Some(allow_user_resources),
             ..
         } => {
             let mut hm = HashMap::new();
@@ -986,6 +1017,7 @@ async fn execute_component(
                     PolicyTriggerableInputs {
                         static_inputs: static_fields,
                         one_of_inputs: one_of_fields,
+                        allow_user_resources,
                     },
                 );
             } else {
@@ -998,6 +1030,7 @@ async fn execute_component(
                     PolicyTriggerableInputs {
                         static_inputs: static_fields,
                         one_of_inputs: one_of_fields,
+                        allow_user_resources,
                     },
                 );
             }
@@ -1027,22 +1060,31 @@ async fn execute_component(
     let (username, permissioned_as, email) = match policy.execution_mode {
         ExecutionMode::Anonymous => {
             let username = opt_authed
-                .map(|a| a.username)
+                .as_ref()
+                .map(|a| a.username.clone())
                 .unwrap_or_else(|| "anonymous".to_string());
             let (permissioned_as, email) = get_on_behalf_of(&policy)?;
             (username, permissioned_as, email)
         }
         ExecutionMode::Publisher => {
-            let username = opt_authed.map(|a| a.username).ok_or_else(|| {
-                Error::BadRequest("publisher execution mode requires authentication".to_string())
-            })?;
+            let username = opt_authed
+                .as_ref()
+                .map(|a| a.username.clone())
+                .ok_or_else(|| {
+                    Error::BadRequest(
+                        "publisher execution mode requires authentication".to_string(),
+                    )
+                })?;
             let (permissioned_as, email) = get_on_behalf_of(&policy)?;
             (username, permissioned_as, email)
         }
         ExecutionMode::Viewer => {
-            let (username, email) = opt_authed.map(|a| (a.username, a.email)).ok_or_else(|| {
-                Error::BadRequest("Required to be authed in viewer mode".to_string())
-            })?;
+            let (username, email) = opt_authed
+                .as_ref()
+                .map(|a| (a.username.clone(), a.email.clone()))
+                .ok_or_else(|| {
+                    Error::BadRequest("Required to be authed in viewer mode".to_string())
+                })?;
             (
                 username.clone(),
                 username_to_permissioned_as(&username),
@@ -1051,17 +1093,37 @@ async fn execute_component(
         }
     };
 
-    let (job_payload, args, tag) = match payload {
+    let (job_payload, (args, job_id), tag) = match payload {
         ExecuteApp { args, component, raw_code: Some(raw_code), path: None, .. } => {
             let content = &raw_code.content;
             let payload = JobPayload::Code(raw_code.clone());
             let path = digest(content);
-            let args = build_args(policy, &component, path, args)?;
+            let args = build_args(
+                policy,
+                &component,
+                path,
+                args,
+                opt_authed.as_ref(),
+                &user_db,
+                &db,
+                &w_id,
+            )
+            .await?;
             (payload, args, None)
         }
         ExecuteApp { args, component, raw_code: None, path: Some(path), .. } => {
             let (payload, tag) = get_payload_tag_from_prefixed_path(&path, &db, &w_id).await?;
-            let args = build_args(policy, &component, path.to_string(), args)?;
+            let args = build_args(
+                policy,
+                &component,
+                path.to_string(),
+                args,
+                opt_authed.as_ref(),
+                &user_db,
+                &db,
+                &w_id,
+            )
+            .await?;
             (payload, args, tag)
         }
         _ => unreachable!(),
@@ -1081,7 +1143,7 @@ async fn execute_component(
         None,
         None,
         None,
-        None,
+        job_id,
         false,
         false,
         None,
@@ -1150,16 +1212,21 @@ async fn exists_app(
     Ok(Json(exists))
 }
 
-fn build_args(
+async fn build_args(
     policy: Policy,
     component: &str,
     path: String,
     args: HashMap<String, Box<RawValue>>,
-) -> Result<PushArgs> {
+    authed: Option<&ApiAuthed>,
+    user_db: &UserDB,
+    db: &DB,
+    w_id: &str,
+) -> Result<(PushArgs, Option<Uuid>)> {
+    let mut job_id: Option<Uuid> = None;
     let key = format!("{}:{}", component, &path);
-    let (static_inputs, one_of_inputs) = match policy {
+    let (static_inputs, one_of_inputs, allow_user_resources) = match policy {
         Policy { triggerables_v2: Some(t), .. } => {
-            let PolicyTriggerableInputs { static_inputs, one_of_inputs } = t
+            let PolicyTriggerableInputs { static_inputs, one_of_inputs, allow_user_resources } = t
                 .get(&key)
                 .or_else(|| t.get(&path))
                 .map(|x| x.clone())
@@ -1168,6 +1235,7 @@ fn build_args(
                         Some(PolicyTriggerableInputs {
                             static_inputs: HashMap::new(),
                             one_of_inputs: HashMap::new(),
+                            allow_user_resources: Vec::new(),
                         })
                     } else {
                         None
@@ -1177,7 +1245,7 @@ fn build_args(
                     Error::BadRequest(format!("path {} is not allowed in the app policy", path))
                 })?;
 
-            (static_inputs, one_of_inputs)
+            (static_inputs, one_of_inputs, allow_user_resources)
         }
         Policy { triggerables: Some(t), .. } => {
             let static_inputs = t
@@ -1195,7 +1263,7 @@ fn build_args(
                     Error::BadRequest(format!("path {} is not allowed in the app policy", path))
                 })?;
 
-            (static_inputs, HashMap::new())
+            (static_inputs, HashMap::new(), Vec::new())
         }
         _ => Err(Error::BadRequest(format!(
             "Policy is missing triggerables for {}",
@@ -1206,7 +1274,58 @@ fn build_args(
     let mut args = args.clone();
     let mut safe_args = HashMap::<String, Box<RawValue>>::new();
 
+    // tracing::error!("{:?}", allow_user_resources);
+    for k in allow_user_resources.iter() {
+        if let Some(arg_val) = args.get(k) {
+            let key = serde_json::from_str::<String>(arg_val.get()).ok();
+            if let Some(path) =
+                key.and_then(|x| x.clone().strip_prefix("$res:").map(|x| x.to_string()))
+            {
+                if let Some(authed) = authed {
+                    let res = get_resource_value_interpolated_internal(
+                        authed,
+                        Some(user_db.clone()),
+                        db,
+                        w_id,
+                        &path,
+                        None,
+                        "",
+                    )
+                    .await?;
+                    if res.is_none() {
+                        return Err(Error::BadRequest(format!(
+                            "Resource {} not found or not allowed for viewer",
+                            path
+                        )));
+                    }
+                    let job_id = if let Some(job_id) = job_id {
+                        job_id
+                    } else {
+                        job_id = Some(ulid::Ulid::new().into());
+                        job_id.unwrap()
+                    };
+                    let mut tx = db.begin().await?;
+                    let mc =
+                        build_crypt_with_key_suffix(&mut tx, &w_id, &job_id.to_string()).await?;
+                    let encrypted = encrypt(&mc, to_raw_value(&res.unwrap()).get());
+                    tx.commit().await?;
+                    safe_args.insert(
+                        k.to_string(),
+                        to_raw_value(&format!("$encrypted:{encrypted}")),
+                    );
+                } else {
+                    return Err(Error::BadRequest(
+                        "User resources are not allowed without being logged in".to_string(),
+                    ));
+                }
+            }
+        }
+    }
+
     for (k, v) in one_of_inputs {
+        if safe_args.contains_key(&k) {
+            continue;
+        }
         if let Some(arg_val) = args.get(&k) {
             let arg_str = arg_val.get();
 
@@ -1239,9 +1358,13 @@ fn build_args(
     }
 
     for (k, v) in args {
-        let arg_str = serde_json::to_string(&v).unwrap_or_else(|_| "".to_string());
+        if safe_args.contains_key(&k) {
+            continue;
+        }
 
-        if !arg_str.contains("$var:") && !arg_str.contains("$res:") {
+        let arg_str = v.get();
+
+        if !arg_str.contains("\"$var:") && !arg_str.contains("\"$res:") {
             safe_args.insert(k.to_string(), v);
         } else {
             safe_args.insert(
@@ -1254,7 +1377,7 @@ fn build_args(
                         )
                         .replace(
                             "$res:",
-                            "The following resource has been omitted for security reasons: ",
+                            "The following resource has been omitted for security reasons, to allow it, toggle: 'Allow resources from users' on that field input: ",
                         ),
                 )
                 .map_err(|e| {
@@ -1270,5 +1393,5 @@ fn build_args(
     for (k, v) in static_inputs {
         extra.insert(k.to_string(), v.to_owned());
     }
-    Ok(PushArgs { extra, args: safe_args })
+    Ok((PushArgs { extra, args: safe_args }, job_id))
 }

backend/windmill-api/src/concurrency_groups.rs

@@ -197,10 +197,10 @@ async fn get_concurrent_intervals(
             args: None,
             result: None,
             tag: None,
-            scheduled_for_before_now: None,
             has_null_parent: None,
             label: None,
-            is_not_schedule: None,
+            scheduled_for_before_now: _,
+            is_not_schedule: _,
             started_before: _,
             started_after: _,
             created_before: _,
@@ -221,28 +221,20 @@ async fn get_concurrent_intervals(
     // the workspace.
     // To avoid infering information through filtering, don't return obscured
     // jobs if the filters are too specific
-    if should_fetch_obscured_jobs {
-        let (sqlb_q, sqlb_c) = if w_id != "admin" {
-            // By default get obscured jobs from all workspaces, unless in
-            // admin workspace where admins can select to get all or not
-            (
-                filter_list_queue_query(
-                    sqlb_q,
-                    &ListQueueQuery { all_workspaces: Some(true), ..lqq.clone() },
-                    "admins",
-                ),
-                filter_list_completed_query(
-                    sqlb_c,
-                    &ListCompletedQuery { all_workspaces: Some(true), ..lq.clone() },
-                    "admins",
-                ),
-            )
-        } else {
-            (
-                filter_list_queue_query(sqlb_q, &lqq, w_id.as_str()),
-                filter_list_completed_query(sqlb_c, &lq, w_id.as_str()),
-            )
-        };
+    if should_fetch_obscured_jobs && w_id != "admins"{
+        // Get the obscured jobs from all workspaces (concurrency key could be global)
+        let (sqlb_q, sqlb_c) = (
+            filter_list_queue_query(
+                sqlb_q,
+                &ListQueueQuery { all_workspaces: Some(true), ..lqq.clone() },
+                "admins",
+            ),
+            filter_list_completed_query(
+                sqlb_c,
+                &ListCompletedQuery { all_workspaces: Some(true), ..lq.clone() },
+                "admins",
+            ),
+        );
 
         sqlb_q_user = filter_list_queue_query(sqlb_q_user, &lqq, w_id.as_str());
         sqlb_c_user = filter_list_completed_query(sqlb_c_user, &lq, w_id.as_str());
@@ -311,6 +303,8 @@ async fn get_concurrent_intervals(
             omitted_obscured_jobs: !should_fetch_obscured_jobs,
         }))
     } else {
+        sqlb_q = filter_list_queue_query(sqlb_q, &lqq, w_id.as_str());
+        sqlb_c = filter_list_completed_query(sqlb_c, &lq, w_id.as_str());
         let sql_q = sqlb_q.query()?;
         let sql_c = sqlb_c.query()?;
 

backend/windmill-api/src/configs.rs

@@ -122,7 +122,7 @@ async fn update_config(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "worker_config.update",
         ActionKind::Update,
         "global",
@@ -149,7 +149,7 @@ async fn delete_config(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "worker_config.delete",
         ActionKind::Delete,
         "global",

backend/windmill-api/src/db.rs

@@ -15,6 +15,7 @@ use sqlx::{
     pool::PoolConnection,
     PgConnection, Pool, Postgres,
 };
+use windmill_audit::audit_ee::{AuditAuthor, AuditAuthorable};
 use windmill_common::{
     db::{Authable, Authed},
     error::Error,
@@ -85,7 +86,7 @@ impl Migrate for CustomMigrator {
                     .fetch_one(&mut *self.inner)
                     .await
                     .map_err(|e| {
-                        tracing::error!("Error acquiring lock: {e}");
+                        tracing::error!("Error acquiring lock: {e:#}");
                         sqlx::migrate::MigrateError::Execute(e)
                     })?
                     .unwrap_or(false);
@@ -174,7 +175,7 @@ pub async fn migrate(db: &DB) -> Result<(), Error> {
         Err(sqlx::migrate::MigrateError::VersionMissing(e)) => {
             tracing::error!("Database had been applied more migrations than this container. 
             This usually mean than another container on a more recent version migrated the database and this one is on an earlier version.
-            Please update the container to latest. Not critical, but may cause issues if migration introduced a breaking change. Version missing: {e}");
+            Please update the container to latest. Not critical, but may cause issues if migration introduced a breaking change. Version missing: {e:#}");
             custom_migrator.unlock().await?;
             Ok(())
         }
@@ -183,7 +184,7 @@ pub async fn migrate(db: &DB) -> Result<(), Error> {
 
     #[cfg(feature = "enterprise")]
     if let Err(e) = windmill_migrations(&mut custom_migrator, db).await {
-        tracing::error!("Could not apply windmill custom migrations: {e}")
+        tracing::error!("Could not apply windmill custom migrations: {e:#}")
     }
 
     Ok(())
@@ -246,12 +247,34 @@ impl From<ApiAuthed> for Authed {
     }
 }
 
+impl From<&ApiAuthed> for AuditAuthor {
+    fn from(value: &ApiAuthed) -> Self {
+        Self {
+            email: value.email.clone(),
+            username: value.username.clone(),
+            username_override: value.username_override.clone(),
+        }
+    }
+}
+
 impl ApiAuthed {
     pub fn display_username(&self) -> &str {
         self.username_override.as_ref().unwrap_or(&self.username)
     }
 }
 
+impl AuditAuthorable for ApiAuthed {
+    fn username(&self) -> &str {
+        self.username.as_str()
+    }
+    fn email(&self) -> &str {
+        self.email.as_str()
+    }
+    fn username_override(&self) -> Option<&str> {
+        self.username_override.as_deref()
+    }
+}
+
 impl Authable for ApiAuthed {
     fn is_admin(&self) -> bool {
         self.is_admin

backend/windmill-api/src/flows.rs

@@ -348,7 +348,7 @@ async fn create_flow(
 
     audit_log(
         &mut tx,
-        &authed.username,
+        &authed,
         "flows.create",
         ActionKind::Create,
         &w_id,
@@ -504,7 +504,7 @@ async fn update_flow(
         nf.visible_to_runner_only.unwrap_or(false),
     )
     .execute(&mut tx)
-    .await?;
+    .await.map_err(|e| error::Error::InternalErr(format!("Error updating flow due to flow update: {e:#}")))?;
 
     if nf.path != flow_path {
         check_schedule_conflict(tx.transaction_mut(), &w_id, &nf.path).await?;
@@ -520,7 +520,7 @@ async fn update_flow(
             .bind(&flow_path)
             .bind(&w_id)
         .fetch_all(&mut tx)
-        .await?;
+        .await.map_err(|e| error::Error::InternalErr(format!("Error updating flow due to related schedules update: {e:#}")))?;
 
     let schedule = sqlx::query_as::<_, Schedule>(
         "UPDATE schedule SET path = $1, script_path = $1 WHERE path = $2 AND workspace_id = $3 AND is_flow IS true RETURNING *")
@@ -528,7 +528,7 @@ async fn update_flow(
         .bind(&flow_path)
         .bind(&w_id)
     .fetch_optional(&mut tx)
-    .await?;
+    .await.map_err(|e| error::Error::InternalErr(format!("Error updating flow due to related schedule update: {e:#}")))?;
 
     if let Some(schedule) = schedule {
         clear_schedule(tx.transaction_mut(), &flow_path, &w_id).await?;
@@ -553,7 +553,7 @@ async fn update_flow(
 
     audit_log(
         &mut tx,
-        &authed.username,
+        &authed,
         "flows.update",
         ActionKind::Create,
         &w_id,
@@ -618,14 +618,24 @@ async fn update_flow(
         w_id
     )
     .execute(&mut new_tx)
-    .await?;
+    .await
+    .map_err(|e| {
+        error::Error::InternalErr(format!(
+            "Error updating flow due to updating dependency job field: {e:#}"
+        ))
+    })?;
     if let Some(old_dep_job) = old_dep_job {
         sqlx::query!(
             "UPDATE queue SET canceled = true WHERE id = $1",
             old_dep_job
         )
         .execute(&mut new_tx)
-        .await?;
+        .await
+        .map_err(|e| {
+            error::Error::InternalErr(format!(
+                "Error updating flow due to cancelling dependency job: {e:#}"
+            ))
+        })?;
     }
 
     new_tx.commit().await?;
@@ -746,7 +756,7 @@ async fn archive_flow_by_path(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "flows.archive",
         ActionKind::Delete,
         &w_id,
@@ -813,7 +823,7 @@ async fn delete_flow_by_path(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "flows.delete",
         ActionKind::Delete,
         &w_id,
@@ -844,7 +854,7 @@ async fn delete_flow_by_path(
     .await
     .map_err(|e| {
         Error::InternalErr(format!(
-            "error deleting deployment metadata for script with path {path} in workspace {w_id}: {e}"
+            "error deleting deployment metadata for script with path {path} in workspace {w_id}: {e:#}"
         ))
     })?;
 

backend/windmill-api/src/folders.rs

@@ -223,7 +223,7 @@ async fn create_folder(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "folder.create",
         ActionKind::Create,
         &w_id,
@@ -356,7 +356,7 @@ async fn update_folder(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "folder.update",
         ActionKind::Update,
         &w_id,
@@ -522,7 +522,7 @@ async fn delete_folder(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "folder.delete",
         ActionKind::Delete,
         &w_id,
@@ -585,7 +585,7 @@ async fn add_owner(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "folder.add_owner",
         ActionKind::Update,
         &w_id,
@@ -664,7 +664,7 @@ async fn remove_owner(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "folder.remove_owner",
         ActionKind::Update,
         &w_id,

backend/windmill-api/src/groups.rs

@@ -252,7 +252,7 @@ async fn create_group(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "group.create",
         ActionKind::Create,
         &w_id,
@@ -296,7 +296,7 @@ async fn create_igroup(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "igroup.create",
         ActionKind::Create,
         "global",
@@ -337,7 +337,7 @@ async fn update_igroup(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "igroup.updated",
         ActionKind::Delete,
         "global",
@@ -367,7 +367,7 @@ async fn delete_igroup(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "igroup.delete",
         ActionKind::Delete,
         "global",
@@ -472,7 +472,7 @@ async fn delete_group(
     .await?;
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "group.delete",
         ActionKind::Delete,
         &w_id,
@@ -522,7 +522,7 @@ async fn update_group(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "group.edit",
         ActionKind::Update,
         &w_id,
@@ -573,7 +573,7 @@ async fn add_user(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "group.adduser",
         ActionKind::Update,
         &w_id,
@@ -624,7 +624,7 @@ async fn add_user_igroup(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "igroup.adduser",
         ActionKind::Update,
         "global",
@@ -693,7 +693,7 @@ async fn remove_user_igroup(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "igroup.removeuser",
         ActionKind::Update,
         "global",
@@ -733,7 +733,7 @@ async fn remove_user(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "group.removeuser",
         ActionKind::Update,
         &w_id,

backend/windmill-api/src/jobs.rs

@@ -51,7 +51,7 @@ use sqlx::types::JsonRawValue;
 use sqlx::{types::Uuid, FromRow, Postgres, Transaction};
 use tower_http::cors::{Any, CorsLayer};
 use urlencoding::encode;
-use windmill_audit::audit_ee::audit_log;
+use windmill_audit::audit_ee::{audit_log, AuditAuthor};
 use windmill_audit::ActionKind;
 use windmill_common::worker::{to_raw_value, CUSTOM_TAGS_PER_WORKSPACE, SERVER_CONFIG};
 use windmill_common::{
@@ -317,18 +317,31 @@ async fn cancel_job_api(
 ) -> error::Result<String> {
     let tx = db.begin().await?;
 
-    let username = match opt_authed {
-        Some(authed) => authed.username,
-        None => "anonymous".to_string(),
+    let audit_author = match opt_authed {
+        Some(authed) => (&authed).into(),
+        None => AuditAuthor {
+            email: "anonymous".to_string(),
+            username: "anonymous".to_string(),
+            username_override: None,
+        },
     };
 
-    let (mut tx, job_option) =
-        windmill_queue::cancel_job(&username, reason, id, &w_id, tx, &db, rsmq, false).await?;
+    let (mut tx, job_option) = windmill_queue::cancel_job(
+        &audit_author.username,
+        reason,
+        id,
+        &w_id,
+        tx,
+        &db,
+        rsmq,
+        false,
+    )
+    .await?;
 
     if let Some(id) = job_option {
         audit_log(
             &mut *tx,
-            &username,
+            &audit_author,
             "jobs.cancel",
             ActionKind::Delete,
             &w_id,
@@ -358,13 +371,17 @@ async fn cancel_persistent_script_api(
     Path((w_id, script_path)): Path<(String, StripPath)>,
     Json(CancelJob { reason }): Json<CancelJob>,
 ) -> error::Result<()> {
-    let username = match opt_authed {
-        Some(authed) => authed.username,
-        None => "anonymous".to_string(),
+    let audit_author = match opt_authed {
+        Some(authed) => (&authed).into(),
+        None => AuditAuthor {
+            email: "anonymous".to_string(),
+            username: "anonymous".to_string(),
+            username_override: None,
+        },
     };
 
     let cancelled_job_ids = windmill_queue::cancel_persistent_script_jobs(
-        &username,
+        &audit_author.username,
         reason,
         script_path.to_path(),
         &w_id,
@@ -375,7 +392,7 @@ async fn cancel_persistent_script_api(
 
     audit_log(
         &db,
-        &username,
+        &audit_author,
         "jobs.cancel_persistent",
         ActionKind::Delete,
         &w_id,
@@ -406,18 +423,31 @@ async fn force_cancel(
 ) -> error::Result<String> {
     let tx = db.begin().await?;
 
-    let username = match opt_authed {
-        Some(authed) => authed.username,
-        None => "anonymous".to_string(),
+    let audit_author = match opt_authed {
+        Some(authed) => (&authed).into(),
+        None => AuditAuthor {
+            email: "anonymous".to_string(),
+            username: "anonymous".to_string(),
+            username_override: None,
+        },
     };
 
-    let (mut tx, job_option) =
-        windmill_queue::cancel_job(&username, reason, id, &w_id, tx, &db, rsmq, true).await?;
+    let (mut tx, job_option) = windmill_queue::cancel_job(
+        &audit_author.username,
+        reason,
+        id,
+        &w_id,
+        tx,
+        &db,
+        rsmq,
+        true,
+    )
+    .await?;
 
     if let Some(id) = job_option {
         audit_log(
             &mut *tx,
-            &username,
+            &audit_author,
             "jobs.force_cancel",
             ActionKind::Delete,
             &w_id,
@@ -454,7 +484,7 @@ pub async fn get_path_for_hash<'c>(
     .await
     .map_err(|e| {
         Error::InternalErr(format!(
-            "querying getting path for hash {hash} in {w_id}: {e}"
+            "querying getting path for hash {hash} in {w_id}: {e:#}"
         ))
     })?;
     Ok(path)
@@ -486,7 +516,7 @@ pub async fn get_path_tag_limits_cache_for_hash(
     .await
     .map_err(|e| {
         Error::InternalErr(format!(
-            "querying getting path for hash {hash} in {w_id}: {e}"
+            "querying getting path for hash {hash} in {w_id}: {e:#}"
         ))
     })?;
     Ok((
@@ -933,7 +963,6 @@ impl From<ListCompletedQuery> for ListQueueQuery {
     }
 }
 
-
 pub fn filter_list_queue_query(
     mut sqlb: SqlBuilder,
     lq: &ListQueueQuery,
@@ -1396,7 +1425,7 @@ async fn resume_suspended_job_internal(
     {
         approver.approver
     } else {
-        authed.map(|x| x.username)
+        authed.as_ref().map(|x| x.username.clone())
     };
     insert_resume_job(
         resume_id,
@@ -1419,9 +1448,17 @@ async fn resume_suspended_job_internal(
     } else {
         resume_immediately_if_relevant(parent_flow_info, job_id, &mut tx).await?;
     }
+
+    let audit_author = match authed {
+        Some(authed) => (&authed).into(),
+        None => {
+            let approver = approver.unwrap_or_else(|| "anonymous".to_string());
+            AuditAuthor { email: approver.clone(), username: approver, username_override: None }
+        }
+    };
     audit_log(
         &mut *tx,
-        &approver.unwrap_or_else(|| "anonymous".to_string()),
+        &audit_author,
         "jobs.approved",
         ActionKind::Update,
         &w_id,
@@ -2047,76 +2084,75 @@ pub struct UnifiedJob {
 }
 
 const CJ_FIELDS: &[&str] = &[
-            "'CompletedJob' as typ",
-            "id",
-            "workspace_id",
-            "parent_job",
-            "created_by",
-            "created_at",
-            "started_at",
-            "null as scheduled_for",
-            "null as running",
-            "script_hash",
-            "script_path",
-            "null as args",
-            "duration_ms",
-            "success",
-            "deleted",
-            "canceled",
-            "canceled_by",
-            "job_kind",
-            "schedule_path",
-            "permissioned_as",
-            "is_flow_step",
-            "language",
-            "is_skipped",
-            "email",
-            "visible_to_owner",
-            "null as suspend",
-            "mem_peak",
-            "tag",
-            "null as concurrent_limit",
-            "null as concurrency_time_window_s",
-            "priority",
-            "result->'wm_labels' as labels",
-        ];
+    "'CompletedJob' as typ",
+    "id",
+    "workspace_id",
+    "parent_job",
+    "created_by",
+    "created_at",
+    "started_at",
+    "null as scheduled_for",
+    "null as running",
+    "script_hash",
+    "script_path",
+    "null as args",
+    "duration_ms",
+    "success",
+    "deleted",
+    "canceled",
+    "canceled_by",
+    "job_kind",
+    "schedule_path",
+    "permissioned_as",
+    "is_flow_step",
+    "language",
+    "is_skipped",
+    "email",
+    "visible_to_owner",
+    "null as suspend",
+    "mem_peak",
+    "tag",
+    "null as concurrent_limit",
+    "null as concurrency_time_window_s",
+    "priority",
+    "result->'wm_labels' as labels",
+];
 const QJ_FIELDS: &[&str] = &[
-            "'QueuedJob' as typ",
-            "id",
-            "workspace_id",
-            "parent_job",
-            "created_by",
-            "created_at",
-            "started_at",
-            "scheduled_for",
-            "running",
-            "script_hash",
-            "script_path",
-            "null as args",
-            "null as duration_ms",
-            "null as success",
-            "false as deleted",
-            "canceled",
-            "canceled_by",
-            "job_kind",
-            "schedule_path",
-            "permissioned_as",
-            "is_flow_step",
-            "language",
-            "false as is_skipped",
-            "email",
-            "visible_to_owner",
-            "suspend",
-            "mem_peak",
-            "tag",
-            "concurrent_limit",
-            "concurrency_time_window_s",
-            "priority",
-            "null as labels",
-        ];
+    "'QueuedJob' as typ",
+    "id",
+    "workspace_id",
+    "parent_job",
+    "created_by",
+    "created_at",
+    "started_at",
+    "scheduled_for",
+    "running",
+    "script_hash",
+    "script_path",
+    "null as args",
+    "null as duration_ms",
+    "null as success",
+    "false as deleted",
+    "canceled",
+    "canceled_by",
+    "job_kind",
+    "schedule_path",
+    "permissioned_as",
+    "is_flow_step",
+    "language",
+    "false as is_skipped",
+    "email",
+    "visible_to_owner",
+    "suspend",
+    "mem_peak",
+    "tag",
+    "concurrent_limit",
+    "concurrency_time_window_s",
+    "priority",
+    "null as labels",
+];
 
 impl UnifiedJob {
-
     pub fn completed_job_fields() -> &'static [&'static str] {
         CJ_FIELDS
     }
@@ -3515,7 +3551,7 @@ async fn add_batch_jobs(
         job_kind,
         language,
         dedicated_worker,
-        _custom_concurrency_key,
+        custom_concurrency_key,
         concurrent_limit,
         concurrent_time_window_s,
         timeout,
@@ -3659,6 +3695,13 @@ async fn add_batch_jobs(
         )
         .fetch_all(&db)
         .await?;
+    sqlx::query!(
+        "INSERT INTO concurrency_key (job_id, key) SELECT id, $1 FROM unnest($2::uuid[]) as id",
+        custom_concurrency_key,
+        &uuids
+    )
+    .execute(&db)
+    .await?;
 
     Ok(Json(uuids))
 }
@@ -3945,7 +3988,7 @@ pub fn filter_list_completed_query(
     }
 
     if let Some(ps) = &lq.script_path_start {
-        sqlb.and_where_like_left("script_path", ps);
+        sqlb.and_where_like_left("script_path", "?".bind(ps));
     }
     if let Some(p) = &lq.script_path_exact {
         sqlb.and_where_eq("script_path", "?".bind(p));
@@ -4318,7 +4361,7 @@ async fn delete_completed_job<'a>(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "jobs.delete",
         ActionKind::Delete,
         &w_id,

backend/windmill-api/src/lib.rs

@@ -130,7 +130,7 @@ pub async fn run_server(
             let r =
                 rsmq_async::RsmqConnection::create_queue(&mut rsmq, &tag, None, None, None).await;
             if let Err(e) = r {
-                tracing::info!("Redis queue {tag} could not be created: {e}");
+                tracing::info!("Redis queue {tag} could not be created: {e:#}");
             } else {
                 tracing::info!("Redis queue {tag} created");
             }

backend/windmill-api/src/openai.rs

@@ -198,7 +198,7 @@ async fn proxy(
         }
 
         let config: OpenaiConfig = serde_json::from_value(resource.unwrap())
-            .map_err(|e| Error::InternalErr(format!("validating openai resource {e}")))?;
+            .map_err(|e| Error::InternalErr(format!("validating openai resource {e:#}")))?;
 
         let mut user = None::<String>;
         let mut resource = match config {
@@ -238,7 +238,7 @@ async fn proxy(
         let azure_base_path = if let Some(azure_base_path) = azure_base_path {
             Some(
                 serde_json::from_value::<String>(azure_base_path).map_err(|e| {
-                    Error::InternalErr(format!("validating openai azure base path {e}"))
+                    Error::InternalErr(format!("validating openai azure base path {e:#}"))
                 })?,
             )
         } else {
@@ -317,7 +317,7 @@ async fn proxy(
     let mut tx = db.begin().await?;
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "openai.request",
         ActionKind::Execute,
         &w_id,

backend/windmill-api/src/raw_apps.rs

@@ -134,9 +134,7 @@ async fn get_data(
     let app = not_found_if_none(app_o, "App", path)?;
     let res = Response::builder().header(header::CONTENT_TYPE, "text/javascript");
 
-    Ok(res
-        .body(Body::from(app))
-        .unwrap())
+    Ok(res.body(Body::from(app)).unwrap())
 }
 
 async fn create_app(
@@ -181,7 +179,7 @@ async fn create_app(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "apps.create",
         ActionKind::Create,
         &w_id,
@@ -217,7 +215,7 @@ async fn delete_app(
     .await?;
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "apps.delete",
         ActionKind::Delete,
         &w_id,
@@ -294,7 +292,7 @@ async fn update_app(
     let npath = app.path.clone().unwrap_or_else(|| path.to_owned());
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "apps.update",
         ActionKind::Update,
         &w_id,

backend/windmill-api/src/resources.rs

@@ -24,7 +24,7 @@ use serde_json::{value::RawValue, Value};
 use sql_builder::{bind::Bind, quote, SqlBuilder};
 use sqlx::{FromRow, Postgres, Transaction};
 use uuid::Uuid;
-use windmill_audit::audit_ee::audit_log;
+use windmill_audit::audit_ee::{audit_log, AuditAuthor};
 use windmill_audit::ActionKind;
 use windmill_common::{
     db::UserDB,
@@ -473,15 +473,20 @@ pub async fn transform_json_value<'c>(
             let path = y.strip_prefix("$var:").unwrap();
             let tx: Transaction<'_, Postgres> =
                 authed_transaction_or_default(authed, user_db.clone(), db).await?;
+
             let v = crate::variables::get_value_internal(
                 tx,
                 db,
                 workspace,
                 path,
-                user_db
+                &user_db
                     .clone()
-                    .map(|_| authed.username.as_str())
-                    .unwrap_or("backend"),
+                    .map(|_| authed.into())
+                    .unwrap_or(AuditAuthor {
+                        email: "backend".to_string(),
+                        username: "backend".to_string(),
+                        username_override: None,
+                    }),
             )
             .await?;
             Ok(Value::String(v))
@@ -650,7 +655,7 @@ async fn create_resource(
     .await?;
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "resources.create",
         ActionKind::Create,
         &w_id,
@@ -710,7 +715,7 @@ async fn delete_resource(
     .await?;
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "resources.delete",
         ActionKind::Delete,
         &w_id,
@@ -796,7 +801,7 @@ async fn update_resource(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "resources.update",
         ActionKind::Update,
         &w_id,
@@ -857,7 +862,7 @@ async fn update_resource_value(
     .await?;
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "resources.update",
         ActionKind::Update,
         &w_id,
@@ -1001,7 +1006,7 @@ async fn create_resource_type(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "resource_types.create",
         ActionKind::Create,
         &w_id,
@@ -1065,7 +1070,7 @@ async fn delete_resource_type(
     .await?;
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "resource_types.delete",
         ActionKind::Delete,
         &w_id,
@@ -1121,7 +1126,7 @@ async fn update_resource_type(
     sqlx::query(&sql).execute(&mut *tx).await?;
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "resource_types.update",
         ActionKind::Update,
         &w_id,

backend/windmill-api/src/schedule.rs

@@ -185,7 +185,7 @@ async fn create_schedule(
         .bind(&ns.tag)
     .fetch_one(&mut tx)
     .await
-    .map_err(|e| Error::InternalErr(format!("inserting schedule in {w_id}: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("inserting schedule in {w_id}: {e:#}")))?;
 
     handle_deployment_metadata(
         &authed.email,
@@ -201,7 +201,7 @@ async fn create_schedule(
 
     audit_log(
         &mut tx,
-        &authed.username,
+        &authed,
         "schedule.create",
         ActionKind::Create,
         &w_id,
@@ -268,7 +268,7 @@ async fn edit_schedule(
         .bind(&w_id)
     .fetch_one(&mut tx)
     .await
-    .map_err(|e| Error::InternalErr(format!("updating schedule in {w_id}: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("updating schedule in {w_id}: {e:#}")))?;
 
     handle_deployment_metadata(
         &authed.email,
@@ -284,7 +284,7 @@ async fn edit_schedule(
 
     audit_log(
         &mut tx,
-        &authed.username,
+        &authed,
         "schedule.edit",
         ActionKind::Update,
         &w_id,
@@ -496,7 +496,7 @@ pub async fn set_enabled(
 
     audit_log(
         &mut tx,
-        &authed.username,
+        &authed,
         "schedule.setenabled",
         ActionKind::Update,
         &w_id,
@@ -544,7 +544,7 @@ pub async fn set_enabled(
 
 //     audit_log(
 //         &mut tx,
-//         &authed.username,
+//         &authed,
 //         "schedule.setenabled",
 //         ActionKind::Update,
 //         &w_id,
@@ -621,7 +621,7 @@ async fn delete_schedule(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "schedule.delete",
         ActionKind::Delete,
         &w_id,
@@ -793,6 +793,7 @@ pub async fn clear_schedule<'c>(
     path: &str,
     w_id: &str,
 ) -> Result<()> {
+    tracing::info!("Clearing schedule {}", path);
     sqlx::query!(
         "DELETE FROM queue WHERE schedule_path = $1 AND running = false AND workspace_id = $2 AND is_flow_step = false",
         path,

backend/windmill-api/src/scripts.rs

@@ -663,7 +663,7 @@ async fn create_script_internal<'c>(
     if p_hashes.is_some() && !p_hashes.unwrap().is_empty() {
         audit_log(
             &mut tx,
-            &authed.username,
+            &authed,
             "scripts.update",
             ActionKind::Update,
             &w_id,
@@ -682,7 +682,7 @@ async fn create_script_internal<'c>(
     } else {
         audit_log(
             &mut tx,
-            &authed.username,
+            &authed,
             "scripts.create",
             ActionKind::Create,
             &w_id,
@@ -1160,10 +1160,10 @@ async fn archive_script_by_path(
     )
     .fetch_one(&db)
     .await
-    .map_err(|e| Error::InternalErr(format!("archiving script in {w_id}: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("archiving script in {w_id}: {e:#}")))?;
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "scripts.archive",
         ActionKind::Delete,
         &w_id,
@@ -1211,11 +1211,11 @@ async fn archive_script_by_hash(
     .bind(&hash.0)
     .fetch_one(&mut *tx)
     .await
-    .map_err(|e| Error::InternalErr(format!("archiving script in {w_id}: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("archiving script in {w_id}: {e:#}")))?;
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "scripts.archive",
         ActionKind::Delete,
         &w_id,
@@ -1251,11 +1251,11 @@ async fn delete_script_by_hash(
     .bind(&w_id)
     .fetch_one(&db)
     .await
-    .map_err(|e| Error::InternalErr(format!("deleting script by hash {w_id}: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("deleting script by hash {w_id}: {e:#}")))?;
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "scripts.delete",
         ActionKind::Delete,
         &w_id,
@@ -1310,7 +1310,7 @@ async fn delete_script_by_path(
     )
     .fetch_one(&db)
     .await
-    .map_err(|e| Error::InternalErr(format!("deleting script by path {w_id}: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("deleting script by path {w_id}: {e:#}")))?;
 
     sqlx::query!(
         "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'script'",
@@ -1322,7 +1322,7 @@ async fn delete_script_by_path(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "scripts.delete",
         ActionKind::Delete,
         &w_id,
@@ -1357,7 +1357,7 @@ async fn delete_script_by_path(
     .await
     .map_err(|e| {
         Error::InternalErr(format!(
-            "error deleting deployment metadata for script with path {path} in workspace {w_id}: {e}"
+            "error deleting deployment metadata for script with path {path} in workspace {w_id}: {e:#}"
         ))
     })?;
 

backend/windmill-api/src/settings.rs

@@ -42,7 +42,12 @@ pub fn global_service() -> Router {
         )
         .route("/test_smtp", post(test_email))
         .route("/test_license_key", post(test_license_key))
-        .route("/send_stats", post(send_stats));
+        .route("/send_stats", post(send_stats))
+        .route(
+            "/latest_key_renewal_attempt",
+            get(get_latest_key_renewal_attempt),
+        )
+        .route("/renew_license_key", post(renew_license_key));
 
     #[cfg(feature = "parquet")]
     {
@@ -115,7 +120,7 @@ pub async fn test_s3_bucket(
         .ok_or_else(|| {
             error::Error::InternalErr("Failed to list files in blob storage".to_string())
         })?
-        .map_err(|e| anyhow::anyhow!("error listing bucket: {e}"))?;
+        .map_err(|e| anyhow::anyhow!("error listing bucket: {e:#}"))?;
     tracing::info!("Listed files: {:?}", first_file);
     let path = object_store::path::Path::from(format!(
         "/test-s3-bucket-{uuid}",
@@ -125,7 +130,7 @@ pub async fn test_s3_bucket(
     client
         .put(&path, Bytes::from_static(b"hello"))
         .await
-        .map_err(|e| anyhow::anyhow!("error writing file to {path}: {e}"))?;
+        .map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?;
     let content = client
         .get(&path)
         .await
@@ -263,3 +268,60 @@ pub async fn send_stats(Extension(db): Extension<DB>, authed: ApiAuthed) -> Resu
 
     Ok("Sent stats".to_string())
 }
+
+#[derive(serde::Serialize)]
+pub struct KeyRenewalAttempt {
+    result: String,
+    attempted_at: chrono::DateTime<chrono::Utc>,
+}
+
+pub async fn get_latest_key_renewal_attempt(
+    Extension(db): Extension<DB>,
+    authed: ApiAuthed,
+) -> JsonResult<Option<KeyRenewalAttempt>> {
+    require_super_admin(&db, &authed.email).await?;
+
+    let last_attempt = sqlx::query!(
+        "SELECT value, created_at FROM metrics WHERE id = $1 ORDER BY created_at DESC LIMIT 1",
+        "license_key_renewal"
+    )
+    .fetch_optional(&db)
+    .await?;
+
+    match last_attempt {
+        Some(last_attempt) => {
+            let last_attempt_result = serde_json::from_value::<String>(last_attempt.value)
+                .map_err(|e| {
+                    error::Error::InternalErr(format!("Failed to parse last attempt: {}", e))
+                })?;
+            Ok(Json(Some(KeyRenewalAttempt {
+                result: last_attempt_result,
+                attempted_at: last_attempt.created_at,
+            })))
+        }
+        None => Ok(Json(None)),
+    }
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn renew_license_key() -> Result<String> {
+    return Err(error::Error::BadRequest(
+        "License key renewal not available on community edition".to_string(),
+    ));
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn renew_license_key(Extension(db): Extension<DB>, authed: ApiAuthed) -> Result<String> {
+    require_super_admin(&db, &authed.email).await?;
+    windmill_common::stats_ee::send_stats(&"manual".to_string(), &HTTP_CLIENT, &db).await?;
+    let result = windmill_common::ee::renew_license_key(&HTTP_CLIENT, &db).await;
+
+    if result != "success" {
+        return Err(error::Error::BadRequest(format!(
+            "Failed to renew license key: {}",
+            result
+        )));
+    } else {
+        return Ok("Renewed license key".to_string());
+    }
+}

backend/windmill-api/src/users.rs

@@ -40,7 +40,7 @@ use sqlx::FromRow;
 use time::OffsetDateTime;
 use tower_cookies::{Cookie, Cookies};
 use tracing::{Instrument, Span};
-use windmill_audit::audit_ee::audit_log;
+use windmill_audit::audit_ee::{audit_log, AuditAuthor};
 use windmill_audit::ActionKind;
 use windmill_common::global_settings::AUTOMATE_USERNAME_CREATION_SETTING;
 use windmill_common::users::truncate_token;
@@ -72,6 +72,7 @@ pub fn workspaced_service() -> Router {
         .route("/whois/:username", get(whois))
         .route("/whoami", get(whoami))
         .route("/leave", post(leave_workspace))
+        .route("/username_to_email/:username", get(username_to_email))
 }
 
 pub fn global_service() -> Router {
@@ -118,6 +119,16 @@ pub fn make_unauthed_service() -> Router {
 fn username_override_from_label(label: Option<String>) -> Option<String> {
     if label.as_ref().is_some_and(|x| x.starts_with("webhook-")) {
         label
+    } else if label
+        .as_ref()
+        .is_some_and(|x| x.starts_with("ephemeral-script-end-user-"))
+    {
+        Some(
+            label
+                .unwrap()
+                .trim_start_matches("ephemeral-script-end-user-")
+                .to_string(),
+        )
     } else {
         None
     }
@@ -919,9 +930,10 @@ async fn logout(
         .fetch_optional(&mut *tx)
         .await?;
     if let Some(email) = email {
+        let email = email.unwrap_or("noemail".to_string());
         audit_log(
             &mut *tx,
-            &email.unwrap_or("noemail".to_string()),
+            &AuditAuthor { email: email.clone(), username: email, username_override: None },
             "users.logout",
             ActionKind::Delete,
             "global",
@@ -985,7 +997,7 @@ async fn global_whoami(
     )
     .fetch_one(&db)
     .await
-    .map_err(|e| Error::InternalErr(format!("fetching global identity: {e}")));
+    .map_err(|e| Error::InternalErr(format!("fetching global identity: {e:#}")));
 
     if let Ok(user) = user {
         Ok(Json(user))
@@ -1275,7 +1287,7 @@ async fn whois(
 // }
 
 async fn decline_invite(
-    ApiAuthed { email, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Extension(db): Extension<DB>,
     Json(nu): Json<DeclineInvite>,
 ) -> Result<(StatusCode, String)> {
@@ -1284,18 +1296,18 @@ async fn decline_invite(
     let is_admin = sqlx::query_scalar!(
         "DELETE FROM workspace_invite WHERE workspace_id = $1 AND email = $2 RETURNING is_admin",
         nu.workspace_id,
-        email,
+        authed.email,
     )
     .fetch_optional(&mut *tx)
     .await?;
 
     audit_log(
         &mut *tx,
-        &email,
+        &authed,
         "users.decline_invite",
         ActionKind::Delete,
         &nu.workspace_id,
-        Some(&email),
+        Some(&authed.email),
         None,
     )
     .await?;
@@ -1306,11 +1318,14 @@ async fn decline_invite(
             StatusCode::OK,
             format!(
                 "user {} declined invite to workspace {}",
-                &email, nu.workspace_id
+                &authed.email, nu.workspace_id
             ),
         ))
     } else {
-        Err(Error::NotFound(format!("invite for {email} not found")))
+        Err(Error::NotFound(format!(
+            "invite for {} not found",
+            authed.email
+        )))
     }
 }
 
@@ -1319,7 +1334,7 @@ lazy_static! {
 }
 
 async fn accept_invite(
-    ApiAuthed { email, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Extension(webhook): Extension<WebhookShared>,
     Extension(db): Extension<DB>,
     Extension(rsmq): Extension<Option<rsmq_async::MultiplexedRsmq>>,
@@ -1330,7 +1345,7 @@ async fn accept_invite(
     let r = sqlx::query!(
         "DELETE FROM workspace_invite WHERE workspace_id = $1 AND email = $2 RETURNING is_admin, operator",
         nu.workspace_id,
-        email,
+        authed.email,
     )
     .fetch_optional(&mut *tx)
     .await?;
@@ -1339,7 +1354,7 @@ async fn accept_invite(
         let already_in_workspace = sqlx::query_scalar!(
             "SELECT EXISTS(SELECT 1 FROM usr WHERE workspace_id = $1 AND email = $2)",
             &nu.workspace_id,
-            &email,
+            &authed.email,
         )
         .fetch_one(&mut *tx)
         .await?
@@ -1351,14 +1366,14 @@ async fn accept_invite(
                 StatusCode::CREATED,
                 format!(
                     "user {} accepted invite to workspace {}",
-                    &email, nu.workspace_id
+                    &authed.email, nu.workspace_id
                 ),
             ));
         }
         let username;
-        (tx, username) = add_user_to_workspace(
+        (tx, username) = join_workspace(
             &nu.workspace_id,
-            &email,
+            &authed,
             nu.username,
             r.is_admin,
             r.operator,
@@ -1368,29 +1383,29 @@ async fn accept_invite(
 
         audit_log(
             &mut *tx,
-            &username,
+            &ApiAuthed { username: username.clone(), ..authed.clone() },
             "users.accept_invite",
             ActionKind::Create,
             &nu.workspace_id,
-            Some(&email),
+            Some(&authed.email),
             None,
         )
         .await?;
         tx.commit().await?;
 
         handle_deployment_metadata(
-            &email,
+            &authed.email,
             &username,
             &db,
             &nu.workspace_id,
-            windmill_git_sync::DeployedObject::User { email: email.clone() },
-            Some(format!("User '{}' accepted invite", &email)),
+            windmill_git_sync::DeployedObject::User { email: authed.email.clone() },
+            Some(format!("User '{}' accepted invite", &authed.email)),
             rsmq,
             true,
         )
         .await?;
         webhook.send_instance_event(InstanceEvent::UserJoinedWorkspace {
-            email: email.clone(),
+            email: authed.email.clone(),
             workspace: nu.workspace_id.clone(),
             username: username,
         });
@@ -1398,17 +1413,20 @@ async fn accept_invite(
             StatusCode::CREATED,
             format!(
                 "user {} accepted invite to workspace {}",
-                &email, nu.workspace_id
+                &authed.email, nu.workspace_id
             ),
         ))
     } else {
-        Err(Error::NotFound(format!("invite for {email} not found")))
+        Err(Error::NotFound(format!(
+            "invite for {} not found",
+            authed.email
+        )))
     }
 }
 
-async fn add_user_to_workspace<'c>(
+async fn join_workspace<'c>(
     w_id: &str,
-    email: &str,
+    authed: &ApiAuthed,
     username: Option<String>,
     is_admin: bool,
     operator: bool,
@@ -1430,7 +1448,7 @@ async fn add_user_to_workspace<'c>(
                 "username is not allowed when username creation is automated".to_string(),
             ));
         }
-        get_instance_username_or_create_pending(&mut tx, &email).await?
+        get_instance_username_or_create_pending(&mut tx, &authed.email).await?
     } else {
         let username = username.ok_or(Error::BadRequest("username is required".to_string()))?;
         let already_exists_username = sqlx::query_scalar!(
@@ -1460,7 +1478,7 @@ async fn add_user_to_workspace<'c>(
     let already_exists_email = sqlx::query_scalar!(
         "SELECT EXISTS(SELECT 1 FROM usr WHERE workspace_id = $1 AND email = $2)",
         &w_id,
-        username,
+        authed.email,
     )
     .fetch_one(&mut *tx)
     .await?
@@ -1469,7 +1487,7 @@ async fn add_user_to_workspace<'c>(
     if already_exists_email {
         return Err(Error::BadRequest(format!(
             "user with email {} already exists in workspace {}",
-            email, w_id
+            authed.email, w_id
         )));
     }
 
@@ -1478,7 +1496,7 @@ async fn add_user_to_workspace<'c>(
             (workspace_id, email, username, is_admin, operator)
             VALUES ($1, $2, $3, $4, $5)",
         &w_id,
-        email,
+        authed.email,
         username,
         is_admin,
         operator
@@ -1496,33 +1514,30 @@ async fn add_user_to_workspace<'c>(
     .await?;
     audit_log(
         &mut *tx,
-        &username,
+        &AuditAuthor { username: username.clone(), ..authed.into() },
         "users.add_to_workspace",
         ActionKind::Create,
         &w_id,
-        Some(email),
+        Some(&authed.email),
         None,
     )
     .await?;
     Ok((tx, username))
 }
 
-async fn leave_instance(
-    Extension(db): Extension<DB>,
-    ApiAuthed { email, username, .. }: ApiAuthed,
-) -> Result<String> {
+async fn leave_instance(Extension(db): Extension<DB>, authed: ApiAuthed) -> Result<String> {
     let mut tx = db.begin().await?;
-    sqlx::query!("DELETE FROM password WHERE email = $1", &email)
+    sqlx::query!("DELETE FROM password WHERE email = $1", &authed.email)
         .execute(&mut *tx)
         .await?;
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "workspaces.leave",
         ActionKind::Delete,
         "global",
-        Some(&email),
+        Some(&authed.email),
         None,
     )
     .await?;
@@ -1553,7 +1568,7 @@ async fn get_workspace_user(
 }
 
 async fn update_workspace_user(
-    ApiAuthed { username, email, is_admin, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Extension(db): Extension<DB>,
     Extension(rsmq): Extension<Option<rsmq_async::MultiplexedRsmq>>,
     Path((w_id, username_to_update)): Path<(String, String)>,
@@ -1561,7 +1576,7 @@ async fn update_workspace_user(
 ) -> Result<String> {
     let mut tx = db.begin().await?;
 
-    require_admin(is_admin, &username)?;
+    require_admin(authed.is_admin, &authed.username)?;
 
     if let Some(a) = eu.is_admin {
         sqlx::query_scalar!(
@@ -1598,7 +1613,7 @@ async fn update_workspace_user(
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "users.update",
         ActionKind::Update,
         &w_id,
@@ -1618,8 +1633,8 @@ async fn update_workspace_user(
     tx.commit().await?;
 
     handle_deployment_metadata(
-        &email,
-        &username,
+        &authed.email,
+        &authed.username,
         &db,
         &w_id,
         windmill_git_sync::DeployedObject::User { email: user_email.clone() },
@@ -1633,12 +1648,12 @@ async fn update_workspace_user(
 }
 
 async fn update_user(
-    ApiAuthed { email, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Path(email_to_update): Path<String>,
     Extension(db): Extension<DB>,
     Json(eu): Json<EditUser>,
 ) -> Result<String> {
-    require_super_admin(&db, &email).await?;
+    require_super_admin(&db, &authed.email).await?;
     let mut tx = db.begin().await?;
 
     if let Some(sa) = eu.is_super_admin {
@@ -1653,7 +1668,7 @@ async fn update_user(
 
     audit_log(
         &mut *tx,
-        &email,
+        &authed,
         "users.update",
         ActionKind::Update,
         "global",
@@ -1666,11 +1681,11 @@ async fn update_user(
 }
 
 async fn delete_user(
-    ApiAuthed { email, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Path(email_to_delete): Path<String>,
     Extension(db): Extension<DB>,
 ) -> Result<String> {
-    require_super_admin(&db, &email).await?;
+    require_super_admin(&db, &authed.email).await?;
     let mut tx = db.begin().await?;
 
     sqlx::query!("DELETE FROM password WHERE email = $1", &email_to_delete)
@@ -1702,7 +1717,7 @@ async fn delete_user(
     }
     audit_log(
         &mut *tx,
-        &email,
+        &authed,
         "users.delete",
         ActionKind::Delete,
         "global",
@@ -1720,14 +1735,14 @@ lazy_static::lazy_static! {
 }
 
 async fn create_user(
-    ApiAuthed { email, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Extension(db): Extension<DB>,
     Extension(webhook): Extension<WebhookShared>,
     Extension(argon2): Extension<Arc<Argon2<'_>>>,
     Extension(rsmq): Extension<Option<rsmq_async::MultiplexedRsmq>>,
     Json(mut nu): Json<NewUser>,
 ) -> Result<(StatusCode, String)> {
-    require_super_admin(&db, &email).await?;
+    require_super_admin(&db, &authed.email).await?;
     let mut tx = db.begin().await?;
 
     nu.email = nu.email.to_lowercase();
@@ -1787,7 +1802,7 @@ async fn create_user(
 
     audit_log(
         &mut *tx,
-        &email,
+        &authed,
         "users.add_global",
         ActionKind::Create,
         "global",
@@ -1795,19 +1810,20 @@ async fn create_user(
         None,
     )
     .await?;
-    tx = add_to_demo_if_exists(tx, &email).await?;
+    tx = add_to_demo_if_exists(tx, &nu.email).await?;
 
     tx.commit().await?;
 
-    invite_user_to_all_auto_invite_worspaces(&db, &nu.email, rsmq).await?;
+    invite_user_to_all_auto_invite_worspaces(&db, &nu.email, rsmq, &authed).await?;
     send_email_if_possible(
         "Invited to Windmill",
         &format!(
-            "You have been granted access to Windmill by {email}.
+            "You have been granted access to Windmill by {}.
 
 Log in and change your password: {}/user/login?email={}&password={}&rd=%2F%23user-settings
 
 You can then join or create a workspace. Happy building!",
+            authed.email,
             BASE_URL.read().await.clone(),
             &nu.email,
             &nu.password
@@ -1837,14 +1853,14 @@ pub async fn send_email_if_possible_intern(subject: &str, content: &str, to: Str
 }
 
 async fn delete_workspace_user(
-    ApiAuthed { username, email, is_admin, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Extension(db): Extension<DB>,
     Extension(rsmq): Extension<Option<rsmq_async::MultiplexedRsmq>>,
     Path((w_id, username_to_delete)): Path<(String, String)>,
 ) -> Result<String> {
     let mut tx = db.begin().await?;
 
-    require_admin(is_admin, &username)?;
+    require_admin(authed.is_admin, &authed.username)?;
 
     let email_to_delete_o = sqlx::query_scalar!(
         "SELECT email FROM usr where username = $1 AND workspace_id = $2",
@@ -1856,17 +1872,17 @@ async fn delete_workspace_user(
 
     let email_to_delete = not_found_if_none(email_to_delete_o, "User", &username_to_delete)?;
 
-    let username = sqlx::query_scalar!(
-        "DELETE FROM usr WHERE email = $1 AND workspace_id = $2 RETURNING username",
+    sqlx::query_scalar!(
+        "DELETE FROM usr WHERE email = $1 AND workspace_id = $2",
         email_to_delete,
         &w_id
     )
-    .fetch_one(&mut *tx)
+    .execute(&mut *tx)
     .await?;
 
     sqlx::query!(
         "DELETE FROM usr_to_group WHERE usr = $1 AND workspace_id = $2",
-        &username,
+        &username_to_delete,
         &w_id
     )
     .execute(&mut *tx)
@@ -1874,7 +1890,7 @@ async fn delete_workspace_user(
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "users.delete",
         ActionKind::Delete,
         &w_id,
@@ -1885,8 +1901,8 @@ async fn delete_workspace_user(
     tx.commit().await?;
 
     handle_deployment_metadata(
-        &email,
-        &username,
+        &authed.email,
+        &authed.username,
         &db,
         &w_id,
         windmill_git_sync::DeployedObject::User { email: email_to_delete.clone() },
@@ -1905,47 +1921,48 @@ async fn delete_workspace_user(
 async fn set_password(
     Extension(db): Extension<DB>,
     Extension(argon2): Extension<Arc<Argon2<'_>>>,
-    ApiAuthed { username, email, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Json(EditPassword { password }): Json<EditPassword>,
 ) -> Result<String> {
     let mut tx = db.begin().await?;
 
     let custom_type = sqlx::query_scalar!(
         "SELECT login_type::TEXT FROM password WHERE email = $1",
-        &email
+        &authed.email
     )
     .fetch_one(&mut *tx)
     .await
-    .map_err(|e| Error::InternalErr(format!("setting password: {e}")))?
+    .map_err(|e| Error::InternalErr(format!("setting password: {e:#}")))?
     .unwrap_or("".to_string());
 
     if custom_type != "password".to_string() {
         return Err(Error::BadRequest(format!(
-            "login type for {email} is of type {custom_type}. Cannot set password."
+            "login type for {} is of type {custom_type}. Cannot set password.",
+            authed.email
         )));
     }
 
     sqlx::query!(
         "UPDATE password SET password_hash = $1 WHERE email = $2",
         &hash_password(argon2, password)?,
-        &email,
+        &authed.email,
     )
     .execute(&mut *tx)
     .await?;
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "users.setpassword",
         ActionKind::Update,
         "global",
-        Some(&email),
+        Some(&authed.email),
         None,
     )
     .await?;
     tx.commit().await?;
 
-    Ok(format!("password of {} updated", email))
+    Ok(format!("password of {} updated", authed.email))
 }
 
 pub fn hash_password(argon2: Arc<Argon2>, password: String) -> Result<String> {
@@ -2141,7 +2158,11 @@ async fn login(
 
             audit_log(
                 &mut *tx,
-                &email,
+                &AuditAuthor {
+                    username: email.clone(),
+                    email: email.clone(),
+                    username_override: None,
+                },
                 "users.login",
                 ActionKind::Create,
                 "global",
@@ -2215,23 +2236,25 @@ pub async fn create_session_token<'c>(
 
 async fn create_token(
     Extension(db): Extension<DB>,
-    ApiAuthed { email, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Json(new_token): Json<NewToken>,
 ) -> Result<(StatusCode, String)> {
     let token = rd_string(30);
     let mut tx = db.begin().await?;
 
-    let is_super_admin =
-        sqlx::query_scalar!("SELECT super_admin FROM password WHERE email = $1", email)
-            .fetch_optional(&mut *tx)
-            .await?
-            .unwrap_or(false);
+    let is_super_admin = sqlx::query_scalar!(
+        "SELECT super_admin FROM password WHERE email = $1",
+        authed.email
+    )
+    .fetch_optional(&mut *tx)
+    .await?
+    .unwrap_or(false);
     sqlx::query!(
         "INSERT INTO token
             (token, email, label, expiration, super_admin, scopes)
             VALUES ($1, $2, $3, $4, $5, $6)",
         token,
-        email,
+        authed.email,
         new_token.label,
         new_token.expiration,
         is_super_admin,
@@ -2242,14 +2265,14 @@ async fn create_token(
 
     audit_log(
         &mut *tx,
-        &email,
+        &authed,
         "users.token.create",
         ActionKind::Create,
         &"global",
         Some(&token[0..10]),
         None,
     )
-    .instrument(tracing::info_span!("token", email = &email))
+    .instrument(tracing::info_span!("token", email = &authed.email))
     .await?;
     tx.commit().await?;
     Ok((StatusCode::CREATED, token))
@@ -2257,11 +2280,11 @@ async fn create_token(
 
 async fn impersonate(
     Extension(db): Extension<DB>,
-    ApiAuthed { email, username, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Json(new_token): Json<NewToken>,
 ) -> Result<(StatusCode, String)> {
     let token = rd_string(30);
-    require_super_admin(&db, &email).await?;
+    require_super_admin(&db, &authed.email).await?;
 
     if new_token.impersonate_email.is_none() {
         return Err(Error::BadRequest(
@@ -2295,7 +2318,7 @@ async fn impersonate(
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "users.impersonate",
         ActionKind::Delete,
         &"global",
@@ -2344,7 +2367,7 @@ async fn list_tokens(
 
 async fn delete_token(
     Extension(db): Extension<DB>,
-    ApiAuthed { email, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Path(token_prefix): Path<String>,
 ) -> Result<String> {
     let mut tx = db.begin().await?;
@@ -2355,14 +2378,14 @@ async fn delete_token(
                  AND token LIKE concat($2::text, '%')
            RETURNING concat(substring(token for 10), '*****')",
     )
-    .bind(&email)
+    .bind(&authed.email)
     .bind(&token_prefix)
     .fetch_all(&mut *tx)
     .await?;
 
     audit_log(
         &mut *tx,
-        &email,
+        &authed,
         "users.token.delete",
         ActionKind::Delete,
         &"global",
@@ -2383,20 +2406,20 @@ async fn delete_token(
 async fn leave_workspace(
     Extension(db): Extension<DB>,
     Path(w_id): Path<String>,
-    ApiAuthed { username, .. }: ApiAuthed,
+    authed: ApiAuthed,
 ) -> Result<String> {
     let mut tx = db.begin().await?;
     sqlx::query!(
         "DELETE FROM usr WHERE workspace_id = $1 AND username = $2",
         &w_id,
-        username
+        authed.username
     )
     .execute(&mut *tx)
     .await?;
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "users.leave_workspace",
         ActionKind::Delete,
         &w_id,
@@ -2584,18 +2607,35 @@ async fn get_instance_username_info(
     }))
 }
 
+async fn username_to_email(
+    Path((w_id, username)): Path<(String, String)>,
+    Extension(db): Extension<DB>,
+) -> Result<String> {
+    let email = sqlx::query_scalar!(
+        "SELECT email FROM usr WHERE username = $1 AND workspace_id = $2",
+        &username,
+        &w_id
+    )
+    .fetch_optional(&db)
+    .await?;
+
+    let email = not_found_if_none(email, "user", username)?;
+
+    Ok(email)
+}
+
 #[derive(Deserialize)]
 struct RenameUser {
     new_username: String,
 }
 
 async fn rename_user(
-    ApiAuthed { email, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Path(user_email): Path<String>,
     Extension(db): Extension<DB>,
     Json(ru): Json<RenameUser>,
 ) -> Result<String> {
-    require_super_admin(&db, &email).await?;
+    require_super_admin(&db, &authed.email).await?;
 
     let mut tx = db.begin().await?;
 
@@ -2652,7 +2692,7 @@ async fn rename_user(
 
     audit_log(
         &mut *tx,
-        &email,
+        &authed,
         "users.rename",
         ActionKind::Update,
         "global",

backend/windmill-api/src/utils.rs

@@ -21,7 +21,7 @@ pub async fn require_super_admin(db: &DB, email: &str) -> error::Result<()> {
     let is_admin = sqlx::query_scalar!("SELECT super_admin FROM password WHERE email = $1", email)
         .fetch_optional(db)
         .await
-        .map_err(|e| Error::InternalErr(format!("fetching super admin: {e}")))?
+        .map_err(|e| Error::InternalErr(format!("fetching super admin: {e:#}")))?
         .unwrap_or(false);
 
     if !is_admin {
@@ -148,7 +148,7 @@ pub async fn get_instance_username_or_create_pending<'c>(
             )
             .execute(&mut **tx)
             .await
-            .map_err(|e| Error::InternalErr(format!("creating pending user: {e}")))?;
+            .map_err(|e| Error::InternalErr(format!("creating pending user: {e:#}")))?;
 
             Ok(username)
         }

backend/windmill-api/src/variables.rs

@@ -20,7 +20,7 @@ use axum::{
 };
 use hyper::StatusCode;
 use serde_json::Value;
-use windmill_audit::audit_ee::audit_log;
+use windmill_audit::audit_ee::{audit_log, AuditAuthorable};
 use windmill_audit::ActionKind;
 use windmill_common::{
     db::UserDB,
@@ -97,9 +97,10 @@ async fn list_variables(
          from variable
          LEFT JOIN account ON variable.account = account.id AND account.workspace_id = $1
          LEFT JOIN resource ON resource.path = variable.path AND resource.workspace_id = $1
-         WHERE variable.workspace_id = $1 ORDER BY path",
+         WHERE variable.workspace_id = $1 AND variable.path NOT LIKE 'u/' || $2 || '/secret_arg/%' ORDER BY path",
     )
     .bind(&w_id)
+    .bind(&authed.username)
     .fetch_all(&mut *tx)
     .await?;
 
@@ -151,7 +152,7 @@ async fn get_variable(
         if decrypt_secret {
             audit_log(
                 &mut *tx,
-                &authed.username,
+                &authed,
                 "variables.decrypt_secret",
                 ActionKind::Execute,
                 &w_id,
@@ -192,7 +193,7 @@ async fn get_value(
 ) -> JsonResult<String> {
     let path = path.to_path();
     let tx = user_db.begin(&authed).await?;
-    return get_value_internal(tx, &db, &w_id, &path, &authed.username)
+    return get_value_internal(tx, &db, &w_id, &path, &authed)
         .await
         .map(Json);
 }
@@ -315,7 +316,7 @@ async fn create_variable(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "variables.create",
         ActionKind::Create,
         &w_id,
@@ -391,7 +392,7 @@ async fn delete_variable(
     .await?;
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "variables.delete",
         ActionKind::Delete,
         &w_id,
@@ -547,7 +548,7 @@ async fn update_variable(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "variables.update",
         ActionKind::Update,
         &w_id,
@@ -603,7 +604,7 @@ pub async fn get_value_internal<'c>(
     db: &DB,
     w_id: &str,
     path: &str,
-    username: &str,
+    audit_author: &impl AuditAuthorable,
 ) -> Result<String> {
     let variable_o = sqlx::query!(
         "SELECT value, account, (now() > account.expires_at) as is_expired, is_secret, path from variable
@@ -622,7 +623,7 @@ pub async fn get_value_internal<'c>(
     let r = if variable.is_secret {
         audit_log(
             &mut *tx,
-            username,
+            audit_author,
             "variables.decrypt_secret",
             ActionKind::Execute,
             &w_id,

backend/windmill-api/src/workers.rs

@@ -51,7 +51,16 @@ struct WorkerPing {
     custom_tags: Option<Vec<String>>,
     worker_group: String,
     wm_version: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
     occupancy_rate: Option<f32>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    memory: Option<i64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    vcpus: Option<i64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    memory_usage: Option<i64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    wm_memory_usage: Option<i64>,
 }
 
 #[derive(Serialize, Deserialize)]
@@ -79,7 +88,7 @@ async fn list_worker_pings(
 
     let rows = sqlx::query_as!(
         WorkerPing,
-        "SELECT worker, worker_instance,  EXTRACT(EPOCH FROM (now() - ping_at))::integer as last_ping, started_at, ip, jobs_executed, CASE WHEN $4 IS TRUE THEN current_job_id ELSE NULL END as current_job_id, CASE WHEN $4 IS TRUE THEN current_job_workspace_id ELSE NULL END as current_job_workspace_id, custom_tags, worker_group, wm_version, occupancy_rate
+        "SELECT worker, worker_instance,  EXTRACT(EPOCH FROM (now() - ping_at))::integer as last_ping, started_at, ip, jobs_executed, CASE WHEN $4 IS TRUE THEN current_job_id ELSE NULL END as current_job_id, CASE WHEN $4 IS TRUE THEN current_job_workspace_id ELSE NULL END as current_job_workspace_id, custom_tags, worker_group, wm_version, occupancy_rate, memory, vcpus, memory_usage, wm_memory_usage
         FROM worker_ping
         WHERE ($1::integer IS NULL AND ping_at > now() - interval '5 minute') OR (ping_at > now() - ($1 || ' seconds')::interval)
         ORDER BY ping_at desc LIMIT $2 OFFSET $3",

backend/windmill-api/src/workspaces.rs

@@ -34,7 +34,7 @@ use itertools::Itertools;
 use regex::Regex;
 
 use uuid::Uuid;
-use windmill_audit::audit_ee::audit_log;
+use windmill_audit::audit_ee::{audit_log, AuditAuthor, AuditAuthorable};
 use windmill_audit::ActionKind;
 use windmill_common::db::UserDB;
 use windmill_common::s3_helpers::LargeFileStorage;
@@ -376,7 +376,7 @@ async fn get_settings(
     )
     .fetch_one(&mut *tx)
     .await
-    .map_err(|e| Error::InternalErr(format!("getting settings: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("getting settings: {e:#}")))?;
 
     tx.commit().await?;
     Ok(Json(settings))
@@ -399,7 +399,7 @@ async fn get_deploy_to(
     )
     .fetch_one(&mut *tx)
     .await
-    .map_err(|e| Error::InternalErr(format!("getting deploy_to: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("getting deploy_to: {e:#}")))?;
 
     tx.commit().await?;
     Ok(Json(settings))
@@ -448,7 +448,7 @@ async fn edit_slack_command(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.edit_command_script",
         ActionKind::Update,
         &w_id,
@@ -533,7 +533,7 @@ async fn edit_deploy_to(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.edit_deploy_to",
         ActionKind::Update,
         &w_id,
@@ -571,7 +571,8 @@ async fn auto_add_user(
     w_id: &str,
     operator: &bool,
     tx: &mut Transaction<'_, Postgres>,
-) -> Result<()> {
+    authorable: &impl AuditAuthorable,
+) -> Result<String> {
     let automate_username_creation = sqlx::query_scalar!(
         "SELECT value FROM global_settings WHERE name = $1",
         AUTOMATE_USERNAME_CREATION_SETTING,
@@ -645,9 +646,24 @@ async fn auto_add_user(
     )
     .execute(&mut **tx)
     .await?;
+    let audit_author = if authorable.username() == authorable.email() && authorable.email() == email
+    {
+        // if the user is auto adding themselves (e.g. by joining the instance), we use their newly created workspace username for audit logs
+        AuditAuthor {
+            username: username.clone(),
+            email: email.to_string(),
+            username_override: None,
+        }
+    } else {
+        AuditAuthor {
+            username: authorable.username().to_string(),
+            email: authorable.email().to_string(),
+            username_override: authorable.username_override().map(|x| x.to_string()),
+        }
+    };
     audit_log(
         &mut **tx,
-        &username,
+        &audit_author,
         "users.auto_invite_add",
         ActionKind::Create,
         &w_id,
@@ -655,7 +671,7 @@ async fn auto_add_user(
         None,
     )
     .await?;
-    Ok(())
+    Ok(username)
 }
 
 async fn edit_auto_invite(
@@ -720,7 +736,7 @@ async fn edit_auto_invite(
             .fetch_all(&mut *tx).await?);
 
             for user in users_to_auto_add.as_ref().unwrap() {
-                auto_add_user(&user.email, &w_id, &operator, &mut tx).await?;
+                auto_add_user(&user.email, &w_id, &operator, &mut tx, &authed).await?;
                 send_email_if_possible(
                     &format!("Added to Windmill's workspace: {w_id}"),
                     &format!(
@@ -757,7 +773,7 @@ async fn edit_auto_invite(
     }
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.edit_auto_invite_domain",
         ActionKind::Update,
         &w_id,
@@ -818,7 +834,7 @@ async fn edit_webhook(
     }
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.edit_webhook",
         ActionKind::Update,
         &w_id,
@@ -862,7 +878,7 @@ async fn edit_copilot_config(
     }
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.edit_copilot_config",
         ActionKind::Update,
         &w_id,
@@ -903,7 +919,7 @@ async fn get_copilot_info(
     )
     .fetch_one(&mut *tx)
     .await
-    .map_err(|e| Error::InternalErr(format!("getting openai_resource_path and code_completion_enabled: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("getting openai_resource_path and code_completion_enabled: {e:#}")))?;
     tx.commit().await?;
 
     Ok(Json(CopilotInfo {
@@ -926,7 +942,7 @@ async fn edit_large_file_storage_config(
     let args_for_audit = format!("{:?}", new_config.large_file_storage);
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.edit_large_file_storage_config",
         ActionKind::Update,
         &w_id,
@@ -994,7 +1010,7 @@ async fn edit_git_sync_config(
     let args_for_audit = format!("{:?}", new_config.git_sync_settings);
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.edit_git_sync_config",
         ActionKind::Update,
         &w_id,
@@ -1058,7 +1074,7 @@ async fn edit_default_scripts(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.edit_default_scripts",
         ActionKind::Update,
         &w_id,
@@ -1128,7 +1144,7 @@ async fn edit_default_app(
     let args_for_audit = format!("{:?}", new_config.default_app_path);
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.edit_default_app",
         ActionKind::Update,
         &w_id,
@@ -1221,7 +1237,7 @@ async fn edit_error_handler(
     }
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.edit_error_handler",
         ActionKind::Update,
         &w_id,
@@ -1263,7 +1279,7 @@ async fn set_environment_variable(
 
             audit_log(
                 &mut *tx,
-                &authed.username,
+                &authed,
                 "workspace.set_environment_variable",
                 ActionKind::Create,
                 &w_id,
@@ -1285,7 +1301,7 @@ async fn set_environment_variable(
 
             audit_log(
                 &mut *tx,
-                &authed.username,
+                &authed,
                 "workspace.delete_environment_variable",
                 ActionKind::Delete,
                 &w_id,
@@ -1611,7 +1627,7 @@ async fn create_workspace(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.create",
         ActionKind::Create,
         &nw.id,
@@ -1643,7 +1659,7 @@ async fn edit_workspace(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspaces.update",
         ActionKind::Update,
         &w_id,
@@ -1659,9 +1675,9 @@ async fn edit_workspace(
 async fn archive_workspace(
     Extension(db): Extension<DB>,
     Path(w_id): Path<String>,
-    ApiAuthed { is_admin, username, email, .. }: ApiAuthed,
+    authed: ApiAuthed,
 ) -> Result<String> {
-    require_admin(is_admin, &username)?;
+    require_admin(authed.is_admin, &authed.username)?;
     let mut tx = db.begin().await?;
     sqlx::query!("UPDATE workspace SET deleted = true WHERE id = $1", &w_id)
         .execute(&mut *tx)
@@ -1669,11 +1685,11 @@ async fn archive_workspace(
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "workspaces.archive",
         ActionKind::Update,
         &w_id,
-        Some(&email),
+        Some(&authed.email),
         None,
     )
     .await?;
@@ -1685,24 +1701,24 @@ async fn archive_workspace(
 async fn leave_workspace(
     Extension(db): Extension<DB>,
     Path(w_id): Path<String>,
-    ApiAuthed { email, username, .. }: ApiAuthed,
+    authed: ApiAuthed,
 ) -> Result<String> {
     let mut tx = db.begin().await?;
     sqlx::query!(
         "DELETE FROM usr WHERE workspace_id = $1 AND email = $2",
         &w_id,
-        &email
+        &authed.email
     )
     .execute(&mut *tx)
     .await?;
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "workspaces.leave",
         ActionKind::Delete,
         &w_id,
-        Some(&email),
+        Some(&authed.email),
         None,
     )
     .await?;
@@ -1714,9 +1730,9 @@ async fn leave_workspace(
 async fn unarchive_workspace(
     Extension(db): Extension<DB>,
     Path(w_id): Path<String>,
-    ApiAuthed { is_admin, username, email, .. }: ApiAuthed,
+    authed: ApiAuthed,
 ) -> Result<String> {
-    require_admin(is_admin, &username)?;
+    require_admin(authed.is_admin, &authed.username)?;
     let mut tx = db.begin().await?;
     sqlx::query!("UPDATE workspace SET deleted = false WHERE id = $1", &w_id)
         .execute(&mut *tx)
@@ -1724,11 +1740,11 @@ async fn unarchive_workspace(
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "workspaces.unarchive",
         ActionKind::Update,
         &w_id,
-        Some(&email),
+        Some(&authed.email),
         None,
     )
     .await?;
@@ -1740,7 +1756,7 @@ async fn unarchive_workspace(
 async fn delete_workspace(
     Extension(db): Extension<DB>,
     Path(w_id): Path<String>,
-    ApiAuthed { username, email, .. }: ApiAuthed,
+    authed: ApiAuthed,
 ) -> Result<String> {
     let w_id = match w_id.as_str() {
         "starter" => Err(Error::BadRequest(
@@ -1752,7 +1768,7 @@ async fn delete_workspace(
         _ => Ok(w_id),
     }?;
     let mut tx = db.begin().await?;
-    require_super_admin(&db, &email).await?;
+    require_super_admin(&db, &authed.email).await?;
 
     sqlx::query!("DELETE FROM dependency_map WHERE workspace_id = $1", &w_id)
         .execute(&mut *tx)
@@ -1859,11 +1875,11 @@ async fn delete_workspace(
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "workspaces.delete",
         ActionKind::Delete,
         &w_id,
-        Some(&email),
+        Some(&authed.email),
         None,
     )
     .await?;
@@ -1876,6 +1892,7 @@ pub async fn invite_user_to_all_auto_invite_worspaces(
     db: &DB,
     email: &str,
     rsmq: Option<rsmq_async::MultiplexedRsmq>,
+    authorable: &impl AuditAuthorable,
 ) -> Result<()> {
     let mut tx = db.begin().await?;
     let domain = email.split('@').last().unwrap();
@@ -1890,14 +1907,8 @@ pub async fn invite_user_to_all_auto_invite_worspaces(
     for r in workspaces {
         if r.auto_add.is_some() && r.auto_add.unwrap() {
             let operator = r.auto_invite_operator.unwrap_or(false);
-            auto_add_user(email, &r.workspace_id, &operator, &mut tx).await?;
-            let username = sqlx::query_scalar!(
-                "SELECT username FROM usr WHERE workspace_id = $1 AND email = $2",
-                r.workspace_id,
-                email
-            )
-            .fetch_one(&mut *tx)
-            .await?;
+            let username =
+                auto_add_user(email, &r.workspace_id, &operator, &mut tx, authorable).await?;
             auto_added_workspace_usernames.push((r.workspace_id, username));
         } else {
             sqlx::query!(
@@ -1999,14 +2010,14 @@ If you do not have an account on {}, login with SSO or ask an admin to create an
 }
 
 async fn add_user(
-    ApiAuthed { username, email, is_admin, .. }: ApiAuthed,
+    authed: ApiAuthed,
     Extension(db): Extension<DB>,
     Extension(webhook): Extension<WebhookShared>,
     Extension(rsmq): Extension<Option<rsmq_async::MultiplexedRsmq>>,
     Path(w_id): Path<String>,
     Json(mut nu): Json<NewWorkspaceUser>,
 ) -> Result<(StatusCode, String)> {
-    require_admin(is_admin, &username)?;
+    require_admin(authed.is_admin, &authed.username)?;
     nu.email = nu.email.to_lowercase();
 
     let mut tx = db.begin().await?;
@@ -2023,7 +2034,7 @@ async fn add_user(
     if already_exists_email {
         return Err(Error::BadRequest(format!(
             "user with email {} already exists in workspace {}",
-            email, w_id
+            nu.email, w_id
         )));
     }
 
@@ -2091,11 +2102,11 @@ async fn add_user(
 
     audit_log(
         &mut *tx,
-        &username,
+        &authed,
         "users.add_to_workspace",
         ActionKind::Create,
         &w_id,
-        Some(&email),
+        Some(&nu.email),
         None,
     )
     .await?;
@@ -2103,8 +2114,8 @@ async fn add_user(
     tx.commit().await?;
 
     handle_deployment_metadata(
-        &email,
-        &username,
+        &authed.email,
+        &authed.username,
         &db,
         &w_id,
         windmill_git_sync::DeployedObject::User { email: nu.email.clone() },
@@ -2117,9 +2128,10 @@ async fn add_user(
     send_email_if_possible(
         &format!("Added to Windmill's workspace: {w_id}"),
         &format!(
-            "You have been granted access to Windmill's workspace {w_id} by {email}
+            "You have been granted access to Windmill's workspace {w_id} by {}
 
 If you do not have an account on {}, login with SSO or ask an admin to create an account for you.",
+            authed.email,
             BASE_URL.read().await.clone()
         ),
         &nu.email,
@@ -2566,9 +2578,9 @@ async fn tarball_workspace(
     if !skip_variables.unwrap_or(false) {
         let variables =
             sqlx::query_as::<_, ExportableListableVariable>(if !skip_secrets.unwrap_or(false) {
-                "SELECT * FROM variable WHERE workspace_id = $1"
+                "SELECT * FROM variable WHERE workspace_id = $1 AND path NOT LIKE 'u/%/secret_arg/%'"
             } else {
-                "SELECT * FROM variable WHERE workspace_id = $1 AND is_secret = false"
+                "SELECT * FROM variable WHERE workspace_id = $1 AND is_secret = false AND path NOT LIKE  'u/%/secret_arg/%'"
             })
             .bind(&w_id)
             .fetch_all(&mut *tx)
@@ -2810,7 +2822,7 @@ async fn change_workspace_name(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspace.change_workspace_name",
         ActionKind::Update,
         &w_id,
@@ -3135,7 +3147,7 @@ async fn change_workspace_id(
 
     audit_log(
         &mut *tx,
-        &authed.username,
+        &authed,
         "workspace.change_workspace_id",
         ActionKind::Update,
         &rw.new_id,

backend/windmill-audit/src/audit_ee.rs

@@ -15,10 +15,37 @@ use windmill_common::{
 use crate::{ActionKind, AuditLog, ListAuditLogQuery};
 use sqlx::{Postgres, Transaction};
 
+#[derive(Clone)]
+pub struct AuditAuthor {
+    pub username: String,
+    pub email: String,
+    pub username_override: Option<String>,
+}
+
+impl AuditAuthorable for AuditAuthor {
+    fn email(&self) -> &str {
+        &self.email
+    }
+
+    fn username(&self) -> &str {
+        &self.username
+    }
+
+    fn username_override(&self) -> Option<&str> {
+        self.username_override.as_deref()
+    }
+}
+
+pub trait AuditAuthorable {
+    fn username(&self) -> &str;
+    fn email(&self) -> &str;
+    fn username_override(&self) -> Option<&str>;
+}
+
 #[tracing::instrument(level = "trace", skip_all)]
 pub async fn audit_log<'c, E: sqlx::Executor<'c, Database = Postgres>>(
     _db: E,
-    _username: &str,
+    _author: &impl AuditAuthorable,
     mut _operation: &str,
     _action_kind: ActionKind,
     _w_id: &str,

backend/windmill-common/Cargo.toml

@@ -7,6 +7,7 @@ edition.workspace = true
 [features]
 default = []
 enterprise = []
+jemalloc = ["dep:tikv-jemalloc-ctl"]
 prometheus = ["dep:prometheus"]
 flamegraph = ["dep:tracing-flame"]
 loki = ["dep:tracing-loki"]
@@ -49,3 +50,7 @@ aws-sdk-sts = { workspace = true, optional = true }
 indexmap.workspace = true
 bytes = { workspace = true, optional = true }
 mail-send.workspace = true
+sysinfo.workspace = true
+
+[target.'cfg(not(target_env = "msvc"))'.dependencies]
+tikv-jemalloc-ctl = { optional = true, workspace = true }

backend/windmill-common/src/ee.rs

@@ -25,3 +25,14 @@ pub async fn get_license_plan() -> LicensePlan {
 pub enum CriticalErrorChannel {}
 
 pub async fn trigger_critical_error_channels(_error_message: String) {}
+
+#[cfg(feature = "enterprise")]
+pub async fn renew_license_key(_http_client: &reqwest::Client, _db: &crate::db::DB) -> String {
+    // Implementation is not open source
+    "".to_string()
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn schedule_key_renewal(_http_client: &reqwest::Client, _db: &crate::db::DB) -> () {
+    // Implementation is not open source
+}

backend/windmill-common/src/flow_status.rs

@@ -124,6 +124,7 @@ struct UntaggedFlowStatusModule {
     parallel: Option<bool>,
     while_loop: Option<bool>,
     approvers: Option<Vec<Approval>>,
+    failed_retries: Option<Vec<Uuid>>,
 }
 
 #[derive(Serialize, Debug, Clone)]
@@ -167,6 +168,8 @@ pub enum FlowStatusModule {
         #[serde(default)]
         #[serde(skip_serializing_if = "Vec::is_empty")]
         approvers: Vec<Approval>,
+        #[serde(skip_serializing_if = "Vec::is_empty")]
+        failed_retries: Vec<Uuid>,
     },
     Failure {
         id: String,
@@ -175,6 +178,8 @@ pub enum FlowStatusModule {
         flow_jobs: Option<Vec<Uuid>>,
         #[serde(skip_serializing_if = "Option::is_none")]
         branch_chosen: Option<BranchChosen>,
+        #[serde(skip_serializing_if = "Vec::is_empty")]
+        failed_retries: Vec<Uuid>,
     },
 }
 
@@ -235,6 +240,7 @@ impl<'de> Deserialize<'de> for FlowStatusModule {
                 flow_jobs: untagged.flow_jobs,
                 branch_chosen: untagged.branch_chosen,
                 approvers: untagged.approvers.unwrap_or_default(),
+                failed_retries: untagged.failed_retries.unwrap_or_default(),
             }),
             "Failure" => Ok(FlowStatusModule::Failure {
                 id: untagged
@@ -245,6 +251,7 @@ impl<'de> Deserialize<'de> for FlowStatusModule {
                     .ok_or_else(|| serde::de::Error::missing_field("job"))?,
                 flow_jobs: untagged.flow_jobs,
                 branch_chosen: untagged.branch_chosen,
+                failed_retries: untagged.failed_retries.unwrap_or_default(),
             }),
             other => Err(serde::de::Error::unknown_variant(
                 other,

backend/windmill-common/src/global_settings.rs

@@ -25,6 +25,7 @@ pub const OBJECT_STORE_CACHE_CONFIG_SETTING: &str = "object_store_cache_config";
 pub const AUTOMATE_USERNAME_CREATION_SETTING: &str = "automate_username_creation";
 pub const HUB_BASE_URL_SETTING: &str = "hub_base_url";
 pub const CRITICAL_ERROR_CHANNELS_SETTING: &str = "critical_error_channels";
+pub const DEV_INSTANCE_SETTING: &str = "dev_instance";
 
 pub const ENV_SETTINGS: [&str; 50] = [
     "DISABLE_NSJAIL",

backend/windmill-common/src/jobs.rs

@@ -434,7 +434,7 @@ pub async fn script_hash_to_tag_and_limits<'c>(
     .await
     .map_err(|e| {
         Error::InternalErr(format!(
-            "querying getting tag for hash {script_hash}: {e}"
+            "querying getting tag for hash {script_hash}: {e:#}"
         ))
     })?;
     Ok((

backend/windmill-common/src/variables.rs

@@ -78,6 +78,20 @@ pub async fn build_crypt<'c>(
     Ok(magic_crypt::new_magic_crypt!(crypt_key, 256))
 }
 
+pub async fn build_crypt_with_key_suffix<'c>(
+    db: &mut Transaction<'c, Postgres>,
+    w_id: &str,
+    key_suffix: &str,
+) -> crate::error::Result<MagicCrypt256> {
+    let key = get_workspace_key(w_id, db).await?;
+    let crypt_key = if let Some(ref salt) = SECRET_SALT.as_ref() {
+        format!("{}{}{}", key, salt, key_suffix)
+    } else {
+        format!("{}{}", key, key_suffix)
+    };
+    Ok(magic_crypt::new_magic_crypt!(crypt_key, 256))
+}
+
 pub async fn get_workspace_key<'c>(
     w_id: &str,
     db: &mut Transaction<'c, Postgres>,
@@ -88,7 +102,7 @@ pub async fn get_workspace_key<'c>(
     )
     .fetch_one(&mut **db)
     .await
-    .map_err(|e| crate::Error::InternalErr(format!("fetching workspace key: {e}")))?;
+    .map_err(|e| crate::Error::InternalErr(format!("fetching workspace key: {e:#}")))?;
     Ok(key)
 }
 
@@ -118,9 +132,7 @@ pub async fn get_secret_value_as_admin(
             let mut tx = db.begin().await?;
             let mc = build_crypt(&mut tx, &w_id).await?;
             tx.commit().await?;
-
-            mc.decrypt_base64_to_string(value)
-                .map_err(|e| crate::Error::InternalErr(e.to_string()))?
+            decrypt_value_with_mc(value, mc).await?
         } else {
             "".to_string()
         }
@@ -131,6 +143,15 @@ pub async fn get_secret_value_as_admin(
     Ok(r)
 }
 
+pub async fn decrypt_value_with_mc(
+    value: String,
+    mc: MagicCrypt256,
+) -> Result<String, crate::error::Error> {
+    Ok(mc
+        .decrypt_base64_to_string(value)
+        .map_err(|e| crate::Error::InternalErr(e.to_string()))?)
+}
+
 pub async fn get_reserved_variables(
     db: &DB,
     w_id: &str,

backend/windmill-common/src/worker.rs

@@ -7,6 +7,7 @@ use std::{
     collections::{HashMap, HashSet},
     sync::{atomic::AtomicBool, Arc},
 };
+use sysinfo::{MemoryRefreshKind, System};
 use tokio::sync::RwLock;
 
 use crate::{error, global_settings::CUSTOM_TAGS_SETTING, server::ServerConfig, DB};
@@ -31,9 +32,9 @@ lazy_static::lazy_static! {
         "snowflake".to_string(),
         "mssql".to_string(),
         "graphql".to_string(),
+        "php".to_string(),
         "dependency".to_string(),
         "flow".to_string(),
-        "hub".to_string(),
         "other".to_string()
     ];
 
@@ -132,16 +133,7 @@ fn process_custom_tags(tags: Vec<String>) -> (Vec<String>, HashMap<String, Vec<S
     (global, specific)
 }
 
-pub async fn update_ping(worker_instance: &str, worker_name: &str, ip: &str, db: &DB) {
-    let (tags, dw) = {
-        let wc = WORKER_CONFIG.read().await.clone();
-        (
-            wc.worker_tags,
-            wc.dedicated_worker
-                .as_ref()
-                .map(|x| format!("{}:{}", x.workspace_id, x.path)),
-        )
-    };
+pub fn get_vcpus() -> Option<i64> {
     let mut vcpus = std::process::Command::new("cat")
         .args(["/sys/fs/cgroup/cpu.max"])
         .output()
@@ -153,7 +145,8 @@ pub async fn update_ping(worker_instance: &str, worker_name: &str, ip: &str, db:
                 .map(|s| s.to_string())
                 .collect::<Vec<String>>()
                 .get(0)
-                .map(|s| s.to_string())
+                .map(|s| s.to_string().trim().parse::<i64>().ok())
+                .flatten()
         })
         .flatten();
 
@@ -162,22 +155,74 @@ pub async fn update_ping(worker_instance: &str, worker_name: &str, ip: &str, db:
             .args(["/sys/fs/cgroup/cpu/cpu.cfs_quota_us"])
             .output()
             .ok()
-            .map(|o| String::from_utf8_lossy(&o.stdout).to_string())
+            .map(|o| {
+                String::from_utf8_lossy(&o.stdout)
+                    .to_string()
+                    .trim()
+                    .parse::<i64>()
+                    .ok()
+            })
+            .flatten()
     }
 
-    let mut memory = std::process::Command::new("cat")
-        .args(["/sys/fs/cgroup/memory.max"])
-        .output()
-        .ok()
-        .map(|o| String::from_utf8_lossy(&o.stdout).to_string());
+    vcpus
+}
 
-    if memory.is_none() {
-        memory = std::process::Command::new("cat")
-            .args(["/sys/fs/cgroup/memory/memory.limit_in_bytes"])
-            .output()
-            .ok()
-            .map(|o| String::from_utf8_lossy(&o.stdout).to_string())
+pub fn get_memory() -> Option<i64> {
+    let mut sys = System::new();
+    sys.refresh_memory();
+    let limits = sys.cgroup_limits();
+
+    if let Some(limits) = limits {
+        i64::try_from(limits.total_memory).ok()
+    } else {
+        None
     }
+}
+
+pub fn get_worker_memory_usage() -> Option<i64> {
+    let mut sys = System::new();
+    sys.refresh_memory_specifics(MemoryRefreshKind::new().with_ram());
+    i64::try_from(sys.used_memory()).ok()
+}
+
+pub fn get_windmill_memory_usage() -> Option<i64> {
+    #[cfg(all(not(target_env = "msvc"), feature = "jemalloc"))]
+    {
+        match tikv_jemalloc_ctl::epoch::advance() {
+            Ok(_) => match tikv_jemalloc_ctl::stats::resident::read() {
+                Ok(resident) => i64::try_from(resident).ok(),
+                Err(e) => {
+                    tracing::error!("jemalloc resident memory read failed: {:?}", e);
+                    None
+                }
+            },
+            Err(e) => {
+                tracing::error!("jemalloc epoch advance failed: {:?}", e);
+                None
+            }
+        }
+    }
+
+    #[cfg(any(target_env = "msvc", not(feature = "jemalloc")))]
+    {
+        None
+    }
+}
+
+pub async fn update_ping(worker_instance: &str, worker_name: &str, ip: &str, db: &DB) {
+    let (tags, dw) = {
+        let wc = WORKER_CONFIG.read().await.clone();
+        (
+            wc.worker_tags,
+            wc.dedicated_worker
+                .as_ref()
+                .map(|x| format!("{}:{}", x.workspace_id, x.path)),
+        )
+    };
+
+    let vcpus = get_vcpus();
+    let memory = get_memory();
 
     sqlx::query!(
         "INSERT INTO worker_ping (worker_instance, worker, ip, custom_tags, worker_group, dedicated_worker, wm_version, vcpus, memory) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) ON CONFLICT (worker) DO UPDATE set ip = $3, custom_tags = $4, worker_group = $5",
@@ -188,8 +233,8 @@ pub async fn update_ping(worker_instance: &str, worker_name: &str, ip: &str, db:
         *WORKER_GROUP,
         dw,
         crate::utils::GIT_VERSION,
-        vcpus.map(|x| x.parse::<i64>().ok()).flatten(),
-        memory.map(|x| x.parse::<i64>().ok()).flatten()
+        vcpus,
+        memory
     )
     .execute(db)
     .await

backend/windmill-queue/src/jobs.rs

@@ -40,7 +40,7 @@ use tokio::{sync::RwLock, time::sleep};
 use tracing::{instrument, Instrument};
 use ulid::Ulid;
 use uuid::Uuid;
-use windmill_audit::audit_ee::audit_log;
+use windmill_audit::audit_ee::{audit_log, AuditAuthor};
 use windmill_audit::ActionKind;
 #[cfg(not(feature = "enterprise"))]
 use windmill_common::worker::PriorityTags;
@@ -59,9 +59,9 @@ use windmill_common::{
     },
     oauth2::WORKSPACE_SLACK_BOT_TOKEN_PATH,
     schedule::Schedule,
-    scripts::{ScriptHash, ScriptLang},
+    scripts::{get_full_hub_script_by_path, ScriptHash, ScriptLang},
     users::{SUPERADMIN_NOTIFICATION_EMAIL, SUPERADMIN_SECRET_EMAIL},
-    utils::{not_found_if_none, report_critical_error},
+    utils::{not_found_if_none, report_critical_error, StripPath},
     worker::{to_raw_value, DEFAULT_TAGS_PER_WORKSPACE, NO_LOGS, WORKER_CONFIG},
     BASE_URL, DB, METRICS_ENABLED,
 };
@@ -608,7 +608,7 @@ pub async fn add_completed_job<
     )
     .fetch_one(&mut tx)
     .await
-    .map_err(|e| Error::InternalErr(format!("Could not add completed job {job_id}: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("Could not add completed job {job_id}: {e:#}")))?;
     // tracing::error!("2 {:?}", start.elapsed());
 
     if !queued_job.is_flow_step {
@@ -793,7 +793,7 @@ pub async fn add_completed_job<
         .await
         .map_err(|e| {
             Error::InternalErr(format!(
-                "Error updating to add ended_at timestamp concurrency_key={concurrency_key}: {e}"
+                "Error updating to add ended_at timestamp concurrency_key={concurrency_key}: {e:#}"
             ))
         }) {
             tracing::error!("Could not update concurrency_key: {}", e);
@@ -827,7 +827,7 @@ pub async fn add_completed_job<
             sqlx::query_scalar!("SELECT premium FROM workspace WHERE id = $1", w_id)
                 .fetch_one(db)
                 .await
-                .map_err(|e| Error::InternalErr(format!("fetching if {w_id} is premium: {e}")))?;
+                .map_err(|e| Error::InternalErr(format!("fetching if {w_id} is premium: {e:#}")))?;
         let _ = sqlx::query!(
                 "INSERT INTO usage (id, is_workspace, month_, usage) 
                 VALUES ($1, TRUE, EXTRACT(YEAR FROM current_date) * 12 + EXTRACT(MONTH FROM current_date), $2) 
@@ -836,7 +836,7 @@ pub async fn add_completed_job<
                 additional_usage as i32)
                 .execute(db)
                 .await
-                .map_err(|e| Error::InternalErr(format!("updating usage: {e}")));
+                .map_err(|e| Error::InternalErr(format!("updating usage: {e:#}")));
 
         if !premium_workspace {
             let _ = sqlx::query!(
@@ -847,7 +847,7 @@ pub async fn add_completed_job<
                 additional_usage as i32)
                 .execute(db)
                 .await
-                .map_err(|e| Error::InternalErr(format!("updating usage: {e}")));
+                .map_err(|e| Error::InternalErr(format!("updating usage: {e:#}")));
         }
     }
 
@@ -1708,7 +1708,7 @@ pub async fn pull<R: rsmq_async::RsmqConnection + Send + Clone>(
         .await
         .map_err(|e| {
             Error::InternalErr(format!(
-                "Error getting concurrency count for script path {job_script_path}: {e}"
+                "Error getting concurrency count for script path {job_script_path}: {e:#}"
             ))
         })?;
         tracing::debug!("running_job: {}", running_job.unwrap_or(0));
@@ -1719,7 +1719,7 @@ pub async fn pull<R: rsmq_async::RsmqConnection + Send + Clone>(
             f64::from(job_custom_concurrency_time_window_s),
         ).fetch_one(&mut tx).await.map_err(|e| {
             Error::InternalErr(format!(
-                "Error getting completed count for key {job_concurrency_key}: {e}"
+                "Error getting completed count for key {job_concurrency_key}: {e:#}"
             ))
         })?;
 
@@ -1736,7 +1736,7 @@ pub async fn pull<R: rsmq_async::RsmqConnection + Send + Clone>(
         .await
         .map_err(|e| {
             Error::InternalErr(format!(
-                "Error getting concurrency count for script path {job_script_path}: {e}"
+                "Error getting concurrency count for script path {job_script_path}: {e:#}"
             ))
         })?;
 
@@ -1764,7 +1764,7 @@ pub async fn pull<R: rsmq_async::RsmqConnection + Send + Clone>(
         .await
         .map_err(|e| {
             Error::InternalErr(format!(
-                "Error decreasing concurrency count for script path {job_script_path}: {e}"
+                "Error decreasing concurrency count for script path {job_script_path}: {e:#}"
             ))
         })?;
 
@@ -1772,26 +1772,52 @@ pub async fn pull<R: rsmq_async::RsmqConnection + Send + Clone>(
 
         let job_uuid: Uuid = pulled_job.id;
         let avg_script_duration: Option<i64> = sqlx::query_scalar!(
-            "SELECT CAST(ROUND(AVG(duration_ms) / 1000, 0) AS BIGINT) AS avg_duration_s FROM
-                (SELECT duration_ms FROM completed_job WHERE script_path = $1
-                ORDER BY started_at
+            "SELECT CAST(ROUND(AVG(duration_ms), 0) AS BIGINT) AS avg_duration_s FROM
+                (SELECT duration_ms FROM concurrency_key LEFT JOIN completed_job ON completed_job.id = concurrency_key.job_id WHERE key = $1 AND ended_at IS NOT NULL
+                ORDER BY ended_at
                 DESC LIMIT 10) AS t",
-            job_script_path
+            job_concurrency_key
         )
         .fetch_one(&mut tx)
         .await?;
+        tracing::info!("avg script duration computed: {:?}", avg_script_duration);
 
+        // let before_me = sqlx::query!(
+        //     "SELECT schedu FROM queue WHERE script_path = $1 AND job_kind != 'dependencies' AND running = true AND workspace_id = $2 AND canceled = false AND started_at < $3 ORDER BY started_at DESC LIMIT 1",
+        //     job_script_path,
+        //     &pulled_job.workspace_id,
+        //     min_started_at.now.unwrap()
+        // )
         // optimal scheduling is: 'older_job_in_concurrency_time_window_started_timestamp + script_avg_duration + concurrency_time_window_s'
-        let estimated_next_schedule_timestamp = ((min_started_at
-            .min_started_at
-            .unwrap_or(min_started_at.now.unwrap()))
-            + Duration::try_seconds(avg_script_duration.map(i64::from).unwrap_or(0))
-                .unwrap_or_default()
-                .max(Duration::try_seconds(5).unwrap_or_default())
+        let inc = Duration::try_milliseconds(
+            avg_script_duration.map(|x| i64::from(x + 100)).unwrap_or(0),
+        )
+        .unwrap_or_default()
+        .max(Duration::try_seconds(1).unwrap_or_default())
             + Duration::try_seconds(i64::from(job_custom_concurrency_time_window_s))
-                .unwrap_or_default()
-                .max(Duration::try_seconds(5).unwrap_or_default()))
-        .max(min_started_at.now.unwrap() + Duration::try_seconds(10).unwrap_or_default());
+                .unwrap_or_default();
+
+        let now = min_started_at.now.unwrap();
+        let min_started_p_inc = (min_started_at.min_started_at.unwrap_or(now) + inc)
+            .max(now + Duration::try_seconds(3).unwrap_or_default());
+
+        let mut estimated_next_schedule_timestamp = min_started_p_inc;
+        loop {
+            let nestimated = estimated_next_schedule_timestamp + inc;
+            let jobs_in_window = sqlx::query_scalar!(
+                "SELECT COUNT(*) FROM queue LEFT JOIN concurrency_key ON concurrency_key.job_id = queue.id
+                 WHERE key = $1 AND running = false AND canceled = false AND scheduled_for >= $2 AND scheduled_for < $3",
+                job_concurrency_key,
+                estimated_next_schedule_timestamp,
+                nestimated
+            ).fetch_optional(&mut tx).await?.flatten().unwrap_or(0) as i32;
+            tracing::info!("estimated_next_schedule_timestamp: {:?}, jobs_in_window: {jobs_in_window}, nestimated: {nestimated}, inc: {inc}", estimated_next_schedule_timestamp);
+            if jobs_in_window < job_custom_concurrent_limit {
+                break;
+            } else {
+                estimated_next_schedule_timestamp = nestimated;
+            }
+        }
 
         tracing::info!("Job '{}' from path '{}' with concurrency key '{}' has reached its concurrency limit of {} jobs run in the last {} seconds. This job will be re-queued for next execution at {}", 
             job_uuid, job_script_path,  job_concurrency_key, job_custom_concurrent_limit, job_custom_concurrency_time_window_s, estimated_next_schedule_timestamp);
@@ -1810,12 +1836,13 @@ pub async fn pull<R: rsmq_async::RsmqConnection + Send + Clone>(
                 SET running = false
                 , started_at = null
                 , scheduled_for = '{estimated_next_schedule_timestamp}'
+                , last_ping = null
                 WHERE id = '{job_uuid}'
                 RETURNING tag"
             ))
             .fetch_one(&mut tx)
             .await
-            .map_err(|e| Error::InternalErr(format!("Could not update and re-queue job {job_uuid}. The job will be marked as running but it is not running: {e}")))?;
+            .map_err(|e| Error::InternalErr(format!("Could not update and re-queue job {job_uuid}. The job will be marked as running but it is not running: {e:#}")))?;
 
             if let Some(ref mut rsmq) = tx.rsmq {
                 rsmq.send_message(
@@ -1827,16 +1854,19 @@ pub async fn pull<R: rsmq_async::RsmqConnection + Send + Clone>(
             tx.commit().await?;
         } else {
             // if using posgtres, then we're able to re-queue the entire batch of scheduled job for this script_path, so we do it
-            sqlx::query(&format!(
+            sqlx::query!(
                 "UPDATE queue
                 SET running = false
                 , started_at = null
-                , scheduled_for = '{estimated_next_schedule_timestamp}'
-                WHERE (id = '{job_uuid}') OR (script_path = '{job_script_path}' AND running = false AND scheduled_for <= now())"
-            ))
+                , scheduled_for = $1
+                , last_ping = null
+                WHERE id = $2",
+                estimated_next_schedule_timestamp,
+                job_uuid,
+            )
             .fetch_all(&mut tx)
             .await
-            .map_err(|e| Error::InternalErr(format!("Could not update and re-queue job {job_uuid}. The job will be marked as running but it is not running: {e}")))?;
+            .map_err(|e| Error::InternalErr(format!("Could not update and re-queue job {job_uuid}. The job will be marked as running but it is not running: {e:#}")))?;
             tx.commit().await?
         }
     }
@@ -2088,7 +2118,8 @@ fn interpolate_args(x: String, args: &PushArgs, workspace_id: &str) -> String {
                 .get(arg_name)
                 .or(args.extra.get(arg_name))
                 .map(|x| x.get())
-                .unwrap_or_default();
+                .unwrap_or_default()
+                .trim_matches('"');
             interpolated =
                 interpolated.replace(format!("$args[{}]", arg_name).as_str(), &arg_value);
         }
@@ -2779,7 +2810,7 @@ pub async fn push<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
                 .await
                 .map_err(|e| {
                     Error::InternalErr(format!(
-                        "fetching if {workspace_id} is premium and overquota: {e}"
+                        "fetching if {workspace_id} is premium and overquota: {e:#}"
                     ))
                 })?;
 
@@ -2797,7 +2828,7 @@ pub async fn push<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
                 )
                 .fetch_one(_db)
                 .await
-                .map_err(|e| Error::InternalErr(format!("updating usage: {e}")))?;
+                .map_err(|e| Error::InternalErr(format!("updating usage: {e:#}")))?;
 
             let user_usage = if !premium_workspace {
                 Some(sqlx::query_scalar!(
@@ -2809,7 +2840,7 @@ pub async fn push<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
                 )
                 .fetch_one(_db)
                 .await
-                .map_err(|e| Error::InternalErr(format!("updating usage: {e}")))?)
+                .map_err(|e| Error::InternalErr(format!("updating usage: {e:#}")))?)
             } else {
                 None
             };
@@ -2991,6 +3022,10 @@ pub async fn push<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
                 permissioned_as = SUPERADMIN_NOTIFICATION_EMAIL.to_string();
                 email = SUPERADMIN_NOTIFICATION_EMAIL;
             }
+
+            let hub_script =
+                get_full_hub_script_by_path(StripPath(path.clone()), &HTTP_CLIENT, _db).await?;
+
             (
                 None,
                 Some(path),
@@ -2999,7 +3034,7 @@ pub async fn push<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
                 JobKind::Script_Hub,
                 None,
                 None,
-                None,
+                Some(hub_script.language),
                 None,
                 None,
                 None,
@@ -3430,12 +3465,6 @@ pub async fn push<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
             },
             script_path.clone().expect("dedicated script has a path")
         )
-    } else if job_kind == JobKind::Script_Hub {
-        if per_workspace {
-            format!("hub-{}", workspace_id)
-        } else {
-            "hub".to_string()
-        }
     } else {
         if tag == Some("".to_string()) {
             tag = None;
@@ -3522,7 +3551,7 @@ pub async fn push<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
         )
         .execute(&mut tx)
         .await
-        .map_err(|e| Error::InternalErr(format!("Could not insert concurrency_key={concurrency_key} for job_id={job_id} script_path={script_path:?} workspace_id={workspace_id}: {e}")))?;
+        .map_err(|e| Error::InternalErr(format!("Could not insert concurrency_key={concurrency_key} for job_id={job_id} script_path={script_path:?} workspace_id={workspace_id}: {e:#}")))?;
     }
 
     let uuid = sqlx::query_scalar!(
@@ -3567,7 +3596,7 @@ pub async fn push<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
     )
     .fetch_one(&mut tx)
     .await
-    .map_err(|e| Error::InternalErr(format!("Could not insert into queue {job_id} with tag {tag}, schedule_path {schedule_path:?}, script_path: {script_path:?}, email {email}, workspace_id {workspace_id}: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("Could not insert into queue {job_id} with tag {tag}, schedule_path {schedule_path:?}, script_path: {script_path:?}, email {email}, workspace_id {workspace_id}: {e:#}")))?;
 
     // TODO: technically the job isn't queued yet, as the transaction can be rolled back. Should be solved when moving these metrics to the queue abstraction.
     #[cfg(feature = "prometheus")]
@@ -3600,9 +3629,23 @@ pub async fn push<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
             JobKind::DeploymentCallback => "jobs.run.deployment_callback",
         };
 
+        let audit_author = if format!("u/{user}") != permissioned_as && user != permissioned_as {
+            AuditAuthor {
+                email: email.to_string(),
+                username: permissioned_as.trim_start_matches("u/").to_string(),
+                username_override: Some(user.to_string()),
+            }
+        } else {
+            AuditAuthor {
+                email: email.to_string(),
+                username: user.to_string(),
+                username_override: None,
+            }
+        };
+
         audit_log(
             &mut tx,
-            &user,
+            &audit_author,
             operation_name,
             ActionKind::Execute,
             workspace_id,
@@ -3797,6 +3840,7 @@ async fn restarted_flows_resolution(
                     flow_jobs: _,
                     branch_chosen: _,
                     approvers: _,
+                    failed_retries: _,
                 } => Ok(truncated_modules.push(module)),
                 _ => Err(Error::InternalErr(format!(
                     "Flow cannot be restarted from a non successful module",

backend/windmill-worker/src/bun_executor.rs

@@ -450,7 +450,7 @@ pub async fn generate_wrapper_mjs(
         format!("{job_dir}/wrapper.mjs"),
     )
     .await
-    .map_err(|e| error::Error::InternalErr(format!("Could not move wrapper to mjs: {e}")))?;
+    .map_err(|e| error::Error::InternalErr(format!("Could not move wrapper to mjs: {e:#}")))?;
     Ok(())
 }
 

backend/windmill-worker/src/common.rs

@@ -26,7 +26,10 @@ use windmill_common::s3_helpers::OBJECT_STORE_CACHE_SETTINGS;
 use windmill_common::s3_helpers::{
     get_etag_or_empty, LargeFileStorage, ObjectStoreResource, S3Object,
 };
-use windmill_common::worker::{CLOUD_HOSTED, TMP_DIR, WORKER_CONFIG};
+use windmill_common::variables::{build_crypt_with_key_suffix, decrypt_value_with_mc};
+use windmill_common::worker::{
+    get_windmill_memory_usage, get_worker_memory_usage, CLOUD_HOSTED, TMP_DIR, WORKER_CONFIG,
+};
 use windmill_common::{
     error::{self, Error},
     jobs::QueuedJob,
@@ -146,7 +149,7 @@ pub async fn write_file_binary(dir: &str, path: &str, content: &[u8]) -> error::
 }
 
 lazy_static::lazy_static! {
-    static ref RE_RES_VAR: Regex = Regex::new(r#"\$(?:var|res)\:"#).unwrap();
+    static ref RE_RES_VAR: Regex = Regex::new(r#"\$(?:var|res|encrypted)\:"#).unwrap();
 }
 
 pub async fn transform_json<'a>(
@@ -172,13 +175,13 @@ pub async fn transform_json<'a>(
         let inner_vs = v.get();
         if (*RE_RES_VAR).is_match(inner_vs) {
             let value = serde_json::from_str(inner_vs).map_err(|e| {
-                error::Error::InternalErr(format!("Error while parsing inner arg: {e}"))
+                error::Error::InternalErr(format!("Error while parsing inner arg: {e:#}"))
             })?;
             let transformed =
                 transform_json_value(&k, &client.get_authed().await, workspace, value, job, db)
                     .await?;
             let as_raw = serde_json::from_value(transformed).map_err(|e| {
-                error::Error::InternalErr(format!("Error while parsing inner arg: {e}"))
+                error::Error::InternalErr(format!("Error while parsing inner arg: {e:#}"))
             })?;
             r.insert(k.to_string(), as_raw);
         } else {
@@ -200,13 +203,13 @@ pub async fn transform_json_as_values<'a>(
         let inner_vs = v.get();
         if (*RE_RES_VAR).is_match(inner_vs) {
             let value = serde_json::from_str(inner_vs).map_err(|e| {
-                error::Error::InternalErr(format!("Error while parsing inner arg: {e}"))
+                error::Error::InternalErr(format!("Error while parsing inner arg: {e:#}"))
             })?;
             let transformed =
                 transform_json_value(&k, &client.get_authed().await, workspace, value, job, db)
                     .await?;
             let as_raw = serde_json::from_value(transformed).map_err(|e| {
-                error::Error::InternalErr(format!("Error while parsing inner arg: {e}"))
+                error::Error::InternalErr(format!("Error while parsing inner arg: {e:#}"))
             })?;
             r.insert(k.to_string(), as_raw);
         } else {
@@ -254,7 +257,7 @@ pub async fn transform_json_value(
                 .await
                 .map(|x| json!(x))
                 .map_err(|e| {
-                    Error::NotFound(format!("Variable {path} not found for `{name}`: {e}"))
+                    Error::NotFound(format!("Variable {path} not found for `{name}`: {e:#}"))
                 })
         }
         Value::String(y) if y.starts_with("$res:") => {
@@ -271,9 +274,23 @@ pub async fn transform_json_value(
                 )
                 .await
                 .map_err(|e| {
-                    Error::NotFound(format!("Resource {path} not found for `{name}`: {e}"))
+                    Error::NotFound(format!("Resource {path} not found for `{name}`: {e:#}"))
                 })
         }
+        Value::String(y) if y.starts_with("$encrypted:") => {
+            let encrypted = y.strip_prefix("$encrypted:").unwrap();
+            let mut tx = db.begin().await?;
+            let mc = build_crypt_with_key_suffix(&mut tx, &job.workspace_id, &job.id.to_string())
+                .await?;
+            tx.commit().await?;
+            decrypt_value_with_mc(encrypted.to_string(), mc)
+                .await
+                .and_then(|x| {
+                    serde_json::from_str(&x).map_err(|e| Error::InternalErr(e.to_string()))
+                })
+
+            // let path = y.strip_prefix("$res:").unwrap();
+        }
         Value::String(y) if y.starts_with("$") => {
             let flow_path = if let Some(uuid) = job.parent_job {
                 sqlx::query_scalar!("SELECT script_path FROM queue WHERE id = $1", uuid)
@@ -567,10 +584,15 @@ where
                 // update the last_ping column every 5 seconds
                 i+=1;
                 if i % 10 == 0 {
+                    let memory_usage = get_worker_memory_usage();
+                    let wm_memory_usage = get_windmill_memory_usage();
+                    tracing::info!("{worker_name}/{job_id} in {w_id} worker memory snapshot {}kB/{}kB", memory_usage.unwrap_or_default()/1024, wm_memory_usage.unwrap_or_default()/1024);
                     sqlx::query!(
-                        "UPDATE worker_ping SET ping_at = now(), current_job_id = $1, current_job_workspace_id = $2 WHERE worker = $3",
+                        "UPDATE worker_ping SET ping_at = now(), current_job_id = $1, current_job_workspace_id = $2, memory_usage = $3, wm_memory_usage = $4 WHERE worker = $5",
                         &job_id,
                         &w_id,
+                        memory_usage,
+                        wm_memory_usage,
                         &worker_name
                     )
                     .execute(&db)
@@ -613,7 +635,7 @@ where
                     .fetch_optional(&db)
                     .await
                     .unwrap_or_else(|e| {
-                        tracing::error!(%e, "error updating job {job_id}: {e}");
+                        tracing::error!(%e, "error updating job {job_id}: {e:#}");
                         Some((false, None, None, false))
                     })
                     .unwrap_or_else(|| {
@@ -886,7 +908,7 @@ pub async fn handle_child(
         if let Some(mut file) = File::create(format!("/proc/{pid}/oom_score_adj"))
             .await
             .map_err(|e| {
-                tracing::error!("Could not create oom_score_file to pid {pid}: {e}");
+                tracing::error!("Could not create oom_score_file to pid {pid}: {e:#}");
                 e
             })
             .ok()
@@ -1202,7 +1224,7 @@ pub async fn resolve_job_timeout(
             .fetch_one(_db)
             .await
             .map_err(|e| {
-                tracing::error!(%e, "error getting premium workspace for job {_job_id}: {e}");
+                tracing::error!(%e, "error getting premium workspace for job {_job_id}: {e:#}");
             })
             .unwrap_or(false);
     #[cfg(not(feature = "cloud"))]
@@ -1558,7 +1580,7 @@ pub async fn save_in_cache(
     .execute(db)
     .await
     {
-        tracing::error!("Error creating cache resource {e}")
+        tracing::error!("Error creating cache resource {e:#}")
     }
 }
 

backend/windmill-worker/src/dedicated_worker.rs

@@ -110,7 +110,7 @@ pub async fn handle_dedicated_process(
             .await
             .expect("child process encountered an error");
         if let Err(e) = process_status(status) {
-            tracing::error!("child exit status was not success: {e}");
+            tracing::error!("child exit status was not success: {e:#}");
         } else {
             tracing::info!("child exist status was success");
         }
@@ -203,7 +203,7 @@ pub async fn handle_dedicated_process(
 
     child
         .await
-        .map_err(|e| anyhow::anyhow!("child process encountered an error: {e}"))?;
+        .map_err(|e| anyhow::anyhow!("child process encountered an error: {e:#}"))?;
     tracing::info!("dedicated worker child process exited successfully");
     Ok(())
 }

backend/windmill-worker/src/mysql_executor.rs

@@ -59,7 +59,7 @@ pub async fn do_mysql(
             .await?;
 
         let as_raw = serde_json::from_value(val)
-            .map_err(|e| Error::InternalErr(format!("Error while parsing inline resource: {e}")))?;
+            .map_err(|e| Error::InternalErr(format!("Error while parsing inline resource: {e:#}")))?;
 
         Some(as_raw)
     } else {

backend/windmill-worker/src/pg_executor.rs

@@ -132,7 +132,7 @@ pub async fn do_postgresql(
             if !root_certificate_pem.is_empty() {
                 connector.add_root_certificate(
                     Certificate::from_pem(root_certificate_pem.as_bytes())
-                        .map_err(|e| error::Error::BadConfig(format!("Invalid Certs: {e}")))?,
+                        .map_err(|e| error::Error::BadConfig(format!("Invalid Certs: {e:#}")))?,
                 );
             } else {
                 connector.danger_accept_invalid_certs(true);

backend/windmill-worker/src/worker.rs

@@ -6,7 +6,7 @@
  * LICENSE-AGPL for a copy of the license.
  */
 
-use windmill_common::worker::TMP_DIR;
+use windmill_common::worker::{get_windmill_memory_usage, get_worker_memory_usage, TMP_DIR};
 
 use anyhow::Result;
 use const_format::concatcp;
@@ -127,12 +127,20 @@ pub async fn create_token_for_owner_in_bg(
         let owner = job.permissioned_as.clone();
         let email = job.email.clone();
         let job_id = job.id.clone();
+
+        let label = if job.permissioned_as != format!("u/{}", job.created_by)
+            && job.permissioned_as != job.created_by
+        {
+            format!("ephemeral-script-end-user-{}", job.created_by)
+        } else {
+            "ephemeral-script".to_string()
+        };
         tokio::spawn(async move {
             let token = create_token_for_owner(
                 &db.clone(),
                 &w_id,
                 &owner,
-                "ephemeral-script",
+                &label,
                 *SCRIPT_TOKEN_EXPIRY,
                 &email,
                 &job_id,
@@ -1234,7 +1242,7 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
                     )
                     .await
                     {
-                        tracing::error!("Error updating flow status after job completion for {flow} on {worker_name2}: {e}");
+                        tracing::error!("Error updating flow status after job completion for {flow} on {worker_name2}: {e:#}");
                     }
                 }
                 SendResult::Kill => {
@@ -1367,7 +1375,7 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
             queue_init_bash_maybe(db, same_worker_tx.clone(), &worker_name, rsmq.clone()).await
         {
             killpill_tx.send(()).unwrap_or_default();
-            tracing::error!("Error queuing init bash script for worker {worker_name}: {e}");
+            tracing::error!("Error queuing init bash script for worker {worker_name}: {e:#}");
             return;
         }
     }
@@ -1418,17 +1426,26 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
         if last_ping.elapsed().as_secs() > NUM_SECS_PING {
             let tags = WORKER_CONFIG.read().await.worker_tags.clone();
 
+            let memory_usage = get_worker_memory_usage();
+            let wm_memory_usage = get_windmill_memory_usage();
+
             if let Err(e) = sqlx::query!(
-                "UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2, occupancy_rate = $3, current_job_id = NULL, current_job_workspace_id = NULL WHERE worker = $4",
+                "UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2, occupancy_rate = $3, memory_usage = $4, wm_memory_usage = $5, current_job_id = NULL, current_job_workspace_id = NULL WHERE worker = $6",
                 jobs_executed,
                 tags.as_slice(),
                 worker_code_execution_metric / start_time.elapsed().as_secs_f32(),
+                memory_usage,
+                wm_memory_usage,
                 &worker_name
             ).execute(db).await {
                 tracing::error!("failed to update worker ping, exiting: {}", e);
                 killpill_tx.send(()).unwrap_or_default();
             }
-            tracing::info!("updating last ping");
+            tracing::info!(
+                "ping update and worker memory snapshot {}kB/{}kB",
+                memory_usage.unwrap_or_default() / 1024,
+                wm_memory_usage.unwrap_or_default() / 1024
+            );
 
             last_ping = Instant::now();
         }
@@ -2103,7 +2120,7 @@ async fn spawn_dedicated_worker(
                     .bind(&w_id)
                     .fetch_optional(&db)
                     .await
-                    .map_err(|e| Error::InternalErr(format!("expected content and lock: {e}")))
+                    .map_err(|e| Error::InternalErr(format!("expected content and lock: {e:#}")))
                     .map(|x| x.map(|y| (y.0, y.1, y.2, y.3, if y.4 { y.5.map(|z| z.to_string()) } else { None })))
                 };
                 if let Ok(q) = q {
@@ -2484,7 +2501,7 @@ pub async fn handle_job_error<R: rsmq_async::RsmqConnection + Send + Sync + Clon
         let (flow, job_status_to_update) = if let Some(parent_job_id) = job.parent_job {
             if let Err(e) = update_job_future().await {
                 tracing::error!(
-                    "error updating job future for job {} for handle_job_error: {e}",
+                    "error updating job future for job {} for handle_job_error: {e:#}",
                     job.id
                 );
             }
@@ -2719,7 +2736,7 @@ async fn handle_queued_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>(
             )
             .fetch_one(db)
             .await
-            .map_err(|e| Error::InternalErr(format!("fetching step flow status: {e}")))?
+            .map_err(|e| Error::InternalErr(format!("fetching step flow status: {e:#}")))?
             .ok_or_else(|| Error::InternalErr(format!("Expected script_path")))?;
             let step = step.unwrap_or(-1);
             Some(format!(

backend/windmill-worker/src/worker_flow.rs

@@ -17,7 +17,7 @@ use crate::common::{hash_args, save_in_cache};
 use crate::js_eval::{eval_timeout, IdContext};
 use crate::{AuthedClient, PreviousResult, SameWorkerPayload, SendResult, KEEP_JOB_DIR};
 use anyhow::Context;
-use serde::Serialize;
+use serde::{Deserialize, Serialize};
 use serde_json::value::RawValue;
 use serde_json::{json, Value};
 use sqlx::types::Json;
@@ -117,7 +117,7 @@ pub async fn update_flow_status_after_job_completion<
         {
             Ok(j) => j,
             Err(e) => {
-                tracing::error!("Error while updating flow status of {} after  completion of {}, updating flow status again with error: {e}", nrec.flow,&nrec.job_id_for_status);
+                tracing::error!("Error while updating flow status of {} after  completion of {}, updating flow status again with error: {e:#}", nrec.flow,&nrec.job_id_for_status);
                 update_flow_status_after_job_completion_internal(
                     db,
                     client,
@@ -158,6 +158,10 @@ pub struct SkipIfStopped {
     pub args: Option<Json<HashMap<String, Box<RawValue>>>>,
 }
 
+#[derive(sqlx::FromRow, Deserialize)]
+pub struct RowFlowStatus {
+    pub flow_status: sqlx::types::Json<Box<serde_json::value::RawValue>>,
+}
 // #[instrument(level = "trace", skip_all)]
 pub async fn update_flow_status_after_job_completion_internal<
     'a,
@@ -190,26 +194,26 @@ pub async fn update_flow_status_after_job_completion_internal<
     ) = {
         // tracing::debug!("UPDATE FLOW STATUS: {flow:?} {success} {result:?} {w_id} {depth}");
 
-        let old_status_json = sqlx::query_scalar!(
+        let old_status_json = sqlx::query_as::<_, RowFlowStatus>(
             "SELECT flow_status FROM queue WHERE id = $1 AND workspace_id = $2",
-            flow,
-            w_id
         )
+        .bind(flow)
+        .bind(w_id)
         .fetch_one(db)
         .await
         .map_err(|e| {
             Error::InternalErr(format!(
-                "fetching flow status {flow} while reporting {success} {result:?}: {e}"
+                "fetching flow status {flow} while reporting {success} {result:?}: {e:#}"
             ))
-        })?
-        .ok_or_else(|| Error::InternalErr(format!("requiring a previous status")))?;
-
-        let old_status = serde_json::from_value::<FlowStatus>(old_status_json).or_else(|e| {
-            Err(Error::InternalErr(format!(
-                "requiring status to be parsable as FlowStatus: {e:?}"
-            )))
         })?;
 
+        let old_status = serde_json::from_str::<FlowStatus>(old_status_json.flow_status.get())
+            .or_else(|e| {
+                Err(Error::InternalErr(format!(
+                    "requiring status to be parsable as FlowStatus: {e:?}"
+                )))
+            })?;
+
         let module_index = usize::try_from(old_status.step).ok();
 
         let module_status = module_index
@@ -265,7 +269,7 @@ pub async fn update_flow_status_after_job_completion_internal<
             .bind(flow)
             .fetch_one(db)
             .await
-            .map_err(|e| Error::InternalErr(format!("retrieval of stop_early_expr from state: {e}")))?;
+            .map_err(|e| Error::InternalErr(format!("retrieval of stop_early_expr from state: {e:#}")))?;
             let r = SkipIfStopped::from_row(&row)?;
 
             let stop_early = success
@@ -327,7 +331,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                         .fetch_one(&mut tx)
                         .await.map_err(|e| {
                             Error::InternalErr(format!(
-                                "error while fetching iterator index: {e}"
+                                "error while fetching iterator index: {e:#}"
                             ))
                         })?
                         .ok_or_else(|| Error::InternalErr(format!("requiring an index in InProgress")))?;
@@ -351,7 +355,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                         .await
                         .map_err(|e| {
                             Error::InternalErr(format!(
-                                "error while fetching branchall index: {e}"
+                                "error while fetching branchall index: {e:#}"
                             ))
                         })?
                         .ok_or_else(|| Error::InternalErr(format!("requiring an index in InProgress")))?;
@@ -371,7 +375,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                         .await
                         .map_err(|e| {
                             Error::InternalErr(format!(
-                                "error while fetching sucess from completed_jobs: {e}"
+                                "error while fetching sucess from completed_jobs: {e:#}"
                             ))
                         })?
                         .into_iter()
@@ -384,6 +388,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                             flow_jobs: Some(jobs.clone()),
                             branch_chosen: None,
                             approvers: vec![],
+                            failed_retries: vec![],
                         }
                     } else {
                         success = false;
@@ -392,6 +397,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                             job: job_id_for_status.clone(),
                             flow_jobs: Some(jobs.clone()),
                             branch_chosen: None,
+                            failed_retries: vec![],
                         }
                     };
                     let r = sqlx::query_scalar!(
@@ -399,7 +405,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                         flow,
                     ).fetch_optional(db).await.map_err(|e| {
                         Error::InternalErr(format!(
-                            "error while deleting parallel_monitor_lock: {e}"
+                            "error while deleting parallel_monitor_lock: {e:#}"
                         ))
                     })?;
                     if r.is_some() {
@@ -425,7 +431,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                         .fetch_all(db)
                         .await
                         .map_err(|e| {
-                            Error::InternalErr(format!("error while locking jobs to decrease parallelism of: {e}"))
+                            Error::InternalErr(format!("error while locking jobs to decrease parallelism of: {e:#}"))
                         })?;
                         for id in ids {
                             sqlx::query!(
@@ -436,7 +442,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                             .await
                             .map_err(|e| {
                                 Error::InternalErr(format!(
-                                    "error decreasing suspend for {id}: {e}"
+                                    "error decreasing suspend for {id}: {e:#}"
                                 ))
                             })?;
                         }
@@ -451,7 +457,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                     .execute(db)
                     .await
                     .map_err(|e| {
-                        Error::InternalErr(format!("error while setting last ping to null: {e}"))
+                        Error::InternalErr(format!("error while setting last ping to null: {e:#}"))
                     })?;
 
                     let r = sqlx::query_scalar!(
@@ -459,7 +465,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                         flow,
                         job_id_for_status
                     ).fetch_optional(db).await.map_err(|e| {
-                        Error::InternalErr(format!("error while removing parallel_monitor_lock: {e}"))
+                        Error::InternalErr(format!("error while removing parallel_monitor_lock: {e:#}"))
                     })?;
                     if r.is_some() {
                         tracing::info!(
@@ -517,6 +523,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                             flow_jobs,
                             branch_chosen,
                             approvers: vec![],
+                            failed_retries: old_status.retry.failed_jobs.clone(),
                         }),
                     )
                 } else {
@@ -529,7 +536,9 @@ pub async fn update_flow_status_after_job_completion_internal<
                         .fetch_optional(&mut tx)
                         .await
                         .map_err(|e| {
-                            Error::InternalErr(format!("error while getting retry fromn step: {e}"))
+                            Error::InternalErr(format!(
+                                "error while getting retry from step: {e:#}"
+                            ))
                         })?
                         .flatten();
 
@@ -549,6 +558,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                             job: job_id_for_status.clone(),
                             flow_jobs,
                             branch_chosen,
+                            failed_retries: old_status.retry.failed_jobs.clone(),
                         }),
                     )
                 }
@@ -566,7 +576,7 @@ pub async fn update_flow_status_after_job_completion_internal<
             .execute(&mut tx)
             .await
             .map_err(|e| {
-                Error::InternalErr(format!("error while setting flow index for {flow}: {e}"))
+                Error::InternalErr(format!("error while setting flow index for {flow}: {e:#}"))
             })?;
             old_status.step + 1
         } else {
@@ -587,7 +597,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                 .fetch_one(&mut tx)
                 .await.map_err(|e| {
                     Error::InternalErr(format!(
-                        "error while fetching failure module: {e}"
+                        "error while fetching failure module: {e:#}"
                     ))
                 })?;
 
@@ -605,7 +615,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                 .await
                 .map_err(|e| {
                     Error::InternalErr(format!(
-                        "error while setting flow status in failure step: {e}"
+                        "error while setting flow status in failure step: {e:#}"
                     ))
                 })?;
             } else {
@@ -620,7 +630,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                 .execute(&mut tx)
                 .await
                 .map_err(|e| {
-                    Error::InternalErr(format!("error while setting new flow status: {e}"))
+                    Error::InternalErr(format!("error while setting new flow status: {e:#}"))
                 })?;
 
                 if let Some(job_result) = new_status.job_result() {
@@ -635,7 +645,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                     .execute(&mut tx)
                     .await.map_err(|e| {
                         Error::InternalErr(format!(
-                            "error while setting leaf jobs: {e}"
+                            "error while setting leaf jobs: {e:#}"
                         ))
                     })?;
                 }
@@ -751,7 +761,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                 .execute(db)
                 .await
                 .map_err(|e| {
-                    Error::InternalErr(format!("error while cleaning up completed_job: {e}"))
+                    Error::InternalErr(format!("error while cleaning up completed_job: {e:#}"))
                 })?;
             }
         }
@@ -813,7 +823,7 @@ pub async fn update_flow_status_after_job_completion_internal<
                     stop_early && skip_if_stop_early,
                     Json(
                         &serde_json::from_str::<Value>(nresult.get()).unwrap_or_else(
-                            |e| json!({"error": format!("Impossible to serialize error: {e}")}),
+                            |e| json!({"error": format!("Impossible to serialize error: {e:#}")}),
                         ),
                     ),
                     0,
@@ -958,7 +968,7 @@ async fn compute_skip_loop_failures_and_parallelism(
     .fetch_one(db)
     .await
     .map(|(v, n)| (v,n))
-    .map_err(|e| Error::InternalErr(format!("error during retrieval of skip_loop_failures: {e}")))
+    .map_err(|e| Error::InternalErr(format!("error during retrieval of skip_loop_failures: {e:#}")))
 }
 
 async fn compute_skip_branchall_failure<'c>(
@@ -978,7 +988,11 @@ async fn compute_skip_branchall_failure<'c>(
     .fetch_one(db)
     .await
     .map(|(v,)| v)
-    .map_err(|e| Error::InternalErr(format!("error during retrieval of skip_loop_failures: {e}")))
+    .map_err(|e| {
+        Error::InternalErr(format!(
+            "error during retrieval of skip_loop_failures: {e:#}"
+        ))
+    })
 }
 
 async fn has_failure_module<'c>(
@@ -993,7 +1007,11 @@ async fn has_failure_module<'c>(
     .bind(flow)
     .fetch_one(&mut **tx)
     .await
-    .map_err(|e| Error::InternalErr(format!("error during retrieval of has_failure_module: {e}")))
+    .map_err(|e| {
+        Error::InternalErr(format!(
+            "error during retrieval of has_failure_module: {e:#}"
+        ))
+    })
     .map(|v| v.unwrap_or(false))
 }
 
@@ -1007,7 +1025,7 @@ async fn has_failure_module<'c>(
 //     )
 //     .fetch_one(db)
 //     .await
-//     .map_err(|e| Error::InternalErr(format!("error during retrieval of cleanup module: {e}")))?;
+//     .map_err(|e| Error::InternalErr(format!("error during retrieval of cleanup module: {e:#}")))?;
 
 //     raw_value
 //         .clone()
@@ -1109,7 +1127,7 @@ pub async fn get_step_of_flow_status(db: &DB, id: Uuid) -> error::Result<Step> {
     )
     .fetch_one(db)
     .await
-    .map_err(|e| Error::InternalErr(format!("fetching step flow status: {e}")))?;
+    .map_err(|e| Error::InternalErr(format!("fetching step flow status: {e:#}")))?;
     if r.step < r.len {
         Ok(Step::Step(r.step.ok_or_else(|| {
             Error::InternalErr("step is null".to_string())
@@ -1168,7 +1186,7 @@ async fn transform_input(
                 .await
                 .map_err(|e| {
                     Error::ExecutionErr(format!(
-                        "Error during isolated evaluation of expression `{expr}`:\n{e}"
+                        "Error during isolated evaluation of expression `{expr}`:\n{e:#}"
                     ))
                 })?;
                 mapped.insert(key.to_string(), v);
@@ -1351,7 +1369,7 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
             .await
             .map_err(|e| {
                 Error::InternalErr(format!(
-                    "error sending update flow message to job completed channel: {e}"
+                    "error sending update flow message to job completed channel: {e:#}"
                 ))
             })?;
 
@@ -1398,7 +1416,7 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                         .await
                         .map_err(|e| {
                             Error::InternalErr(format!(
-                                "error sending update flow message to job completed channel: {e}"
+                                "error sending update flow message to job completed channel: {e:#}"
                             ))
                         })?;
 
@@ -1430,7 +1448,7 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                     .await
                     .map_err(|e| {
                         Error::InternalErr(format!(
-                            "error sending update flow message to job completed channel: {e}"
+                            "error sending update flow message to job completed channel: {e:#}"
                         ))
                     })?;
 
@@ -1539,7 +1557,7 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                                 .await
                                 .map_err(|e| {
                                     Error::ExecutionErr(format!(
-                                        "Error during isolated evaluation of expression `{expr}`:\n{e}"
+                                        "Error during isolated evaluation of expression `{expr}`:\n{e:#}"
                                     ))
                                 })?
                                 .get(),
@@ -1549,7 +1567,7 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                             } else {
                                 let e = eval_result.err().unwrap();
                                 return Err(Error::ExecutionErr(format!(
-                                    "Result returned by input transform invalid `{e}`"
+                                    "Result returned by input transform invalid `{e:#}`"
                                 )));
                             }
                         }
@@ -1678,7 +1696,7 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                     .await
                     .map_err(|e| {
                         Error::InternalErr(format!(
-                            "error sending update flow message to job completed channel: {e}"
+                            "error sending update flow message to job completed channel: {e:#}"
                         ))
                     })?;
 
@@ -1734,7 +1752,7 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                             .await
                             .map_err(|e| {
                                 Error::ExecutionErr(format!(
-                                    "Error during isolated evaluation of expression `{expr}`:\n{e}"
+                                    "Error during isolated evaluation of expression `{expr}`:\n{e:#}"
                                 ))
                             })?
                             .get(),
@@ -1788,11 +1806,13 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                 status.retry.failed_jobs.push(job.clone());
                 sqlx::query(
                     "UPDATE queue
-                    SET flow_status = JSONB_SET(flow_status, ARRAY['retry'], $1)
+                    SET flow_status = JSONB_SET(JSONB_SET(flow_status, ARRAY['retry'], $1), ARRAY['modules', $3::TEXT, 'failed_retries'], $4)
                     WHERE id = $2",
                 )
                 .bind(json!(RetryStatus { fail_count, ..status.retry.clone() }))
                 .bind(flow_job.id)
+                .bind(status.step)
+                .bind(json!(status.retry.failed_jobs))
                 .execute(db)
                 .await
                 .context("update flow retry")?;
@@ -1819,9 +1839,7 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                     .context("missing failure module")?;
                 status_module = status.failure_module.module_status.clone();
 
-                /* (retry feature) save the previous_result the first time this step is run */
-                let retry = &module.retry.clone().unwrap_or_default();
-                if retry.has_attempts() {
+                if module.retry.as_ref().is_some_and(|x| x.has_attempts()) {
                     sqlx::query(
                         "UPDATE queue
                         SET flow_status = JSONB_SET(flow_status, ARRAY['retry'], $1)
@@ -1835,28 +1853,6 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                 };
                 None
             }
-
-            /* (retry feature) save the previous_result the first time this step is run */
-        }
-        FlowStatusModule::WaitingForPriorSteps { .. }
-            if module
-                .retry
-                .as_ref()
-                .map(|x| x.has_attempts())
-                .unwrap_or(false)
-                && status.retry.fail_count == 0 =>
-        {
-            sqlx::query(
-                "UPDATE queue
-                SET flow_status = JSONB_SET(flow_status, ARRAY['retry'], $1)
-                WHERE id = $2",
-            )
-            .bind(json!(RetryStatus { fail_count: 0, failed_jobs: vec![] }))
-            .bind(flow_job.id)
-            .execute(db)
-            .await
-            .context("update flow retry")?;
-            None
         }
         _ => None,
     };
@@ -1895,7 +1891,7 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
         if let Some(row) = row {
             RawArgs::from_row(&row)
                 .map(|x| x.args.map(|x| x.0).unwrap_or_else(HashMap::new))
-                .map_err(|e| error::Error::InternalErr(format!("Impossible to build args: {e}")))
+                .map_err(|e| error::Error::InternalErr(format!("Impossible to build args: {e:#}")))
         } else {
             Ok(HashMap::new())
         }
@@ -1927,7 +1923,7 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                 })
                 .unwrap(),
             )
-            .map_err(|e| error::Error::InternalErr(format!("identity: {e}"))),
+            .map_err(|e| error::Error::InternalErr(format!("identity: {e:#}"))),
 
             Ok(_) => Ok(flow_job_args),
             Err(e) => {
@@ -1972,7 +1968,8 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                 job: Uuid::nil(),
                 flow_jobs: Some(vec![]),
                 branch_chosen: None,
-                approvers: vec![]
+                approvers: vec![],
+                failed_retries: vec![],
             }))
             .bind(flow_job.id)
             .execute(db)
@@ -2164,8 +2161,9 @@ async fn push_next_flow_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
         }
 
         if payload_tag.delete_after_use {
-            let uuid_singleton_json = serde_json::to_value(&[uuid])
-                .map_err(|e| error::Error::InternalErr(format!("Unable to serialize uuid: {e}")))?;
+            let uuid_singleton_json = serde_json::to_value(&[uuid]).map_err(|e| {
+                error::Error::InternalErr(format!("Unable to serialize uuid: {e:#}"))
+            })?;
 
             sqlx::query(
                 "UPDATE queue
@@ -2963,6 +2961,7 @@ fn is_simple_modules(modules: &Vec<FlowModule>, flow: &FlowValue) -> bool {
         && modules[0].sleep.is_none()
         && modules[0].suspend.is_none()
         && modules[0].cache_ttl.is_none()
+        && modules[0].retry.is_none()
         && modules[0].stop_after_if.is_none()
         && (modules[0].mock.is_none() || modules[0].mock.as_ref().is_some_and(|m| !m.enabled))
         && flow.failure_module.is_none();

benchmarks/lib.ts

@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
 import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
 import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
 
-export const VERSION = "v1.335.0";
+export const VERSION = "v1.339.0";
 
 export async function login(email: string, password: string): Promise<string> {
   return await windmill.UserService.login({

cli/main.ts

@@ -33,7 +33,7 @@ addEventListener("error", (event) => {
   }
 });
 
-export const VERSION = "v1.335.0";
+export const VERSION = "v1.339.0";
 
 let command: any = new Command()
   .name("wmill")

frontend/package-lock.json

@@ -1,12 +1,12 @@
 {
 	"name": "windmill-components",
-	"version": "1.335.0",
+	"version": "1.339.0",
 	"lockfileVersion": 3,
 	"requires": true,
 	"packages": {
 		"": {
 			"name": "windmill-components",
-			"version": "1.335.0",
+			"version": "1.339.0",
 			"license": "AGPL-3.0",
 			"dependencies": {
 				"@aws-crypto/sha256-js": "^4.0.0",
@@ -38,7 +38,7 @@
 				"monaco-graphql": "^1.5.1",
 				"monaco-languageclient": "~7.0.1",
 				"ol": "^7.4.0",
-				"openai": "^4.3.0",
+				"openai": "^4.47.1",
 				"pdfjs-dist": "^3.8.162",
 				"quill": "^1.3.7",
 				"svelte-carousel": "^1.0.25",
@@ -53,7 +53,7 @@
 				"vscode-languageclient": "~9.0.1",
 				"vscode-uri": "~3.0.8",
 				"vscode-ws-jsonrpc": "~3.1.0",
-				"windmill-parser-wasm": "^1.327.0",
+				"windmill-parser-wasm": "^1.335.0",
 				"windmill-sql-datatype-parser-wasm": "^1.318.0",
 				"y-monaco": "^0.1.4",
 				"y-websocket": "^1.5.0",
@@ -2514,11 +2514,6 @@
 			"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
 			"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw=="
 		},
-		"node_modules/base-64": {
-			"version": "0.1.0",
-			"resolved": "https://registry.npmjs.org/base-64/-/base-64-0.1.0.tgz",
-			"integrity": "sha512-Y5gU45svrR5tI2Vt/X9GPd3L0HNIKzGu202EjxrXMpuc2V2CiKgemAbUUsqYmZJvPtCXoUKjNZwBJzsNScUbXA=="
-		},
 		"node_modules/base64-js": {
 			"version": "1.5.1",
 			"resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
@@ -2831,14 +2826,6 @@
 				"url": "https://github.com/sponsors/wooorm"
 			}
 		},
-		"node_modules/charenc": {
-			"version": "0.0.2",
-			"resolved": "https://registry.npmjs.org/charenc/-/charenc-0.0.2.tgz",
-			"integrity": "sha512-yrLQ/yVUFXkzg7EDQsPieE/53+0RlaWTs+wBrvW36cyilJ2SaDWfl4Yj7MtLTXleV9uEKefbAGUPv2/iWSooRA==",
-			"engines": {
-				"node": "*"
-			}
-		},
 		"node_modules/chart.js": {
 			"version": "4.4.0",
 			"resolved": "https://registry.npmjs.org/chart.js/-/chart.js-4.4.0.tgz",
@@ -3053,14 +3040,6 @@
 				"node": ">= 8"
 			}
 		},
-		"node_modules/crypt": {
-			"version": "0.0.2",
-			"resolved": "https://registry.npmjs.org/crypt/-/crypt-0.0.2.tgz",
-			"integrity": "sha512-mCxBlsHFYh9C+HVpiEacem8FEBnMXgU9gy4zmNC+SXAZNB/1idgp/aulFJ4FgCi7GPEVbfyng092GqL2k2rmow==",
-			"engines": {
-				"node": "*"
-			}
-		},
 		"node_modules/css-declaration-sorter": {
 			"version": "6.4.1",
 			"resolved": "https://registry.npmjs.org/css-declaration-sorter/-/css-declaration-sorter-6.4.1.tgz",
@@ -3639,15 +3618,6 @@
 				"node": ">=0.3.1"
 			}
 		},
-		"node_modules/digest-fetch": {
-			"version": "1.3.0",
-			"resolved": "https://registry.npmjs.org/digest-fetch/-/digest-fetch-1.3.0.tgz",
-			"integrity": "sha512-CGJuv6iKNM7QyZlM2T3sPAdZWd/p9zQiRNS9G+9COUCwzWFTs0Xp8NF5iePx7wtvhDykReiRRrSeNb4oMmB8lA==",
-			"dependencies": {
-				"base-64": "^0.1.0",
-				"md5": "^2.3.0"
-			}
-		},
 		"node_modules/dir-glob": {
 			"version": "3.0.1",
 			"resolved": "https://registry.npmjs.org/dir-glob/-/dir-glob-3.0.1.tgz",
@@ -5113,11 +5083,6 @@
 				"node": ">=8"
 			}
 		},
-		"node_modules/is-buffer": {
-			"version": "1.1.6",
-			"resolved": "https://registry.npmjs.org/is-buffer/-/is-buffer-1.1.6.tgz",
-			"integrity": "sha512-NcdALwpXkTm5Zvvbk7owOUSvVvBKDgKP5/ewfXEznmQFfs4ZRmanOeKBTjRVjka3QFoN6XJ+9F3USqfHqTaU5w=="
-		},
 		"node_modules/is-core-module": {
 			"version": "2.13.1",
 			"resolved": "https://registry.npmjs.org/is-core-module/-/is-core-module-2.13.1.tgz",
@@ -5749,16 +5714,6 @@
 				"url": "https://github.com/sponsors/wooorm"
 			}
 		},
-		"node_modules/md5": {
-			"version": "2.3.0",
-			"resolved": "https://registry.npmjs.org/md5/-/md5-2.3.0.tgz",
-			"integrity": "sha512-T1GITYmFaKuO91vxyoQMFETst+O71VUPEU3ze5GNzDm0OWdP8v1ziTaAEPUr/3kLsY3Sftgz242A1SetQiDL7g==",
-			"dependencies": {
-				"charenc": "0.0.2",
-				"crypt": "0.0.2",
-				"is-buffer": "~1.1.6"
-			}
-		},
 		"node_modules/mdast-util-find-and-replace": {
 			"version": "3.0.1",
 			"resolved": "https://registry.npmjs.org/mdast-util-find-and-replace/-/mdast-util-find-and-replace-3.0.1.tgz",
@@ -7152,15 +7107,14 @@
 			}
 		},
 		"node_modules/openai": {
-			"version": "4.19.0",
-			"resolved": "https://registry.npmjs.org/openai/-/openai-4.19.0.tgz",
-			"integrity": "sha512-cJbl0noZyAaXVKBTMMq6X5BAvP1pm2rWYDBnZes99NL+Zh5/4NmlAwyuhTZEru5SqGGZIoiYKeMPXy4bm9DI0w==",
+			"version": "4.47.1",
+			"resolved": "https://registry.npmjs.org/openai/-/openai-4.47.1.tgz",
+			"integrity": "sha512-WWSxhC/69ZhYWxH/OBsLEirIjUcfpQ5+ihkXKp06hmeYXgBBIUCa9IptMzYx6NdkiOCsSGYCnTIsxaic3AjRCQ==",
 			"dependencies": {
 				"@types/node": "^18.11.18",
 				"@types/node-fetch": "^2.6.4",
 				"abort-controller": "^3.0.0",
 				"agentkeepalive": "^4.2.1",
-				"digest-fetch": "^1.3.0",
 				"form-data-encoder": "1.7.2",
 				"formdata-node": "^4.3.2",
 				"node-fetch": "^2.6.7",
@@ -10310,9 +10264,9 @@
 			}
 		},
 		"node_modules/windmill-parser-wasm": {
-			"version": "1.327.0",
-			"resolved": "https://registry.npmjs.org/windmill-parser-wasm/-/windmill-parser-wasm-1.327.0.tgz",
-			"integrity": "sha512-lliouG5syBSwaPDN3C5fd2whqZmD5kW7S+gXBtunnMwIydPfXiHqWTL4F2zvbHhcHhiIgN04ATxL7gpxcMP1wg=="
+			"version": "1.335.0",
+			"resolved": "https://registry.npmjs.org/windmill-parser-wasm/-/windmill-parser-wasm-1.335.0.tgz",
+			"integrity": "sha512-a6UzDKRt1InP0FgZHZ/ZhkoEI+AgCXirZ4yq9hTgL4o9PGRId9pl8Nk18R3CEJ9XaCd1LX6+GbRppnBtThyrvA=="
 		},
 		"node_modules/windmill-sql-datatype-parser-wasm": {
 			"version": "1.318.0",

frontend/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "windmill-components",
-	"version": "1.335.0",
+	"version": "1.339.0",
 	"scripts": {
 		"dev": "vite dev",
 		"build": "vite build",
@@ -118,7 +118,7 @@
 		"monaco-graphql": "^1.5.1",
 		"monaco-languageclient": "~7.0.1",
 		"ol": "^7.4.0",
-		"openai": "^4.3.0",
+		"openai": "^4.47.1",
 		"pdfjs-dist": "^3.8.162",
 		"quill": "^1.3.7",
 		"svelte-carousel": "^1.0.25",
@@ -133,7 +133,7 @@
 		"vscode-languageclient": "~9.0.1",
 		"vscode-uri": "~3.0.8",
 		"vscode-ws-jsonrpc": "~3.1.0",
-		"windmill-parser-wasm": "^1.327.0",
+		"windmill-parser-wasm": "^1.335.0",
 		"windmill-sql-datatype-parser-wasm": "^1.318.0",
 		"y-monaco": "^0.1.4",
 		"y-websocket": "^1.5.0",

frontend/src/lib/components/ApiConnectForm.svelte

@@ -37,9 +37,8 @@
 			notFound = true
 		}
 	}
-	$: {
-		$workspaceStore && loadSchema()
-	}
+	$: $workspaceStore && loadSchema()
+
 	$: notFound && rawCode && parseJson()
 
 	function parseJson() {

frontend/src/lib/components/AppConnect.svelte

@@ -1,631 +0,0 @@
-<script lang="ts" context="module">
-	const apiTokenApps: Record<
-		string,
-		{ img?: string; instructions: string[]; linkedSecret?: string }
-	> = {
-		airtable: {
-			img: '/airtable_connect.png',
-			instructions: [
-				'Go to <a href="https://airtable.com/create/tokens" target="_blank" rel=”noopener noreferrer”>https://airtable.com/create/tokens</a>',
-				'Click on "Create new token"',
-				'Set a name, specify the scopes or the access level and click on "Create token"',
-				'Copy the token'
-			]
-		},
-		discord_webhook: {
-			img: '/discord_webhook.png',
-			instructions: ['Click on Server Settings', 'Click on Integration', 'Find "Webhooks"'],
-			linkedSecret: 'webhook_url'
-		},
-		toggl: {
-			img: '/toggl_connect.png',
-			instructions: [
-				'Go to <a href="https://track.toggl.com/profile" target="_blank" rel=”noopener noreferrer”>https://track.toggl.com/profile</a>',
-				'Find "API Token"'
-			]
-		},
-		mailchimp: {
-			img: '/mailchimp_connect.png',
-			instructions: [
-				'Go to <a href="https://admin.mailchimp.com/account/api" target="_blank" rel=”noopener noreferrer”>https://admin.mailchimp.com/account/api</a>',
-				'Find "Your API Keys"'
-			]
-		},
-		sendgrid: {
-			img: '/sendgrid_connect.png',
-			instructions: [
-				'Go to <a href="https://app.sendgrid.com/settings/api_keys" target="_blank" rel=”noopener noreferrer”>https://app.sendgrid.com/settings/api_keys</a>',
-				'Create an API key',
-				'Copy your key'
-			]
-		},
-		supabase: {
-			img: '/supabase_connect.png',
-			instructions: ['Go to the API Settings of your app to find the project URL and key']
-		},
-
-		square: {
-			img: '/square_connect.gif',
-			instructions: [
-				'Go to <a href="https://developer.squareup.com/apps" target="_blank" rel=”noopener noreferrer”>https://developer.squareup.com/apps</a>',
-				'In the left pane, choose Credentials',
-				'At the top of the page, choose Production mode for a production access token or Sandbox mode for a Sandbox access token.'
-			]
-		}
-	}
-</script>
-
-<script lang="ts">
-	import { oauthStore, workspaceStore } from '$lib/stores'
-	import IconedResourceType from './IconedResourceType.svelte'
-	import {
-		OauthService,
-		ResourceService,
-		VariableService,
-		type TokenResponse,
-		type ResourceType
-	} from '$lib/gen'
-	import { emptyString, truncateRev, urlize } from '$lib/utils'
-	import { createEventDispatcher } from 'svelte'
-	import Path from './Path.svelte'
-	import { Button, Drawer, Skeleton } from './common'
-	import DrawerContent from './common/drawer/DrawerContent.svelte'
-	import ApiConnectForm from './ApiConnectForm.svelte'
-	import SearchItems from './SearchItems.svelte'
-	import WhitelistIp from './WhitelistIp.svelte'
-	import { sendUserToast } from '$lib/toast'
-	import OauthScopes from './OauthScopes.svelte'
-	import DarkModeObserver from './DarkModeObserver.svelte'
-	import Markdown from 'svelte-exmarkdown'
-	import autosize from '$lib/autosize'
-	import Required from './Required.svelte'
-	import Toggle from './Toggle.svelte'
-	import { Pen } from 'lucide-svelte'
-	import GfmMarkdown from './GfmMarkdown.svelte'
-
-	export let newPageOAuth = false
-
-	const nativeLanguagesCategory = [
-		'postgresql',
-		'mysql',
-		'bigquery',
-		'snowflake',
-		'mssql',
-		'graphql'
-	]
-
-	let filter = ''
-	let manual = false
-	let value: string = ''
-	let valueToken: TokenResponse | undefined = undefined
-	let connects:
-		| Record<string, { scopes: string[]; extra_params?: Record<string, string> }>
-		| undefined = undefined
-	let connectsManual:
-		| [string, { img?: string; instructions: string[]; key?: string }][]
-		| undefined = undefined
-	let args: any = {}
-	let renderDescription = true
-
-	$: linkedSecretCandidates = apiTokenApps[resourceType]?.linkedSecret
-		? ([apiTokenApps[resourceType]?.linkedSecret] as string[])
-		: args != undefined
-		? Object.keys(args).filter((x) =>
-				['token', 'secret', 'key', 'pass', 'private'].some((y) => x.toLowerCase().includes(y))
-		  )
-		: undefined
-
-	function linkedSecretValue(x: string) {
-		let r = 0
-		let lowerCasedX = x.toLowerCase()
-		if (lowerCasedX.includes('secret')) {
-			r += 10
-		}
-		if (lowerCasedX.includes('password')) {
-			r += 5
-		}
-		if (lowerCasedX.includes('private')) {
-			r += 4
-		}
-		if (lowerCasedX.includes('key')) {
-			r += 3
-		}
-		if (lowerCasedX.includes('token')) {
-			r += 2
-		}
-		if (lowerCasedX.includes('pass')) {
-			r += 1
-		}
-		return r
-	}
-
-	function forceSecretValue(resourceType: string): string | undefined {
-		if (resourceType == 'git_repository') {
-			return 'url'
-		}
-	}
-	$: linkedSecret =
-		forceSecretValue(resourceType) ??
-		linkedSecretCandidates?.sort((ua, ub) => linkedSecretValue(ub) - linkedSecretValue(ua))?.[0]
-
-	let scopes: string[] = []
-	let extra_params: [string, string][] = []
-
-	let path: string
-	let description = ''
-
-	let drawer: Drawer
-	let resourceType = ''
-	let resourceTypeInfo: ResourceType | undefined = undefined
-	let step = 1
-
-	let no_back = false
-
-	let pathError = ''
-
-	export async function open(rt?: string) {
-		step = 1
-		value = ''
-		description = ''
-		no_back = false
-		resourceType = rt ?? ''
-		valueToken = undefined
-		await loadConnects()
-
-		const connect = connects?.[resourceType]
-		if (connect) {
-			scopes = connect.scopes
-			extra_params = Object.entries(connect.extra_params ?? {})
-		} else {
-			manual = true
-			if (rt) {
-				next()
-			}
-		}
-		drawer.openDrawer?.()
-	}
-
-	export function openFromOauth(rt: string) {
-		resourceType = rt
-		value = $oauthStore?.access_token!
-		valueToken = $oauthStore!
-		$oauthStore = undefined
-		manual = false
-		step = 3
-		no_back = true
-		drawer.openDrawer?.()
-	}
-
-	async function loadConnects() {
-		const nconnects = (await OauthService.listOauthConnects()) as any
-		if (nconnects['supabase_wizard']) {
-			delete nconnects['supabase_wizard']
-		}
-		connects = nconnects
-	}
-
-	const connectAndManual = ['gitlab']
-
-	async function loadResources() {
-		await loadConnects()
-		const availableRts = await ResourceService.listResourceTypeNames({
-			workspace: $workspaceStore!
-		})
-
-		connectsManual = availableRts
-			.filter((x) => connectAndManual.includes(x) || !Object.keys(connects ?? {}).includes(x))
-			.map((x) => [
-				x,
-				apiTokenApps[x] ?? {
-					instructions: '',
-					img: undefined,
-					linkedSecret: undefined
-				}
-			])
-		const filteredNativeLanguages = filteredConnectsManual?.filter(
-			(o) => nativeLanguagesCategory?.includes(o[0]) ?? false
-		)
-
-		try {
-			filteredConnectsManual = [
-				...(filteredNativeLanguages ?? []),
-				...(filteredConnectsManual ?? []).filter(
-					([key, _]) => !nativeLanguagesCategory.includes(key)
-				)
-			]
-		} catch (e) {}
-	}
-
-	async function next() {
-		if (step == 1 && manual) {
-			resourceTypeInfo = await ResourceService.getResourceType({
-				workspace: $workspaceStore!,
-				path: resourceType
-			})
-			step += 1
-			args = {}
-		} else if (step == 1 && !manual) {
-			const url = new URL(`/api/oauth/connect/${resourceType}`, window.location.origin)
-			url.searchParams.append('scopes', scopes.join('+'))
-			if (extra_params.length > 0) {
-				extra_params.forEach(([key, value]) => url.searchParams.append(key, value))
-			}
-			if (!newPageOAuth) {
-				window.location.href = url.toString()
-			} else {
-				window.open(url.toString(), '_blank')
-				drawer.closeDrawer()
-			}
-		} else {
-			let exists = await VariableService.existsVariable({
-				workspace: $workspaceStore!,
-				path
-			})
-			if (exists) {
-				throw Error(`Variable at path ${path} already exists. Delete it or pick another path`)
-			}
-			exists = await ResourceService.existsResource({
-				workspace: $workspaceStore!,
-				path
-			})
-
-			if (exists) {
-				throw Error(`Resource at path ${path} already exists. Delete it or pick another path`)
-			}
-
-			let account: number | undefined = undefined
-			if (valueToken?.expires_in != undefined) {
-				account = Number(
-					await OauthService.createAccount({
-						workspace: $workspaceStore!,
-						requestBody: {
-							refresh_token: valueToken.refresh_token ?? '',
-							expires_in: valueToken.expires_in,
-							client: resourceType
-						}
-					})
-				)
-			}
-
-			const resourceValue = args
-
-			let saveVariable = false
-			if (!manual || linkedSecret != undefined) {
-				let v = manual ? args[linkedSecret ?? ''] : value
-				if (typeof v == 'string' && v != '' && !v.startsWith('$var:')) {
-					saveVariable = true
-					await VariableService.createVariable({
-						workspace: $workspaceStore!,
-						requestBody: {
-							path,
-							value: v,
-							is_secret: true,
-							description: emptyString(description)
-								? `${manual ? 'Token' : 'OAuth token'} for ${resourceType}`
-								: description,
-							is_oauth: !manual,
-							account: account
-						}
-					})
-					resourceValue[linkedSecret ?? 'token'] = `$var:${path}`
-				}
-			}
-
-			await ResourceService.createResource({
-				workspace: $workspaceStore!,
-				requestBody: {
-					resource_type: resourceType,
-					path,
-					value: resourceValue,
-					description
-				}
-			})
-			dispatch('refresh', path)
-			sendUserToast(`Saved resource${saveVariable ? ' and variable' : ''} path: ${path}`)
-			drawer.closeDrawer?.()
-		}
-	}
-
-	async function back() {
-		if (step > 1) {
-			step -= 1
-		}
-	}
-
-	const dispatch = createEventDispatcher()
-
-	$: isGoogleSignin =
-		step == 1 &&
-		(resourceType == 'google' ||
-			resourceType == 'gmail' ||
-			resourceType == 'gcal' ||
-			resourceType == 'gdrive' ||
-			resourceType == 'gsheets')
-
-	$: disabled =
-		(step == 1 && resourceType == '') ||
-		(step == 2 &&
-			value == '' &&
-			args &&
-			args['token'] == '' &&
-			args['password'] == '' &&
-			args['api_key'] == '' &&
-			args['key'] == '' &&
-			linkedSecret != undefined) ||
-		(step == 3 && pathError != '') ||
-		!isValid
-
-	let isValid = true
-	let filteredConnects: [string, { scopes: string[]; extra_params?: Record<string, string> }][] = []
-	let filteredConnectsManual: [string, { img?: string; instructions: string[]; key?: string }][] =
-		[]
-
-	let darkMode: boolean = false
-</script>
-
-<DarkModeObserver bind:darkMode />
-
-<SearchItems
-	{filter}
-	items={connects ? Object.entries(connects).sort((a, b) => a[0].localeCompare(b[0])) : undefined}
-	bind:filteredItems={filteredConnects}
-	f={(x) => x[0]}
-/>
-<SearchItems
-	{filter}
-	items={connectsManual?.sort((a, b) => a[0].localeCompare(b[0]))}
-	bind:filteredItems={filteredConnectsManual}
-	f={(x) => x[0]}
-/>
-
-<Drawer
-	bind:this={drawer}
-	on:close={() => {
-		dispatch('close')
-	}}
-	on:open={() => {
-		loadResources()
-	}}
-	size="800px"
->
-	<DrawerContent
-		title="Add a Resource"
-		on:close={drawer.closeDrawer}
-		tooltip="Resources represent connections to third party systems. Learn more on how to integrate external APIs."
-		documentationLink="https://www.windmill.dev/docs/integrations/integrations_on_windmill"
-	>
-		{#if step == 1}
-			<div class="w-12/12 pb-2 flex flex-row my-1 gap-1">
-				<input
-					type="text"
-					placeholder="Search resource type"
-					bind:value={filter}
-					class="text-2xl grow"
-				/>
-			</div>
-
-			<h2 class="mb-4">OAuth APIs</h2>
-			<div class="grid sm:grid-cols-2 md:grid-cols-3 gap-x-2 gap-y-1 items-center mb-2">
-				{#if filteredConnects}
-					{#each filteredConnects as [key, values]}
-						<Button
-							size="sm"
-							variant="border"
-							color={key === resourceType ? 'blue' : 'light'}
-							btnClasses={key === resourceType ? '!border-2' : 'm-[1px]'}
-							on:click={() => {
-								manual = false
-								resourceType = key
-								scopes = values.scopes
-								extra_params = Object.entries(values.extra_params ?? {})
-							}}
-						>
-							<IconedResourceType name={key} after={true} width="20px" height="20px" />
-						</Button>
-					{/each}
-				{:else}
-					{#each new Array(3) as _}
-						<Skeleton layout={[[2]]} />
-					{/each}
-				{/if}
-			</div>
-			{#if connects && Object.keys(connects).length == 0}
-				<div class="text-secondary text-sm w-full"
-					>No OAuth APIs has been setup on the instance. To add oauth APIs, first sync the resource
-					types with the hub, then add oauth configuration. See <a
-						href="https://www.windmill.dev/docs/misc/setup_oauth">documentation</a
-					>
-				</div>
-			{/if}
-			{#if manual == false && resourceType != ''}
-				<h3>Scopes</h3>
-				{#if !manual && resourceType != ''}
-					<OauthScopes bind:scopes />
-				{/if}
-			{/if}
-
-			<h2 class="mt-8 mb-4">Others</h2>
-
-			{#if connectsManual && connectsManual?.length < 10}
-				<div class="text-secondary p-2">
-					Resource Types have not been synced with the hub. Go to the admins workspace to sync them
-					(and add a schedule to do daily):
-					<p class="mt-4"
-						>1. Go to the "admins" workspaces:
-						<img src="/sync_resource_types.png" alt="sync resource types" class="mt-2" />
-					</p>
-					<p class="mt-4">
-						2: Run the synchronization script:
-						<img src="/sync_resource_types2.png" alt="sync resource types" class="mt-2" />
-					</p>
-				</div>
-			{/if}
-
-			<div class="grid sm:grid-cols-2 md:grid-cols-3 gap-x-2 gap-y-1 items-center mb-2">
-				{#if filteredConnectsManual}
-					{#each filteredConnectsManual as [key, _]}
-						{#if nativeLanguagesCategory.includes(key)}
-							<Button
-								size="sm"
-								variant="border"
-								color={key === resourceType ? 'blue' : 'light'}
-								btnClasses={key === resourceType ? '!border-2 !bg-blue-50/75' : 'm-[1px]'}
-								on:click={() => {
-									manual = true
-									resourceType = key
-									next()
-									dispatch('click')
-								}}
-							>
-								<IconedResourceType name={key} after={true} width="20px" height="20px" />
-							</Button>
-						{/if}
-					{/each}
-				{/if}
-			</div>
-
-			<div class="mt-8 mb-4" />
-			<div class="grid sm:grid-cols-2 md:grid-cols-3 gap-x-2 gap-y-1 items-center mb-2">
-				{#if filteredConnectsManual}
-					{#each filteredConnectsManual as [key, _]}
-						{#if !nativeLanguagesCategory.includes(key)}
-							<!-- Exclude specific items -->
-							<Button
-								size="sm"
-								variant="border"
-								color={key === resourceType ? 'blue' : 'light'}
-								btnClasses={key === resourceType ? '!border-2 !bg-blue-50/75' : 'm-[1px]'}
-								on:click={() => {
-									manual = true
-									resourceType = key
-									next()
-									dispatch('click')
-								}}
-							>
-								<IconedResourceType name={key} after={true} width="20px" height="20px" />
-							</Button>
-						{/if}
-					{/each}
-				{:else}
-					{#each new Array(9) as _}
-						<Skeleton layout={[[2]]} />
-					{/each}
-				{/if}
-			</div>
-		{:else if step == 2 && manual}
-			<Path
-				bind:error={pathError}
-				bind:path
-				initialPath=""
-				namePlaceholder={resourceType}
-				kind="resource"
-			/>
-
-			{#if apiTokenApps[resourceType]}
-				<h2 class="mt-4 mb-2">Instructions</h2>
-				<div class="pl-10">
-					<ol class="list-decimal">
-						{#each apiTokenApps[resourceType].instructions as step}
-							<li>
-								{@html step}
-							</li>
-						{/each}
-					</ol>
-				</div>
-				{#if apiTokenApps[resourceType].img}
-					<div class="mt-4 w-full overflow-hidden">
-						<img class="m-auto max-h-60" alt="connect" src={apiTokenApps[resourceType].img} />
-					</div>
-				{/if}
-			{:else if !emptyString(resourceTypeInfo?.description)}
-				<h4 class="mt-8 mb-2">{resourceTypeInfo?.name} description</h4>
-				<div class="text-sm">
-					<Markdown md={urlize(resourceTypeInfo?.description ?? '', 'md')} />
-				</div>
-			{/if}
-			{#if resourceType == 'postgresql' || resourceType == 'mysql' || resourceType == 'mongodb'}
-				<WhitelistIp />
-			{/if}
-
-			<h4 class="mt-8 inline-flex items-center gap-4"
-				>Resource description <Required required={false} />
-				<div class="flex gap-1 items-center">
-					<Toggle size="xs" bind:checked={renderDescription} />
-					<Pen size={14} />
-				</div>
-			</h4>
-			{#if renderDescription}
-				<div>
-					<div class="flex flex-row-reverse text-2xs text-tertiary -mt-1">GH Markdown</div>
-					<textarea use:autosize bind:value={description} placeholder={'Resource description'} />
-				</div>
-			{:else if description == undefined || description == ''}
-				<div class="text-sm text-tertiary">No description provided</div>
-			{:else}
-				<div class="mt-2" />
-				<GfmMarkdown md={description} />
-			{/if}
-			<div class="mt-12">
-				{#key resourceTypeInfo}
-					<ApiConnectForm
-						{linkedSecret}
-						{linkedSecretCandidates}
-						{resourceType}
-						{resourceTypeInfo}
-						bind:args
-						bind:isValid
-					/>
-				{/key}
-			</div>
-		{:else}
-			<Path
-				initialPath=""
-				namePlaceholder={resourceType}
-				bind:error={pathError}
-				bind:path
-				kind="resource"
-			/>
-			{#if apiTokenApps[resourceType] || !manual}
-				<ul class="mt-10">
-					<li>
-						1. A secret variable containing the {apiTokenApps[resourceType]?.linkedSecret ??
-							'token'}
-						<span class="font-bold">{truncateRev(value, 5, '*****')}</span>
-						will be stored a
-						<span class="font-mono whitespace-nowrap">{path}</span>.
-					</li>
-					<li class="mt-4">
-						2. The resource containing that token will be stored at the same path <span
-							class="font-mono whitespace-nowrap">{path}</span
-						>. The Variable and Resource will be "linked together", they will be deleted and renamed
-						together.
-					</li></ul
-				>
-			{/if}
-		{/if}
-		<div slot="actions" class="flex gap-1">
-			{#if step > 1 && !no_back}
-				<Button variant="border" on:click={back}>Back</Button>
-			{/if}
-			{#if isGoogleSignin}
-				<button {disabled} on:click={next}>
-					<img
-						class="h-10 w-auto object-contain"
-						src={darkMode ? '/google_signin_dark.png' : '/google_signin_light.png'}
-						alt="Google sign-in"
-					/>
-				</button>
-			{:else}
-				<Button {disabled} on:click={next}>
-					{#if step == 1 && !manual}
-						Connect
-					{:else if step == 1 && manual}
-						Next
-					{:else}
-						Save
-					{/if}
-				</Button>
-			{/if}
-		</div>
-	</DrawerContent>
-</Drawer>

frontend/src/lib/components/AppConnectDrawer.svelte

@@ -0,0 +1,82 @@
+<script lang="ts">
+	import { createEventDispatcher } from 'svelte'
+	import { Button, Drawer } from './common'
+	import DrawerContent from './common/drawer/DrawerContent.svelte'
+
+	import AppConnectInner from './AppConnectInner.svelte'
+	import DarkModeObserver from './DarkModeObserver.svelte'
+
+	let drawer: Drawer
+	let resourceType = ''
+	let step = 1
+	let disabled = false
+	let isGoogleSignin = false
+	let manual = false
+
+	let appConnectInner: AppConnectInner | undefined = undefined
+
+	let rtToLoad: string | undefined = ''
+	export async function open(rt?: string) {
+		rtToLoad = rt
+		drawer.openDrawer?.()
+	}
+
+	$: appConnectInner?.open(rtToLoad)
+
+	const dispatch = createEventDispatcher()
+
+	let darkMode: boolean = false
+</script>
+
+<DarkModeObserver bind:darkMode />
+
+<Drawer
+	bind:this={drawer}
+	on:close={() => {
+		step = 1
+		dispatch('close')
+	}}
+	size="800px"
+>
+	<DrawerContent
+		title="Add a Resource"
+		on:close={drawer.closeDrawer}
+		tooltip="Resources represent connections to third party systems. Learn more on how to integrate external APIs."
+		documentationLink="https://www.windmill.dev/docs/integrations/integrations_on_windmill"
+	>
+		<AppConnectInner
+			bind:this={appConnectInner}
+			bind:step
+			bind:resourceType
+			bind:isGoogleSignin
+			bind:disabled
+			bind:manual
+			on:close={drawer?.closeDrawer}
+			on:refresh
+		/>
+		<div slot="actions" class="flex gap-1">
+			{#if step > 1}
+				<Button variant="border" on:click={appConnectInner?.back}>Back</Button>
+			{/if}
+			{#if isGoogleSignin}
+				<button {disabled} on:click={appConnectInner?.next}>
+					<img
+						class="h-10 w-auto object-contain"
+						src={darkMode ? '/google_signin_dark.png' : '/google_signin_light.png'}
+						alt="Google sign-in"
+					/>
+				</button>
+			{:else}
+				<Button {disabled} on:click={appConnectInner?.next}>
+					{#if step == 2 && !manual}
+						Connect
+					{:else if step == 1}
+						Next
+					{:else}
+						Save
+					{/if}
+				</Button>
+			{/if}
+		</div>
+	</DrawerContent>
+</Drawer>

frontend/src/lib/components/AppConnectInner.svelte

@@ -0,0 +1,556 @@
+<script lang="ts">
+	import { workspaceStore } from '$lib/stores'
+	import IconedResourceType from './IconedResourceType.svelte'
+	import {
+		OauthService,
+		ResourceService,
+		VariableService,
+		type TokenResponse,
+		type ResourceType
+	} from '$lib/gen'
+	import { emptyString, truncateRev, urlize } from '$lib/utils'
+	import { createEventDispatcher } from 'svelte'
+	import Path from './Path.svelte'
+	import { Button, Skeleton } from './common'
+	import ApiConnectForm from './ApiConnectForm.svelte'
+	import SearchItems from './SearchItems.svelte'
+	import WhitelistIp from './WhitelistIp.svelte'
+	import { sendUserToast } from '$lib/toast'
+	import OauthScopes from './OauthScopes.svelte'
+	import Markdown from 'svelte-exmarkdown'
+	import autosize from '$lib/autosize'
+	import Required from './Required.svelte'
+	import Toggle from './Toggle.svelte'
+	import { Pen } from 'lucide-svelte'
+	import GfmMarkdown from './GfmMarkdown.svelte'
+	import { apiTokenApps, forceSecretValue, linkedSecretValue } from './app_connect'
+
+	export let step = 1
+	export let resourceType = ''
+	export let isGoogleSignin = false
+	export let disabled = false
+	export let manual = false
+
+	let isValid = true
+
+	const nativeLanguagesCategory = [
+		'postgresql',
+		'mysql',
+		'bigquery',
+		'snowflake',
+		'mssql',
+		'graphql'
+	]
+
+	let filter = ''
+	let value: string = ''
+	let valueToken: TokenResponse | undefined = undefined
+	let connects: string[] | undefined = undefined
+	let connectsManual:
+		| [string, { img?: string; instructions: string[]; key?: string }][]
+		| undefined = undefined
+	let args: any = {}
+	let renderDescription = true
+
+	$: linkedSecretCandidates = apiTokenApps[resourceType]?.linkedSecret
+		? ([apiTokenApps[resourceType]?.linkedSecret] as string[])
+		: args != undefined
+		? Object.keys(args).filter((x) =>
+				['token', 'secret', 'key', 'pass', 'private'].some((y) => x.toLowerCase().includes(y))
+		  )
+		: undefined
+
+	$: linkedSecret =
+		forceSecretValue(resourceType) ??
+		linkedSecretCandidates?.sort((ua, ub) => linkedSecretValue(ub) - linkedSecretValue(ua))?.[0]
+
+	let scopes: string[] = []
+	let extra_params: [string, string][] = []
+
+	let path: string
+	let description = ''
+
+	let resourceTypeInfo: ResourceType | undefined = undefined
+
+	let pathError = ''
+
+	export async function open(rt?: string) {
+		step = 1
+		value = ''
+		description = ''
+		resourceType = rt ?? ''
+		valueToken = undefined
+		await loadConnects()
+
+		const isConnect = connects?.includes(resourceType)
+		if (isConnect) {
+			manual = false
+			next()
+		} else {
+			manual = true
+			if (rt) {
+				next()
+			} else {
+				loadResourceTypes()
+			}
+		}
+	}
+
+	async function loadConnects() {
+		if (!connects) {
+			connects = (await OauthService.listOauthConnects()).filter((x) => x != 'supabase_wizard')
+		}
+	}
+
+	const connectAndManual = ['gitlab']
+
+	$: isGoogleSignin =
+		step == 1 &&
+		(resourceType == 'google' ||
+			resourceType == 'gmail' ||
+			resourceType == 'gcal' ||
+			resourceType == 'gdrive' ||
+			resourceType == 'gsheets')
+
+	$: disabled =
+		(step == 1 && resourceType == '') ||
+		(step == 2 &&
+			value == '' &&
+			args &&
+			args['token'] == '' &&
+			args['password'] == '' &&
+			args['api_key'] == '' &&
+			args['key'] == '' &&
+			linkedSecret != undefined) ||
+		step == 3 ||
+		(step == 4 && pathError != '') ||
+		!isValid
+
+	export async function loadResourceTypes() {
+		if (connectsManual) {
+			return
+		}
+		const availableRts = await ResourceService.listResourceTypeNames({
+			workspace: $workspaceStore!
+		})
+
+		connectsManual = availableRts
+			.filter((x) => connectAndManual.includes(x) || !Object.keys(connects ?? {}).includes(x))
+			.map((x) => [
+				x,
+				apiTokenApps[x] ?? {
+					instructions: '',
+					img: undefined,
+					linkedSecret: undefined
+				}
+			])
+		const filteredNativeLanguages = filteredConnectsManual?.filter(
+			(o) => nativeLanguagesCategory?.includes(o[0]) ?? false
+		)
+
+		try {
+			filteredConnectsManual = [
+				...(filteredNativeLanguages ?? []),
+				...(filteredConnectsManual ?? []).filter(
+					([key, _]) => !nativeLanguagesCategory.includes(key)
+				)
+			]
+		} catch (e) {}
+	}
+
+	function popupListener(event) {
+		let data = event.data
+		if (event.origin !== window.location.origin) {
+			return
+		}
+
+		if (data.type === 'error') {
+			sendUserToast(event.data.error, true)
+			step = 2
+		} else if (data.type === 'success') {
+			resourceType = data.resource_type
+			value = data.res.access_token!
+			valueToken = data.res
+			step = 4
+		}
+	}
+
+	async function getScopesAndParams() {
+		const connect = await OauthService.getOauthConnect({ client: resourceType })
+		scopes = connect.scopes ?? []
+		extra_params = Object.entries(connect.extra_params ?? {})
+	}
+
+	async function getResourceTypeInfo() {
+		resourceTypeInfo = await ResourceService.getResourceType({
+			workspace: $workspaceStore!,
+			path: resourceType
+		})
+	}
+	export async function next() {
+		if (step == 1) {
+			if (manual) {
+				await getResourceTypeInfo()
+				args = {}
+			} else {
+				await Promise.all([getScopesAndParams(), getResourceTypeInfo()])
+			}
+			step += 1
+		} else if (step == 2 && !manual) {
+			const url = new URL(`/api/oauth/connect/${resourceType}`, window.location.origin)
+			url.searchParams.append('scopes', scopes.join('+'))
+			if (extra_params.length > 0) {
+				extra_params.forEach(([key, value]) => url.searchParams.append(key, value))
+			}
+			// if (!newPageOAuth) {
+			// 	window.location.href = url.toString()
+			// } else {
+			window.addEventListener('message', popupListener, { once: true })
+			window.open(url.toString(), '_blank', 'popup=true')
+			step += 1
+
+			// 	dispatch('close')
+			// }
+		} else {
+			let exists = await VariableService.existsVariable({
+				workspace: $workspaceStore!,
+				path
+			})
+			if (exists) {
+				throw Error(`Variable at path ${path} already exists. Delete it or pick another path`)
+			}
+			exists = await ResourceService.existsResource({
+				workspace: $workspaceStore!,
+				path
+			})
+
+			if (exists) {
+				throw Error(`Resource at path ${path} already exists. Delete it or pick another path`)
+			}
+
+			let account: number | undefined = undefined
+			if (valueToken?.expires_in != undefined) {
+				account = Number(
+					await OauthService.createAccount({
+						workspace: $workspaceStore!,
+						requestBody: {
+							refresh_token: valueToken.refresh_token ?? '',
+							expires_in: valueToken.expires_in,
+							client: resourceType
+						}
+					})
+				)
+			}
+
+			const resourceValue = args
+
+			let saveVariable = false
+			if (!manual || linkedSecret != undefined) {
+				let v = manual ? args[linkedSecret ?? ''] : value
+				if (typeof v == 'string' && v != '' && !v.startsWith('$var:')) {
+					saveVariable = true
+					await VariableService.createVariable({
+						workspace: $workspaceStore!,
+						requestBody: {
+							path,
+							value: v,
+							is_secret: true,
+							description: emptyString(description)
+								? `${manual ? 'Token' : 'OAuth token'} for ${resourceType}`
+								: description,
+							is_oauth: !manual,
+							account: account
+						}
+					})
+					resourceValue[linkedSecret ?? 'token'] = `$var:${path}`
+				}
+			}
+
+			await ResourceService.createResource({
+				workspace: $workspaceStore!,
+				requestBody: {
+					resource_type: resourceType,
+					path,
+					value: resourceValue,
+					description
+				}
+			})
+			dispatch('refresh', path)
+			dispatch('close')
+			sendUserToast(`Saved resource${saveVariable ? ' and variable' : ''} path: ${path}`)
+			step = 1
+			resourceType = ''
+		}
+	}
+
+	export async function back() {
+		if (step == 4) {
+			step -= 2
+		} else if (step > 1) {
+			step -= 1
+		}
+		if (step == 1) {
+			loadConnects()
+			loadResourceTypes()
+		}
+	}
+
+	const dispatch = createEventDispatcher()
+
+	let filteredConnects: { key: string }[] = []
+	let filteredConnectsManual: [string, { img?: string; instructions: string[]; key?: string }][] =
+		[]
+
+	let editScopes = false
+</script>
+
+<SearchItems
+	{filter}
+	items={connects
+		? connects
+				.sort((a, b) => a.localeCompare(b))
+				.map((key) => ({
+					key
+				}))
+		: undefined}
+	bind:filteredItems={filteredConnects}
+	f={(x) => x.key}
+/>
+<SearchItems
+	{filter}
+	items={connectsManual?.sort((a, b) => a[0].localeCompare(b[0]))}
+	bind:filteredItems={filteredConnectsManual}
+	f={(x) => x[0]}
+/>
+
+{#if step == 1}
+	<div class="w-12/12 pb-2 flex flex-row my-1 gap-1">
+		<input
+			type="text"
+			placeholder="Search resource type"
+			bind:value={filter}
+			class="text-2xl grow"
+		/>
+	</div>
+
+	<h2 class="mb-4">OAuth APIs</h2>
+	<div class="grid sm:grid-cols-2 md:grid-cols-3 gap-x-2 gap-y-1 items-center mb-2">
+		{#if filteredConnects}
+			{#each filteredConnects as { key }}
+				<Button
+					size="sm"
+					variant="border"
+					color={key === resourceType ? 'blue' : 'light'}
+					btnClasses={key === resourceType ? '!border-2' : 'm-[1px]'}
+					on:click={() => {
+						manual = false
+						resourceType = key
+						next()
+					}}
+				>
+					<IconedResourceType name={key} after={true} width="20px" height="20px" />
+				</Button>
+			{/each}
+		{:else}
+			{#each new Array(3) as _}
+				<Skeleton layout={[[2]]} />
+			{/each}
+		{/if}
+	</div>
+	{#if connects && connects.length == 0}
+		<div class="text-secondary text-sm w-full"
+			>No OAuth APIs has been setup on the instance. To add oauth APIs, first sync the resource
+			types with the hub, then add oauth configuration. See <a
+				href="https://www.windmill.dev/docs/misc/setup_oauth">documentation</a
+			>
+		</div>
+	{/if}
+
+	<h2 class="mt-8 mb-4">Others</h2>
+
+	{#if connectsManual && connectsManual?.length < 10}
+		<div class="text-secondary p-2">
+			Resource Types have not been synced with the hub. Go to the admins workspace to sync them (and
+			add a schedule to do daily):
+			<p class="mt-4"
+				>1. Go to the "admins" workspaces:
+				<img src="/sync_resource_types.png" alt="sync resource types" class="mt-2" />
+			</p>
+			<p class="mt-4">
+				2: Run the synchronization script:
+				<img src="/sync_resource_types2.png" alt="sync resource types" class="mt-2" />
+			</p>
+		</div>
+	{/if}
+
+	<div class="grid sm:grid-cols-2 md:grid-cols-3 gap-x-2 gap-y-1 items-center mb-2">
+		{#if filteredConnectsManual}
+			{#each filteredConnectsManual as [key, _]}
+				{#if nativeLanguagesCategory.includes(key)}
+					<Button
+						size="sm"
+						variant="border"
+						color={key === resourceType ? 'blue' : 'light'}
+						btnClasses={key === resourceType ? '!border-2' : 'm-[1px]'}
+						on:click={() => {
+							manual = true
+							resourceType = key
+							next()
+						}}
+					>
+						<IconedResourceType name={key} after={true} width="20px" height="20px" />
+					</Button>
+				{/if}
+			{/each}
+		{/if}
+	</div>
+
+	<div class="mt-8 mb-4" />
+	<div class="grid sm:grid-cols-2 md:grid-cols-3 gap-x-2 gap-y-1 items-center mb-2">
+		{#if filteredConnectsManual}
+			{#each filteredConnectsManual as [key, _]}
+				{#if !nativeLanguagesCategory.includes(key)}
+					<!-- Exclude specific items -->
+					<Button
+						size="sm"
+						variant="border"
+						color={key === resourceType ? 'blue' : 'light'}
+						btnClasses={key === resourceType ? '!border-2' : 'm-[1px]'}
+						on:click={() => {
+							manual = true
+							resourceType = key
+							next()
+						}}
+					>
+						<IconedResourceType name={key} after={true} width="20px" height="20px" />
+					</Button>
+				{/if}
+			{/each}
+		{:else}
+			{#each new Array(9) as _}
+				<Skeleton layout={[[2]]} />
+			{/each}
+		{/if}
+	</div>
+{:else if step == 2 && manual}
+	<Path
+		bind:error={pathError}
+		bind:path
+		initialPath=""
+		namePlaceholder={resourceType}
+		kind="resource"
+	/>
+
+	{#if apiTokenApps[resourceType]}
+		<h2 class="mt-4 mb-2">Instructions</h2>
+		<div class="pl-10">
+			<ol class="list-decimal">
+				{#each apiTokenApps[resourceType].instructions as step}
+					<li>
+						{@html step}
+					</li>
+				{/each}
+			</ol>
+		</div>
+		{#if apiTokenApps[resourceType].img}
+			<div class="mt-4 w-full overflow-hidden">
+				<img class="m-auto max-h-60" alt="connect" src={apiTokenApps[resourceType].img} />
+			</div>
+		{/if}
+	{:else if !emptyString(resourceTypeInfo?.description)}
+		<h4 class="mt-8 mb-2">{resourceTypeInfo?.name} description</h4>
+		<div class="text-sm">
+			<Markdown md={urlize(resourceTypeInfo?.description ?? '', 'md')} />
+		</div>
+	{/if}
+	{#if resourceType == 'postgresql' || resourceType == 'mysql' || resourceType == 'mongodb'}
+		<WhitelistIp />
+	{/if}
+
+	<h4 class="mt-8 inline-flex items-center gap-4"
+		>Resource description <Required required={false} />
+		<div class="flex gap-1 items-center">
+			<Toggle size="xs" bind:checked={renderDescription} />
+			<Pen size={14} />
+		</div>
+	</h4>
+	{#if renderDescription}
+		<div>
+			<div class="flex flex-row-reverse text-2xs text-tertiary -mt-1">GH Markdown</div>
+			<textarea use:autosize bind:value={description} placeholder={'Resource description'} />
+		</div>
+	{:else if description == undefined || description == ''}
+		<div class="text-sm text-tertiary">No description provided</div>
+	{:else}
+		<div class="mt-2" />
+		<GfmMarkdown md={description} />
+	{/if}
+	<div class="mt-12">
+		{#key resourceTypeInfo}
+			<ApiConnectForm
+				{linkedSecret}
+				{linkedSecretCandidates}
+				{resourceType}
+				{resourceTypeInfo}
+				bind:args
+				bind:isValid
+			/>
+		{/key}
+	</div>
+{:else if step == 2 && !manual}
+	{#if manual == false && resourceType != ''}
+		<h1 class="mb-4">{resourceType}</h1>
+		<div class="my-4 text-secondary"
+			>Click connect to create a resource backed by an oauth connection, whose token is fetched from
+			the external services and refreshed automatically if needed before expiration (using its
+			refresh token)</div
+		>
+		<h4 class="mb-2">Description</h4>
+		<div class="text-sm mb-8">
+			<Markdown md={urlize(resourceTypeInfo?.description ?? '', 'md')} />
+		</div>
+		<h3 class="mb-4 flex gap-4"
+			>Scopes <button
+				on:click={() => {
+					editScopes = !editScopes
+				}}><Pen size={14} /></button
+			></h3
+		>
+
+		{#if editScopes}
+			<OauthScopes bind:scopes />
+		{:else}
+			<div class="flex flex-col gap-1">
+				{#each scopes as scope}
+					<div class="py-0.5 pl-2 text-xs">- {scope}</div>
+				{/each}
+			</div>
+		{/if}
+	{/if}
+{:else if step == 3 && !manual}
+	Finish connection in popup window
+{:else}
+	<Path
+		initialPath=""
+		namePlaceholder={resourceType}
+		bind:error={pathError}
+		bind:path
+		kind="resource"
+	/>
+	{#if apiTokenApps[resourceType] || !manual}
+		<ul class="mt-10">
+			<li>
+				1. A secret variable containing the {apiTokenApps[resourceType]?.linkedSecret ?? 'token'}
+				<span class="font-bold">{truncateRev(value, 5, '*****')}</span>
+				will be stored a
+				<span class="font-mono whitespace-nowrap">{path}</span>.
+			</li>
+			<li class="mt-4">
+				2. The resource containing that token will be stored at the same path <span
+					class="font-mono whitespace-nowrap">{path}</span
+				>. The Variable and Resource will be "linked together", they will be deleted and renamed
+				together.
+			</li></ul
+		>
+	{/if}
+{/if}

frontend/src/lib/components/ArgInput.svelte

@@ -230,7 +230,6 @@
 	}}
 	readOnlyMode={false}
 />
-
 <!-- svelte-ignore a11y-autofocus -->
 <div class="flex flex-col w-full {minW ? 'min-w-[250px]' : ''}">
 	<div>

frontend/src/lib/components/CustomOauth.svelte

@@ -67,8 +67,8 @@
 <label class="block pb-2">
 	<span class="text-primary font-semibold text-sm"
 		>Payload <Tooltip
-			>Auth is passed in query most commonly. LinkedIn is an example of OAuth using
-			x-www-form-urlencoded
+			>Auth (client id/client secret) is passed as basic auth most commonly but can be passed in the
+			body x-www-form-urlencoded. Some LinkedIn is an example of OAuth using x-www-form-urlencoded
 		</Tooltip></span
 	>
 	<div>

frontend/src/lib/components/Dev.svelte

@@ -374,7 +374,6 @@
 		} catch (e) {
 			console.error(e)
 			validCode = false
-			schema = emptySchema()
 		}
 	}
 

frontend/src/lib/components/DisplayResult.svelte

@@ -637,7 +637,7 @@
 							title="Large result detected"
 							type="warning"
 							tooltip="We recommend using persistent object storage for large result. See docs for setting up an object storage service integration using s3 or any other s3 compatible services."
-							documentationLink="https://www.windmill.dev/docs/core_concepts/persistent_storage#large-data-files-s3-r2-minio-azure-blob"
+							documentationLink="https://www.windmill.dev/docs/core_concepts/persistent_storage#object-storage-for-large-data-s3-r2-minio-azure-blob"
 						/>
 					</div>
 					{#if result && result != 'WINDMILL_TOO_BIG'}

frontend/src/lib/components/Editor.svelte

@@ -94,7 +94,6 @@
 	export let automaticLayout = true
 	export let websocketAlive = {
 		pyright: false,
-		black: false,
 		ruff: false,
 		deno: false,
 		go: false,
@@ -803,22 +802,6 @@
 					{},
 					undefined
 				)
-				connectToLanguageServer(
-					`${wsProtocol}://${window.location.host}/ws/diagnostic`,
-					'black',
-					{
-						formatters: {
-							black: {
-								command: 'black',
-								args: ['--quiet', '-']
-							}
-						},
-						formatFiletypes: {
-							python: 'black'
-						}
-					},
-					undefined
-				)
 			} else if (lang === 'go') {
 				connectToLanguageServer(
 					`${wsProtocol}://${window.location.host}/ws/go`,
@@ -876,7 +859,6 @@
 						(new Date().getTime() - lastWsAttempt.getTime() > 60000 && nbWsAttempt < 2)
 					) {
 						if (
-							!websocketAlive.black &&
 							!websocketAlive.deno &&
 							!websocketAlive.pyright &&
 							!websocketAlive.go &&
@@ -1126,7 +1108,6 @@
 			})
 
 			if (
-				!websocketAlive.black &&
 				!websocketAlive.deno &&
 				!websocketAlive.pyright &&
 				!websocketAlive.ruff &&

frontend/src/lib/components/EditorBar.svelte

@@ -49,7 +49,6 @@
 	export let editor: Editor | undefined
 	export let websocketAlive: {
 		pyright: boolean
-		black: boolean
 		ruff: boolean
 		deno: boolean
 		go: boolean
@@ -212,7 +211,7 @@
 	}
 
 	function phpCompile(schema: Schema) {
-		let res = '	'
+		let res = '  '
 		const entries = Object.entries(schema.properties)
 		if (entries.length === 0) {
 			return 'array'
@@ -603,7 +602,6 @@ $res = json_decode(curl_exec($ch));`)
 							(<span class={websocketAlive.go ? 'green' : 'text-red-700'}>Go</span>)
 						{:else if lang == 'python3'}
 							(<span class={websocketAlive.pyright ? 'green' : 'text-red-700'}>Pyright</span>
-							<span class={websocketAlive.black ? 'green' : 'text-red-700'}>Black</span>
 							<span class={websocketAlive.ruff ? 'green' : 'text-red-700'}>Ruff</span>)
 						{:else if lang == 'bash'}
 							(<span class={websocketAlive.shellcheck ? 'green' : 'text-red-700'}>Shellcheck</span>)

frontend/src/lib/components/FlowStatusViewerInner.svelte

@@ -66,6 +66,7 @@
 	let jobFailures: boolean[] = []
 
 	let forloop_selected = ''
+	let retry_selected = ''
 	let timeout: NodeJS.Timeout
 
 	let localModuleStates: Writable<Record<string, GraphModuleState>> = writable({})
@@ -296,7 +297,8 @@
 					duration_ms: job['duration_ms'],
 					started_at: started_at,
 					iteration: mod.iterator?.itered?.length,
-					iteration_total: mod.iterator?.itered?.length
+					iteration_total: mod.iterator?.itered?.length,
+					retries: mod?.failed_retries?.length
 					// retries: $flowStateStore?.raw_flow
 				})
 				setDurationStatusByJob(mod.id, job.id, {
@@ -653,6 +655,43 @@
 							<div class="line w-8 h-10" />
 						{/if}
 						<li class="w-full border p-6 space-y-2 bg-blue-50/50 dark:bg-frost-900/50">
+							{#if render && Array.isArray(mod.failed_retries)}
+								{#each mod.failed_retries as failedRetry, j}
+									<Button
+										variant={retry_selected === failedRetry ? 'contained' : 'border'}
+										color="red"
+										btnClasses="w-full flex justify-start"
+										on:click={() => {
+											if (retry_selected == failedRetry) {
+												retry_selected = ''
+											} else {
+												retry_selected = failedRetry
+											}
+										}}
+										endIcon={{
+											icon: ChevronDown,
+											classes: retry_selected == failedRetry ? '!rotate-180' : ''
+										}}
+									>
+										<span class="truncate font-mono">
+											# Attempt {j + 1}: {failedRetry}
+										</span>
+									</Button>
+
+									<!-- <LogId id={loopJobId} /> -->
+									<div class="border p-6" class:hidden={retry_selected != failedRetry}>
+										<svelte:self
+											{childFlow}
+											globalModuleStates={[localModuleStates, ...globalModuleStates]}
+											globalDurationStatuses={[localDurationStatuses, ...globalDurationStatuses]}
+											render={failedRetry == retry_selected}
+											reducedPolling={false}
+											{workspaceId}
+											jobId={failedRetry}
+										/>
+									</div>
+								{/each}
+							{/if}
 							{#if ['InProgress', 'Success', 'Failure'].includes(mod.type)}
 								{#if job.raw_flow?.modules[i]?.value.type == 'flow'}
 									<svelte:self

frontend/src/lib/components/InstanceSettings.svelte

@@ -13,10 +13,10 @@
 	import KeycloakSetting from './KeycloakSetting.svelte'
 	import Alert from './common/alert/Alert.svelte'
 	import { isCloudHosted } from '$lib/cloud'
-	import { capitalize } from '$lib/utils'
+	import { capitalize, classNames } from '$lib/utils'
 	import { enterpriseLicense } from '$lib/stores'
 	import CustomOauth from './CustomOauth.svelte'
-	import { AlertTriangle, Plus, X } from 'lucide-svelte'
+	import { AlertCircle, AlertTriangle, BadgeCheck, Info, Plus, X, BadgeX } from 'lucide-svelte'
 	import CustomSso from './CustomSso.svelte'
 	import AuthentikSetting from '$lib/components/AuthentikSetting.svelte'
 	import AutheliaSetting from '$lib/components/AutheliaSetting.svelte'
@@ -25,6 +25,7 @@
 	import Password from './Password.svelte'
 	import ObjectStoreConfigSettings from './ObjectStoreConfigSettings.svelte'
 	import { fade } from 'svelte/transition'
+	import Popover from './Popover.svelte'
 
 	export let tab: string = 'Core'
 	export let hideTabs: boolean = false
@@ -35,6 +36,10 @@
 	let initialRequirePreexistingUserForOauth: boolean = false
 	let requirePreexistingUserForOauth: boolean = false
 	let ssoOrOauth: 'sso' | 'oauth' = 'sso'
+	let latestKeyRenewalAttempt: {
+		result: string
+		attempted_at: string
+	} | null = null
 
 	let serverConfig = {}
 	let initialValues: Record<string, any> = {}
@@ -79,6 +84,7 @@
 		if (values['base_url'] == undefined) {
 			values['base_url'] = 'http://localhost'
 		}
+		latestKeyRenewalAttempt = await SettingService.getLatestKeyRenewalAttempt()
 	}
 
 	export async function saveSettings() {
@@ -144,7 +150,7 @@
 			try {
 				let i = parseInt(splitted[1])
 				let date = new Date(i * 1000)
-				return date.toDateString()
+				return date.toLocaleDateString()
 			} catch {}
 		}
 		return undefined
@@ -180,6 +186,17 @@
 	let clientName = ''
 
 	let licenseKeyChanged = false
+
+	export async function renewLicenseKey() {
+		try {
+			await SettingService.renewLicenseKey()
+			sendUserToast('Key renewal successful')
+			loadSettings()
+		} catch (err) {
+			latestKeyRenewalAttempt = await SettingService.getLatestKeyRenewalAttempt()
+			throw err
+		}
+	}
 </script>
 
 <div class="pb-8">
@@ -477,22 +494,81 @@
 																requestBody: { license_key: values[setting.key] }
 															})
 															sendUserToast('Valid key')
-														}}>Test Key</Button
+														}}
 													>
-												</div>
-												{#if values[setting.key]?.length > 0}
-													{#if parseDate(values[setting.key])}
-														<span class="text-tertiary text-2xs"
-															>License key expires on {parseDate(values[setting.key])}</span
-														>
+														Test Key
+													</Button>
+													{#if $enterpriseLicense}
+														<Button on:click={renewLicenseKey} size="xs">Renew key</Button>
 													{/if}
-												{/if}
-												{#if licenseKeyChanged}
-													<div class="text-yellow-600"
-														>Refresh page after setting license key and saving to unlock all
-														features</div
-													>
-												{/if}
+												</div>
+												<div class="mt-1 flex flex-col gap-1 items-start">
+													{#if values[setting.key]?.length > 0}
+														{#if parseDate(values[setting.key])}
+															<div class="flex flex-row gap-1 items-center">
+																<Info size={12} class="text-tertiary" />
+																<span class="text-tertiary text-xs"
+																	>License key expires on {parseDate(values[setting.key])}</span
+																>
+															</div>
+														{/if}
+													{/if}
+													{#if latestKeyRenewalAttempt}
+														{@const attemptedAt = new Date(
+															latestKeyRenewalAttempt.attempted_at
+														).toLocaleString()}
+														<div class="relative">
+															<Popover notClickable>
+																<div class="flex flex-row items-center gap-1">
+																	{#if latestKeyRenewalAttempt.result === 'success'}
+																		<BadgeCheck class="text-green-600" size={12} />
+																	{:else}
+																		<BadgeX class="text-red-600" size={12} />
+																	{/if}
+																	<span
+																		class={classNames(
+																			'text-xs',
+																			latestKeyRenewalAttempt.result === 'success'
+																				? 'text-green-600'
+																				: 'text-red-600'
+																		)}
+																	>
+																		{latestKeyRenewalAttempt.result === 'success'
+																			? 'Latest key renewal succeeded'
+																			: 'Latest key renewal failed'}
+																		on {attemptedAt}
+																	</span>
+																</div>
+																<div slot="text">
+																	{#if latestKeyRenewalAttempt.result === 'success'}
+																		<span class="text-green-300">
+																			Latest key renewal succeeded on {attemptedAt}
+																		</span>
+																	{:else}
+																		<span class="text-red-300">
+																			Latest key renewal failed on {attemptedAt}: {latestKeyRenewalAttempt.result.replace(
+																				'error: ',
+																				''
+																			)}
+																		</span>
+																	{/if}
+																	<br />
+																	As long as invoices are paid and usage corresponds to the subscription,
+																	the key is renewed daily with a validity of 35 days (grace period).
+																</div>
+															</Popover>
+														</div>
+													{/if}
+													{#if licenseKeyChanged && !$enterpriseLicense}
+														<div class="flex flex-row items-center gap-1">
+															<AlertCircle size={12} class="text-yellow-600" />
+															<span class="text-xs text-yellow-600">
+																Refresh page after setting and saving license key to unlock all
+																features
+															</span>
+														</div>
+													{/if}
+												</div>
 											{:else if setting.fieldType == 'email'}
 												<input
 													type="email"