feat: Support mistral anthropic for ai (#4692 )

* wip: openai proxy and other ai proxy integration * fixing migration script * wip: support different ai provider in front, fix proxy openai * wip: adding frontend ai provider * updated copilot types * wip: working on anthropic integration * done AI proxy front * adding new type and support for anthropic * updating gitignore * adding streaming response * added streaming prompt * push lib/gen * wip: fixing anthropic * anthropic fully supported * fix backend missing var error and fully support stream event for anthropic * remove gen directory * fixing openapi file * add support for mistral, and update create workspace components * remove deref.json * remove package-json * openapi * fix ui enable code * added utility function for init workspace ai provider * fix workspace switch bug * update anthropic property and fixed frontend error * fix workspace settings * update error message and fix typo migration file * chore: update openapi file * fix dev file * add .sqlx * all * update sqlx --------- Co-authored-by: dieriba <t.dieriba@gmail.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
updating ai system prompt
2026-09-20 00:02:28 +00:00 · 2024-11-15 12:01:20 -05:00 · 2024-11-15 12:00:38 -05:00 · 2024-11-15 11:58:58 -05:00 · 2024-11-15 11:58:58 -05:00 · 2024-11-15 11:58:58 -05:00
139 changed files with 5085 additions and 2556 deletions
@@ -6,4 +6,4 @@ frontend/src/routes/test.svelte
 CaddyfileRemoteMalo
 *.swp
 **/.idea/
-.direnv
+.direnv
@@ -1,5 +1,22 @@
 # Changelog

+## [1.424.0](https://github.com/windmill-labs/windmill/compare/v1.423.2...v1.424.0) (2024-11-14)
+
+
+### Features
+
+* allow setting password and login type from superadmin UI ([2a1bff3](https://github.com/windmill-labs/windmill/commit/2a1bff3160b65eadba399229b5f53950ec2c0d48))
+* **backend:** monitor minimal version of living workers ([#4704](https://github.com/windmill-labs/windmill/issues/4704)) ([50ff183](https://github.com/windmill-labs/windmill/commit/50ff183baeaa2a39c2b0188ee8dd6172b08ae801))
+* Support mistral anthropic for ai ([#4692](https://github.com/windmill-labs/windmill/issues/4692)) ([556b4a4](https://github.com/windmill-labs/windmill/commit/556b4a41a1d010bb8a1796f485d343c1e412d278))
+
+
+### Bug Fixes
+
+* add countCompletedJobs api ([2a78359](https://github.com/windmill-labs/windmill/commit/2a78359af7b8e4700634c2ad2745c1d14d2da4f7))
+* add queue_couts api ([10b6b1d](https://github.com/windmill-labs/windmill/commit/10b6b1dc04cb2b2d4ec5ceb26a5dbd2ac7bd1394))
+* autoscaling when count &lt; min worker set to min_workers ([180bb82](https://github.com/windmill-labs/windmill/commit/180bb826436f9960def8c6cf3e9692714ffdf917))
+* deployment callbacks have a concurrency limit of 1 on same path ([c44fc4f](https://github.com/windmill-labs/windmill/commit/c44fc4f5ab8bcb5dffec08d5ed4ccd61feb1cb13))
+
 ## [1.423.2](https://github.com/windmill-labs/windmill/compare/v1.423.1...v1.423.2) (2024-11-13)


@@ -2,7 +2,7 @@ target/
 .env
 oauth.json
 oauth2.json
-windmill-api/openapi-deref.yaml
 tracing.folded
 heaptrack*
 index/
+windmill-api/openapi-*.*
@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE alerts SET acknowledged_workspace = true, acknowledged = true WHERE workspace_id = $1",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "00588a40dde5189ac1c61505f17acb0f4c244c60477427505bf5bd1b104d3bf9"
+}
@@ -1,12 +1,12 @@
 {
  "db_name": "PostgreSQL",
-  "query": "SELECT openai_resource_path, code_completion_enabled FROM workspace_settings WHERE workspace_id = $1",
+  "query": "SELECT ai_resource, code_completion_enabled FROM workspace_settings WHERE workspace_id = $1",
  "describe": {
    "columns": [
      {
        "ordinal": 0,
-        "name": "openai_resource_path",
-        "type_info": "Varchar"
+        "name": "ai_resource",
+        "type_info": "Jsonb"
      },
      {
        "ordinal": 1,
@@ -24,5 +24,5 @@
      false
    ]
  },
-  "hash": "ceb97024ebf1a1c00ea1ed4952f66b229ca4622c537cc252d8ed52b4d24270ee"
+  "hash": "0230bd64d2b6719c3536a106e18a68c7b43b3a9a9efa92b5517132e9c0d8a25b"
 }
@@ -1,6 +1,6 @@
 {
  "db_name": "PostgreSQL",
-  "query": "SELECT\n                -- slack_team_id, \n                -- slack_name, \n                -- slack_command_script, \n                -- CASE WHEN slack_email = 'missing@email.xyz' THEN NULL ELSE slack_email END AS slack_email,\n                auto_invite_domain IS NOT NULL AS \"auto_invite_enabled!\",\n                CASE WHEN auto_invite_operator IS TRUE THEN 'operator' ELSE 'developer' END AS \"auto_invite_as!\", \n                CASE WHEN auto_add IS TRUE THEN 'add' ELSE 'invite' END AS \"auto_invite_mode!\", \n                webhook, \n                deploy_to, \n                error_handler, \n                openai_resource_path, \n                code_completion_enabled, \n                error_handler_extra_args, \n                error_handler_muted_on_cancel, \n                large_file_storage, \n                git_sync,\n                default_app,\n                default_scripts,\n                workspace.name\n            FROM workspace_settings\n            LEFT JOIN workspace ON workspace.id = workspace_settings.workspace_id\n            WHERE workspace_id = $1",
+  "query": "SELECT\n                -- slack_team_id, \n                -- slack_name, \n                -- slack_command_script, \n                -- CASE WHEN slack_email = 'missing@email.xyz' THEN NULL ELSE slack_email END AS slack_email,\n                auto_invite_domain IS NOT NULL AS \"auto_invite_enabled!\",\n                CASE WHEN auto_invite_operator IS TRUE THEN 'operator' ELSE 'developer' END AS \"auto_invite_as!\", \n                CASE WHEN auto_add IS TRUE THEN 'add' ELSE 'invite' END AS \"auto_invite_mode!\", \n                webhook, \n                deploy_to, \n                error_handler, \n                ai_resource, \n                code_completion_enabled, \n                error_handler_extra_args, \n                error_handler_muted_on_cancel, \n                large_file_storage, \n                git_sync,\n                default_app,\n                default_scripts,\n                workspace.name\n            FROM workspace_settings\n            LEFT JOIN workspace ON workspace.id = workspace_settings.workspace_id\n            WHERE workspace_id = $1",
  "describe": {
    "columns": [
      {
@@ -35,8 +35,8 @@
      },
      {
        "ordinal": 6,
-        "name": "openai_resource_path",
-        "type_info": "Varchar"
+        "name": "ai_resource",
+        "type_info": "Jsonb"
      },
      {
        "ordinal": 7,
@@ -102,5 +102,5 @@
      false
    ]
  },
-  "hash": "188534f4b29f6461b1a6214d060f183c830b19a403ebb7b8be55a691675010c3"
+  "hash": "0331a81262e2d3c1bcfaeb64617b11eb68ab4599d64e5e5af639a9ac5d791fd0"
 }
@@ -1,16 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "UPDATE workspace_settings SET openai_resource_path = $1, code_completion_enabled = $2 WHERE workspace_id = $3",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Varchar",
-        "Bool",
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "034583442e6f8ae38d6c4e4aac26f17c8d9d0e657f28276228fc90d3e22e1304"
-}
@@ -0,0 +1,18 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO alerts (alert_type, message, acknowledged, acknowledged_workspace, workspace_id, resource)\n        VALUES ('critical_error', $1, $2, $3, $4, $5)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Bool",
+        "Bool",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "044e2b428ee6e2dd4543c87ad8835e239cf7567d18b8b3fa6608ea3a9d206ca7"
+}
@@ -1,83 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "WITH uuid_table as (\n            select gen_random_uuid() as uuid from generate_series(1, $11)\n        )\n        INSERT INTO queue \n            (id, script_hash, script_path, job_kind, language, args, tag, created_by, permissioned_as, email, scheduled_for, workspace_id, concurrent_limit, concurrency_time_window_s, timeout)\n            (SELECT uuid, $1, $2, $3, $4, ('{ \"uuid\": \"' || uuid || '\" }')::jsonb, $5, $6, $7, $8, $9, $10, $12, $13, $14 FROM uuid_table) \n        RETURNING id",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "id",
-        "type_info": "Uuid"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Int8",
-        "Varchar",
-        {
-          "Custom": {
-            "name": "job_kind",
-            "kind": {
-              "Enum": [
-                "script",
-                "preview",
-                "flow",
-                "dependencies",
-                "flowpreview",
-                "script_hub",
-                "identity",
-                "flowdependencies",
-                "http",
-                "graphql",
-                "postgresql",
-                "noop",
-                "appdependencies",
-                "deploymentcallback",
-                "singlescriptflow"
-              ]
-            }
-          }
-        },
-        {
-          "Custom": {
-            "name": "script_lang",
-            "kind": {
-              "Enum": [
-                "python3",
-                "deno",
-                "go",
-                "bash",
-                "postgresql",
-                "nativets",
-                "bun",
-                "mysql",
-                "bigquery",
-                "snowflake",
-                "graphql",
-                "powershell",
-                "mssql",
-                "php",
-                "bunnative",
-                "rust",
-                "ansible"
-              ]
-            }
-          }
-        },
-        "Varchar",
-        "Varchar",
-        "Varchar",
-        "Varchar",
-        "Timestamptz",
-        "Varchar",
-        "Int4",
-        "Int4",
-        "Int4",
-        "Int4"
-      ]
-    },
-    "nullable": [
-      false
-    ]
-  },
-  "hash": "0a686ca61444d7ad7484071727aa039a6ea6697e5a49a633b767c052aa3e0a18"
-}
@@ -1,14 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "DELETE FROM healthchecks WHERE check_type = $1",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "0ee63ef2dd5c88edba2a1f56d31f29876724922f148dad7af35b36efbf70207a"
-}
@@ -0,0 +1,16 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE workspace_settings SET ai_resource = $1, code_completion_enabled = $2 WHERE workspace_id = $3",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Jsonb",
+        "Bool",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "1610c79b8d238e3a35d795f34a2a0100b933b7f199d41cc2f554d57c811d55f3"
+}
@@ -65,8 +65,8 @@
      },
      {
        "ordinal": 12,
-        "name": "openai_resource_path",
-        "type_info": "Varchar"
+        "name": "ai_resource",
+        "type_info": "Jsonb"
      },
      {
        "ordinal": 13,
@@ -117,6 +117,11 @@
        "ordinal": 22,
        "name": "deploy_ui",
        "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 23,
+        "name": "mute_critical_alerts",
+        "type_info": "Bool"
      }
    ],
    "parameters": {
@@ -147,6 +152,7 @@
      true,
      false,
      true,
+      true,
      true
    ]
  },
@@ -0,0 +1,29 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT\n                flow_status AS \"flow_status!: Json<Box<RawValue>>\",\n                raw_flow->'modules'->(flow_status->'step')::int AS \"module: Json<Box<RawValue>>\"\n            FROM queue WHERE id = $1 AND workspace_id = $2 LIMIT 1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "flow_status!: Json<Box<RawValue>>",
+        "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 1,
+        "name": "module: Json<Box<RawValue>>",
+        "type_info": "Jsonb"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Uuid",
+        "Text"
+      ]
+    },
+    "nullable": [
+      true,
+      null
+    ]
+  },
+  "hash": "1e7ce0c140410ae799f9c0c5772e4be4506bfd238c88f1b1c3ddf39b29071446"
+}
@@ -1,6 +1,6 @@
 {
  "db_name": "PostgreSQL",
-  "query": "SELECT value\n            FROM resource\n            WHERE path = $1 AND workspace_id = $2",
+  "query": "SELECT value\n                FROM resource\n                WHERE path = $1 AND workspace_id = $2",
  "describe": {
    "columns": [
      {
@@ -19,5 +19,5 @@
      true
    ]
  },
-  "hash": "acdaa5151f8f7f37bb8c8c5a7d146789887e47db9695fc26b1dfaedd735e1e60"
+  "hash": "286fa00c088df146c08c1934556f06a4243c66787ab8759a8045e60effd2fb77"
 }
@@ -0,0 +1,18 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO alerts (alert_type, message, acknowledged, acknowledged_workspace, workspace_id, resource)\n        VALUES ('recovered_critical_error', $1, $2, $3, $4, $5)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Bool",
+        "Bool",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "28987898c7e4b172d466bf08f33c2da733e71f8a253b0122c17b9e021919809e"
+}
@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT raw_flow->'failure_module' != 'null'::jsonb\n            FROM completed_job\n            WHERE id = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "?column?",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Uuid"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "31dd23f6768052e486668172e15eba1a3f9b6a8e5678a7ee8e5456ff4405d6f9"
+}
@@ -1,20 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT value::BOOLEAN as value FROM global_settings WHERE name = 'critical_alert_mute_ui'",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "value",
-        "type_info": "Bool"
-      }
-    ],
-    "parameters": {
-      "Left": []
-    },
-    "nullable": [
-      null
-    ]
-  },
-  "hash": "33cfb00f61dbae09467bd3732ce8e83c9835ccd30c90cb24788cdb54d1ceb575"
-}
@@ -1,6 +1,6 @@
 {
  "db_name": "PostgreSQL",
-  "query": "SELECT id, alert_type, message, created_at, acknowledged \n             FROM alerts \n             WHERE acknowledged = $1\n             ORDER BY created_at DESC \n             LIMIT $2 OFFSET $3",
+  "query": "SELECT id, alert_type, message, created_at, COALESCE(acknowledged, false) AS acknowledged, workspace_id\n                 FROM alerts\n                 WHERE COALESCE(acknowledged, false) = $1\n                 ORDER BY created_at DESC\n                 LIMIT $2 OFFSET $3",
  "describe": {
    "columns": [
      {
@@ -27,6 +27,11 @@
        "ordinal": 4,
        "name": "acknowledged",
        "type_info": "Bool"
+      },
+      {
+        "ordinal": 5,
+        "name": "workspace_id",
+        "type_info": "Text"
      }
    ],
    "parameters": {
@@ -41,8 +46,9 @@
      false,
      false,
      false,
+      null,
      true
    ]
  },
-  "hash": "0b955f2cff82a2d4ba3840588143e08952f029480d4a42503ecc3c5e70437995"
+  "hash": "344b3a5d9683273a956b5156fed5ab9fdff1a7252198e4ae290d8c8f937ff177"
 }
@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE password SET login_type = $1 WHERE email = $2",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "349396e8fdd96d45875110bc06767e6eb876792ec8f83e9b03c2fb46bb12e0b9"
+}
@@ -1,15 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO alerts (alert_type, message, acknowledged) VALUES ('recovered_critical_error', $1, $2)",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Bool"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "3a94ad52c6b7cde844fa868167248cd9ff63e5fdfa1d93d8fbec32a257b6b05e"
-}
@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE alerts SET acknowledged = true, acknowledged_workspace = true WHERE resource = $1 AND alert_type = 'critical_error'",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "4a3a8207627418ba7b7eabaccd7ec72884f2a1eacf0ab0e0c6ccbf56c654c14a"
+}
@@ -1,15 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO alerts (alert_type, message, acknowledged) VALUES ('critical_error', $1, $2)",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Bool"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "4d22084a5d9860832f30e8f08cbfa1848ed3c1336fa4790f45b8189c4ac97d91"
-}
@@ -1,6 +1,6 @@
 {
  "db_name": "PostgreSQL",
-  "query": "SELECT id, alert_type, message, created_at, acknowledged \n             FROM alerts \n             ORDER BY created_at DESC \n             LIMIT $1 OFFSET $2",
+  "query": "SELECT id, alert_type, message, created_at, COALESCE(acknowledged, false) AS acknowledged, workspace_id\n                 FROM alerts\n                 ORDER BY created_at DESC\n                 LIMIT $1 OFFSET $2",
  "describe": {
    "columns": [
      {
@@ -27,6 +27,11 @@
        "ordinal": 4,
        "name": "acknowledged",
        "type_info": "Bool"
+      },
+      {
+        "ordinal": 5,
+        "name": "workspace_id",
+        "type_info": "Text"
      }
    ],
    "parameters": {
@@ -40,8 +45,9 @@
      false,
      false,
      false,
+      null,
      true
    ]
  },
-  "hash": "cc5ab80241b88c5befea279f16c4ec68cec17b31dcd277b321f652917346496b"
+  "hash": "4d30c5a2894d655741d167f5589f5816583f0bae78a0dd3270dedb377dfa944a"
 }
@@ -65,8 +65,8 @@
      },
      {
        "ordinal": 12,
-        "name": "openai_resource_path",
-        "type_info": "Varchar"
+        "name": "ai_resource",
+        "type_info": "Jsonb"
      },
      {
        "ordinal": 13,
@@ -117,6 +117,11 @@
        "ordinal": 22,
        "name": "deploy_ui",
        "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 23,
+        "name": "mute_critical_alerts",
+        "type_info": "Bool"
      }
    ],
    "parameters": {
@@ -147,6 +152,7 @@
      true,
      false,
      true,
+      true,
      true
    ]
  },
@@ -1,15 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "UPDATE workspace_settings SET openai_resource_path = NULL, code_completion_enabled = $1 WHERE workspace_id = $2",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Bool",
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "6268eabd561502a44e273d6391102278974623f7a7bcad6891d7abadf4d3ea03"
-}
@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE alerts\n         SET\n           acknowledged = true,\n           acknowledged_workspace = CASE\n             WHEN $2::text IS NOT NULL AND workspace_id = $2 THEN true\n             ELSE acknowledged_workspace\n           END\n         WHERE id = $1",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Int4",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "65da41c7ded54cdee8d33211561c068b72294cc99ff44ed0a13179df508ebc6a"
+}
@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE workspace_settings SET ai_resource = NULL, code_completion_enabled = $1 WHERE workspace_id = $2",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Bool",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "6940ddc5ee8d2a1048213d46f52d964b199604ba66dedbe9f89cea727d726a2d"
+}
@@ -0,0 +1,54 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT id, alert_type, message, created_at, COALESCE(acknowledged_workspace, false) AS acknowledged, workspace_id\n                 FROM alerts\n                 WHERE workspace_id = $1\n                 ORDER BY created_at DESC\n                 LIMIT $2 OFFSET $3",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 1,
+        "name": "alert_type",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 2,
+        "name": "message",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 3,
+        "name": "created_at",
+        "type_info": "Timestamptz"
+      },
+      {
+        "ordinal": 4,
+        "name": "acknowledged",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 5,
+        "name": "workspace_id",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Int8",
+        "Int8"
+      ]
+    },
+    "nullable": [
+      false,
+      false,
+      false,
+      false,
+      null,
+      true
+    ]
+  },
+  "hash": "777e7084edf9a5d14f299ff479ae43c8ead47a6a531d4b031a1342c9b2f16506"
+}
@@ -0,0 +1,14 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE alerts \n         SET\n           acknowledged = true,\n           acknowledged_workspace = CASE\n             WHEN $1::text IS NOT NULL THEN true\n             ELSE acknowledged_workspace\n           END\n         WHERE ($1::text IS NOT NULL AND workspace_id = $1)\n            OR ($1::text IS NULL)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "7c32176755c6ea2b6ae531860d436caae3fa256fc0803749ec5107632669adb3"
+}
@@ -0,0 +1,55 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT id, alert_type, message, created_at, COALESCE(acknowledged_workspace, false) AS acknowledged, workspace_id\n                 FROM alerts\n                 WHERE workspace_id = $1 AND COALESCE(acknowledged_workspace, false) = $2\n                 ORDER BY created_at DESC\n                 LIMIT $3 OFFSET $4",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 1,
+        "name": "alert_type",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 2,
+        "name": "message",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 3,
+        "name": "created_at",
+        "type_info": "Timestamptz"
+      },
+      {
+        "ordinal": 4,
+        "name": "acknowledged",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 5,
+        "name": "workspace_id",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Bool",
+        "Int8",
+        "Int8"
+      ]
+    },
+    "nullable": [
+      false,
+      false,
+      false,
+      false,
+      null,
+      true
+    ]
+  },
+  "hash": "7c5a29de07cbe42326a15d4d7fd9714c20b767508e27368be6de38d7b18a3a4d"
+}
@@ -0,0 +1,12 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE alerts SET acknowledged = true",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": []
+  },
+  "hash": "a59fae29ebcc9aa53308b777ead3d2b652618ac454f139db738f4499cb4eb079"
+}
@@ -1,12 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "UPDATE alerts SET acknowledged = true WHERE acknowledged = false",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": []
-    },
-    "nullable": []
-  },
-  "hash": "a7c5008aa7ea43d0afac7d9f19846976ed7af2e90270902f001115e023cb947d"
-}
@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT raw_flow->'failure_module' != 'null'::jsonb\n            FROM queue\n            WHERE id = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "?column?",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Uuid"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "aa6907b7266ee437dfbd77a9bf6c047fda88f3e72a0d10323a5e4020cf857c42"
+}
@@ -1,11 +1,11 @@
 {
  "db_name": "PostgreSQL",
-  "query": "SELECT openai_resource_path FROM workspace_settings WHERE workspace_id = $1",
+  "query": "SELECT wm_version FROM worker_ping WHERE wm_version != $1 AND ping_at > now() - interval '5 minutes'",
  "describe": {
    "columns": [
      {
        "ordinal": 0,
-        "name": "openai_resource_path",
+        "name": "wm_version",
        "type_info": "Varchar"
      }
    ],
@@ -15,8 +15,8 @@
      ]
    },
    "nullable": [
-      true
+      false
    ]
  },
-  "hash": "f8b689ec3e09f14dc02ac9e9f98d252a14d3a2acfadcd9cd6ba27ca9799f4706"
+  "hash": "ad03e5acf10ef94abc37cb9f56b1775c67f075c8bc83458e8be2e242347218d6"
 }
@@ -1,14 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "UPDATE alerts SET acknowledged = true WHERE id = $1",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Int4"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "be3ae231557e794172336bc27d725f862dcf039bbb3d75ced9d54c86f53d2580"
-}
@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT ai_resource FROM workspace_settings WHERE workspace_id = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "ai_resource",
+        "type_info": "Jsonb"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      true
+    ]
+  },
+  "hash": "ceda377fa534656ac12a7f41db77db1f054ec751855c8db7352b0e9c20b63ef8"
+}
@@ -1,6 +1,6 @@
 {
  "db_name": "PostgreSQL",
-  "query": "SELECT raw_flow->'modules'->($1)->'value'->>'type' = 'flow' FROM queue WHERE id = $2",
+  "query": "SELECT raw_flow->'modules'->($1)::text->'value'->>'type' = 'flow' FROM queue WHERE id = $2 LIMIT 1",
  "describe": {
    "columns": [
      {
@@ -19,5 +19,5 @@
      null
    ]
  },
-  "hash": "3e539fef054ad31bc1736e27276087775a721a6ee7ae35b03fd4ce3563ea3838"
+  "hash": "de1abe57b6aa61155f747a3bcb98359f70eade6654b5a884915f07f3ef3fe15e"
 }
@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE workspace_settings SET mute_critical_alerts = $1 WHERE workspace_id = $2",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Bool",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "f22168826350797e88153b65a5b1e906a7868f34c062f154b2ee4f24c57aa5c3"
+}
@@ -843,9 +843,9 @@ dependencies = [

 [[package]]
 name = "aws-sdk-sts"
-version = "1.49.0"
+version = "1.50.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "53dcf5e7d9bd1517b8b998e170e650047cea8a2b85fe1835abe3210713e541b7"
+checksum = "6ada54e5f26ac246dc79727def52f7f8ed38915cb47781e2a72213957dc3a7d5"
 dependencies = [
 "aws-credential-types",
 "aws-runtime",
@@ -1598,9 +1598,9 @@ dependencies = [

 [[package]]
 name = "cc"
-version = "1.2.0"
+version = "1.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1aeb932158bd710538c73702db6945cb68a8fb08c519e6e12706b94263b36db8"
+checksum = "fd9de9f2205d5ef3fd67e685b0df337994ddd4495e2a28d185500d0e1edfea47"
 dependencies = [
 "jobserver",
 "libc",
@@ -1783,9 +1783,9 @@ dependencies = [

 [[package]]
 name = "comfy-table"
-version = "7.1.2"
+version = "7.1.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e0d05af1e006a2407bedef5af410552494ce5be9090444dbbcb57258c1af3d56"
+checksum = "24f165e7b643266ea80cb858aed492ad9280e3e05ce24d4a99d7d7b889b6a4d9"
 dependencies = [
 "strum 0.26.3",
 "strum_macros 0.26.4",
@@ -1994,6 +1994,15 @@ dependencies = [
 "once_cell",
 ]

+[[package]]
+name = "croner"
+version = "2.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09b42c74d1c9fc4680c245ca7287bd631fe73eb364657268b0e65bafdec83d47"
+dependencies = [
+ "chrono",
+]
+
 [[package]]
 name = "crossbeam"
 version = "0.8.4"
@@ -3355,9 +3364,9 @@ dependencies = [

 [[package]]
 name = "flate2"
-version = "1.0.34"
+version = "1.0.35"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a1b589b4dc103969ad3cf85c950899926ec64300a1a46d76c03a6072957036f0"
+checksum = "c936bfdafb507ebbf50b8074c54fa31c5be9a1e7e5f467dd659697041407d07c"
 dependencies = [
 "crc32fast",
 "libz-sys",
@@ -10610,7 +10619,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"

 [[package]]
 name = "windmill"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "axum",
@@ -10652,7 +10661,7 @@ dependencies = [

 [[package]]
 name = "windmill-api"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "argon2",
@@ -10672,6 +10681,7 @@ dependencies = [
 "chrono-tz 0.10.0",
 "cookie 0.17.0",
 "cron",
+ "croner",
 "datafusion",
 "futures",
 "git-version",
@@ -10737,7 +10747,7 @@ dependencies = [

 [[package]]
 name = "windmill-api-client"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "base64 0.22.1",
 "chrono",
@@ -10755,7 +10765,7 @@ dependencies = [

 [[package]]
 name = "windmill-audit"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "chrono",
 "serde",
@@ -10768,7 +10778,7 @@ dependencies = [

 [[package]]
 name = "windmill-autoscaling"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "rsmq_async",
@@ -10783,7 +10793,7 @@ dependencies = [

 [[package]]
 name = "windmill-common"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "async-stream",
@@ -10811,6 +10821,7 @@ dependencies = [
 "rand 0.8.5",
 "regex",
 "reqwest 0.12.9",
+ "semver 1.0.23",
 "serde",
 "serde_json",
 "sha2 0.10.8",
@@ -10829,7 +10840,7 @@ dependencies = [

 [[package]]
 name = "windmill-git-sync"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "regex",
 "rsmq_async",
@@ -10844,7 +10855,7 @@ dependencies = [

 [[package]]
 name = "windmill-indexer"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "bytes",
@@ -10866,7 +10877,7 @@ dependencies = [

 [[package]]
 name = "windmill-macros"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "itertools 0.13.0",
 "lazy_static",
@@ -10878,7 +10889,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "convert_case 0.6.0",
 "serde",
@@ -10887,7 +10898,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-bash"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -10899,7 +10910,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-go"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "gosyn",
@@ -10911,7 +10922,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-graphql"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -10923,7 +10934,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-php"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "itertools 0.13.0",
@@ -10934,7 +10945,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-py"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "itertools 0.13.0",
@@ -10945,7 +10956,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-py-imports"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "async-recursion",
@@ -10963,7 +10974,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-rust"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "convert_case 0.6.0",
@@ -10980,7 +10991,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-sql"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -10992,7 +11003,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-ts"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "lazy_static",
@@ -11010,7 +11021,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-wasm"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "getrandom 0.2.15",
@@ -11031,7 +11042,7 @@ dependencies = [

 [[package]]
 name = "windmill-parser-yaml"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "serde_json",
@@ -11041,7 +11052,7 @@ dependencies = [

 [[package]]
 name = "windmill-queue"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "async-recursion",
@@ -11050,6 +11061,7 @@ dependencies = [
 "chrono",
 "chrono-tz 0.10.0",
 "cron",
+ "croner",
 "futures-core",
 "hex",
 "hmac",
@@ -11074,7 +11086,7 @@ dependencies = [

 [[package]]
 name = "windmill-sql-datatype-parser-wasm"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "wasm-bindgen",
 "wasm-bindgen-test",
@@ -11084,7 +11096,7 @@ dependencies = [

 [[package]]
 name = "windmill-worker"
-version = "1.423.2"
+version = "1.424.0"
 dependencies = [
 "anyhow",
 "async-recursion",
@@ -1,6 +1,6 @@
 [package]
 name = "windmill"
-version = "1.423.2"
+version = "1.424.0"
 authors.workspace = true
 edition.workspace = true

@@ -29,7 +29,7 @@ members = [
 ]

 [workspace.package]
-version = "1.423.2"
+version = "1.424.0"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]
 edition = "2021"

@@ -180,6 +180,7 @@ tokio-util = { version = "^0", features = ["io"] }
 json-pointer = "^0"
 itertools = "^0"
 regex = "^1"
+semver = "^1"

 deno_fetch = "0.195.0"
 deno_tls = "0.158.0"
@@ -1 +1 @@
-e3df64af864f9d3546a97ff21f719c553e34ff1c
+51dcbf93b0d127af9f33fa346cc63fcd2475d4fa
@@ -0,0 +1,7 @@
+-- Add down migration script here
+ALTER TABLE workspace_settings
+ALTER COLUMN ai_resource TYPE text
+USING ai_resource->>'path';
+
+ALTER TABLE workspace_settings
+RENAME COLUMN ai_resource to openai_resource_path;
@@ -0,0 +1,7 @@
+-- Add up migration script here
+ALTER TABLE workspace_settings
+ALTER COLUMN openai_resource_path TYPE jsonb
+USING jsonb_build_object('provider', 'openai', 'path', openai_resource_path);
+
+ALTER TABLE workspace_settings
+RENAME COLUMN openai_resource_path TO ai_resource;
@@ -0,0 +1,4 @@
+ALTER TABLE alerts DROP COLUMN workspace_id;
+ALTER TABLE alerts DROP COLUMN acknowledged_workspace;
+ALTER TABLE alerts DROP COLUMN resource;
+ALTER TABLE workspace_settings DROP COLUMN mute_critical_alerts;
@@ -0,0 +1,4 @@
+ALTER TABLE alerts ADD COLUMN workspace_id TEXT DEFAULT NULL;
+ALTER TABLE alerts ADD COLUMN acknowledged_workspace BOOL DEFAULT NULL;
+ALTER TABLE alerts ADD COLUMN resource TEXT DEFAULT NULL;
+ALTER TABLE workspace_settings ADD COLUMN mute_critical_alerts BOOL DEFAULT NULL;
@@ -7,7 +7,6 @@
 */

 use anyhow::Context;
-use git_version::git_version;
 use monitor::{
    reload_timeout_wait_result_setting, send_current_log_file_to_object_store,
    send_logs_to_object_store,
@@ -31,18 +30,19 @@ use windmill_common::ee::{maybe_renew_license_key_on_start, LICENSE_KEY_ID, LICE

 use windmill_common::{
    global_settings::{
-        BASE_URL_SETTING, BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ERROR_CHANNELS_SETTING,
-        CUSTOM_TAGS_SETTING, DEFAULT_TAGS_PER_WORKSPACE_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING,
-        ENV_SETTINGS, EXPOSE_DEBUG_METRICS_SETTING, EXPOSE_METRICS_SETTING,
-        EXTRA_PIP_INDEX_URL_SETTING, HUB_BASE_URL_SETTING, JOB_DEFAULT_TIMEOUT_SECS_SETTING,
-        JWT_SECRET_SETTING, KEEP_JOB_DIR_SETTING, LICENSE_KEY_SETTING, NPM_CONFIG_REGISTRY_SETTING,
-        OAUTH_SETTING, PIP_INDEX_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING, CRITICAL_ALERT_MUTE_UI_SETTING,
-        REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RETENTION_PERIOD_SECS_SETTING,
-        SAML_METADATA_SETTING, SCIM_TOKEN_SETTING, SMTP_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
+        BASE_URL_SETTING, BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ALERT_MUTE_UI_SETTING,
+        CRITICAL_ERROR_CHANNELS_SETTING, CUSTOM_TAGS_SETTING, DEFAULT_TAGS_PER_WORKSPACE_SETTING,
+        DEFAULT_TAGS_WORKSPACES_SETTING, ENV_SETTINGS, EXPOSE_DEBUG_METRICS_SETTING,
+        EXPOSE_METRICS_SETTING, EXTRA_PIP_INDEX_URL_SETTING, HUB_BASE_URL_SETTING,
+        JOB_DEFAULT_TIMEOUT_SECS_SETTING, JWT_SECRET_SETTING, KEEP_JOB_DIR_SETTING,
+        LICENSE_KEY_SETTING, NPM_CONFIG_REGISTRY_SETTING, OAUTH_SETTING, PIP_INDEX_URL_SETTING,
+        REQUEST_SIZE_LIMIT_SETTING, REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING,
+        RETENTION_PERIOD_SECS_SETTING, SAML_METADATA_SETTING, SCIM_TOKEN_SETTING, SMTP_SETTING,
+        TIMEOUT_WAIT_RESULT_SETTING,
    },
    scripts::ScriptLang,
    stats_ee::schedule_stats,
-    utils::{hostname, rd_string, Mode},
+    utils::{hostname, rd_string, Mode, GIT_VERSION},
    worker::{reload_custom_tags_setting, HUB_CACHE_DIR, TMP_DIR, WORKER_GROUP},
    DB, METRICS_ENABLED,
 };
@@ -67,24 +67,24 @@ use windmill_worker::{
    get_hub_script_content_and_requirements, BUN_BUNDLE_CACHE_DIR, BUN_CACHE_DIR,
    BUN_DEPSTAR_CACHE_DIR, DENO_CACHE_DIR, DENO_CACHE_DIR_DEPS, DENO_CACHE_DIR_NPM,
    GO_BIN_CACHE_DIR, GO_CACHE_DIR, LOCK_CACHE_DIR, PIP_CACHE_DIR, POWERSHELL_CACHE_DIR,
-    RUST_CACHE_DIR, TAR_PIP_CACHE_DIR, TMP_LOGS_DIR, UV_CACHE_DIR,
+    PY311_CACHE_DIR, RUST_CACHE_DIR, TAR_PIP_CACHE_DIR, TMP_LOGS_DIR, UV_CACHE_DIR,
 };

 use crate::monitor::{
    initial_load, load_keep_job_dir, load_metrics_debug_enabled, load_require_preexisting_user,
    load_tag_per_workspace_enabled, load_tag_per_workspace_workspaces, monitor_db, monitor_pool,
    reload_base_url_setting, reload_bunfig_install_scopes_setting,
-    reload_critical_error_channels_setting, reload_extra_pip_index_url_setting,
-    reload_hub_base_url_setting, reload_job_default_timeout_setting, reload_jwt_secret_setting,
-    reload_license_key, reload_npm_config_registry_setting, reload_pip_index_url_setting,
+    reload_critical_alert_mute_ui_setting, reload_critical_error_channels_setting,
+    reload_extra_pip_index_url_setting, reload_hub_base_url_setting,
+    reload_job_default_timeout_setting, reload_jwt_secret_setting, reload_license_key,
+    reload_npm_config_registry_setting, reload_pip_index_url_setting,
    reload_retention_period_setting, reload_scim_token_setting, reload_smtp_config,
-    reload_worker_config, reload_critical_alert_mute_ui_setting,
+    reload_worker_config,
 };

 #[cfg(feature = "parquet")]
 use crate::monitor::reload_s3_cache_setting;

-const GIT_VERSION: &str = git_version!(args = ["--tag", "--always"], fallback = "unknown-version");
 const DEFAULT_NUM_WORKERS: usize = 1;
 const DEFAULT_PORT: u16 = 8000;
 const DEFAULT_SERVER_BIND_ADDR: Ipv4Addr = Ipv4Addr::new(0, 0, 0, 0);
@@ -997,13 +997,14 @@ pub async fn run_workers<R: rsmq_async::RsmqConnection + Send + Sync + Clone + '
    for x in [
        LOCK_CACHE_DIR,
        TMP_LOGS_DIR,
-        PIP_CACHE_DIR,
        UV_CACHE_DIR,
        TAR_PIP_CACHE_DIR,
        DENO_CACHE_DIR,
        DENO_CACHE_DIR_DEPS,
        DENO_CACHE_DIR_NPM,
        BUN_CACHE_DIR,
+        PY311_CACHE_DIR,
+        PIP_CACHE_DIR,
        BUN_DEPSTAR_CACHE_DIR,
        BUN_BUNDLE_CACHE_DIR,
        GO_CACHE_DIR,
@@ -34,12 +34,12 @@ use windmill_common::{
    error,
    flow_status::FlowStatusModule,
    global_settings::{
-        BASE_URL_SETTING, BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ERROR_CHANNELS_SETTING,
-        DEFAULT_TAGS_PER_WORKSPACE_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING,
-        EXPOSE_DEBUG_METRICS_SETTING, EXPOSE_METRICS_SETTING, EXTRA_PIP_INDEX_URL_SETTING,
-        HUB_BASE_URL_SETTING, JOB_DEFAULT_TIMEOUT_SECS_SETTING, JWT_SECRET_SETTING,
-        KEEP_JOB_DIR_SETTING, LICENSE_KEY_SETTING, NPM_CONFIG_REGISTRY_SETTING, OAUTH_SETTING,
-        PIP_INDEX_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING, CRITICAL_ALERT_MUTE_UI_SETTING,
+        BASE_URL_SETTING, BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ALERT_MUTE_UI_SETTING,
+        CRITICAL_ERROR_CHANNELS_SETTING, DEFAULT_TAGS_PER_WORKSPACE_SETTING,
+        DEFAULT_TAGS_WORKSPACES_SETTING, EXPOSE_DEBUG_METRICS_SETTING, EXPOSE_METRICS_SETTING,
+        EXTRA_PIP_INDEX_URL_SETTING, HUB_BASE_URL_SETTING, JOB_DEFAULT_TIMEOUT_SECS_SETTING,
+        JWT_SECRET_SETTING, KEEP_JOB_DIR_SETTING, LICENSE_KEY_SETTING, NPM_CONFIG_REGISTRY_SETTING,
+        OAUTH_SETTING, PIP_INDEX_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING,
        REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RETENTION_PERIOD_SECS_SETTING,
        SAML_METADATA_SETTING, SCIM_TOKEN_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
    },
@@ -51,11 +51,11 @@ use windmill_common::{
    utils::{now_from_db, rd_string, report_critical_error, Mode},
    worker::{
        load_worker_config, make_pull_query, make_suspended_pull_query, reload_custom_tags_setting,
-        DEFAULT_TAGS_PER_WORKSPACE, DEFAULT_TAGS_WORKSPACES, SMTP_CONFIG, WORKER_CONFIG,
-        WORKER_GROUP,
+        update_min_version, DEFAULT_TAGS_PER_WORKSPACE, DEFAULT_TAGS_WORKSPACES, SMTP_CONFIG,
+        WORKER_CONFIG, WORKER_GROUP,
    },
-    BASE_URL, CRITICAL_ERROR_CHANNELS, DB, DEFAULT_HUB_BASE_URL, HUB_BASE_URL, JOB_RETENTION_SECS,
-    METRICS_DEBUG_ENABLED, METRICS_ENABLED, CRITICAL_ALERT_MUTE_UI_ENABLED
+    BASE_URL, CRITICAL_ALERT_MUTE_UI_ENABLED, CRITICAL_ERROR_CHANNELS, DB, DEFAULT_HUB_BASE_URL,
+    HUB_BASE_URL, JOB_RETENTION_SECS, METRICS_DEBUG_ENABLED, METRICS_ENABLED,
 };
 use windmill_queue::cancel_job;
 use windmill_worker::{
@@ -231,13 +231,20 @@ pub async fn load_tag_per_workspace_workspaces(db: &DB) -> error::Result<()> {
 }

 pub async fn reload_critical_alert_mute_ui_setting(db: &DB) -> error::Result<()> {
-    let mute = load_value_from_global_settings(db, CRITICAL_ALERT_MUTE_UI_SETTING).await;
-    match mute {
-        Ok(Some(serde_json::Value::Bool(t))) => {
-            CRITICAL_ALERT_MUTE_UI_ENABLED.store(t, Ordering::Relaxed);
+    if let Ok(Some(serde_json::Value::Bool(t))) =
+        load_value_from_global_settings(db, CRITICAL_ALERT_MUTE_UI_SETTING).await
+    {
+        CRITICAL_ALERT_MUTE_UI_ENABLED.store(t, Ordering::Relaxed);
+
+        if t {
+            if let Err(e) = sqlx::query!("UPDATE alerts SET acknowledged = true")
+                .execute(db)
+                .await
+            {
+                tracing::error!("Error updating alerts: {}", e.to_string());
+            }
        }
-        _ => (),
-    };
+    }
    Ok(())
 }

@@ -1080,6 +1087,10 @@ pub async fn monitor_db(
        }
    };

+    let update_min_worker_version_f = async {
+        update_min_version(db).await;
+    };
+
    join!(
        expired_items_f,
        zombie_jobs_f,
@@ -1088,6 +1099,7 @@ pub async fn monitor_db(
        worker_groups_alerts_f,
        jobs_waiting_alerts_f,
        apply_autoscaling_f,
+        update_min_worker_version_f,
    );
 }

@@ -1318,7 +1330,7 @@ async fn handle_zombie_jobs<R: rsmq_async::RsmqConnection + Send + Sync + Clone>
                ON CONFLICT (job_id) DO UPDATE SET logs = job_logs.logs || '\nRestarted job after not receiving job''s ping for too long the ' || now() || '\n\n' WHERE job_logs.job_id = $1", r.id)
                .execute(db).await;
            tracing::error!(error_message);
-            report_critical_error(error_message, db.clone()).await;
+            report_critical_error(error_message, db.clone(), Some(&r.workspace_id), None).await;
        }
    }

@@ -1432,7 +1444,7 @@ async fn handle_zombie_flows(
                flow.id, flow.workspace_id
            );
            tracing::error!(error_message);
-            report_critical_error(error_message, db.clone()).await;
+            report_critical_error(error_message, db.clone(), Some(&flow.workspace_id), None).await;
            // if the flow hasn't started and is a zombie, we can simply restart it
            sqlx::query!(
                "UPDATE queue SET running = false, started_at = null WHERE id = $1 AND canceled = false",
@@ -1452,7 +1464,7 @@ async fn handle_zombie_flows(
                    format!("Flow {id} was cancelled because it")
                }
            );
-            report_critical_error(reason.clone(), db.clone()).await;
+            report_critical_error(reason.clone(), db.clone(), Some(&flow.workspace_id), None).await;
            cancel_zombie_flow_job(db, flow, &rsmq, reason).await?;
        }
    }
@@ -101,3 +101,4 @@ pin-project.workspace = true
 http.workspace = true
 async-stream.workspace = true
 ulid.workspace = true
+croner = "2.0.6"
@@ -1,7 +1,7 @@
 {
  "openapi": "3.0.3",
  "info": {
-    "version": "1.418.0",
+    "version": "1.423.2",
    "title": "Windmill API",
    "contact": {
      "name": "Windmill Team",
@@ -286,7 +286,7 @@
        }
      }
    },
-    "/w/{workspace}/users/{username}": {
+    "/w/{workspace}/users/get/{username}": {
      "get": {
        "summary": "get user (require admin privilege)",
        "operationId": "getUser",
@@ -436,6 +436,106 @@
        }
      }
    },
+    "/users/set_password_of/{user}": {
+      "post": {
+        "summary": "set password for a specific user (require super admin)",
+        "operationId": "setPasswordForUser",
+        "tags": [
+          "user"
+        ],
+        "parameters": [
+          {
+            "name": "user",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string"
+            }
+          }
+        ],
+        "requestBody": {
+          "description": "set password",
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "type": "object",
+                "properties": {
+                  "password": {
+                    "type": "string"
+                  }
+                },
+                "required": [
+                  "password"
+                ]
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "password set",
+            "content": {
+              "text/plain": {
+                "schema": {
+                  "type": "string"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/users/set_login_type/{user}": {
+      "post": {
+        "summary": "set login type for a specific user (require super admin)",
+        "operationId": "setLoginTypeForUser",
+        "tags": [
+          "user"
+        ],
+        "parameters": [
+          {
+            "name": "user",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string"
+            }
+          }
+        ],
+        "requestBody": {
+          "description": "set login type",
+          "required": true,
+          "content": {
+            "application/json": {
+              "schema": {
+                "type": "object",
+                "properties": {
+                  "login_type": {
+                    "type": "string"
+                  }
+                },
+                "required": [
+                  "login_type"
+                ]
+              }
+            }
+          }
+        },
+        "responses": {
+          "200": {
+            "description": "login type set",
+            "content": {
+              "text/plain": {
+                "schema": {
+                  "type": "string"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
    "/users/create": {
      "post": {
        "summary": "create user",
@@ -1105,6 +1205,9 @@
                      },
                      "tls_implicit": {
                        "type": "boolean"
+                      },
+                      "disable_tls": {
+                        "type": "boolean"
                      }
                    },
                    "required": [
@@ -1113,7 +1216,8 @@
                      "password",
                      "port",
                      "from",
-                      "tls_implicit"
+                      "tls_implicit",
+                      "disable_tls"
                    ]
                  }
                },
@@ -2460,8 +2564,8 @@
                    "deploy_to": {
                      "type": "string"
                    },
-                    "openai_resource_path": {
-                      "type": "string"
+                    "ai_resource": {
+                      "$ref": "#/components/schemas/AiResource"
                    },
                    "code_completion_enabled": {
                      "type": "boolean"
@@ -2901,8 +3005,8 @@
                  "code_completion_enabled"
                ],
                "properties": {
-                  "openai_resource_path": {
-                    "type": "string"
+                  "ai_resource": {
+                    "$ref": "#/components/schemas/AiResource"
                  },
                  "code_completion_enabled": {
                    "type": "boolean"
@@ -2946,7 +3050,10 @@
                "schema": {
                  "type": "object",
                  "properties": {
-                    "exists_openai_resource_path": {
+                    "ai_provider": {
+                      "type": "string"
+                    },
+                    "exists_ai_resource": {
                      "type": "boolean"
                    },
                    "code_completion_enabled": {
@@ -2954,7 +3061,8 @@
                    }
                  },
                  "required": [
-                    "exists_openai_resource_path",
+                    "ai_provider",
+                    "exists_ai_resource",
                    "code_completion_enabled"
                  ]
                }
@@ -13160,6 +13268,30 @@
        }
      }
    },
+    "/workers/queue_counts": {
+      "get": {
+        "summary": "get counts of jobs waiting for an executor per tag",
+        "operationId": "getCountsOfJobsWaitingPerTag",
+        "tags": [
+          "worker"
+        ],
+        "responses": {
+          "200": {
+            "description": "queue counts",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "object",
+                  "additionalProperties": {
+                    "type": "integer"
+                  }
+                }
+              }
+            }
+          }
+        }
+      }
+    },
    "/configs/list_worker_groups": {
      "get": {
        "summary": "list worker groups",
@@ -14810,6 +14942,20 @@
            "schema": {
              "type": "string"
            }
+          },
+          {
+            "name": "content_type",
+            "in": "query",
+            "schema": {
+              "type": "string"
+            }
+          },
+          {
+            "name": "content_disposition",
+            "in": "query",
+            "schema": {
+              "type": "string"
+            }
          }
        ],
        "requestBody": {
@@ -16116,6 +16262,21 @@
      }
    },
    "schemas": {
+      "AiResource": {
+        "type": "object",
+        "properties": {
+          "path": {
+            "type": "string"
+          },
+          "provider": {
+            "type": "string"
+          }
+        },
+        "required": [
+          "path",
+          "provider"
+        ]
+      },
      "Script": {
        "type": "object",
        "properties": {
@@ -17226,7 +17387,7 @@
              "jobs.disapproval",
              "jobs.delete",
              "account.delete",
-              "openai.request",
+              "ai.request",
              "resources.create",
              "resources.update",
              "resources.delete",
@@ -18067,6 +18228,23 @@
          "route_path": {
            "type": "string"
          },
+          "static_asset_config": {
+            "type": "object",
+            "properties": {
+              "s3": {
+                "type": "string"
+              },
+              "storage": {
+                "type": "string"
+              },
+              "filename": {
+                "type": "string"
+              }
+            },
+            "required": [
+              "s3"
+            ]
+          },
          "is_flow": {
            "type": "boolean"
          },
@@ -18126,6 +18304,23 @@
          "route_path": {
            "type": "string"
          },
+          "static_asset_config": {
+            "type": "object",
+            "properties": {
+              "s3": {
+                "type": "string"
+              },
+              "storage": {
+                "type": "string"
+              },
+              "filename": {
+                "type": "string"
+              }
+            },
+            "required": [
+              "s3"
+            ]
+          },
          "is_flow": {
            "type": "boolean"
          },
@@ -18168,6 +18363,23 @@
          "route_path": {
            "type": "string"
          },
+          "static_asset_config": {
+            "type": "object",
+            "properties": {
+              "s3": {
+                "type": "string"
+              },
+              "storage": {
+                "type": "string"
+              },
+              "filename": {
+                "type": "string"
+              }
+            },
+            "required": [
+              "s3"
+            ]
+          },
          "is_flow": {
            "type": "boolean"
          },
@@ -18917,6 +19129,12 @@
              "type": "object"
            }
          },
+          "s3_inputs": {
+            "type": "array",
+            "items": {
+              "type": "object"
+            }
+          },
          "execution_mode": {
            "type": "string",
            "enum": [
@@ -1,7 +1,7 @@
 openapi: "3.0.3"

 info:
-  version: 1.423.2
+  version: 1.424.0
  title: Windmill API

  contact:
@@ -283,6 +283,71 @@ paths:
            text/plain:
              schema:
                type: string
+                
+  /users/set_password_of/{user}:
+    post:
+      summary: set password for a specific user (require super admin)
+      operationId: setPasswordForUser
+      tags:
+        - user
+      parameters:
+        - name: user
+          in: path
+          required: true
+          schema:
+            type: string
+      requestBody:
+        description: set password
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                password:
+                  type: string
+              required:
+                - password
+      responses:
+        "200":
+          description: password set
+          content:
+            text/plain:
+              schema:
+                type: string
+
+  /users/set_login_type/{user}:
+    post:
+      summary: set login type for a specific user (require super admin)
+      operationId: setLoginTypeForUser
+      tags:
+        - user
+      parameters:
+        - name: user
+          in: path
+          required: true
+          schema:
+            type: string
+      requestBody:
+        description: set login type
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                login_type:
+                  type: string
+              required:
+                - login_type
+      responses:
+        "200":
+          description: login type set
+          content:
+            text/plain:
+              schema:
+                type: string
+

  /users/create:
    post:
@@ -1616,8 +1681,8 @@ paths:
                    type: string
                  deploy_to:
                    type: string
-                  openai_resource_path:
-                    type: string
+                  ai_resource:
+                    $ref: "#/components/schemas/AiResource"
                  code_completion_enabled:
                    type: boolean
                  error_handler:
@@ -1636,6 +1701,8 @@ paths:
                    type: string
                  default_scripts:
                    $ref: "#/components/schemas/WorkspaceDefaultScripts"
+                  mute_critical_alerts:
+                    type: boolean
                required:
                  - code_completion_enabled
                  - automatic_billing
@@ -1901,8 +1968,8 @@ paths:
              required:
                - code_completion_enabled
              properties:
-                openai_resource_path:
-                  type: string
+                ai_resource:
+                  $ref: "#/components/schemas/AiResource"
                code_completion_enabled:
                  type: boolean
      responses:
@@ -1930,12 +1997,15 @@ paths:
              schema:
                type: object
                properties:
-                  exists_openai_resource_path:
+                  ai_provider:
+                    type: string
+                  exists_ai_resource:
                    type: boolean
                  code_completion_enabled:
                    type: boolean
                required:
-                  - exists_openai_resource_path
+                  - ai_provider
+                  - exists_ai_resource
                  - code_completion_enabled

  /w/{workspace}/workspaces/edit_error_handler:
@@ -2649,6 +2719,112 @@ paths:
                items:
                  $ref: "#/components/schemas/ContextualVariable"

+  /w/{workspace}/workspaces/critical_alerts:
+    get:
+      summary: Get all critical alerts for this workspace
+      operationId: workspaceGetCriticalAlerts
+      tags:
+        - setting
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+        - in: query
+          name: page
+          schema:
+            type: integer
+            default: 1
+            description: The page number to retrieve (minimum value is 1)
+        - in: query
+          name: page_size
+          schema:
+            type: integer
+            default: 10
+            maximum: 100
+            description: Number of alerts per page (maximum is 100)
+        - in: query
+          name: acknowledged
+          schema:
+            type: boolean
+            nullable: true
+            description: Filter by acknowledgment status; true for acknowledged, false for unacknowledged, and omit for all alerts
+      responses:
+        "200":
+          description: Successfully retrieved all critical alerts
+          content:
+            application/json:
+              schema:
+                type: array
+                items:
+                  $ref: '#/components/schemas/CriticalAlert'
+
+  /w/{workspace}/workspaces/critical_alerts/{id}/acknowledge:
+    post:
+      summary: Acknowledge a critical alert for this workspace
+      operationId: workspaceAcknowledgeCriticalAlert
+      tags:
+        - setting
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+        - in: path
+          name: id
+          required: true
+          schema:
+            type: integer
+          description: The ID of the critical alert to acknowledge
+      responses:
+        "200":
+          description: Successfully acknowledged the critical alert
+          content:
+            application/json:
+              schema:
+                type: string
+                example: "Critical alert acknowledged"
+
+  /w/{workspace}/workspaces/critical_alerts/acknowledge_all:
+    post:
+      summary: Acknowledge all unacknowledged critical alerts for this workspace
+      operationId: workspaceAcknowledgeAllCriticalAlerts
+      tags:
+        - setting
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+      responses:
+        "200":
+          description: Successfully acknowledged all unacknowledged critical alerts.
+          content:
+            application/json:
+              schema:
+                type: string
+                example: "All unacknowledged critical alerts acknowledged"
+
+  /w/{workspace}/workspaces/critical_alerts/mute:
+    post:
+      summary: Mute critical alert UI for this workspace
+      operationId: workspaceMuteCriticalAlertsUI
+      tags:
+        - setting
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+      requestBody:
+        description: Boolean flag to mute critical alerts.
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                mute_critical_alerts:
+                  type: boolean
+                  description: Whether critical alerts should be muted.
+                  example: true
+      responses:
+        '200':
+          description: Successfully updated mute critical alert settings.
+          content:
+            application/json:
+              schema:
+                type: string
+                example: "Updated mute critical alert UI settings for workspace: workspace_id"
+
  /oauth/login_callback/{client_name}:
    post:
      security: []
@@ -5986,6 +6162,39 @@ paths:
                required:
                  - database_length

+  /w/{workspace}/jobs/completed/count_jobs:
+    get:
+      summary: count number of completed jobs with filter
+      operationId: countCompletedJobs
+      tags:
+        - job
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+        - name: completed_after_s_ago
+          in: query
+          schema:
+            type: integer
+        - name: success
+          in: query
+          schema:
+            type: boolean
+        - name: tags
+          in: query
+          schema:
+            type: string
+        - name: all_workspaces
+          in: query
+          schema:
+            type: boolean
+      responses:
+        "200":
+          description: Count of completed jobs
+          content:
+            application/json:
+              schema:
+                type: integer
+
+
  /w/{workspace}/jobs/queue/list_filtered_uuids:
    get:
      summary: get the ids of all jobs matching the given filters
@@ -8188,6 +8397,23 @@ paths:
                    - id
                    - values

+  /workers/queue_counts:
+    get:
+      summary: get counts of jobs waiting for an executor per tag
+      operationId: getCountsOfJobsWaitingPerTag
+      tags:
+        - worker
+      responses:
+        "200":
+          description: queue counts
+          content:
+            application/json:
+              schema: 
+                type: object
+                additionalProperties:
+                  type: integer
+
+              
  /configs/list_worker_groups:
    get:
      summary: list worker groups
@@ -10112,6 +10338,16 @@ components:
  schemas:
    $ref: "../../openflow.openapi.yaml#/components/schemas"

+    AiResource:
+      type: object
+      properties:
+        path:
+          type: string
+        provider:
+          type: string
+      required:
+        - path
+        - provider
    Script:
      type: object
      properties:
@@ -10930,7 +11166,7 @@ components:
            - "jobs.disapproval"
            - "jobs.delete"
            - "account.delete"
-            - "openai.request"
+            - "ai.request"
            - "resources.create"
            - "resources.update"
            - "resources.delete"
@@ -12817,3 +13053,7 @@ components:
          type: boolean
          nullable: true
          description: Acknowledgment status of the alert, can be true, false, or null if not set
+        workspace_id:
+          type: string
+          nullable: true
+          description: Workspace id if the alert is in the scope of a workspace
@@ -0,0 +1,507 @@
+use crate::{
+    db::{ApiAuthed, DB},
+    variables::decrypt,
+};
+use anthropic::AnthropicCache;
+use axum::{
+    body::Bytes,
+    extract::{Path, Query},
+    response::IntoResponse,
+    routing::post,
+    Extension, Router,
+};
+use lazy_static::lazy_static;
+use mistral::MistralCache;
+use openai::OpenaiCache;
+use quick_cache::sync::Cache;
+use reqwest::{Client, RequestBuilder};
+use serde::{Deserialize, Deserializer};
+use windmill_audit::audit_ee::audit_log;
+use windmill_audit::ActionKind;
+use windmill_common::error::{to_anyhow, Result};
+use windmill_common::variables::build_crypt;
+
+use windmill_common::error::Error;
+
+use serde_json::value::{RawValue, Value};
+use std::collections::HashMap;
+
+lazy_static::lazy_static! {
+    static ref HTTP_CLIENT: Client = reqwest::ClientBuilder::new()
+        .timeout(std::time::Duration::from_secs(60 * 5))
+        .user_agent("windmill/beta")
+        .build().unwrap();
+}
+
+trait AiRequest {
+    fn prepare_request(self, path: &str, body: Bytes) -> Result<RequestBuilder>;
+}
+
+mod openai {
+    use super::*;
+
+    use super::{get_variable_or_self, KeyCache};
+
+    const API_VERSION: &str = "2023-05-15";
+
+    #[derive(Deserialize, Debug)]
+    struct OpenaiResource {
+        api_key: String,
+        organization_id: Option<String>,
+    }
+
+    #[derive(Deserialize, Debug)]
+    struct OpenaiClientCredentialsOauthResource {
+        client_id: String,
+        client_secret: String,
+        token_url: String,
+        user: Option<String>,
+    }
+
+    #[derive(Deserialize, Debug)]
+    #[serde(untagged, rename_all = "snake_case")]
+    enum OpenaiConfig {
+        Resource(OpenaiResource),
+        ClientCredentialsOauthResource(OpenaiClientCredentialsOauthResource),
+    }
+
+    lazy_static::lazy_static! {
+        pub static ref OPENAI_AZURE_BASE_PATH: Option<String> = std::env::var("OPENAI_AZURE_BASE_PATH").ok();
+    }
+
+    #[derive(Deserialize, Debug)]
+    struct OpenaiCredentials {
+        access_token: String,
+    }
+
+    #[derive(Clone, Debug, Deserialize)]
+    pub struct OpenaiCache {
+        api_key: String,
+        organization_id: Option<String>,
+        azure_base_path: Option<String>,
+        user: Option<String>,
+    }
+
+    impl OpenaiCache {
+        pub fn new(
+            api_key: String,
+            organization_id: Option<String>,
+            azure_base_path: Option<String>,
+            user: Option<String>,
+        ) -> Self {
+            Self { api_key, organization_id, azure_base_path, user }
+        }
+    }
+
+    const BASE_URL: &str = "https://api.openai.com/v1";
+    impl AiRequest for OpenaiCache {
+        fn prepare_request(self, openai_path: &str, mut body: Bytes) -> Result<RequestBuilder> {
+            let OpenaiCache { api_key, azure_base_path, organization_id, user } = self;
+            if user.is_some() {
+                tracing::debug!("Adding user to request body");
+                let mut json_body: HashMap<String, Box<RawValue>> = serde_json::from_slice(&body)
+                    .map_err(|e| {
+                    Error::InternalErr(format!("Failed to parse request body: {}", e))
+                })?;
+
+                let user_json_string = serde_json::Value::String(user.unwrap()).to_string(); // makes sure to escape characters
+
+                json_body.insert(
+                    "user".to_string(),
+                    RawValue::from_string(user_json_string)
+                        .map_err(|e| Error::InternalErr(format!("Failed to parse user: {}", e)))?,
+                );
+
+                body = serde_json::to_vec(&json_body)
+                    .map_err(|e| {
+                        Error::InternalErr(format!("Failed to reserialize request body: {}", e))
+                    })?
+                    .into();
+            }
+
+            let base_url = if let Some(base_url) = azure_base_path {
+                base_url
+            } else {
+                BASE_URL.to_string()
+            };
+            let url = format!("{}/{}", base_url, openai_path);
+            let mut request = HTTP_CLIENT
+                .post(url)
+                .header("content-type", "application/json")
+                .body(body);
+
+            if base_url != BASE_URL {
+                request = request
+                    .header("api-key", api_key)
+                    .query(&[("api-version", API_VERSION)])
+            } else {
+                request = request.header("authorization", format!("Bearer {}", api_key))
+            }
+
+            if let Some(org_id) = organization_id {
+                request = request.header("OpenAI-Organization", org_id);
+            }
+
+            Ok(request)
+        }
+    }
+
+    async fn get_openai_key_using_credentials_flow(
+        mut resource: OpenaiClientCredentialsOauthResource,
+        db: &DB,
+        w_id: &str,
+    ) -> Result<String> {
+        resource.client_id = get_variable_or_self(resource.client_id, db, w_id).await?;
+        resource.client_secret = get_variable_or_self(resource.client_secret, db, w_id).await?;
+        resource.token_url = get_variable_or_self(resource.token_url, db, w_id).await?;
+        let mut params = HashMap::new();
+        params.insert("grant_type", "client_credentials");
+        let response = HTTP_CLIENT
+            .post(resource.token_url)
+            .form(&params)
+            .basic_auth(resource.client_id, Some(resource.client_secret))
+            .send()
+            .await
+            .map_err(|err| {
+                Error::InternalErr(format!(
+                    "Failed to get OpenAI credentials using credentials flow: {}",
+                    err
+                ))
+            })?;
+        let response = response.json::<OpenaiCredentials>().await.map_err(|err| {
+            Error::InternalErr(format!(
+                "Failed to parse OpenAI credentials from credentials flow: {}",
+                err
+            ))
+        })?;
+        Ok(response.access_token)
+    }
+
+    pub async fn get_cached_value(db: &DB, w_id: &str, resource: Value) -> Result<KeyCache> {
+        let config = serde_json::from_value(resource)
+            .map_err(|e| Error::InternalErr(format!("validating openai resource {e:#}")))?;
+
+        let mut user = None::<String>;
+        let mut resource = match config {
+            OpenaiConfig::Resource(resource) => {
+                tracing::debug!("Getting OpenAI key from static resource");
+                resource
+            }
+            OpenaiConfig::ClientCredentialsOauthResource(resource) => {
+                tracing::debug!("Getting OpenAI key with client credentials flow");
+                user = resource.user.clone();
+                let token = get_openai_key_using_credentials_flow(resource, db, w_id).await?;
+                OpenaiResource { api_key: token, organization_id: None }
+            }
+        };
+
+        resource.api_key = get_variable_or_self(resource.api_key, db, w_id).await?;
+
+        if let Some(organization_id) = resource.organization_id {
+            resource.organization_id = Some(get_variable_or_self(organization_id, db, w_id).await?);
+        }
+
+        if user.is_some() {
+            user = Some(get_variable_or_self(user.unwrap(), db, w_id).await?);
+        }
+
+        let azure_base_path = sqlx::query_scalar!(
+            "SELECT value
+            FROM global_settings
+            WHERE name = 'openai_azure_base_path'",
+        )
+        .fetch_optional(db)
+        .await?;
+
+        let azure_base_path = if let Some(azure_base_path) = azure_base_path {
+            Some(
+                serde_json::from_value::<String>(azure_base_path).map_err(|e| {
+                    Error::InternalErr(format!("validating openai azure base path {e:#}"))
+                })?,
+            )
+        } else {
+            OPENAI_AZURE_BASE_PATH.clone()
+        };
+
+        let workspace_cache = OpenaiCache::new(
+            resource.api_key.clone(),
+            resource.organization_id.clone(),
+            azure_base_path.clone(),
+            user.clone(),
+        );
+        Ok(KeyCache::Openai(workspace_cache))
+    }
+}
+
+mod anthropic {
+
+    use super::*;
+
+    #[derive(Clone, Deserialize, Debug)]
+    pub struct AnthropicCache {
+        #[serde(rename = "apiKey")]
+        pub api_key: String,
+    }
+
+    const API_VERSION: &str = "2023-06-01";
+
+    impl AnthropicCache {
+        pub fn new(api_key: String) -> Self {
+            Self { api_key }
+        }
+    }
+
+    const BASE_URL: &str = "https://api.anthropic.com";
+    impl AiRequest for AnthropicCache {
+        fn prepare_request(self, anthropic_path: &str, body: Bytes) -> Result<RequestBuilder> {
+            let AnthropicCache { api_key } = self;
+            let url = format!("{}/{}", BASE_URL, anthropic_path);
+            let request = HTTP_CLIENT
+                .post(url)
+                .header("x-api-key", api_key)
+                .header("anthropic-version", API_VERSION)
+                .header("content-type", "application/json")
+                .body(body);
+            Ok(request)
+        }
+    }
+
+    pub async fn get_cached_value(db: &DB, w_id: &str, resource: Value) -> Result<KeyCache> {
+        let mut resource: AnthropicCache = serde_json::from_value(resource)
+            .map_err(|e| Error::InternalErr(format!("validating anthropic resource {e:#}")))?;
+        resource.api_key = get_variable_or_self(resource.api_key, db, w_id).await?;
+        let workspace_cache = AnthropicCache::new(resource.api_key);
+        Ok(KeyCache::Anthropic(workspace_cache))
+    }
+}
+
+mod mistral {
+    use super::*;
+    #[derive(Deserialize, Clone, Debug)]
+    pub struct MistralCache {
+        #[serde(rename = "apiKey")]
+        pub api_key: String,
+    }
+
+    impl MistralCache {
+        pub fn new(api_key: String) -> Self {
+            Self { api_key }
+        }
+    }
+
+    const BASE_URL: &str = "https://api.mistral.ai";
+    impl AiRequest for MistralCache {
+        fn prepare_request(self, mistral_path: &str, body: Bytes) -> Result<RequestBuilder> {
+            let MistralCache { api_key } = self;
+
+            let url = format!("{}/{}", BASE_URL, mistral_path);
+            let request = HTTP_CLIENT
+                .post(url)
+                .header("content-type", "application/json")
+                .header("Accept", "application/json")
+                .header("authorization", format!("Bearer {}", api_key))
+                .body(body);
+            Ok(request)
+        }
+    }
+
+    pub async fn get_cached_value(db: &DB, w_id: &str, resource: Value) -> Result<KeyCache> {
+        let mut resource: MistralCache = serde_json::from_value(resource)
+            .map_err(|e| Error::InternalErr(format!("validating mistral resource {e:#}")))?;
+        resource.api_key = get_variable_or_self(resource.api_key, db, w_id).await?;
+
+        let workspace_cache = MistralCache::new(resource.api_key);
+        Ok(KeyCache::Mistral(workspace_cache))
+    }
+}
+
+#[derive(Clone, Debug)]
+pub enum KeyCache {
+    Openai(OpenaiCache),
+    Anthropic(AnthropicCache),
+    Mistral(MistralCache),
+}
+
+#[derive(Clone, Debug)]
+pub struct AiCache {
+    pub path: String,
+    pub cached_key: KeyCache,
+    pub expires_at: std::time::Instant,
+}
+
+impl AiCache {
+    pub fn new(path: String, cached_key: KeyCache) -> Self {
+        Self {
+            path,
+            cached_key,
+            expires_at: std::time::Instant::now() + std::time::Duration::from_secs(60),
+        }
+    }
+    fn is_expired(&self) -> bool {
+        self.expires_at < std::time::Instant::now()
+    }
+}
+
+lazy_static! {
+    pub static ref AI_KEY_CACHE: Cache<String, AiCache> = Cache::new(500);
+}
+
+struct Variable {
+    value: String,
+    is_secret: bool,
+}
+
+#[derive(Deserialize, Debug)]
+struct ProxyQueryParams {
+    no_cache: Option<bool>,
+}
+
+async fn get_variable_or_self(path: String, db: &DB, w_id: &str) -> Result<String> {
+    if !path.starts_with("$var:") {
+        return Ok(path);
+    }
+    let path = path.strip_prefix("$var:").unwrap().to_string();
+    let mut variable = sqlx::query_as!(
+        Variable,
+        "SELECT value, is_secret
+        FROM variable
+        WHERE path = $1 AND workspace_id = $2",
+        &path,
+        &w_id
+    )
+    .fetch_one(db)
+    .await?;
+    if variable.is_secret {
+        let mc = build_crypt(db, w_id).await?;
+        variable.value = decrypt(&mc, variable.value)?;
+    }
+    Ok(variable.value)
+}
+
+#[derive(Deserialize, Debug)]
+pub struct AiResource {
+    pub path: String,
+    #[serde(deserialize_with = "check_if_valid_ai_provider")]
+    pub provider: String,
+}
+
+fn check_if_valid_ai_provider<'de, D>(provider: D) -> std::result::Result<String, D::Error>
+where
+    D: Deserializer<'de>,
+{
+    let provider = String::deserialize(provider)?;
+    match provider.as_str() {
+        "anthropic" | "openai" | "mistral" => Ok(provider),
+        _ => Err(serde::de::Error::custom(
+            "Only the following Ai providers are supported: openai, anthropic and mistral"
+                .to_string(),
+        )),
+    }
+}
+
+pub fn workspaced_service() -> Router {
+    let router = Router::new().route("/proxy/*ai", post(proxy));
+
+    router
+}
+
+async fn proxy(
+    authed: ApiAuthed,
+    Extension(db): Extension<DB>,
+    Path((w_id, ai_path)): Path<(String, String)>,
+    Query(query_params): Query<ProxyQueryParams>,
+    body: Bytes,
+) -> impl IntoResponse {
+    let workspace_cache = AI_KEY_CACHE.get(&w_id);
+    let ai_cache = match workspace_cache {
+        Some(cache) if !cache.is_expired() && !query_params.no_cache.unwrap_or(false) => {
+            cache.cached_key
+        }
+        _ => {
+            let ai_resource = sqlx::query_scalar!(
+                "SELECT ai_resource FROM workspace_settings WHERE workspace_id = $1",
+                &w_id
+            )
+            .fetch_one(&db)
+            .await?;
+
+            if ai_resource.is_none() {
+                return Err(Error::InternalErr("AI resource not configured".to_string()));
+            }
+
+            let ai_resource = serde_json::from_value::<AiResource>(ai_resource.unwrap())
+                .map_err(|e| Error::BadRequest(e.to_string()))?;
+            let ai_resource_path = ai_resource.path;
+
+            let resource = sqlx::query_scalar!(
+                "SELECT value
+                FROM resource
+                WHERE path = $1 AND workspace_id = $2",
+                &ai_resource_path,
+                &w_id
+            )
+            .fetch_optional(&db)
+            .await?
+            .ok_or_else(|| {
+                Error::InternalErr(format!(
+                    "Could not find the {} resource at path {ai_resource_path}, update the resource path in the workspace settings", ai_resource.provider
+                ))
+            })?;
+
+            if resource.is_none() {
+                return Err(Error::InternalErr(format!(
+                    "{} resource missing value",
+                    ai_resource.provider
+                )));
+            }
+
+            let resource = resource.unwrap();
+
+            let ai_cache = match ai_resource.provider.as_str() {
+                "openai" => openai::get_cached_value(&db, &w_id, resource).await,
+                "anthropic" => anthropic::get_cached_value(&db, &w_id, resource).await,
+                "mistral" => mistral::get_cached_value(&db, &w_id, resource).await,
+                provider => {
+                    return Err(Error::BadRequest(format!("{} is not supported", provider)))
+                }
+            };
+            let ai_cache = ai_cache?;
+            AI_KEY_CACHE.insert(
+                w_id.clone(),
+                AiCache::new(ai_resource_path, ai_cache.clone()),
+            );
+            ai_cache
+        }
+    };
+    let (path, request) = match ai_cache {
+        KeyCache::Openai(cached) => ("openai_path", cached.prepare_request(&ai_path, body)),
+        KeyCache::Anthropic(cached) => ("anthropic_path", cached.prepare_request(&ai_path, body)),
+        KeyCache::Mistral(cached) => ("mistral_path", cached.prepare_request(&ai_path, body)),
+    };
+
+    let response = request?.send().await.map_err(to_anyhow)?;
+
+    let mut tx = db.begin().await?;
+
+    audit_log(
+        &mut *tx,
+        &authed,
+        "ai.request",
+        ActionKind::Execute,
+        &w_id,
+        Some(&authed.email),
+        Some([(path, &format!("{:?}", ai_path)[..])].into()),
+    )
+    .await?;
+    tx.commit().await?;
+
+    if response.error_for_status_ref().is_err() {
+        let err_msg = response.text().await.unwrap_or("".to_string());
+        return Err(Error::AiError(err_msg));
+    }
+
+    let status_code = response.status();
+    let headers = response.headers().clone();
+    let stream = response.bytes_stream();
+    Ok((status_code, headers, axum::body::Body::from_stream(stream)))
+}
@@ -12,6 +12,7 @@ use quick_cache::sync::Cache;
 use serde_json::value::RawValue;
 use sqlx::Pool;
 use std::collections::HashMap;
+use std::ops::{Deref, DerefMut};
 #[cfg(feature = "prometheus")]
 use std::sync::atomic::Ordering;
 use tokio::io::AsyncReadExt;
@@ -205,6 +206,7 @@ pub fn workspaced_service() -> Router {
        .route("/queue/list_filtered_uuids", get(list_filtered_uuids))
        .route("/queue/cancel_selection", post(cancel_selection))
        .route("/completed/count", get(count_completed_jobs))
+        .route("/completed/count_jobs", get(count_completed_jobs_detail))
        .route(
            "/completed/list",
            get(list_completed_jobs).layer(cors.clone()),
@@ -600,7 +602,7 @@ async fn get_flow_job_debug_info(
    Extension(db): Extension<DB>,
    Path((w_id, id)): Path<(String, Uuid)>,
 ) -> error::Result<Response> {
-    let job = get_queued_job(&id, w_id.as_str(), &db).await?;
+    let job = get_queued_job_ex(&db, &w_id, id, false, None).await?;
    if let Some(job) = job {
        let is_flow = job.is_flow();
        if job.is_flow_step || !is_flow {
@@ -734,6 +736,67 @@ fn generate_get_job_query(no_logs: bool, table: &str) -> String {
    {join}
    WHERE id = $1 AND {table}.workspace_id = $2");
 }
+pub async fn get_queued_job_ex(
+    db: &DB,
+    workspace_id: &str,
+    job_id: Uuid,
+    no_logs: bool,
+    // first optional is if authed need to be checked, second is the opt_authed itself
+    opt_authed: Option<&Option<ApiAuthed>>,
+) -> error::Result<Option<JobExtended<QueuedJob>>> {
+    let query = if no_logs { &*GET_QUEUED_JOB_QUERY_NO_LOGS } else { &*GET_QUEUED_JOB_QUERY };
+    let job = sqlx::query_as::<_, JobExtended<QueuedJob>>(query)
+        .bind(job_id)
+        .bind(workspace_id)
+        .fetch_optional(db)
+        .await?;
+
+    if let Some(job) = job.as_ref() {
+        if opt_authed.is_some_and(|x| x.is_none()) && job.created_by != "anonymous" {
+            return Err(Error::BadRequest(
+                "As a non logged in user, you can only see jobs ran by anonymous users".to_string(),
+            ));
+        }
+    }
+
+    Ok(job)
+}
+pub async fn get_completed_job_ex(
+    db: &DB,
+    workspace_id: &str,
+    job_id: Uuid,
+    no_logs: bool,
+    // first optional is if authed need to be checked, second is the opt_authed itself
+    opt_authed: Option<&Option<ApiAuthed>>,
+) -> error::Result<Option<JobExtended<CompletedJob>>> {
+    let query = if no_logs { &*GET_COMPLETED_JOB_QUERY_NO_LOGS } else { &*GET_COMPLETED_JOB_QUERY };
+    let cjob = sqlx::query_as::<_, JobExtended<CompletedJob>>(query)
+        .bind(job_id)
+        .bind(workspace_id)
+        .fetch_optional(db)
+        .await?;
+
+    if let Some(job) = cjob.as_ref() {
+        if opt_authed.is_some_and(|x| x.is_none()) && job.created_by != "anonymous" {
+            return Err(Error::BadRequest(
+                "As a non logged in user, you can only see jobs ran by anonymous users".to_string(),
+            ));
+        }
+    }
+
+    if let Some(mut cjob) = cjob {
+        let CompletedJobWithFormattedResult { mut cj, result } = format_completed_job_result(cjob.inner);
+        cj.result = match result {
+            Some(FormattedResult::RawValue(rv)) => rv,
+            Some(FormattedResult::Vec(v)) => Some(to_raw_value(&v)),
+            None => None,
+        }.map(sqlx::types::Json);
+        cjob.inner = cj;
+        return Ok(Some(cjob));
+    }
+
+    Ok(cjob)
+}
 pub async fn get_job_internal(
    db: &DB,
    workspace_id: &str,
@@ -742,48 +805,18 @@ pub async fn get_job_internal(
    // first optional is if authed need to be checked, second is the opt_authed itself
    opt_authed: Option<&Option<ApiAuthed>>,
 ) -> error::Result<Job> {
-    let cjob_maybe = sqlx::query_as::<_, CompletedJob>(if no_logs {
-        &*GET_COMPLETED_JOB_QUERY_NO_LOGS
-    } else {
-        &*GET_COMPLETED_JOB_QUERY
-    })
-    .bind(job_id)
-    .bind(workspace_id)
-    .fetch_optional(db)
-    .await?
-    .map(Job::CompletedJob);
-
-    if let Some(cjob) = cjob_maybe {
-        Ok(match cjob {
-            Job::CompletedJob(cjob) => {
-                if opt_authed.is_some_and(|x| x.is_none()) && cjob.created_by != "anonymous" {
-                    return Err(Error::BadRequest(
-                        "As a non logged in user, you can only see jobs ran by anonymous users"
-                            .to_string(),
-                    ));
-                }
-                Job::CompletedJobWithFormattedResult(format_completed_job_result(cjob))
-            }
-            cjob => cjob,
-        })
-    } else {
-        let job_o = sqlx::query_as::<_, QueuedJob>(if no_logs {
-            &*GET_QUEUED_JOB_QUERY_NO_LOGS
-        } else {
-            &*GET_QUEUED_JOB_QUERY
-        })
-        .bind(job_id)
-        .bind(workspace_id)
-        .fetch_optional(db)
+    let cjob = get_completed_job_ex(db, workspace_id, job_id, no_logs, opt_authed.clone())
        .await?
-        .map(Job::QueuedJob);
-        let job: Job = not_found_if_none(job_o, "Job", job_id.to_string())?;
-        if opt_authed.is_some_and(|x| x.is_none()) && job.created_by() != "anonymous" {
-            return Err(Error::BadRequest(
-                "As a non logged in user, you can only see jobs ran by anonymous users".to_string(),
-            ));
+        .map(Job::CompletedJob);
+
+    match cjob {
+        Some(cjob) => Ok(cjob),
+        None => {
+            let job_maybe = get_queued_job_ex(db, workspace_id, job_id, no_logs, opt_authed)
+                .await?
+                .map(Job::QueuedJob);
+            not_found_if_none(job_maybe, "Job", job_id.to_string())
        }
-        Ok(job)
    }
 }

@@ -1560,6 +1593,50 @@ async fn count_queue_jobs(
    ))
 }

+#[derive(Deserialize)]
+pub struct CountCompletedJobsQuery {
+    completed_after_s_ago: Option<i64>,
+    success: Option<bool>,
+    tags: Option<String>,
+    all_workspaces: Option<bool>,
+}
+
+async fn count_completed_jobs_detail(
+    Extension(db): Extension<DB>,
+    Path(w_id): Path<String>,
+    Query(query): Query<CountCompletedJobsQuery>,
+) -> error::JsonResult<i64> {
+    let mut sqlb = SqlBuilder::select_from("completed_job");
+    sqlb
+        .field("COUNT(*) as count");
+
+    if !query.all_workspaces.unwrap_or(false) {
+        sqlb.and_where_eq("workspace_id", "?".bind(&w_id));
+    }
+
+    if let Some(after_s_ago) = query.completed_after_s_ago {
+        let after = Utc::now() - chrono::Duration::seconds(after_s_ago);
+        sqlb.and_where_gt("started_at + duration_ms / 1000 * interval '1 second'", "?".bind(&after.to_rfc3339()));
+    }
+
+    if let Some(success) = query.success {
+        sqlb.and_where_eq("success", "?".bind(&success));
+    }
+
+    if let Some(tags) = query.tags {
+        sqlb.and_where_in("tag", &tags.split(",").map(|t| format!("'{}'", t)).collect::<Vec<_>>());
+    }
+
+    let sql = sqlb.sql()?;
+    let stats = sqlx::query_scalar::<_, i64>(&sql)
+        .fetch_one(&db)
+        .await?;
+
+    Ok(Json(stats))
+}
+
+
+
 async fn count_completed_jobs(
    Extension(db): Extension<DB>,
    Path(w_id): Path<String>,
@@ -1735,7 +1812,6 @@ async fn resume_suspended_job_internal(
    let trigger_email = match &parent_flow {
        Job::CompletedJob(job) => &job.email,
        Job::QueuedJob(job) => &job.email,
-        Job::CompletedJobWithFormattedResult(job) => &job.cj.email,
    };
    conditionally_require_authed_user(authed.clone(), flow_status, trigger_email)?;

@@ -2010,7 +2086,6 @@ pub async fn get_suspended_job_flow(
    let trigger_email = match &flow {
        Job::CompletedJob(job) => &job.email,
        Job::QueuedJob(job) => &job.email,
-        Job::CompletedJobWithFormattedResult(job) => &job.cj.email,
    };
    conditionally_require_authed_user(authed.clone(), flow_status.clone(), trigger_email)?;

@@ -2225,13 +2300,45 @@ pub async fn get_resume_urls(
    Ok(Json(res))
 }

+#[derive(sqlx::FromRow, Debug, Serialize)]
+pub struct JobExtended<T> {
+    #[sqlx(flatten)]
+    #[serde(flatten)]
+    inner: T,
+
+    #[sqlx(skip)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub self_wait_time_ms: Option<i64>,
+    #[sqlx(skip)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub aggregate_wait_time_ms: Option<i64>,
+}
+
+impl<T> JobExtended<T> {
+    pub fn new(self_wait_time_ms: Option<i64>, aggregate_wait_time_ms: Option<i64>, inner: T) -> Self {
+        Self { inner, self_wait_time_ms, aggregate_wait_time_ms }
+    }
+}
+
+impl<T> Deref for JobExtended<T> {
+    type Target = T;
+
+    fn deref(&self) -> &Self::Target {
+        &self.inner
+    }
+}
+
+impl<T> DerefMut for JobExtended<T> {
+    fn deref_mut(&mut self) -> &mut Self::Target {
+        &mut self.inner
+    }
+}
+
 #[derive(Serialize, Debug)]
 #[serde(tag = "type")]
 pub enum Job {
-    QueuedJob(QueuedJob),
-    CompletedJob(CompletedJob),
-    #[serde(rename = "CompletedJob")]
-    CompletedJobWithFormattedResult(CompletedJobWithFormattedResult),
+    QueuedJob(JobExtended<QueuedJob>),
+    CompletedJob(JobExtended<CompletedJob>),
 }

 impl Job {
@@ -2239,27 +2346,6 @@ impl Job {
        match self {
            Job::QueuedJob(job) => &job.created_by,
            Job::CompletedJob(job) => &job.created_by,
-            Job::CompletedJobWithFormattedResult(job) => &job.cj.created_by,
-        }
-    }
-    pub fn raw_flow(&self) -> Option<FlowValue> {
-        match self {
-            Job::QueuedJob(job) => job
-                .raw_flow
-                .as_ref()
-                .map(|rf| serde_json::from_str(rf.0.get()).ok())
-                .flatten(),
-            Job::CompletedJob(job) => job
-                .raw_flow
-                .as_ref()
-                .map(|rf| serde_json::from_str(rf.0.get()).ok())
-                .flatten(),
-            Job::CompletedJobWithFormattedResult(job) => job
-                .cj
-                .raw_flow
-                .as_ref()
-                .map(|rf| serde_json::from_str(rf.0.get()).ok())
-                .flatten(),
        }
    }

@@ -2279,13 +2365,6 @@ impl Job {
                    job.logs = Some(logs.to_string());
                }
            }
-            Job::CompletedJobWithFormattedResult(job) => {
-                if let Some(ref mut l) = job.cj.logs {
-                    l.push_str(logs);
-                } else {
-                    job.cj.logs = Some(logs.to_string());
-                }
-            }
        }
    }

@@ -2293,7 +2372,6 @@ impl Job {
        match self {
            Job::QueuedJob(job) => job.logs.as_ref().map(|l| l.len()),
            Job::CompletedJob(job) => job.logs.as_ref().map(|l| l.len()),
-            Job::CompletedJobWithFormattedResult(job) => job.cj.logs.as_ref().map(|l| l.len()),
        }
    }

@@ -2301,7 +2379,6 @@ impl Job {
        match self {
            Job::QueuedJob(job) => job.logs.clone(),
            Job::CompletedJob(job) => job.logs.clone(),
-            Job::CompletedJobWithFormattedResult(job) => job.cj.logs.clone(),
        }
    }
    pub fn flow_status(&self) -> Option<FlowStatus> {
@@ -2316,19 +2393,12 @@ impl Job {
                .as_ref()
                .map(|rf| serde_json::from_str(rf.0.get()).ok())
                .flatten(),
-            Job::CompletedJobWithFormattedResult(job) => job
-                .cj
-                .flow_status
-                .as_ref()
-                .map(|rf| serde_json::from_str(rf.0.get()).ok())
-                .flatten(),
        }
    }
    pub fn is_flow_step(&self) -> bool {
        match self {
            Job::QueuedJob(job) => job.is_flow_step,
            Job::CompletedJob(job) => job.is_flow_step,
-            Job::CompletedJobWithFormattedResult(job) => job.cj.is_flow_step,
        }
    }

@@ -2343,7 +2413,6 @@ impl Job {
        match self {
            Job::QueuedJob(job) => &job.job_kind,
            Job::CompletedJob(job) => &job.job_kind,
-            Job::CompletedJobWithFormattedResult(job) => &job.cj.job_kind,
        }
    }

@@ -2351,7 +2420,6 @@ impl Job {
        match self {
            Job::QueuedJob(job) => job.id,
            Job::CompletedJob(job) => job.id,
-            Job::CompletedJobWithFormattedResult(job) => job.cj.id,
        }
    }

@@ -2359,7 +2427,6 @@ impl Job {
        match self {
            Job::QueuedJob(job) => &job.workspace_id,
            Job::CompletedJob(job) => &job.workspace_id,
-            Job::CompletedJobWithFormattedResult(job) => &job.cj.workspace_id,
        }
    }

@@ -2367,7 +2434,6 @@ impl Job {
        match self {
            Job::QueuedJob(job) => job.script_path.as_ref(),
            Job::CompletedJob(job) => job.script_path.as_ref(),
-            Job::CompletedJobWithFormattedResult(job) => job.cj.script_path.as_ref(),
        }
        .map(String::as_str)
        .unwrap_or("tmp/main")
@@ -2377,7 +2443,6 @@ impl Job {
        match self {
            Job::QueuedJob(job) => job.args.as_ref(),
            Job::CompletedJob(job) => job.args.as_ref(),
-            Job::CompletedJobWithFormattedResult(job) => job.cj.args.as_ref(),
        }
    }

@@ -2426,10 +2491,6 @@ impl Job {
                job.self_wait_time_ms = self_wait_time;
                job.aggregate_wait_time_ms = aggregate_wait_time;
            }
-            Job::CompletedJobWithFormattedResult(job) => {
-                job.cj.self_wait_time_ms = self_wait_time;
-                job.cj.aggregate_wait_time_ms = aggregate_wait_time;
-            }
        }
        Ok(())
    }
@@ -2557,86 +2618,82 @@ impl UnifiedJob {
 impl<'a> From<UnifiedJob> for Job {
    fn from(uj: UnifiedJob) -> Self {
        match uj.typ.as_ref() {
-            "CompletedJob" => Job::CompletedJob(CompletedJob {
-                workspace_id: uj.workspace_id,
-                id: uj.id,
-                parent_job: uj.parent_job,
-                created_by: uj.created_by,
-                created_at: uj.created_at,
-                started_at: uj.started_at.unwrap_or(uj.created_at),
-                duration_ms: uj.duration_ms.unwrap(),
-                success: uj.success.unwrap(),
-                script_hash: uj.script_hash,
-                script_path: uj.script_path,
-                args: None,
-                result: None,
-                logs: None,
-                flow_status: None,
-                deleted: uj.deleted,
-                canceled: uj.canceled,
-                canceled_by: uj.canceled_by,
-                raw_code: None,
-                canceled_reason: None,
-                job_kind: uj.job_kind,
-                schedule_path: uj.schedule_path,
-                permissioned_as: uj.permissioned_as,
-                raw_flow: None,
-                is_flow_step: uj.is_flow_step,
-                language: uj.language,
-                is_skipped: uj.is_skipped,
-                email: uj.email,
-                visible_to_owner: uj.visible_to_owner,
-                mem_peak: uj.mem_peak,
-                tag: uj.tag,
-                priority: uj.priority,
-                labels: uj.labels,
-                self_wait_time_ms: uj.self_wait_time_ms,
-                aggregate_wait_time_ms: uj.aggregate_wait_time_ms,
-            }),
-            "QueuedJob" => Job::QueuedJob(QueuedJob {
-                workspace_id: uj.workspace_id,
-                id: uj.id,
-                parent_job: uj.parent_job,
-                created_by: uj.created_by,
-                created_at: uj.created_at,
-                started_at: uj.started_at,
-                script_hash: uj.script_hash,
-                script_path: uj.script_path,
-                args: None,
-                running: uj.running.unwrap(),
-                scheduled_for: uj.scheduled_for.unwrap(),
-                logs: None,
-                flow_status: None,
-                raw_code: None,
-                raw_lock: None,
-                canceled: uj.canceled,
-                canceled_by: uj.canceled_by,
-                canceled_reason: None,
-                last_ping: None,
-                job_kind: uj.job_kind,
-                schedule_path: uj.schedule_path,
-                permissioned_as: uj.permissioned_as,
-                raw_flow: None,
-                is_flow_step: uj.is_flow_step,
-                language: uj.language,
-                same_worker: false,
-                pre_run_error: None,
-                email: uj.email,
-                visible_to_owner: uj.visible_to_owner,
-                suspend: uj.suspend,
-                mem_peak: uj.mem_peak,
-                root_job: None,
-                leaf_jobs: None,
-                tag: uj.tag,
-                concurrent_limit: uj.concurrent_limit,
-                concurrency_time_window_s: uj.concurrency_time_window_s,
-                timeout: None,
-                flow_step_id: None,
-                cache_ttl: None,
-                priority: uj.priority,
-                self_wait_time_ms: uj.self_wait_time_ms,
-                aggregate_wait_time_ms: uj.aggregate_wait_time_ms,
-            }),
+            "CompletedJob" => Job::CompletedJob(JobExtended::new(uj.self_wait_time_ms, uj.aggregate_wait_time_ms, CompletedJob {
+                    workspace_id: uj.workspace_id,
+                    id: uj.id,
+                    parent_job: uj.parent_job,
+                    created_by: uj.created_by,
+                    created_at: uj.created_at,
+                    started_at: uj.started_at.unwrap_or(uj.created_at),
+                    duration_ms: uj.duration_ms.unwrap(),
+                    success: uj.success.unwrap(),
+                    script_hash: uj.script_hash,
+                    script_path: uj.script_path,
+                    args: None,
+                    result: None,
+                    logs: None,
+                    flow_status: None,
+                    deleted: uj.deleted,
+                    canceled: uj.canceled,
+                    canceled_by: uj.canceled_by,
+                    raw_code: None,
+                    canceled_reason: None,
+                    job_kind: uj.job_kind,
+                    schedule_path: uj.schedule_path,
+                    permissioned_as: uj.permissioned_as,
+                    raw_flow: None,
+                    is_flow_step: uj.is_flow_step,
+                    language: uj.language,
+                    is_skipped: uj.is_skipped,
+                    email: uj.email,
+                    visible_to_owner: uj.visible_to_owner,
+                    mem_peak: uj.mem_peak,
+                    tag: uj.tag,
+                    priority: uj.priority,
+                    labels: uj.labels,
+            })),
+            "QueuedJob" => Job::QueuedJob(JobExtended::new(uj.self_wait_time_ms, uj.aggregate_wait_time_ms, QueuedJob {
+                    workspace_id: uj.workspace_id,
+                    id: uj.id,
+                    parent_job: uj.parent_job,
+                    created_by: uj.created_by,
+                    created_at: uj.created_at,
+                    started_at: uj.started_at,
+                    script_hash: uj.script_hash,
+                    script_path: uj.script_path,
+                    args: None,
+                    running: uj.running.unwrap(),
+                    scheduled_for: uj.scheduled_for.unwrap(),
+                    logs: None,
+                    flow_status: None,
+                    raw_code: None,
+                    raw_lock: None,
+                    canceled: uj.canceled,
+                    canceled_by: uj.canceled_by,
+                    canceled_reason: None,
+                    last_ping: None,
+                    job_kind: uj.job_kind,
+                    schedule_path: uj.schedule_path,
+                    permissioned_as: uj.permissioned_as,
+                    raw_flow: None,
+                    is_flow_step: uj.is_flow_step,
+                    language: uj.language,
+                    same_worker: false,
+                    pre_run_error: None,
+                    email: uj.email,
+                    visible_to_owner: uj.visible_to_owner,
+                    suspend: uj.suspend,
+                    mem_peak: uj.mem_peak,
+                    root_job: None,
+                    leaf_jobs: None,
+                    tag: uj.tag,
+                    concurrent_limit: uj.concurrent_limit,
+                    concurrency_time_window_s: uj.concurrency_time_window_s,
+                    timeout: None,
+                    flow_step_id: None,
+                    cache_ttl: None,
+                    priority: uj.priority,
+            })),
            t => panic!("job type {} not valid", t),
        }
    }
@@ -3079,7 +3136,7 @@ pub async fn run_workflow_as_code(
        i += 1;
    }

-    let job = get_queued_job(&job_id, &w_id, &db).await?;
+    let job = get_queued_job_ex(&db, &w_id, job_id, true, None).await?;

    if *CLOUD_HOSTED {
        tracing::info!("workflow_as_code_tracing id {i} ");
@@ -3087,6 +3144,7 @@ pub async fn run_workflow_as_code(
    }

    let job = not_found_if_none(job, "Queued Job", &job_id.to_string())?;
+    let JobExtended { inner: job, .. } = job;
    let (job_payload, tag, _delete_after_use, timeout) = match job.job_kind {
        JobKind::Preview => (
            JobPayload::Code(RawCode {
@@ -5058,24 +5116,10 @@ async fn get_completed_job<'a>(
    Extension(db): Extension<DB>,
    Path((w_id, id)): Path<(String, Uuid)>,
 ) -> error::Result<Response> {
-    let job_o = sqlx::query_as::<_, CompletedJob>("SELECT id, workspace_id, parent_job, created_by, created_at, duration_ms, success, script_hash, script_path, 
-    CASE WHEN args is null or pg_column_size(args) < 90000 THEN args ELSE '\"WINDMILL_TOO_BIG\"'::jsonb END as args, CASE WHEN result is null or pg_column_size(result) < 90000 THEN result ELSE '\"WINDMILL_TOO_BIG\"'::jsonb END as result, logs, deleted, raw_code, canceled, canceled_by, canceled_reason, job_kind,
-    schedule_path, permissioned_as, flow_status, raw_flow, is_flow_step, language, started_at, is_skipped,
-    raw_lock, email, visible_to_owner, mem_peak, tag, priority, result->'wm_labels' as labels FROM completed_job WHERE id = $1 AND workspace_id = $2")
-        .bind(id)
-        .bind(&w_id)
-        .fetch_optional(&db)
+    let job_o = get_completed_job_ex(&db, &w_id, id, false, Some(&opt_authed))
        .await?;

    let cj = not_found_if_none(job_o, "Completed Job", id.to_string())?;
-
-    if opt_authed.is_none() && cj.created_by != "anonymous" {
-        return Err(Error::BadRequest(
-            "As a non logged in user, you can only see jobs ran by anonymous users".to_string(),
-        ));
-    }
-    let cj = format_completed_job_result(cj);
-
    let response = Json(cj).into_response();
    // let extra_log = query_scalar!(
    //     "SELECT substr(logs, $1) as logs FROM large_logs WHERE workspace_id = $2 AND job_id = $3",
@@ -25,7 +25,6 @@ use argon2::Argon2;
 use axum::extract::DefaultBodyLimit;
 use axum::{middleware::from_extractor, routing::get, Extension, Router};
 use db::DB;
-use git_version::git_version;
 use http::HeaderValue;
 use reqwest::Client;
 use std::collections::HashMap;
@@ -40,7 +39,7 @@ use tower_http::{
 };
 use windmill_common::db::UserDB;
 use windmill_common::worker::{ALL_TAGS, CLOUD_HOSTED};
-use windmill_common::{BASE_URL, INSTANCE_NAME};
+use windmill_common::{BASE_URL, INSTANCE_NAME, utils::GIT_VERSION};

 use crate::scim_ee::has_scim_token;
 use windmill_common::error::AppError;
@@ -63,13 +62,14 @@ mod http_triggers;
 mod indexer_ee;
 mod inputs;
 mod integration;
+mod ai;
+
 #[cfg(feature = "parquet")]
 mod job_helpers_ee;
 pub mod job_metrics;
 pub mod jobs;
 pub mod oauth2_ee;
 mod oidc_ee;
-mod openai;
 mod raw_apps;
 mod resources;
 mod saml_ee;
@@ -93,11 +93,9 @@ mod workers;
 mod workspaces;
 mod workspaces_ee;

-pub const GIT_VERSION: &str =
-    git_version!(args = ["--tag", "--always"], fallback = "unknown-version");
-
 pub const DEFAULT_BODY_LIMIT: usize = 2097152 * 100; // 200MB

+
 lazy_static::lazy_static! {

    pub static ref REQUEST_SIZE_LIMIT: Arc<RwLock<usize>> = Arc::new(RwLock::new(DEFAULT_BODY_LIMIT));
@@ -282,7 +280,7 @@ pub async fn run_server(
                        .nest("/job_helpers", job_helpers_service)
                        .nest("/jobs", jobs::workspaced_service())
                        .nest("/oauth", oauth2_ee::workspaced_service())
-                        .nest("/openai", openai::workspaced_service())
+                        .nest("/ai", ai::workspaced_service())
                        .nest("/raw_apps", raw_apps::workspaced_service())
                        .nest("/resources", resources::workspaced_service())
                        .nest("/schedules", schedule::workspaced_service())
@@ -1,344 +0,0 @@
-use std::collections::HashMap;
-
-use crate::db::{ApiAuthed, DB};
-
-use axum::{
-    body::Bytes,
-    extract::{Extension, Path, Query},
-    response::IntoResponse,
-    routing::post,
-    Router,
-};
-use quick_cache::sync::Cache;
-use reqwest::Client;
-use serde_json::value::RawValue;
-use windmill_audit::audit_ee::audit_log;
-use windmill_audit::ActionKind;
-use windmill_common::{
-    error::{to_anyhow, Error},
-    variables::build_crypt,
-};
-
-use crate::variables::decrypt;
-use serde::Deserialize;
-
-lazy_static::lazy_static! {
-    static ref HTTP_CLIENT: Client = reqwest::ClientBuilder::new()
-        .timeout(std::time::Duration::from_secs(60 * 5))
-        .user_agent("windmill/beta")
-        .build().unwrap();
-}
-
-pub fn workspaced_service() -> Router {
-    let router = Router::new().route("/proxy/*openai_path", post(proxy));
-
-    router
-}
-
-#[derive(Deserialize)]
-struct OpenaiResource {
-    api_key: String,
-    organization_id: Option<String>,
-}
-
-#[derive(Deserialize)]
-struct OpenaiClientCredentialsOauthResource {
-    client_id: String,
-    client_secret: String,
-    token_url: String,
-    user: Option<String>,
-}
-
-#[derive(Deserialize)]
-#[serde(untagged)]
-enum OpenaiConfig {
-    Resource(OpenaiResource),
-    ClientCredentialsOauthResource(OpenaiClientCredentialsOauthResource),
-}
-
-struct Variable {
-    value: String,
-    is_secret: bool,
-}
-async fn get_variable_or_self(path: String, db: &DB, w_id: &String) -> Result<String, Error> {
-    if !path.starts_with("$var:") {
-        return Ok(path);
-    }
-    let path = path.strip_prefix("$var:").unwrap().to_string();
-    let mut variable = sqlx::query_as!(
-        Variable,
-        "SELECT value, is_secret
-        FROM variable
-        WHERE path = $1 AND workspace_id = $2",
-        &path,
-        &w_id
-    )
-    .fetch_one(db)
-    .await?;
-    if variable.is_secret {
-        let mc = build_crypt(&db, &w_id).await?;
-        variable.value = decrypt(&mc, variable.value)?;
-    }
-    Ok(variable.value)
-}
-
-lazy_static::lazy_static! {
-    pub static ref OPENAI_AZURE_BASE_PATH: Option<String> = std::env::var("OPENAI_AZURE_BASE_PATH").ok();
-    static ref OPENAI_KEY_CACHE: Cache<String, OpenaiKeyCache> = Cache::new(500);
-}
-
-#[derive(Deserialize)]
-struct OpenaiCredentials {
-    access_token: String,
-}
-async fn get_openai_key_using_credentials_flow(
-    mut resource: OpenaiClientCredentialsOauthResource,
-    db: &DB,
-    w_id: &String,
-) -> Result<String, Error> {
-    resource.client_id = get_variable_or_self(resource.client_id, &db, &w_id).await?;
-    resource.client_secret = get_variable_or_self(resource.client_secret, &db, &w_id).await?;
-    resource.token_url = get_variable_or_self(resource.token_url, &db, &w_id).await?;
-    let mut params = HashMap::new();
-    params.insert("grant_type", "client_credentials");
-    let response = HTTP_CLIENT
-        .post(resource.token_url)
-        .form(&params)
-        .basic_auth(resource.client_id, Some(resource.client_secret))
-        .send()
-        .await
-        .map_err(|err| {
-            Error::InternalErr(format!(
-                "Failed to get OpenAI credentials using credentials flow: {}",
-                err
-            ))
-        })?;
-    let response = response.json::<OpenaiCredentials>().await.map_err(|err| {
-        Error::InternalErr(format!(
-            "Failed to parse OpenAI credentials from credentials flow: {}",
-            err
-        ))
-    })?;
-    Ok(response.access_token)
-}
-
-#[derive(Clone)]
-struct OpenaiKeyCache {
-    api_key: String,
-    organization_id: Option<String>,
-    azure_base_path: Option<String>,
-    user: Option<String>,
-    expires_at: std::time::Instant,
-}
-
-impl OpenaiKeyCache {
-    fn new(
-        api_key: String,
-        organization_id: Option<String>,
-        azure_base_path: Option<String>,
-        expires_at: std::time::Instant,
-        user: Option<String>,
-    ) -> Self {
-        Self { api_key, organization_id, azure_base_path, expires_at, user }
-    }
-    fn is_expired(&self) -> bool {
-        self.expires_at < std::time::Instant::now()
-    }
-}
-
-#[derive(Deserialize)]
-struct ProxyQueryParams {
-    no_cache: Option<bool>,
-}
-async fn proxy(
-    authed: ApiAuthed,
-    Extension(db): Extension<DB>,
-    Path((w_id, openai_path)): Path<(String, String)>,
-    Query(query_params): Query<ProxyQueryParams>,
-    mut body: Bytes,
-) -> impl IntoResponse {
-    let workspace_cache = OPENAI_KEY_CACHE.get(&w_id);
-    let (api_key, organization_id, azure_base_path, user) = if query_params
-        .no_cache
-        .unwrap_or(false)
-        || workspace_cache.is_none()
-        || workspace_cache.clone().unwrap().is_expired()
-    {
-        let openai_resource_path = sqlx::query_scalar!(
-            "SELECT openai_resource_path FROM workspace_settings WHERE workspace_id = $1",
-            &w_id
-        )
-        .fetch_one(&db)
-        .await?;
-
-        if openai_resource_path.is_none() {
-            return Err(Error::InternalErr(
-                "OpenAI resource not configured".to_string(),
-            ));
-        }
-
-        let openai_resource_path = openai_resource_path.unwrap();
-
-        let resource = sqlx::query_scalar!(
-            "SELECT value
-            FROM resource
-            WHERE path = $1 AND workspace_id = $2",
-            &openai_resource_path,
-            &w_id
-        )
-        .fetch_optional(&db)
-        .await?
-        .ok_or_else(|| {
-            Error::InternalErr(format!(
-                "Could not find the OpenAI resource at path {openai_resource_path}, update the resource path in the workspace settings"
-            ))
-        })?;
-
-        if resource.is_none() {
-            return Err(Error::InternalErr(
-                "OpenAI resource missing value".to_string(),
-            ));
-        }
-
-        let config: OpenaiConfig = serde_json::from_value(resource.unwrap())
-            .map_err(|e| Error::InternalErr(format!("validating openai resource {e:#}")))?;
-
-        let mut user = None::<String>;
-        let mut resource = match config {
-            OpenaiConfig::Resource(resource) => {
-                tracing::debug!("Getting OpenAI key from static resource");
-                resource
-            }
-            OpenaiConfig::ClientCredentialsOauthResource(resource) => {
-                tracing::debug!("Getting OpenAI key with client credentials flow");
-                user = resource.user.clone();
-                let token = get_openai_key_using_credentials_flow(resource, &db, &w_id).await?;
-                OpenaiResource { api_key: token, organization_id: None }
-            }
-        };
-
-        resource.api_key = get_variable_or_self(resource.api_key, &db, &w_id).await?;
-
-        if resource.organization_id.is_some() {
-            resource.organization_id =
-                Some(get_variable_or_self(resource.organization_id.unwrap(), &db, &w_id).await?);
-        }
-
-        if user.is_some() {
-            user = Some(get_variable_or_self(user.unwrap(), &db, &w_id).await?);
-        }
-
-        let expires_at = std::time::Instant::now() + std::time::Duration::from_secs(60);
-
-        let azure_base_path = sqlx::query_scalar!(
-            "SELECT value
-            FROM global_settings
-            WHERE name = 'openai_azure_base_path'",
-        )
-        .fetch_optional(&db)
-        .await?;
-
-        let azure_base_path = if let Some(azure_base_path) = azure_base_path {
-            Some(
-                serde_json::from_value::<String>(azure_base_path).map_err(|e| {
-                    Error::InternalErr(format!("validating openai azure base path {e:#}"))
-                })?,
-            )
-        } else {
-            OPENAI_AZURE_BASE_PATH.clone()
-        };
-
-        let workspace_cache = OpenaiKeyCache::new(
-            resource.api_key.clone(),
-            resource.organization_id.clone(),
-            azure_base_path.clone(),
-            expires_at,
-            user.clone(),
-        );
-        OPENAI_KEY_CACHE.insert(w_id.clone(), workspace_cache);
-        (
-            resource.api_key,
-            resource.organization_id,
-            azure_base_path,
-            user,
-        )
-    } else {
-        tracing::debug!("Using cached OpenAI key");
-        let workspace_cache = workspace_cache.unwrap();
-        (
-            workspace_cache.api_key.clone(),
-            workspace_cache.organization_id.clone(),
-            workspace_cache.azure_base_path.clone(),
-            workspace_cache.user.clone(),
-        )
-    };
-
-    if user.is_some() {
-        tracing::debug!("Adding user to request body");
-        let mut json_body: HashMap<String, Box<RawValue>> = serde_json::from_slice(&body)
-            .map_err(|e| Error::InternalErr(format!("Failed to parse request body: {}", e)))?;
-
-        let user_json_string = serde_json::Value::String(user.unwrap()).to_string(); // makes sure to escape characters
-
-        json_body.insert(
-            "user".to_string(),
-            RawValue::from_string(user_json_string)
-                .map_err(|e| Error::InternalErr(format!("Failed to parse user: {}", e)))?,
-        );
-
-        body = serde_json::to_vec(&json_body)
-            .map_err(|e| Error::InternalErr(format!("Failed to reserialize request body: {}", e)))?
-            .into();
-    }
-
-    let base_url = if let Some(base_url) = azure_base_path {
-        base_url
-    } else {
-        "https://api.openai.com/v1".to_string()
-    };
-
-    let url = format!("{}/{}", base_url, openai_path);
-    let mut request = HTTP_CLIENT
-        .post(url)
-        .header("content-type", "application/json")
-        .body(body);
-
-    if base_url != "https://api.openai.com/v1" {
-        request = request
-            .header("api-key", api_key)
-            .query(&[("api-version", "2023-05-15")])
-    } else {
-        request = request.header("authorization", format!("Bearer {}", api_key))
-    }
-
-    if let Some(org_id) = organization_id {
-        request = request.header("OpenAI-Organization", org_id);
-    }
-
-    let response = request.send().await.map_err(to_anyhow)?;
-
-    let mut tx = db.begin().await?;
-    audit_log(
-        &mut *tx,
-        &authed,
-        "openai.request",
-        ActionKind::Execute,
-        &w_id,
-        Some(&authed.email),
-        Some([("openai_path", &format!("{:?}", openai_path)[..])].into()),
-    )
-    .await?;
-    tx.commit().await?;
-
-    if response.error_for_status_ref().is_err() {
-        return Err(Error::OpenAIError(
-            response.text().await.unwrap_or("".to_string()),
-        ));
-    }
-
-    let status_code = response.status();
-    let headers = response.headers().clone();
-    let stream = response.bytes_stream();
-
-    Ok((status_code, headers, axum::body::Body::from_stream(stream)))
-}
@@ -300,7 +300,7 @@ async fn get_resource(
    .await?;
    tx.commit().await?;
    if resource_o.is_none() {
-        explain_resource_perm_error(&path, &w_id, &db).await?;
+        explain_resource_perm_error(&path, &w_id, &db, &authed).await?;
    }
    let resource = not_found_if_none(resource_o, "Resource", path)?;
    Ok(Json(resource))
@@ -343,7 +343,7 @@ async fn get_resource_value(

    tx.commit().await?;
    if value_o.is_none() {
-        explain_resource_perm_error(&path, &w_id, &db).await?;
+        explain_resource_perm_error(&path, &w_id, &db, &authed).await?;
    }

    let value = not_found_if_none(value_o, "Resource", path)?;
@@ -354,6 +354,7 @@ async fn explain_resource_perm_error(
    path: &str,
    w_id: &str,
    db: &sqlx::Pool<Postgres>,
+    authed: &ApiAuthed,
 ) -> windmill_common::error::Result<()> {
    let extra_perms = sqlx::query_scalar!(
        "SELECT extra_perms from resource WHERE path = $1 AND workspace_id = $2",
@@ -378,12 +379,12 @@ async fn explain_resource_perm_error(
        .fetch_optional(db)
        .await?;
        return Err(Error::NotAuthorized(format!(
-            "Resource exists but you don't have access to it:\nresource perms: {}\nfolder perms: {}",
+            "Resource exists but you don't have access to it:\nresource perms: {}\nfolder perms: {}\nauthed as: {authed:?}",
            serde_json::to_string_pretty(&extra_perms).unwrap_or_default(), serde_json::to_string_pretty(&folder_extra_perms).unwrap_or_default()
        )));
    } else {
        return Err(Error::NotAuthorized(format!(
-            "Resource exists but you don't have access to it:\nresource perms: {}",
+            "Resource exists but you don't have access to it:\nresource perms: {}\nauthed as: {authed:?}",
            serde_json::to_string_pretty(&extra_perms).unwrap_or_default()
        )));
    }
@@ -457,7 +458,7 @@ pub async fn get_resource_value_interpolated_internal(
    .await?;
    tx.commit().await?;
    if value_o.is_none() {
-        explain_resource_perm_error(path, workspace, db).await?;
+        explain_resource_perm_error(path, workspace, db, &authed).await?;
    }

    let value = not_found_if_none(value_o, "Resource", path)?;
@@ -32,6 +32,7 @@ use windmill_common::{
 };
 use windmill_git_sync::{handle_deployment_metadata, DeployedObject};
 use windmill_queue::{schedule::push_scheduled_job, QueueTransaction};
+use croner::Cron;

 pub fn workspaced_service() -> Router {
    Router::new()
@@ -153,7 +154,11 @@ async fn create_schedule(

    let mut tx: QueueTransaction<'_, _> = (rsmq.clone(), user_db.begin(&authed).await?).into();

-    cron::Schedule::from_str(&ns.schedule).map_err(|e| Error::BadRequest(e.to_string()))?;
+    Cron::new(&ns.schedule)
+        .with_seconds_optional()
+        .parse()
+        .map_err(|e| Error::BadRequest(format!("cron: {}", e.to_string())))?;
+
    check_path_conflict(tx.transaction_mut(), &w_id, &ns.path).await?;
    check_flow_conflict(
        tx.transaction_mut(),
@@ -255,7 +260,10 @@ async fn edit_schedule(
    let mut tx: QueueTransaction<'_, rsmq_async::MultiplexedRsmq> =
        (rsmq.clone(), user_db.begin(&authed).await?).into();

-    cron::Schedule::from_str(&es.schedule).map_err(|e| Error::BadRequest(e.to_string()))?;
+    Cron::new(&es.schedule)
+        .with_seconds_optional()
+        .parse()
+        .map_err(|e| Error::BadRequest(format!("cron: {}", e.to_string())))?;

    clear_schedule(tx.transaction_mut(), path, &w_id).await?;
    let schedule = sqlx::query_as::<_, Schedule>(
@@ -468,17 +476,19 @@ pub struct PreviewPayload {
 pub async fn preview_schedule(
    Json(payload): Json<PreviewPayload>,
 ) -> JsonResult<Vec<DateTime<Utc>>> {
-    let schedule = cron::Schedule::from_str(&payload.schedule)
-        .map_err(|e| Error::BadRequest(e.to_string()))?;
+
+    let schedule = Cron::new(&payload.schedule)
+        .with_seconds_optional()
+        .parse()
+        .map_err(|e| Error::BadRequest(format!("cron: {}", e.to_string())))?;

    let tz =
        chrono_tz::Tz::from_str(&payload.timezone).map_err(|e| Error::BadRequest(e.to_string()))?;

    let upcoming: Vec<DateTime<Utc>> = schedule
-        .upcoming(tz)
+        .iter_from(Utc::now().with_timezone(&tz))
        .take(5)
-        // Convert back to UTC for a standardised API response. The client will convert to the local timezone.
-        .map(|x| x.with_timezone(&Utc))
+        .map(|dt| dt.with_timezone(&Utc))
        .collect();

    Ok(Json(upcoming))
@@ -434,67 +434,15 @@ pub async fn test_critical_channels() -> Result<String> {
    Ok("Critical channels require EE".to_string())
 }

-use serde::Serialize;
-
-#[derive(Serialize)]
-pub struct CriticalAlert {
-    id: i32,
-    alert_type: String,
-    message: String,
-    created_at: chrono::DateTime<chrono::Utc>,
-    acknowledged: Option<bool>,
-}
-
-#[cfg(feature = "enterprise")]
-#[derive(Deserialize)]
-pub struct AlertQueryParams {
-    pub page: Option<i32>,
-    pub page_size: Option<i32>,
-    pub acknowledged: Option<bool>,
-}
-
 #[cfg(feature = "enterprise")]
 pub async fn get_critical_alerts(
    Extension(db): Extension<DB>,
    authed: ApiAuthed,
-    Query(params): Query<AlertQueryParams>,
-) -> JsonResult<Vec<CriticalAlert>> {
+    Query(params): Query<crate::utils::AlertQueryParams>,
+) -> JsonResult<Vec<crate::utils::CriticalAlert>> {
    require_super_admin(&db, &authed.email).await?;

-    // Default pagination values if not provided
-    let page = params.page.unwrap_or(1).max(1);
-    let page_size = params.page_size.unwrap_or(10).min(100) as i64;
-    let offset = ((page - 1) * page_size as i32) as i64;
-
-    let alerts = if let Some(acknowledged) = params.acknowledged {
-        sqlx::query_as!(
-            CriticalAlert,
-            "SELECT id, alert_type, message, created_at, acknowledged 
-             FROM alerts 
-             WHERE acknowledged = $1
-             ORDER BY created_at DESC 
-             LIMIT $2 OFFSET $3",
-            acknowledged,
-            page_size,
-            offset
-        )
-        .fetch_all(&db)
-        .await?
-    } else {
-        sqlx::query_as!(
-            CriticalAlert,
-            "SELECT id, alert_type, message, created_at, acknowledged 
-             FROM alerts 
-             ORDER BY created_at DESC 
-             LIMIT $1 OFFSET $2",
-            page_size,
-            offset
-        )
-        .fetch_all(&db)
-        .await?
-    };
-
-    Ok(Json(alerts))
+    crate::utils::get_critical_alerts(db, params, None).await
 }

 #[cfg(not(feature = "enterprise"))]
@@ -510,15 +458,7 @@ pub async fn acknowledge_critical_alert(
 ) -> error::Result<String> {
    require_super_admin(&db, &authed.email).await?;
    
-    sqlx::query!(
-        "UPDATE alerts SET acknowledged = true WHERE id = $1",
-        id
-    )
-    .execute(&db)
-    .await?;
-    
-    tracing::info!("Acknowledged critical alert with id: {}", id);
-    Ok("Critical alert acknowledged".to_string())
+    crate::utils::acknowledge_critical_alert(db, None, id).await
 }

 #[cfg(not(feature = "enterprise"))]
@@ -533,14 +473,7 @@ pub async fn acknowledge_all_critical_alerts(
 ) -> error::Result<String> {
    require_super_admin(&db, &authed.email).await?;

-    sqlx::query!(
-        "UPDATE alerts SET acknowledged = true WHERE acknowledged = false"
-    )
-    .execute(&db)
-    .await?;
-
-    tracing::info!("Acknowledged all unacknowledged critical alerts");
-    Ok("All unacknowledged critical alerts acknowledged".to_string())
+    crate::utils::acknowledge_all_critical_alerts(db, None).await
 }

 #[cfg(not(feature = "enterprise"))]
@@ -90,6 +90,9 @@ pub fn global_service() -> Router {
        .route("/accept_invite", post(accept_invite))
        .route("/list_as_super_admin", get(list_users_as_super_admin))
        .route("/setpassword", post(set_password))
+        .route("/set_password_of/:user", post(set_password_of_user))
+        .route("/set_login_type/:user", post(set_login_type))
+
        .route("/create", post(create_user))
        .route("/update/:user", post(update_user))
        .route("/delete/:user", delete(delete_user))
@@ -866,6 +869,12 @@ pub struct EditPassword {
    pub password: String,
 }

+#[derive(Deserialize)]
+pub struct EditLoginType {
+    pub login_type: String,
+}
+
+
 #[derive(FromRow, Serialize)]
 pub struct TruncatedToken {
    pub label: Option<String>,
@@ -2028,7 +2037,52 @@ async fn set_password(
    authed: ApiAuthed,
    Json(ep): Json<EditPassword>,
 ) -> Result<String> {
-    crate::users_ee::set_password(db, argon2, authed, ep).await
+    let email = authed.email.clone();
+    crate::users_ee::set_password(db, argon2, authed, &email, ep).await
+}
+
+async fn set_password_of_user(
+    Extension(db): Extension<DB>,
+    Extension(argon2): Extension<Arc<Argon2<'_>>>,
+    Path(email): Path<String>,
+    authed: ApiAuthed,
+    Json(ep): Json<EditPassword>,
+) -> Result<String> {
+    require_super_admin(&db, &authed.email).await?;
+    crate::users_ee::set_password(db, argon2, authed, &email, ep).await
+}
+
+async fn set_login_type(
+    Extension(db): Extension<DB>,
+    Path(email): Path<String>,
+    authed: ApiAuthed,
+    Json(et): Json<EditLoginType>,
+) -> Result<String> {
+    require_super_admin(&db, &authed.email).await?;
+    let mut tx = db.begin().await?;
+
+    sqlx::query!(
+        "UPDATE password SET login_type = $1 WHERE email = $2",
+        et.login_type,
+        email
+    )
+    .execute(&mut *tx)
+    .await?;
+
+    audit_log(
+        &mut *tx,
+        &authed,
+        "users.set_login_type",
+        ActionKind::Update,
+        "global",
+        Some(&email),
+        None,
+    )
+    .await?;
+
+    tx.commit().await?;
+    Ok(format!("login type of {} updated to {}", email, et.login_type))
+    
 }

 async fn login(
@@ -27,6 +27,7 @@ pub async fn set_password(
    _db: DB,
    _argon2: Arc<Argon2<'_>>,
    _authed: ApiAuthed,
+    _user_email: &str,
    _ep: EditPassword,
 ) -> Result<String> {
    Err(Error::InternalErr(
@@ -16,6 +16,12 @@ use windmill_common::{
    DB,
 };

+#[cfg(feature = "enterprise")]
+use windmill_common::error::JsonResult;
+
+#[cfg(feature = "enterprise")]
+use axum::Json;
+
 #[derive(Deserialize)]
 pub struct WithStarredInfoQuery {
    pub with_starred_info: Option<bool>,
@@ -162,3 +168,159 @@ pub fn content_plain(body: Body) -> Response {
        .body(body)
        .unwrap()
 }
+
+use serde::Serialize;
+
+#[derive(Serialize)]
+pub struct CriticalAlert {
+    id: i32,
+    alert_type: String,
+    message: String,
+    created_at: chrono::DateTime<chrono::Utc>,
+    acknowledged: Option<bool>,
+    workspace_id: Option<String>,
+}
+
+#[cfg(feature = "enterprise")]
+#[derive(Deserialize, Debug)]
+pub struct AlertQueryParams {
+    pub page: Option<i32>,
+    pub page_size: Option<i32>,
+    pub acknowledged: Option<bool>,
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn get_critical_alerts(
+    db: DB,
+    params: AlertQueryParams,
+    workspace_id: Option<String>,
+) -> JsonResult<Vec<CriticalAlert>> {
+    let page = params.page.unwrap_or(1).max(1);
+    let page_size = params.page_size.unwrap_or(10).min(100) as i64;
+    let offset = ((page - 1) * page_size as i32) as i64;
+
+    let alerts = if let Some(workspace_id) = workspace_id {
+        // `workspace_id` is provided => workspace admin
+        if params.acknowledged.is_none() {
+            // Case: return all rows where `workspace_id` matches
+            sqlx::query_as!(
+                CriticalAlert,
+                "SELECT id, alert_type, message, created_at, COALESCE(acknowledged_workspace, false) AS acknowledged, workspace_id
+                 FROM alerts
+                 WHERE workspace_id = $1
+                 ORDER BY created_at DESC
+                 LIMIT $2 OFFSET $3",
+                workspace_id,
+                page_size,
+                offset
+            )
+            .fetch_all(&db)
+            .await?
+        } else {
+            // Case: return rows where `acknowledged_workspace` matches `params.acknowledged`
+            sqlx::query_as!(
+                CriticalAlert,
+                "SELECT id, alert_type, message, created_at, COALESCE(acknowledged_workspace, false) AS acknowledged, workspace_id
+                 FROM alerts
+                 WHERE workspace_id = $1 AND COALESCE(acknowledged_workspace, false) = $2
+                 ORDER BY created_at DESC
+                 LIMIT $3 OFFSET $4",
+                workspace_id,
+                params.acknowledged,
+                page_size,
+                offset
+            )
+            .fetch_all(&db)
+            .await?
+        }
+    } else {
+        // `workspace_id` is not provided => superadmin
+        if params.acknowledged.is_none() {
+            // Case: Return all rows unfiltered with global acknowledged as acknowledged
+            sqlx::query_as!(
+                CriticalAlert,
+                "SELECT id, alert_type, message, created_at, COALESCE(acknowledged, false) AS acknowledged, workspace_id
+                 FROM alerts
+                 ORDER BY created_at DESC
+                 LIMIT $1 OFFSET $2",
+                page_size,
+                offset
+            )
+            .fetch_all(&db)
+            .await?
+        } else {
+            // Case: Return rows where global acknowledged matches params.acknowledged
+            sqlx::query_as!(
+                CriticalAlert,
+                "SELECT id, alert_type, message, created_at, COALESCE(acknowledged, false) AS acknowledged, workspace_id
+                 FROM alerts
+                 WHERE COALESCE(acknowledged, false) = $1
+                 ORDER BY created_at DESC
+                 LIMIT $2 OFFSET $3",
+                params.acknowledged,
+                page_size,
+                offset
+            )
+            .fetch_all(&db)
+            .await?
+        }
+    };
+
+    Ok(Json(alerts))
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn acknowledge_critical_alert(
+    db: DB,
+    workspace_id: Option<String>,
+    id: i32,
+) -> error::Result<String> {
+    sqlx::query!(
+        "UPDATE alerts
+         SET
+           acknowledged = true,
+           acknowledged_workspace = CASE
+             WHEN $2::text IS NOT NULL AND workspace_id = $2 THEN true
+             ELSE acknowledged_workspace
+           END
+         WHERE id = $1",
+        id,
+        workspace_id
+    )
+    .execute(&db)
+    .await?;
+
+    tracing::info!(
+        "Acknowledged critical alert with id: {}{}",
+        id,
+        workspace_id.map_or_else(|| "".to_string(), |w| format!(" for workspace_id: {}", w))
+    );
+    Ok("Critical alert acknowledged".to_string())
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn acknowledge_all_critical_alerts(
+    db: DB,
+    workspace_id: Option<String>,
+) -> error::Result<String> {
+    sqlx::query!(
+        "UPDATE alerts 
+         SET
+           acknowledged = true,
+           acknowledged_workspace = CASE
+             WHEN $1::text IS NOT NULL THEN true
+             ELSE acknowledged_workspace
+           END
+         WHERE ($1::text IS NOT NULL AND workspace_id = $1)
+            OR ($1::text IS NULL)",
+        workspace_id
+    )
+    .execute(&db)
+    .await?;
+
+    tracing::info!(
+        "Acknowledged all unacknowledged critical alerts{}",
+        workspace_id.map_or_else(|| "".to_string(), |w| format!(" for workspace_id: {}", w))
+    );
+    Ok("All unacknowledged critical alerts acknowledged".to_string())
+}
@@ -754,12 +754,14 @@ async fn disable_with_error(db: &DB, ws_trigger: &WebsocketTrigger, error: Strin
    )
    .execute(db).await {
        Ok(_) => {
-            report_critical_error(format!("Disabling websocket {} because of error: {}", ws_trigger.url, error), db.clone()).await;
+            report_critical_error(format!("Disabling websocket {} because of error: {}", ws_trigger.url, error), db.clone(), Some(&ws_trigger.workspace_id), None).await;
        },
        Err(disable_err) => {
            report_critical_error(
                format!("Could not disable websocket {} with err {}, disabling because of error {}", ws_trigger.path, disable_err, error), 
-                db.clone()
+                db.clone(),
+                Some(&ws_trigger.workspace_id),
+                None,
            ).await;
        }
    }
@@ -896,7 +898,7 @@ async fn listen_to_websocket(
                                                                }
                                                                if should_handle {
                                                                    if let Err(err) = run_job(&db, rsmq.clone(), &ws_trigger, text).await {
-                                                                        report_critical_error(format!("Failed to trigger job from websocket {}: {:?}", ws_trigger.url, err), db.clone()).await;
+                                                                        report_critical_error(format!("Failed to trigger job from websocket {}: {:?}", ws_trigger.url, err), db.clone(), Some(&ws_trigger.workspace_id), None).await;
                                                                    };
                                                                }
                                                            },
@@ -36,6 +36,7 @@ pub fn global_service() -> Router {
        )
        .route("/get_default_tags", get(get_default_tags))
        .route("/queue_metrics", get(get_queue_metrics))
+        .route("/queue_counts", get(get_queue_counts))
 }

 #[derive(FromRow, Serialize, Deserialize)]
@@ -176,3 +177,12 @@ async fn get_queue_metrics(

    Ok(Json(queue_metrics))
 }
+
+async fn get_queue_counts(
+    authed: ApiAuthed,
+    Extension(db): Extension<DB>,
+) -> JsonResult<std::collections::HashMap<String, u32>> {
+    require_super_admin(&db, &authed.email).await?;
+    let queue_counts = windmill_common::queue::get_queue_counts(&db).await;
+    Ok(Json(queue_counts))
+}
@@ -8,6 +8,7 @@

 use std::collections::HashMap;

+use crate::ai::{AiResource, AI_KEY_CACHE};
 use crate::db::ApiAuthed;
 use crate::users_ee::send_email_if_possible;
 use crate::utils::get_instance_username_or_create_pending;
@@ -118,7 +119,11 @@ pub fn workspaced_service() -> Router {
        .route("/change_workspace_name", post(change_workspace_name))
        .route("/change_workspace_id", post(change_workspace_id))
        .route("/usage", get(get_usage))
-        .route("/used_triggers", get(get_used_triggers));
+        .route("/used_triggers", get(get_used_triggers))
+        .route("/critical_alerts", get(get_critical_alerts))
+        .route("/critical_alerts/:id/acknowledge", post(acknowledge_critical_alert))
+        .route("/critical_alerts/acknowledge_all", post(acknowledge_all_critical_alerts))
+        .route("/critical_alerts/mute", post(mute_critical_alerts));

    #[cfg(feature = "stripe")]
    {
@@ -168,7 +173,7 @@ pub struct WorkspaceSettings {
    pub plan: Option<String>,
    pub webhook: Option<String>,
    pub deploy_to: Option<String>,
-    pub openai_resource_path: Option<String>,
+    pub ai_resource: Option<serde_json::Value>,
    pub code_completion_enabled: bool,
    pub error_handler: Option<String>,
    pub error_handler_extra_args: Option<serde_json::Value>,
@@ -179,6 +184,7 @@ pub struct WorkspaceSettings {
    pub default_app: Option<String>,
    pub automatic_billing: bool,
    pub default_scripts: Option<serde_json::Value>,
+    pub mute_critical_alerts: Option<bool>,
 }

 #[derive(FromRow, Serialize, Debug)]
@@ -234,7 +240,7 @@ struct EditWebhook {

 #[derive(Deserialize)]
 struct EditCopilotConfig {
-    openai_resource_path: Option<String>,
+    ai_resource: Option<serde_json::Value>,
    code_completion_enabled: bool,
 }

@@ -645,23 +651,33 @@ async fn edit_copilot_config(

    let mut tx = db.begin().await?;

-    if let Some(openai_resource_path) = &eo.openai_resource_path {
+    if let Some(ai_resource) = &eo.ai_resource {
+        let path = serde_json::from_value::<AiResource>(ai_resource.clone())
+            .map_err(|e| Error::BadRequest(e.to_string()))?
+            .path;
        sqlx::query!(
-            "UPDATE workspace_settings SET openai_resource_path = $1, code_completion_enabled = $2 WHERE workspace_id = $3",
-            openai_resource_path,
+            "UPDATE workspace_settings SET ai_resource = $1, code_completion_enabled = $2 WHERE workspace_id = $3",
+            ai_resource,
            eo.code_completion_enabled,
            &w_id
        )
        .execute(&mut *tx)
        .await?;
+
+        if let Some(cached) = AI_KEY_CACHE.get(&w_id) {
+            if cached.path != path {
+                AI_KEY_CACHE.remove(&w_id);
+            }
+        }
    } else {
        sqlx::query!(
-            "UPDATE workspace_settings SET openai_resource_path = NULL, code_completion_enabled = $1 WHERE workspace_id = $2",
+            "UPDATE workspace_settings SET ai_resource = NULL, code_completion_enabled = $1 WHERE workspace_id = $2",
            eo.code_completion_enabled,
            &w_id,
        )
        .execute(&mut *tx)
        .await?;
+        AI_KEY_CACHE.remove(&w_id);
    }
    audit_log(
        &mut *tx,
@@ -672,10 +688,7 @@ async fn edit_copilot_config(
        Some(&authed.email),
        Some(
            [
-                (
-                    "openai_resource_path",
-                    &format!("{:?}", eo.openai_resource_path)[..],
-                ),
+                ("ai_resource", &format!("{:?}", eo.ai_resource)[..]),
                (
                    "code_completion_enabled",
                    &format!("{:?}", eo.code_completion_enabled)[..],
@@ -692,7 +705,8 @@ async fn edit_copilot_config(

 #[derive(Serialize)]
 struct CopilotInfo {
-    pub exists_openai_resource_path: bool,
+    pub ai_provider: String,
+    pub exists_ai_resource: bool,
    pub code_completion_enabled: bool,
 }
 async fn get_copilot_info(
@@ -701,16 +715,32 @@ async fn get_copilot_info(
 ) -> JsonResult<CopilotInfo> {
    let mut tx = db.begin().await?;
    let record = sqlx::query!(
-        "SELECT openai_resource_path, code_completion_enabled FROM workspace_settings WHERE workspace_id = $1",
+        "SELECT ai_resource, code_completion_enabled FROM workspace_settings WHERE workspace_id = $1",
        &w_id
    )
    .fetch_one(&mut *tx)
    .await
-    .map_err(|e| Error::InternalErr(format!("getting openai_resource_path and code_completion_enabled: {e:#}")))?;
+    .map_err(|e| Error::InternalErr(format!("getting ai_resource and code_completion_enabled: {e:#}")))?;
    tx.commit().await?;

+    let (ai_provider, exists_ai_resource) = if let Some(ai_resource) = record.ai_resource {
+        let ai_resource = serde_json::from_value::<AiResource>(ai_resource);
+        let exist = ai_resource.is_ok();
+        (
+            if exist {
+                ai_resource.unwrap().provider
+            } else {
+                "".to_string()
+            },
+            exist,
+        )
+    } else {
+        ("".to_string(), false)
+    };
+
    Ok(Json(CopilotInfo {
-        exists_openai_resource_path: record.openai_resource_path.is_some(),
+        ai_provider,
+        exists_ai_resource,
        code_completion_enabled: record.code_completion_enabled,
    }))
 }
@@ -2223,7 +2253,7 @@ struct SimplifiedSettings {
    error_handler: Option<String>,
    error_handler_extra_args: Option<Value>,
    error_handler_muted_on_cancel: bool,
-    openai_resource_path: Option<String>,
+    ai_resource: Option<serde_json::Value>,
    code_completion_enabled: bool,
    large_file_storage: Option<Value>,
    git_sync: Option<Value>,
@@ -2588,7 +2618,7 @@ async fn tarball_workspace(
                webhook, 
                deploy_to, 
                error_handler, 
-                openai_resource_path, 
+                ai_resource, 
                code_completion_enabled, 
                error_handler_extra_args, 
                error_handler_muted_on_cancel, 
@@ -3050,3 +3080,93 @@ async fn get_usage(Extension(db): Extension<DB>, Path(w_id): Path<String>) -> Re
    .unwrap_or(0);
    Ok(usage.to_string())
 }
+
+#[cfg(feature = "enterprise")]
+pub async fn get_critical_alerts(
+    Extension(db): Extension<DB>,
+    Path(w_id): Path<String>,
+    authed: ApiAuthed,
+    Query(params): Query<crate::utils::AlertQueryParams>,
+) -> JsonResult<Vec<crate::utils::CriticalAlert>> {
+    require_admin(authed.is_admin, &authed.username)?;
+
+    crate::utils::get_critical_alerts(db, params, Some(w_id)).await
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn get_critical_alerts() -> Error {
+    Error::NotFound("Critical Alerts require EE".to_string())
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn acknowledge_critical_alert(
+    Extension(db): Extension<DB>,
+    Path((w_id, id)): Path<(String, i32)>,
+    authed: ApiAuthed,
+) -> Result<String> {
+    require_admin(authed.is_admin, &authed.username)?;
+    crate::utils::acknowledge_critical_alert(db, Some(w_id), id).await 
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn acknowledge_critical_alert() -> Error {
+    Error::NotFound("Critical Alerts require EE".to_string())
+}
+
+#[cfg(feature = "enterprise")]
+pub async fn acknowledge_all_critical_alerts(
+    Extension(db): Extension<DB>,
+    Path(w_id): Path<String>,
+    authed: ApiAuthed,
+) -> Result<String> {
+    require_admin(authed.is_admin, &authed.username)?;
+    crate::utils::acknowledge_all_critical_alerts(db, Some(w_id)).await
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn acknowledge_all_critical_alerts() -> Error {
+    Error::NotFound("Critical Alerts require EE".to_string())
+}
+
+
+#[cfg(feature = "enterprise")]
+#[derive(Deserialize)]
+pub struct MuteCriticalAlertRequest {
+    pub mute_critical_alerts: Option<bool>,
+}
+
+#[cfg(feature = "enterprise")]
+async fn mute_critical_alerts(
+    Extension(db): Extension<DB>,
+    Path(w_id): Path<String>,
+    ApiAuthed { is_admin, username, .. }: ApiAuthed,
+    Json(m_r): Json<MuteCriticalAlertRequest>,
+) -> Result<String> {
+    require_admin(is_admin, &username)?;
+
+    let mute_alerts = m_r.mute_critical_alerts.unwrap_or(false);
+
+    if mute_alerts {
+        sqlx::query!(
+            "UPDATE alerts SET acknowledged_workspace = true, acknowledged = true WHERE workspace_id = $1",
+            &w_id
+        )
+    .execute(&db)
+    .await?;
+    }
+
+    sqlx::query!(
+        "UPDATE workspace_settings SET mute_critical_alerts = $1 WHERE workspace_id = $2",
+        mute_alerts,
+        &w_id
+    )
+    .execute(&db)
+    .await?;
+
+    Ok(format!("Updated mute criticital alert ui settings for workspace: {}", &w_id))
+}
+
+#[cfg(not(feature = "enterprise"))]
+pub async fn mute_critical_alerts() -> Error {
+    Error::NotFound("Critical Alerts require EE".to_string())
+}
@@ -58,6 +58,7 @@ async-stream.workspace = true
 const_format.workspace = true
 crc.workspace = true
 windmill-macros.workspace = true
+semver.workspace = true

 [target.'cfg(not(target_env = "msvc"))'.dependencies]
 tikv-jemalloc-ctl = { optional = true, workspace = true }
@@ -59,7 +59,7 @@ pub enum Error {
    #[error("Error: {0:#?}")]
    JsonErr(serde_json::Value),
    #[error("{0}")]
-    OpenAIError(String),
+    AiError(String),
    #[error("{0}")]
    AlreadyCompleted(String),
 }
@@ -90,7 +90,7 @@ impl IntoResponse for Error {
            Self::RequireAdmin(_) => axum::http::StatusCode::FORBIDDEN,
            Self::SqlErr(_)
            | Self::BadRequest(_)
-            | Self::OpenAIError(_)
+            | Self::AiError(_)
            | Self::QuotaExceeded(_) => axum::http::StatusCode::BAD_REQUEST,
            _ => axum::http::StatusCode::INTERNAL_SERVER_ERROR,
        };
@@ -16,6 +16,7 @@ use rand::Rng;
 use serde::{Deserialize, Serialize, Serializer};

 use crate::{
+    error::Error,
    more_serde::{default_empty_string, default_id, default_null, default_true, is_default},
    scripts::{Schema, ScriptHash, ScriptLang},
 };
@@ -651,3 +652,29 @@ pub fn add_virtual_items_if_necessary(modules: &mut Vec<FlowModule>) {
        });
    }
 }
+
+pub async fn has_failure_module<'c>(flow: sqlx::types::Uuid, db: &sqlx::Pool<sqlx::Postgres>, completed: bool) -> Result<bool, Error> {
+    if completed {
+        sqlx::query_scalar!(
+            "SELECT raw_flow->'failure_module' != 'null'::jsonb
+            FROM completed_job
+            WHERE id = $1",
+            flow
+        )
+    } else {
+        sqlx::query_scalar!(
+            "SELECT raw_flow->'failure_module' != 'null'::jsonb
+            FROM queue
+            WHERE id = $1",
+            flow
+        )
+    }
+    .fetch_one(db)
+    .await
+    .map_err(|e| {
+        Error::InternalErr(format!(
+            "error during retrieval of has_failure_module: {e:#}"
+        ))
+    })
+    .map(|v| v.unwrap_or(false))
+}
@@ -108,13 +108,6 @@ pub struct QueuedJob {
    pub cache_ttl: Option<i32>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub priority: Option<i16>,
-
-    #[serde(skip_serializing_if = "Option::is_none")]
-    #[sqlx(skip)]
-    pub self_wait_time_ms: Option<i64>,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    #[sqlx(skip)]
-    pub aggregate_wait_time_ms: Option<i64>,
 }

 impl QueuedJob {
@@ -198,8 +191,6 @@ impl Default for QueuedJob {
            flow_step_id: None,
            cache_ttl: None,
            priority: None,
-            self_wait_time_ms: None,
-            aggregate_wait_time_ms: None,
        }
    }
 }
@@ -253,13 +244,6 @@ pub struct CompletedJob {
    pub priority: Option<i16>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub labels: Option<serde_json::Value>,
-
-    #[serde(skip_serializing_if = "Option::is_none")]
-    #[sqlx(skip)]
-    pub self_wait_time_ms: Option<i64>,
-    #[serde(skip_serializing_if = "Option::is_none")]
-    #[sqlx(skip)]
-    pub aggregate_wait_time_ms: Option<i64>,
 }

 impl CompletedJob {
@@ -21,6 +21,7 @@ use reqwest::Client;
 use serde::{Deserialize, Serialize};
 use sha2::{Digest, Sha256};
 use sqlx::{Pool, Postgres};
+use semver::Version;

 pub const MAX_PER_PAGE: usize = 10000;
 pub const DEFAULT_PER_PAGE: usize = 1000;
@@ -28,8 +29,11 @@ pub const DEFAULT_PER_PAGE: usize = 1000;
 pub const GIT_VERSION: &str =
    git_version!(args = ["--tag", "--always"], fallback = "unknown-version");

-use std::sync::atomic::Ordering;
 use crate::CRITICAL_ALERT_MUTE_UI_ENABLED;
+use std::sync::atomic::Ordering;
+
+#[cfg(feature = "enterprise")]
+use crate::worker::CLOUD_HOSTED;

 lazy_static::lazy_static! {
    pub static ref HTTP_CLIENT: Client = reqwest::ClientBuilder::new()
@@ -37,6 +41,10 @@ lazy_static::lazy_static! {
        .timeout(std::time::Duration::from_secs(20))
        .connect_timeout(std::time::Duration::from_secs(10))
        .build().unwrap();
+    pub static ref GIT_SEM_VERSION: Version = Version::parse(
+        // skip first `v` character.
+        GIT_VERSION.split_at(1).1
+    ).unwrap_or(Version::new(0, 1, 0));
 }

 #[derive(Deserialize, Clone)]
@@ -244,15 +252,40 @@ pub fn generate_lock_id(database_name: &str) -> i64 {
    0x3d32ad9e * (CRC_IEEE.checksum(database_name.as_bytes()) as i64)
 }

-pub async fn report_critical_error(error_message: String, _db: DB) -> () {
+pub async fn report_critical_error(
+    error_message: String,
+    _db: DB,
+    workspace_id: Option<&str>,
+    resource: Option<&str>,
+) -> () {
    tracing::error!("CRITICAL ERROR: {error_message}");

-    let mute = CRITICAL_ALERT_MUTE_UI_ENABLED.load(Ordering::Relaxed);
+    let mute_global = CRITICAL_ALERT_MUTE_UI_ENABLED.load(Ordering::Relaxed);
+    let mute_workspace = if let Some(workspace_id) = workspace_id {
+        match fetch_mute_workspace(&_db, workspace_id).await {
+            Ok(flag) => flag,
+            Err(err) => {
+                tracing::error!("Error fetching mute_workspace: {}", err);
+                false
+            }
+        }
+    } else {
+        false
+    };
+
+    // we ack_global if mute_global is true, or if mute_workspace is true
+    // but we ignore global mute setting for ack_workspace
+    let acknowledge_workspace = mute_workspace;
+    let acknowledge_global = mute_global || mute_workspace || ( workspace_id.is_some() && *CLOUD_HOSTED);

    if let Err(err) = sqlx::query!(
-        "INSERT INTO alerts (alert_type, message, acknowledged) VALUES ('critical_error', $1, $2)",
+        "INSERT INTO alerts (alert_type, message, acknowledged, acknowledged_workspace, workspace_id, resource)
+        VALUES ('critical_error', $1, $2, $3, $4, $5)",
        error_message,
-        mute
+        acknowledge_global,
+        acknowledge_workspace,
+        workspace_id,
+        resource,
    )
    .execute(&_db)
    .await
@@ -261,30 +294,86 @@ pub async fn report_critical_error(error_message: String, _db: DB) -> () {
    }

    #[cfg(feature = "enterprise")]
-    send_critical_alert(error_message, &_db, CriticalAlertKind::CriticalError, None).await;
+    if *CLOUD_HOSTED && workspace_id.is_some() {
+        tracing::error!(error_message)
+    } else {
+        send_critical_alert(error_message, &_db, CriticalAlertKind::CriticalError, None).await;
+    }
 }

-pub async fn report_recovered_critical_error(message: String, _db: DB) -> () {
+pub async fn report_recovered_critical_error(
+    message: String,
+    _db: DB,
+    workspace_id: Option<&str>,
+    resource: Option<&str>,
+) -> () {
    tracing::info!("RECOVERED CRITICAL ERROR: {message}");

-    let mute = CRITICAL_ALERT_MUTE_UI_ENABLED.load(Ordering::Relaxed);
-
    if let Err(err) = sqlx::query!(
-        "INSERT INTO alerts (alert_type, message, acknowledged) VALUES ('recovered_critical_error', $1, $2)",
+        "INSERT INTO alerts (alert_type, message, acknowledged, acknowledged_workspace, workspace_id, resource)
+        VALUES ('recovered_critical_error', $1, $2, $3, $4, $5)",
        message,
-        mute
+        true,
+        true,
+        workspace_id,
+        resource,
    )
    .execute(&_db)
    .await
    {
-        tracing::error!("Failed to save critical error to database: {}", err);
+        tracing::error!("Failed to save recovered critical error to database: {}", err);
    }
+
+    // acknowledge all alerts with the same resource
+    if let Some(resource) = resource {
+        if let Err(err) = sqlx::query!(
+            "UPDATE alerts SET acknowledged = true, acknowledged_workspace = true WHERE resource = $1 AND alert_type = 'critical_error'",
+            resource,
+        )
+        .execute(&_db)
+        .await
+        {
+            tracing::error!("Failed to acknowledge critical error alerts for resource {}: {}", resource, err);
+        }
+    }
+
    #[cfg(feature = "enterprise")]
-    send_critical_alert(
-        message,
-        &_db,
-        CriticalAlertKind::RecoveredCriticalError,
-        None,
+    if *CLOUD_HOSTED && workspace_id.is_some() {
+        tracing::error!(message);
+    } else {
+        send_critical_alert(
+            message,
+            &_db,
+            CriticalAlertKind::RecoveredCriticalError,
+            None,
+        )
+        .await;
+    }
+}
+
+pub async fn fetch_mute_workspace(_db: &DB, workspace_id: &str) -> Result<bool> {
+    match sqlx::query!(
+        "SELECT mute_critical_alerts FROM workspace_settings WHERE workspace_id = $1",
+        workspace_id
    )
-    .await;
+    .fetch_optional(_db)
+    .await
+    {
+        Ok(Some(record)) => Ok(record.mute_critical_alerts.unwrap_or(false)),
+        Ok(None) => {
+            tracing::warn!(
+                "Workspace ID {} not found in workspace_settings table",
+                workspace_id
+            );
+            Ok(false)
+        }
+        Err(err) => {
+            tracing::error!(
+                "Error querying workspace_settings for workspace_id {}: {}",
+                workspace_id,
+                err
+            );
+            Err(Error::SqlErr(err))
+        }
+    }
 }
@@ -1,6 +1,7 @@
 use const_format::concatcp;
 use itertools::Itertools;
 use regex::Regex;
+use semver::Version;
 use serde::{Deserialize, Serialize};
 use serde_json::value::RawValue;
 use std::{
@@ -88,6 +89,7 @@ lazy_static::lazy_static! {
    .and_then(|x| x.parse::<bool>().ok())
    .unwrap_or(false);

+    pub static ref MIN_VERSION: Arc<RwLock<Version>> = Arc::new(RwLock::new(Version::new(0, 0, 0)));
 }

 pub async fn make_suspended_pull_query(wc: &WorkerConfig) {
@@ -308,6 +310,8 @@ fn parse_file<T: FromStr>(path: &str) -> Option<T> {
 pub struct PythonAnnotations {
    pub no_cache: bool,
    pub no_uv: bool,
+    pub no_uv_install: bool,
+    pub no_uv_compile: bool,
 }

 #[annotations("//")]
@@ -548,6 +552,31 @@ pub fn get_windmill_memory_usage() -> Option<i64> {
    }
 }

+pub async fn update_min_version<'c, E: sqlx::Executor<'c, Database = sqlx::Postgres>>(executor: E) -> bool {
+    use crate::utils::{GIT_VERSION, GIT_SEM_VERSION};
+
+    // fetch all pings with a different version than self from the last 5 minutes.
+    let pings = sqlx::query_scalar!(
+        "SELECT wm_version FROM worker_ping WHERE wm_version != $1 AND ping_at > now() - interval '5 minutes'",
+        GIT_VERSION
+    ).fetch_all(executor).await.unwrap_or_default();
+
+    let cur_version = GIT_SEM_VERSION.clone();
+    let min_version = pings
+        .iter()
+        .filter(|x| !x.is_empty())
+        .filter_map(|x| semver::Version::parse(x.split_at(1).1).ok())
+        .min()
+        .unwrap_or_else(|| cur_version.clone());
+
+    if min_version != cur_version {
+        tracing::info!("Minimal worker version: {min_version}");
+    }
+
+    *MIN_VERSION.write().await = min_version.clone();
+    min_version >= cur_version
+}
+
 pub async fn update_ping(worker_instance: &str, worker_name: &str, ip: &str, db: &DB) {
    let (tags, dw) = {
        let wc = WORKER_CONFIG.read().await.clone();
@@ -42,4 +42,5 @@ async-recursion.workspace = true
 bigdecimal.workspace = true
 axum.workspace = true
 serde_urlencoded.workspace = true
-regex.workspace = true
+regex.workspace = true
+croner = "2.0.6"
@@ -71,6 +71,9 @@ use windmill_common::BASE_URL;
 #[cfg(feature = "cloud")]
 use windmill_common::users::SUPERADMIN_SYNC_EMAIL;

+#[cfg(feature = "enterprise")]
+use windmill_common::flows::has_failure_module;
+
 #[cfg(feature = "enterprise")]
 use windmill_common::worker::CLOUD_HOSTED;

@@ -482,13 +485,6 @@ where
    }
 }

-#[cfg(feature = "enterprise")]
-#[derive(Deserialize)]
-struct RawFlowFailureModule {
-    #[cfg(feature = "enterprise")]
-    failure_module: Option<Box<RawValue>>,
-}
-
 #[instrument(level = "trace", skip_all)]
 pub async fn add_completed_job_error<R: rsmq_async::RsmqConnection + Clone + Send>(
    db: &Pool<Postgres>,
@@ -928,6 +924,8 @@ pub async fn add_completed_job<
                        .unwrap_or("".to_string()),
                ),
                db.clone(),
+                Some(&w_id),
+                None,
            )
            .await;
        } else if queued_job.email == SCHEDULE_ERROR_HANDLER_USER_EMAIL {
@@ -952,15 +950,7 @@ pub async fn add_completed_job<
        } else if !skip_downstream_error_handlers
            && (matches!(queued_job.job_kind, JobKind::Script)
                || matches!(queued_job.job_kind, JobKind::Flow)
-                    && queued_job
-                        .raw_flow
-                        .as_ref()
-                        .and_then(|v| {
-                            serde_json::from_str::<RawFlowFailureModule>((**v).get())
-                                .ok()
-                                .and_then(|v| v.failure_module)
-                        })
-                        .is_none())
+                    && !has_failure_module(job_id, db, true).await.unwrap_or(false))
            && queued_job.parent_job.is_none()
        {
            let result = serde_json::from_str(
@@ -1004,7 +994,7 @@ pub async fn add_completed_job<
                            "Could not push workspace error handler for failed job ({base_url}/run/{}?workspace={w_id}): {}",
                            queued_job.id,
                            err
-                        ), db.clone())
+                        ), db.clone(), Some(&w_id), None)
                        .await;
                    }
                }
@@ -1187,10 +1177,10 @@ pub async fn report_error_to_workspace_handler_or_critical_side_channel<
                queued_job.id,
                err
            );
-            report_critical_error(error_message, db.clone()).await;
+            report_critical_error(error_message, db.clone(), Some(&w_id), None).await;
        }
    } else {
-        report_critical_error(error_message, db.clone()).await;
+        report_critical_error(error_message, db.clone(), Some(&w_id), None).await;
    }
 }

@@ -3682,15 +3672,15 @@ pub async fn push<'c, 'd, R: rsmq_async::RsmqConnection + Send + 'c>(
        }
        JobPayload::DeploymentCallback { path } => (
            None,
-            Some(path),
+            Some(path.clone()),
            None,
            JobKind::DeploymentCallback,
            None,
            None,
            None,
-            None,
-            None,
-            None,
+            Some(format!("{workspace_id}:git_sync")),
+            Some(1),
+            Some(0),
            None,
            None,
            None,
@@ -25,6 +25,7 @@ use windmill_common::{
    users::username_to_permissioned_as,
    utils::{now_from_db, StripPath},
 };
+use croner::Cron;

 pub async fn push_scheduled_job<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
    db: &DB,
@@ -39,7 +40,9 @@ pub async fn push_scheduled_job<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
        ));
    }

-    let sched = cron::Schedule::from_str(schedule.schedule.as_ref())
+    let sched = Cron::new(schedule.schedule.as_ref())
+        .with_seconds_optional()
+        .parse()
        .map_err(|e| error::Error::BadRequest(e.to_string()))?;

    let tz = chrono_tz::Tz::from_str(&schedule.timezone)
@@ -65,8 +68,8 @@ pub async fn push_scheduled_job<'c, R: rsmq_async::RsmqConnection + Send + 'c>(
    };

    let next = sched
-        .after(&starting_from)
-        .next()
+        // `false` means not inclusive of `starting_from`
+        .find_next_occurrence(&starting_from, false)
        .expect("a schedule should have a next event");

    // println!("next event ({:?}): {}", tz, next);
@@ -0,0 +1,93 @@
+name: "python download pip"
+
+mode: ONCE
+hostname: "python"
+log_level: ERROR
+time_limit: 900
+
+rlimit_as: 2048
+rlimit_cpu: 1000
+rlimit_fsize: 1024
+rlimit_nofile: 64
+
+envar: "HOME=/user"
+envar: "LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH"
+
+cwd: "/tmp"
+
+clone_newnet: false
+clone_newuser: {CLONE_NEWUSER}
+
+keep_caps: true
+keep_env: true
+
+mount {
+    src: "/bin"
+    dst: "/bin"
+	is_bind: true
+}
+
+mount {
+    src: "/lib"
+    dst: "/lib"
+	is_bind: true
+}
+
+mount {
+    src: "/lib64"
+    dst: "/lib64"
+	is_bind: true
+    mandatory: false
+}
+
+mount {
+    src: "/usr"
+    dst: "/usr"
+	is_bind: true
+}
+
+mount {
+    src: "/etc"
+    dst: "/etc"
+	is_bind: true
+}
+
+mount {
+	src: "/dev/null"
+	dst: "/dev/null"
+	is_bind: true
+	rw: true
+}
+
+mount {
+	dst: "/tmp"
+	fstype: "tmpfs"
+	rw: true
+    options: "size=500000000"
+}
+
+
+mount {
+    src: "{WORKER_DIR}/download_deps.py.pip.sh"
+    dst: "/download_deps.sh"
+    is_bind: true
+}
+
+mount {
+    src: "{CACHE_DIR}"
+    dst: "{CACHE_DIR}"
+    is_bind: true
+    rw: true
+}
+
+mount {
+    src: "/dev/urandom"
+    dst: "/dev/urandom"
+    is_bind: true
+}
+
+exec_bin {
+ 	path: "/bin/sh"
+ 	arg: "/download_deps.sh"
+}
+
@@ -0,0 +1,24 @@
+#/bin/sh
+
+INDEX_URL_ARG=$([ -z "$INDEX_URL" ] && echo ""|| echo "--index-url $INDEX_URL" )
+EXTRA_INDEX_URL_ARG=$([ -z "$EXTRA_INDEX_URL" ] && echo ""|| echo "--extra-index-url $EXTRA_INDEX_URL" )
+TRUSTED_HOST_ARG=$([ -z "$TRUSTED_HOST" ] &&  echo "" || echo "--trusted-host $TRUSTED_HOST")
+
+if [ ! -z "$INDEX_URL" ]
+then
+      echo "\$INDEX_URL is set to $INDEX_URL"
+fi
+
+if [ ! -z "$EXTRA_INDEX_URL" ]
+then
+      echo "\$EXTRA_INDEX_URL is set to $EXTRA_INDEX_URL"
+fi
+
+if [ ! -z "$TRUSTED_HOST" ]
+then
+      echo "\$TRUSTED_HOST is set to $TRUSTED_HOST"
+fi
+
+CMD="/usr/local/bin/python3 -m pip install -v \"$REQ\" -I -t \"$TARGET\" --no-cache --no-color --no-deps --isolated --no-warn-conflicts --disable-pip-version-check $INDEX_URL_ARG $EXTRA_INDEX_URL_ARG $TRUSTED_HOST_ARG"
+echo $CMD
+eval $CMD
@@ -19,6 +19,18 @@ then
      echo "\$TRUSTED_HOST is set to $TRUSTED_HOST"
 fi

-CMD="/usr/local/bin/python3 -m pip install -v \"$REQ\" -I -t \"$TARGET\" --no-cache --no-color --no-deps --isolated --no-warn-conflicts --disable-pip-version-check $INDEX_URL_ARG $EXTRA_INDEX_URL_ARG $TRUSTED_HOST_ARG"
+CMD="/usr/local/bin/uv pip install 
+\"$REQ\" 
+--target \"$TARGET\" 
+--no-cache 
+--no-config
+--no-color 
+--no-deps 
+--link-mode=copy
+$INDEX_URL_ARG $EXTRA_INDEX_URL_ARG $TRUSTED_HOST_ARG
+--index-strategy unsafe-best-match
+--system
+"
+
 echo $CMD
 eval $CMD
@@ -116,6 +116,8 @@ async fn handle_ansible_python_deps(
            job_dir,
            worker_dir,
            &mut Some(occupancy_metrics),
+            true,
+            true,
        )
        .await?;
        additional_python_paths.append(&mut venv_path);
@@ -41,7 +41,7 @@ use tokio::{io::AsyncWriteExt, process::Child, time::Instant};

 use crate::{
    AuthedClient, AuthedClientBackgroundTask, JOB_DEFAULT_TIMEOUT, MAX_RESULT_SIZE,
-    MAX_TIMEOUT_DURATION,
+    MAX_TIMEOUT_DURATION, PATH_ENV,
 };

 pub async fn build_args_map<'a>(
@@ -860,11 +860,17 @@ pub async fn save_in_cache(
 }

 fn tentatively_improve_error(err: Error, executable: &str) -> Error {
+    #[cfg(unix)]
+    let err_msg = "No such file or directory (os error 2)";
+
+    #[cfg(windows)]
+    let err_msg = "program not found";
+
    if err
        .to_string()
-        .contains("No such file or directory (os error 2)")
+        .contains(&err_msg)
    {
-        return Error::InternalErr(format!("Executable {executable} not found on worker"));
+        return Error::InternalErr(format!("Executable {executable} not found on worker. PATH: {}", *PATH_ENV));
    }
    return err;
 }
@@ -20,13 +20,21 @@ use std::sync::Arc;
 pub async fn build_tar_and_push(
    s3_client: Arc<dyn ObjectStore>,
    folder: String,
+    no_uv: bool,
 ) -> error::Result<()> {
    use object_store::path::Path;

+    use crate::PY311_CACHE_DIR;
+
    tracing::info!("Started building and pushing piptar {folder}");
    let start = Instant::now();
    let folder_name = folder.split("/").last().unwrap();
-    let tar_path = format!("{PIP_CACHE_DIR}/{folder_name}_tar.tar",);
+    let prefix = if no_uv {
+        PIP_CACHE_DIR
+    } else {
+        PY311_CACHE_DIR
+    };
+    let tar_path = format!("{prefix}/{folder_name}_tar.tar",);

    let tar_file = std::fs::File::create(&tar_path)?;
    let mut tar = tar::Builder::new(tar_file);
@@ -46,7 +54,10 @@ pub async fn build_tar_and_push(
    // })?;
    if let Err(e) = s3_client
        .put(
-            &Path::from(format!("/tar/pip/{folder_name}.tar")),
+            &Path::from(format!(
+                "/tar/{}/{folder_name}.tar",
+                if no_uv { "pip" } else { "python_311" }
+            )),
            std::fs::read(&tar_path)?.into(),
        )
        .await
@@ -71,7 +82,11 @@ pub async fn build_tar_and_push(
 }

 #[cfg(all(feature = "enterprise", feature = "parquet"))]
-pub async fn pull_from_tar(client: Arc<dyn ObjectStore>, folder: String) -> error::Result<()> {
+pub async fn pull_from_tar(
+    client: Arc<dyn ObjectStore>,
+    folder: String,
+    no_uv: bool,
+) -> error::Result<()> {
    use windmill_common::s3_helpers::attempt_fetch_bytes;

    let folder_name = folder.split("/").last().unwrap();
@@ -79,7 +94,10 @@ pub async fn pull_from_tar(client: Arc<dyn ObjectStore>, folder: String) -> erro
    tracing::info!("Attempting to pull piptar {folder_name} from bucket");

    let start = Instant::now();
-    let tar_path = format!("tar/pip/{folder_name}.tar");
+    let tar_path = format!(
+        "tar/{}/{folder_name}.tar",
+        if no_uv { "pip" } else { "python_311" }
+    );
    let bytes = attempt_fetch_bytes(client, &tar_path).await?;

    // tracing::info!("B: {target} {folder}");
@@ -1,4 +1,4 @@
-use std::{collections::HashMap, process::Stdio};
+use std::{collections::HashMap, fs, process::Stdio};

 use itertools::Itertools;
 use regex::Regex;
@@ -42,6 +42,10 @@ lazy_static::lazy_static! {
    static ref USE_PIP_COMPILE: bool = std::env::var("USE_PIP_COMPILE")
        .ok().map(|flag| flag == "true").unwrap_or(false);

+    /// Use pip install
+    static ref USE_PIP_INSTALL: bool = std::env::var("USE_PIP_INSTALL")
+        .ok().map(|flag| flag == "true").unwrap_or(false);
+

    static ref RELATIVE_IMPORT_REGEX: Regex = Regex::new(r#"(import|from)\s(((u|f)\.)|\.)"#).unwrap();

@@ -50,6 +54,8 @@ lazy_static::lazy_static! {
 }

 const NSJAIL_CONFIG_DOWNLOAD_PY_CONTENT: &str = include_str!("../nsjail/download.py.config.proto");
+const NSJAIL_CONFIG_DOWNLOAD_PY_CONTENT_FALLBACK: &str =
+    include_str!("../nsjail/download.py.pip.config.proto");
 const NSJAIL_CONFIG_RUN_PYTHON3_CONTENT: &str = include_str!("../nsjail/run.python3.config.proto");
 const RELATIVE_PYTHON_LOADER: &str = include_str!("../loader.py");

@@ -66,8 +72,8 @@ use crate::{
    },
    handle_child::handle_child,
    AuthedClientBackgroundTask, DISABLE_NSJAIL, DISABLE_NUSER, HOME_ENV, LOCK_CACHE_DIR,
-    NSJAIL_PATH, PATH_ENV, PIP_CACHE_DIR, PIP_EXTRA_INDEX_URL, PIP_INDEX_URL, PROXY_ENVS, TZ_ENV,
-    UV_CACHE_DIR,
+    NSJAIL_PATH, PATH_ENV, PIP_CACHE_DIR, PIP_EXTRA_INDEX_URL, PIP_INDEX_URL, PROXY_ENVS,
+    PY311_CACHE_DIR, TZ_ENV, UV_CACHE_DIR,
 };

 #[cfg(windows)]
@@ -314,7 +320,7 @@ pub async fn uv_pip_compile(
            .args(args)
            .stdout(Stdio::piped())
            .stderr(Stdio::piped());
-        let child_process = start_child_process(child_cmd, "/usr/local/bin/uv").await?;
+        let child_process = start_child_process(child_cmd, uv_cmd).await?;
        append_logs(&job_id, &w_id, logs, db).await;
        handle_child(
            job_id,
@@ -901,10 +907,10 @@ async fn handle_python_deps(
        .unwrap_or_else(|| vec![])
        .clone();

+    let annotations = windmill_common::worker::PythonAnnotations::parse(inner_content);
    let requirements = match requirements_o {
        Some(r) => r,
        None => {
-            let annotation = windmill_common::worker::PythonAnnotations::parse(inner_content);
            let mut already_visited = vec![];

            let requirements = windmill_parser_py_imports::parse_python_imports(
@@ -929,8 +935,8 @@ async fn handle_python_deps(
                    worker_name,
                    w_id,
                    occupancy_metrics,
-                    annotation.no_uv,
-                    annotation.no_cache,
+                    annotations.no_uv || annotations.no_uv_compile,
+                    annotations.no_cache,
                )
                .await
                .map_err(|e| {
@@ -955,6 +961,8 @@ async fn handle_python_deps(
            job_dir,
            worker_dir,
            occupancy_metrics,
+            annotations.no_uv || annotations.no_uv_install,
+            false,
        )
        .await?;
        additional_python_paths.append(&mut venv_path);
@@ -966,6 +974,7 @@ lazy_static::lazy_static! {
    static ref PIP_SECRET_VARIABLE: Regex = Regex::new(r"\$\{PIP_SECRET:([^\s\}]+)\}").unwrap();
 }

+/// pip install, include cached or pull from S3
 pub async fn handle_python_reqs(
    requirements: Vec<&str>,
    job_id: &Uuid,
@@ -977,12 +986,22 @@ pub async fn handle_python_reqs(
    job_dir: &str,
    worker_dir: &str,
    occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
+    // TODO: Remove (Deprecated)
+    mut no_uv_install: bool,
+    is_ansible: bool,
 ) -> error::Result<Vec<String>> {
    let mut req_paths: Vec<String> = vec![];
    let mut vars = vec![("PATH", PATH_ENV.as_str())];
    let pip_extra_index_url;
    let pip_index_url;

+    no_uv_install |= *USE_PIP_INSTALL;
+
+    if no_uv_install && !is_ansible {
+        append_logs(&job_id, w_id, "\nFallback to pip (Deprecated!)\n", db).await;
+        tracing::warn!("Fallback to pip");
+    }
+
    if !*DISABLE_NSJAIL {
        pip_extra_index_url = PIP_EXTRA_INDEX_URL
            .read()
@@ -1013,10 +1032,21 @@ pub async fn handle_python_reqs(
        let _ = write_file(
            job_dir,
            "download.config.proto",
-            &NSJAIL_CONFIG_DOWNLOAD_PY_CONTENT
-                .replace("{WORKER_DIR}", &worker_dir)
-                .replace("{CACHE_DIR}", PIP_CACHE_DIR)
-                .replace("{CLONE_NEWUSER}", &(!*DISABLE_NUSER).to_string()),
+            &(if no_uv_install {
+                NSJAIL_CONFIG_DOWNLOAD_PY_CONTENT_FALLBACK
+            } else {
+                NSJAIL_CONFIG_DOWNLOAD_PY_CONTENT
+            })
+            .replace("{WORKER_DIR}", &worker_dir)
+            .replace(
+                "{CACHE_DIR}",
+                if no_uv_install {
+                    PIP_CACHE_DIR
+                } else {
+                    PY311_CACHE_DIR
+                },
+            )
+            .replace("{CLONE_NEWUSER}", &(!*DISABLE_NUSER).to_string()),
        )?;
    };

@@ -1026,8 +1056,14 @@ pub async fn handle_python_reqs(
        if req.starts_with('#') {
            continue;
        }
+        let py_prefix = if no_uv_install {
+            PIP_CACHE_DIR
+        } else {
+            PY311_CACHE_DIR
+        };
+
        let venv_p = format!(
-            "{PIP_CACHE_DIR}/{}",
+            "{py_prefix}/{}",
            req.replace(' ', "").replace('/', "").replace(':', "")
        );
        if metadata(&venv_p).await.is_ok() {
@@ -1073,7 +1109,10 @@ pub async fn handle_python_reqs(
                .map(|(req, venv_p)| {
                    let os = os.clone();
                    async move {
-                        if pull_from_tar(os, venv_p.clone()).await.is_ok() {
+                        if pull_from_tar(os, venv_p.clone(), no_uv_install)
+                            .await
+                            .is_ok()
+                        {
                            PullFromTar::Pulled(venv_p.to_string())
                        } else {
                            PullFromTar::NotPulled(req.to_string(), venv_p.to_string())
@@ -1114,7 +1153,11 @@ pub async fn handle_python_reqs(

    for (req, venv_p) in req_with_penv {
        let mut logs1 = String::new();
-        logs1.push_str("\n\n--- PIP INSTALL ---\n");
+        if no_uv_install {
+            logs1.push_str("\n\n--- PIP INSTALL ---\n");
+        } else {
+            logs1.push_str("\n\n--- UV PIP INSTALL ---\n");
+        }
        logs1.push_str(&format!("\n{req} is being installed for the first time.\n It will be cached for all ulterior uses."));
        append_logs(&job_id, w_id, logs1, db).await;

@@ -1150,21 +1193,47 @@ pub async fn handle_python_reqs(
            #[cfg(windows)]
            let req = format!("{}", req);

-            let mut command_args = vec![
-                PYTHON_PATH.as_str(),
-                "-m",
-                "pip",
-                "install",
-                &req,
-                "-I",
-                "--no-deps",
-                "--no-color",
-                "--isolated",
-                "--no-warn-conflicts",
-                "--disable-pip-version-check",
-                "-t",
-                venv_p.as_str(),
-            ];
+            let mut command_args = if no_uv_install {
+                vec![
+                    PYTHON_PATH.as_str(),
+                    "-m",
+                    "pip",
+                    "install",
+                    &req,
+                    "-I",
+                    "--no-deps",
+                    "--no-color",
+                    "--isolated",
+                    "--no-warn-conflicts",
+                    "--disable-pip-version-check",
+                    "-t",
+                    venv_p.as_str(),
+                ]
+            } else {
+                vec![
+                    UV_PATH.as_str(),
+                    "pip",
+                    "install",
+                    &req,
+                    "--no-deps",
+                    "--no-color",
+                    // "-p",
+                    // "3.11",
+                    // Prevent uv from discovering configuration files.
+                    "--no-config",
+                    "--link-mode=copy",
+                    // TODO: Doublecheck it
+                    "--system",
+                    // Prefer main index over extra
+                    // https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes
+                    // TODO: Use env variable that can be toggled from UI
+                    "--index-strategy",
+                    "unsafe-best-match",
+                    "--target",
+                    venv_p.as_str(),
+                    "--no-cache",
+                ]
+            };
            let pip_extra_index_url = PIP_EXTRA_INDEX_URL
                .read()
                .await
@@ -1196,7 +1265,7 @@ pub async fn handle_python_reqs(

            envs.push(("HOME", HOME_ENV.as_str()));

-            tracing::debug!("pip install command: {:?}", command_args);
+            tracing::debug!("uv pip install command: {:?}", command_args);

            #[cfg(unix)]
            {
@@ -1207,7 +1276,12 @@ pub async fn handle_python_reqs(
                    .envs(envs)
                    .args([
                        "-x",
-                        &format!("{}/pip-{}.lock", LOCK_CACHE_DIR, fssafe_req),
+                        &format!(
+                            "{}/{}-{}.lock",
+                            LOCK_CACHE_DIR,
+                            if no_uv_install { "pip" } else { "py311" },
+                            fssafe_req
+                        ),
                        "--command",
                        &command_args.join(" "),
                    ])
@@ -1218,16 +1292,20 @@ pub async fn handle_python_reqs(

            #[cfg(windows)]
            {
-                let mut pip_cmd = Command::new(PYTHON_PATH.as_str());
-                pip_cmd
-                    .env_clear()
+                let installer_path = if no_uv_install { command_args[0] } else { "uv" };
+                let mut cmd: Command = Command::new(&installer_path);
+                cmd.env_clear()
                    .envs(envs)
                    .envs(PROXY_ENVS.clone())
                    .env("SystemRoot", SYSTEM_ROOT.as_str())
+                    .env(
+                        "TMP",
+                        std::env::var("TMP").unwrap_or_else(|_| String::from("/tmp")),
+                    )
                    .args(&command_args[1..])
                    .stdout(Stdio::piped())
                    .stderr(Stdio::piped());
-                start_child_process(pip_cmd, PYTHON_PATH.as_str()).await?
+                start_child_process(cmd, installer_path).await?
            }
        };

@@ -1240,7 +1318,7 @@ pub async fn handle_python_reqs(
            false,
            worker_name,
            &w_id,
-            &format!("pip install {req}"),
+            &format!("uv pip install {req}"),
            None,
            false,
            occupancy_metrics,
@@ -1252,6 +1330,16 @@ pub async fn handle_python_reqs(
            "finished setting up python dependencies {}",
            job_id
        );
+        if child.is_err() {
+            
+            if let Err(e) = fs::remove_dir_all(&venv_p) {
+                tracing::warn!(
+                    workspace_id = %w_id,
+                    "failed to remove cache dir: {:?}",
+                    e
+                );
+            }
+        }
        child?;

        #[cfg(all(feature = "enterprise", feature = "parquet"))]
@@ -1260,7 +1348,7 @@ pub async fn handle_python_reqs(
                tracing::warn!("S3 cache not available in the pro plan");
            } else {
                let venv_p = venv_p.clone();
-                tokio::spawn(build_tar_and_push(os, venv_p));
+                tokio::spawn(build_tar_and_push(os, venv_p, no_uv_install));
            }
        }
        req_paths.push(venv_p);
@@ -236,7 +236,14 @@ pub const TMP_LOGS_DIR: &str = concatcp!(TMP_DIR, "/logs");
 pub const ROOT_CACHE_NOMOUNT_DIR: &str = concatcp!(TMP_DIR, "/cache_nomount/");

 pub const LOCK_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "lock");
+// Used as fallback now
 pub const PIP_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "pip");
+
+// pub const PY310_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "python_310");
+pub const PY311_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "python_311");
+// pub const PY312_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "python_312");
+// pub const PY313_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "python_313");
+
 pub const UV_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "uv");
 pub const TAR_PIP_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "tar/pip");
 pub const DENO_CACHE_DIR: &str = concatcp!(ROOT_CACHE_DIR, "deno");
@@ -257,6 +264,7 @@ const NUM_SECS_PING: u64 = 5;
 const NUM_SECS_READINGS: u64 = 60;

 const INCLUDE_DEPS_PY_SH_CONTENT: &str = include_str!("../nsjail/download_deps.py.sh");
+const INCLUDE_DEPS_PY_SH_CONTENT_FALLBACK: &str = include_str!("../nsjail/download_deps.py.pip.sh");

 pub const DEFAULT_CLOUD_TIMEOUT: u64 = 900;
 pub const DEFAULT_SELFHOSTED_TIMEOUT: u64 = 604800; // 7 days
@@ -311,6 +319,7 @@ lazy_static::lazy_static! {
    .and_then(|x| x.parse::<bool>().ok())
    .unwrap_or(false);

+    // pub static ref DISABLE_NSJAIL: bool = false;
    pub static ref DISABLE_NSJAIL: bool = std::env::var("DISABLE_NSJAIL")
        .ok()
        .and_then(|x| x.parse::<bool>().ok())
@@ -740,6 +749,13 @@ pub async fn run_worker<R: rsmq_async::RsmqConnection + Send + Sync + Clone + 's
            "download_deps.py.sh",
            INCLUDE_DEPS_PY_SH_CONTENT,
        );
+
+        // TODO: Remove (Deprecated)
+        let _ = write_file(
+            &worker_dir,
+            "download_deps.py.pip.sh",
+            INCLUDE_DEPS_PY_SH_CONTENT_FALLBACK,
+        );
    }

    let mut last_ping = Instant::now() - Duration::from_secs(NUM_SECS_PING + 1);
@@ -1903,8 +1919,10 @@ async fn handle_queued_job<R: rsmq_async::RsmqConnection + Send + Sync + Clone>(
        }
    };
    if job.is_flow() {
+        let flow = job.parse_raw_flow();
        handle_flow(
            job,
+            flow,
            db,
            &client.get_authed().await,
            None,
@@ -8,7 +8,6 @@

 use std::collections::hash_map::DefaultHasher;
 use std::collections::HashMap;
-use std::hash::Hash;
 use std::sync::atomic::{AtomicUsize, Ordering};
 use std::sync::Arc;
 use std::time::Duration;
@@ -49,7 +48,7 @@ use windmill_common::{
        Approval, BranchAllStatus, BranchChosen, FlowStatus, FlowStatusModule, RetryStatus,
        MAX_RETRY_ATTEMPTS, MAX_RETRY_INTERVAL,
    },
-    flows::{FlowModule, FlowModuleValue, FlowValue, InputTransform, Retry, Suspend},
+    flows::{has_failure_module, FlowModule, FlowModuleValue, FlowValue, InputTransform, Retry, Suspend},
 };
 use windmill_queue::schedule::get_schedule_opt;
 use windmill_queue::{
@@ -177,11 +176,6 @@ struct RecoveryObject {
    recover: Option<bool>,
 }

-#[derive(sqlx::FromRow, Deserialize)]
-pub struct RowFlowStatus {
-    pub flow_status: sqlx::types::Json<Box<serde_json::value::RawValue>>,
-    pub current_module: Option<sqlx::types::Json<Box<serde_json::value::RawValue>>>,
-}
 // #[instrument(level = "trace", skip_all)]
 pub async fn update_flow_status_after_job_completion_internal<
    R: rsmq_async::RsmqConnection + Send + Sync + Clone,
@@ -207,6 +201,7 @@ pub async fn update_flow_status_after_job_completion_internal<
    let (
        should_continue_flow,
        flow_job,
+        flow_value,
        stop_early,
        skip_if_stop_early,
        nresult,
@@ -215,35 +210,28 @@ pub async fn update_flow_status_after_job_completion_internal<
    ) = {
        // tracing::debug!("UPDATE FLOW STATUS: {flow:?} {success} {result:?} {w_id} {depth}");

-        let old_status_json = sqlx::query_as::<_, RowFlowStatus>(
-            "SELECT flow_status, raw_flow->'modules'->(flow_status->'step')::int as current_module FROM queue WHERE id = $1 AND workspace_id = $2",
+        let (old_status, current_module) = sqlx::query!(
+            "SELECT
+                flow_status AS \"flow_status!: Json<Box<RawValue>>\",
+                raw_flow->'modules'->(flow_status->'step')::int AS \"module: Json<Box<RawValue>>\"
+            FROM queue WHERE id = $1 AND workspace_id = $2 LIMIT 1",
+            flow, w_id
        )
-        .bind(flow)
-        .bind(w_id)
        .fetch_one(db)
        .await
-        .map_err(|e| {
-            Error::InternalErr(format!(
-                "fetching flow status {flow} while reporting {success} {result:?}: {e:#}"
-            ))
-        })?;
-
-        let old_status = serde_json::from_str::<FlowStatus>(old_status_json.flow_status.get())
-            .or_else(|e| {
-                Err(Error::InternalErr(format!(
-                    "requiring status to be parsable as FlowStatus: {e:?}"
-                )))
-            })?;
-
-        let current_module = if let Some(x) = old_status_json.current_module {
-            Some(serde_json::from_str::<FlowModule>(x.0.get()).or_else(|e| {
-                Err(Error::InternalErr(format!(
+        .map_err(|e| Error::InternalErr(
+            format!("fetching flow status {flow} while reporting {success} {result:?}: {e:#}")
+        ))
+        .and_then(|record| Ok((
+            serde_json::from_str::<FlowStatus>(record.flow_status.0.get()).map_err(|e| Error::InternalErr(
+                format!("requiring current module to be parsable as FlowStatus: {e:?}")
+            ))?,
+            record.module.map(|json| {
+                serde_json::from_str::<FlowModule>(json.0.get()).map_err(|e| Error::InternalErr(format!(
                    "requiring current module to be parsable as FlowModule: {e:?}"
                )))
-            })?)
-        } else {
-            None
-        };
+            }).transpose()?,
+        )))?;

        let module_step = Step::from_i32_and_len(old_status.step, old_status.modules.len());

@@ -303,16 +291,15 @@ pub async fn update_flow_status_after_job_completion_internal<

            let is_flow = if let Some(step) = step {
                sqlx::query_scalar!(
-                    "SELECT raw_flow->'modules'->($1)->'value'->>'type' = 'flow' FROM queue WHERE id = $2",
-                    step as i32,
-                    &flow
+                    "SELECT raw_flow->'modules'->($1)::text->'value'->>'type' = 'flow' FROM queue WHERE id = $2 LIMIT 1",
+                    step as i32, flow
                )
-                    .fetch_one(db)
-                    .await
-                    .map_err(|e| {
-                        Error::InternalErr(format!("error during retrieval of step's type: {e:#}"))
-                    })?
-                    .unwrap_or(false)
+                .fetch_one(db)
+                .await
+                .map_err(|e| {
+                    Error::InternalErr(format!("error during retrieval of step's type: {e:#}"))
+                })?
+                .unwrap_or(false)
            } else {
                false
            };
@@ -938,10 +925,7 @@ pub async fn update_flow_status_after_job_completion_internal<
            .unwrap_or_else(|| "none".to_string());
        tracing::info!(id = %flow_job.id, root_id = %job_root, "update flow status");

-        let module = get_module(&flow_job, &module_step);
-        // tracing::error!(
-        //     "UPDATE FLOW STATUS 3: {module:#?} {unrecoverable} {} {is_last_step} {success} {skip_error_handler} is_failure_step {is_failure_step}", flow_job.canceled
-        // );
+        let flow_value = flow_job.parse_raw_flow();

        let should_continue_flow = match success {
            _ if stop_early => false,
@@ -953,7 +937,14 @@ pub async fn update_flow_status_after_job_completion_internal<
            }
            false
                if next_retry(
-                    &module.and_then(|m| m.retry.clone()).unwrap_or_default(),
+                    flow_value
+                        .as_ref()
+                        .and_then(|value| match module_step {
+                            Step::PreprocessorStep => value.preprocessor_module.as_ref().and_then(|m| m.retry.as_ref()),
+                            Step::Step(i) => value.modules.get(i).as_ref().and_then(|m| m.retry.as_ref()),
+                            Step::FailureStep => value.failure_module.as_ref().and_then(|m| m.retry.as_ref()),
+                        })
+                        .unwrap_or(&Retry::default()),
                    &old_status.retry,
                )
                .is_some() =>
@@ -963,7 +954,7 @@ pub async fn update_flow_status_after_job_completion_internal<
            false
                if !is_failure_step
                    && !skip_error_handler
-                    && has_failure_module(flow, db).await? =>
+                    && has_failure_module(flow, db, false).await? =>
            {
                true
            }
@@ -975,6 +966,7 @@ pub async fn update_flow_status_after_job_completion_internal<
        (
            should_continue_flow,
            flow_job,
+            flow_value,
            stop_early,
            skip_if_stop_early,
            nresult,
@@ -1042,13 +1034,11 @@ pub async fn update_flow_status_after_job_completion_internal<
                    let args_hash =
                        hash_args(db, client, w_id, job_id_for_status, &flow_job.args).await;
                    let flow_path = flow_job.script_path();
-                    let version_hash = if let Some(rc) = flow_job.raw_flow.as_ref() {
-                        use std::hash::Hasher;
-                        let mut s = DefaultHasher::new();
-                        serde_json::to_string(&rc.0)
-                            .unwrap_or_default()
-                            .hash(&mut s);
-                        format!("flow_{}", hex::encode(s.finish().to_be_bytes()))
+                    let version_hash = if let Some(sqlx::types::Json(s)) = flow_job.raw_flow.as_ref() {
+                        use std::hash::{Hash, Hasher};
+                        let mut h = DefaultHasher::new();
+                        s.get().hash(&mut h);
+                        format!("flow_{}", hex::encode(h.finish().to_be_bytes()))
                    } else {
                        "flow_unknown".to_string()
                    };
@@ -1107,6 +1097,7 @@ pub async fn update_flow_status_after_job_completion_internal<
        tracing::debug!(id = %flow_job.id,  "start handle flow");
        match handle_flow(
            flow_job.clone(),
+            flow_value,
            db,
            client,
            Some(nresult.clone()),
@@ -1239,19 +1230,6 @@ async fn retrieve_flow_jobs_results(
    Ok(to_raw_value(&results))
 }

-fn get_module(flow_job: &QueuedJob, module_step: &Step) -> Option<FlowModule> {
-    let raw_flow = flow_job.parse_raw_flow();
-    if let Some(raw_flow) = raw_flow {
-        match module_step {
-            Step::PreprocessorStep => raw_flow.preprocessor_module.map(|x| *x.clone()),
-            Step::Step(i) => raw_flow.modules.get(*i).map(|x| x.clone()),
-            Step::FailureStep => raw_flow.failure_module.map(|x| *x.clone()),
-        }
-    } else {
-        None
-    }
-}
-
 async fn compute_skip_branchall_failure<'c>(
    job: &Uuid,
    branch: usize,
@@ -1290,23 +1268,6 @@ async fn compute_skip_branchall_failure<'c>(
        }))
 }

-async fn has_failure_module<'c>(flow: Uuid, db: &DB) -> Result<bool, Error> {
-    sqlx::query_scalar::<_, Option<bool>>(
-        "SELECT raw_flow->'failure_module' != 'null'::jsonb
-        FROM queue
-        WHERE id = $1",
-    )
-    .bind(flow)
-    .fetch_one(db)
-    .await
-    .map_err(|e| {
-        Error::InternalErr(format!(
-            "error during retrieval of has_failure_module: {e:#}"
-        ))
-    })
-    .map(|v| v.unwrap_or(false))
-}
-
 // async fn retrieve_cleanup_module<'c>(flow_uuid: Uuid, db: &DB) -> Result<FlowCleanupModule, Error> {
 //     tracing::warn!("Retrieving cleanup module of flow {}", flow_uuid);
 //     let raw_value = sqlx::query_scalar!(
@@ -1523,6 +1484,7 @@ async fn transform_input(
 #[instrument(level = "trace", skip_all)]
 pub async fn handle_flow<R: rsmq_async::RsmqConnection + Send + Sync + Clone>(
    flow_job: Arc<QueuedJob>,
+    flow_value: Option<FlowValue>,
    db: &sqlx::Pool<sqlx::Postgres>,
    client: &AuthedClient,
    last_result: Option<Arc<Box<RawValue>>>,
@@ -1531,8 +1493,7 @@ pub async fn handle_flow<R: rsmq_async::RsmqConnection + Send + Sync + Clone>(
    rsmq: Option<R>,
    job_completed_tx: Sender<SendResult>,
 ) -> anyhow::Result<()> {
-    let flow = flow_job
-        .parse_raw_flow()
+    let flow = flow_value
        .with_context(|| "Unable to parse flow definition")?;
    let status = flow_job
        .parse_flow_status()
@@ -1312,6 +1312,8 @@ async fn python_dep(
            job_dir,
            worker_dir,
            occupancy_metrics,
+            false,
+            false,
        )
        .await;

@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
 import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
 import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";

-export const VERSION = "v1.423.2";
+export const VERSION = "v1.424.0";

 export async function login(email: string, password: string): Promise<string> {
  return await windmill.UserService.login({
@@ -33,10 +33,10 @@
  },
  {
    "kind": "bigrawscript",
-    "jobs": 10000
+    "jobs": 4000
  },
  {
    "kind": "bigscriptinflow",
-    "jobs": 10000
+    "jobs": 4000
  }
 ]
@@ -60,7 +60,7 @@ export {
 //   }
 // });

-export const VERSION = "1.423.2";
+export const VERSION = "1.424.0";

 const command = new Command()
  .name("wmill")
@@ -26,7 +26,10 @@
      ]);
    in {
      devShell = pkgs.mkShell {
-        buildInputs = buildInputs ++ (with pkgs; [ git go xcaddy sqlx-cli ]);
+        buildInputs = buildInputs ++ (with pkgs; [
+          git xcaddy sqlx-cli flock rust-analyzer
+          deno python3 python3Packages.pip go bun uv
+        ]);
        packages = [
          (pkgs.writeScriptBin "wm-caddy" ''
            cd ./frontend
@@ -56,6 +59,9 @@
            sqlx database create
            wm-migrate
          '')
+          (pkgs.writeScriptBin "wm-bench" ''
+            deno run -A benchmarks/main.ts -e admin@windmill.dev -p changeme $*
+          '')
          (pkgs.writeScriptBin "wm" ''
            cd ./frontend
            npm run dev $*
@@ -68,6 +74,12 @@
        DATABASE_URL = "postgres://postgres:changeme@127.0.0.1:5432/";
        REMOTE = "http://127.0.0.1:8000";
        REMOTE_LSP = "http://127.0.0.1:3001";
+        DENO_PATH = "${pkgs.deno}/bin/deno";
+        PYTHON_PATH = "${pkgs.python3}/bin/python3";
+        GO_PATH = "${pkgs.go}/bin/go";
+        BUN_PATH = "${pkgs.bun}/bin/bun";
+        UV_PATH = "${pkgs.uv}/bin/uv";
+        FLOCK_PATH = "${pkgs.flock}/bin/flock";
      };
      packages.default = self.packages.${system}.windmill;
      packages.windmill-client = pkgs.stdenv.mkDerivation {
@@ -82,7 +94,7 @@
          npm config set strict-ssl false
          cd frontend
          npm install --verbose
-          npm run generate-backend-client-mac
+          npm run ${if pkgs.stdenv.isDarwin then "generate-backend-client-mac" else "generate-backend-client"}
          NODE_OPTIONS="--max-old-space-size=8192" npm run build
        '';

@@ -1,14 +1,15 @@
 {
 	"name": "windmill-components",
-	"version": "1.423.2",
+	"version": "1.424.0",
 	"lockfileVersion": 3,
 	"requires": true,
 	"packages": {
 		"": {
 			"name": "windmill-components",
-			"version": "1.423.2",
+			"version": "1.424.0",
 			"license": "AGPL-3.0",
 			"dependencies": {
+				"@anthropic-ai/sdk": "^0.32.1",
 				"@aws-crypto/sha256-js": "^4.0.0",
 				"@codingame/monaco-vscode-configuration-service-override": "~8.0.2",
 				"@codingame/monaco-vscode-files-service-override": "~8.0.2",
@@ -21,6 +22,7 @@
 				"@codingame/monaco-vscode-standalone-typescript-language-features": "~8.0.2",
 				"@json2csv/plainjs": "^7.0.6",
 				"@leeoniya/ufuzzy": "^1.0.8",
+				"@mistralai/mistralai": "^1.3.0",
 				"@popperjs/core": "^2.11.6",
 				"@redocly/json-to-json-schema": "^0.0.1",
 				"@tanstack/svelte-table": "^8.9.9",
@@ -175,6 +177,30 @@
 				"node": ">=6.0.0"
 			}
 		},
+		"node_modules/@anthropic-ai/sdk": {
+			"version": "0.32.1",
+			"resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.32.1.tgz",
+			"integrity": "sha512-U9JwTrDvdQ9iWuABVsMLj8nJVwAyQz6QXvgLsVhryhCEPkLsbcP/MXxm+jYcAwLoV8ESbaTTjnD4kuAFa+Hyjg==",
+			"license": "MIT",
+			"dependencies": {
+				"@types/node": "^18.11.18",
+				"@types/node-fetch": "^2.6.4",
+				"abort-controller": "^3.0.0",
+				"agentkeepalive": "^4.2.1",
+				"form-data-encoder": "1.7.2",
+				"formdata-node": "^4.3.2",
+				"node-fetch": "^2.6.7"
+			}
+		},
+		"node_modules/@anthropic-ai/sdk/node_modules/@types/node": {
+			"version": "18.19.64",
+			"resolved": "https://registry.npmjs.org/@types/node/-/node-18.19.64.tgz",
+			"integrity": "sha512-955mDqvO2vFf/oL7V3WiUtiz+BugyX8uVbaT2H8oj3+8dRyH2FLiNdowe7eNqRM7IOIZvzDH76EoAT+gwm6aIQ==",
+			"license": "MIT",
+			"dependencies": {
+				"undici-types": "~5.26.4"
+			}
+		},
 		"node_modules/@apidevtools/json-schema-ref-parser": {
 			"version": "11.6.1",
 			"resolved": "https://registry.npmjs.org/@apidevtools/json-schema-ref-parser/-/json-schema-ref-parser-11.6.1.tgz",
@@ -3659,6 +3685,14 @@
 			"resolved": "https://registry.npmjs.org/@mapbox/unitbezier/-/unitbezier-0.0.0.tgz",
 			"integrity": "sha512-HPnRdYO0WjFjRTSwO3frz1wKaU649OBFPX3Zo/2WZvuRi6zMiRGui8SnPQiQABgqCf8YikDe5t3HViTVw1WUzA=="
 		},
+		"node_modules/@mistralai/mistralai": {
+			"version": "1.3.0",
+			"resolved": "https://registry.npmjs.org/@mistralai/mistralai/-/mistralai-1.3.0.tgz",
+			"integrity": "sha512-G5DPCSC8sEhG3LUEZDYLD7qEZWDuZXgaX3IcoC3a/ydm9jFuh2pRZtknsgMx2sU8d7kxRuxblY3fPH5C38wnhQ==",
+			"peerDependencies": {
+				"zod": ">= 3"
+			}
+		},
 		"node_modules/@nodelib/fs.scandir": {
 			"version": "2.1.5",
 			"resolved": "https://registry.npmjs.org/@nodelib/fs.scandir/-/fs.scandir-2.1.5.tgz",
@@ -13738,6 +13772,16 @@
 			"integrity": "sha512-A7AMeJfGefk317I/3tBoUYRcDcNavKEkpiPN/nQsBz/viI2GvT7BtrqdPD6rGqBFN8Ax7v4obf+Cl32JF9DDVw==",
 			"dev": true
 		},
+		"node_modules/zod": {
+			"version": "3.23.8",
+			"resolved": "https://registry.npmjs.org/zod/-/zod-3.23.8.tgz",
+			"integrity": "sha512-XBx9AXhXktjUqnepgTiE5flcKIYWi/rme0Eaj+5Y0lftuGBq+jyRu/md4WnuxqgP1ubdpNCsYEYPxrzVHD8d6g==",
+			"license": "MIT",
+			"peer": true,
+			"funding": {
+				"url": "https://github.com/sponsors/colinhacks"
+			}
+		},
 		"node_modules/zstddec": {
 			"version": "0.1.0",
 			"resolved": "https://registry.npmjs.org/zstddec/-/zstddec-0.1.0.tgz",
@@ -1,6 +1,6 @@
 {
 	"name": "windmill-components",
-	"version": "1.423.2",
+	"version": "1.424.0",
 	"scripts": {
 		"dev": "vite dev",
 		"build": "vite build",
@@ -82,6 +82,7 @@
 	},
 	"type": "module",
 	"dependencies": {
+		"@anthropic-ai/sdk": "^0.32.1",
 		"@aws-crypto/sha256-js": "^4.0.0",
 		"@codingame/monaco-vscode-configuration-service-override": "~8.0.2",
 		"@codingame/monaco-vscode-files-service-override": "~8.0.2",
@@ -94,6 +95,7 @@
 		"@codingame/monaco-vscode-standalone-typescript-language-features": "~8.0.2",
 		"@json2csv/plainjs": "^7.0.6",
 		"@leeoniya/ufuzzy": "^1.0.8",
+		"@mistralai/mistralai": "^1.3.0",
 		"@popperjs/core": "^2.11.6",
 		"@redocly/json-to-json-schema": "^0.0.1",
 		"@tanstack/svelte-table": "^8.9.9",
@@ -72,11 +72,13 @@
 </script>

 <div class="flex flex-col max-w-2xl p-2">
+	{#if isConflict} 
 	<span class="text-sm mb-2 leading-6 font-semibold"
-		>{isConflict ? 'Fix username conflict' : 'Change username'}</span
+		>Fix username conflict</span
 	>
+	{/if}

-	<span class="text-xs mb-1 leading-6"
+	<span class="text-sm font-semibold mb-1 leading-6"
 		>{isConflict ? 'Auto-generated instance username' : 'New username'}</span
 	>
 	<input
@@ -39,7 +39,6 @@
 	import { CornerDownLeft, Play } from 'lucide-svelte'
 	import Toggle from './Toggle.svelte'
 	import { setLicense } from '$lib/enterpriseUtils'
-	import { workspacedOpenai } from './copilot/lib'
 	import type { FlowCopilotContext, FlowCopilotModule } from './copilot/flow'
 	import { pickScript } from './flows/flowStateUtils'
 	import {
@@ -49,6 +48,7 @@
 	} from '$lib/relative_imports'
 	import Tooltip from './Tooltip.svelte'
 	import type { ScheduleTrigger, TriggerContext } from './triggers'
+	import { initAllAiWorkspace } from './copilot/lib'
 	import type { FlowPropPickerConfig, PropPickerContext } from './prop_picker'
 	import type { PickableProperties } from './flows/previousResults'
 	$: token = $page.url.searchParams.get('wm_token') ?? undefined
@@ -110,12 +110,13 @@

 	async function setCopilotInfo() {
 		if (workspace) {
-			workspacedOpenai.init(workspace, token)
+			initAllAiWorkspace(workspace)
 			try {
 				copilotInfo.set(await WorkspaceService.getCopilotInfo({ workspace }))
 			} catch (err) {
 				copilotInfo.set({
-					exists_openai_resource_path: false,
+					ai_provider: '',
+					exists_ai_resource: false,
 					code_completion_enabled: false
 				})

@@ -171,6 +171,7 @@
 	import { initVim } from './monaco_keybindings'
 	import { buildWorkerDefinition } from '$lib/monaco_workers/build_workers'
 	import { parseTypescriptDeps } from '$lib/relative_imports'
+	import type { AiProviderTypes } from './copilot/lib'

 	// import EditorTheme from './EditorTheme.svelte'

@@ -593,11 +594,13 @@
 							token.onCancellationRequested(() => {
 								abortController?.abort()
 							})
+							const aiProvider = $copilotInfo.ai_provider
 							const insertText = await editorCodeCompletion(
 								textUntilPosition,
 								textAfterPosition,
 								lang,
-								abortController
+								abortController,
+								aiProvider as AiProviderTypes
 							)
 							if (insertText) {
 								items = [
@@ -624,7 +627,7 @@
 		)
 	}

-	$: $copilotInfo.exists_openai_resource_path &&
+	$: $copilotInfo.exists_ai_resource &&
 		$copilotInfo.code_completion_enabled &&
 		$codeCompletionSessionEnabled &&
 		initialized &&
@@ -85,6 +85,7 @@
 	import type { FlowBuilderWhitelabelCustomUi } from './custom_ui'
 	import FlowYamlEditor from './flows/header/FlowYamlEditor.svelte'
 	import { type TriggerContext, type ScheduleTrigger } from './triggers'
+	import type { AiProviderTypes } from './copilot/lib'

 	export let initialPath: string = ''
 	export let pathStoreInit: string | undefined = undefined
@@ -789,6 +790,7 @@
 		try {
 			push(history, $flowStore)
 			let module = stepOnly ? $copilotModulesStore[0] : $copilotModulesStore[idx]
+			const aiProvider = $copilotInfo.ai_provider as AiProviderTypes

 			copilotLoading = true
 			copilotStatus = "Generating code for step '" + module.id + "'..."
@@ -918,7 +920,8 @@
 						  })
 						: undefined,
 					isFirstInLoop,
-					abortController
+					abortController,
+					aiProvider
 				)
 				unsubscribe()
 			}
@@ -934,7 +937,7 @@
 						pastModule.value.type === 'script')
 				) {
 					const stepSchema: Schema = JSON.parse(JSON.stringify($flowStateStore[module.id].schema)) // deep copy
-					if (isHubStep && pastModule !== undefined && $copilotInfo.exists_openai_resource_path) {
+					if (isHubStep && pastModule !== undefined && $copilotInfo.exists_ai_resource) {
 						// ask AI to set step inputs
 						abortController = new AbortController()
 						const { inputs, allExprs } = await glueCopilot(
@@ -944,7 +947,8 @@
 								value: RawScript | PathScript
 							},
 							isFirstInLoop,
-							abortController
+							abortController,
+							aiProvider
 						)

 						// create flow inputs used by AI for autocompletion
@@ -992,11 +996,7 @@
 							$shouldUpdatePropertyType[key] = 'javascript'
 						})
 					} else {
-						if (
-							isHubStep &&
-							pastModule !== undefined &&
-							!$copilotInfo.exists_openai_resource_path
-						) {
+						if (isHubStep && pastModule !== undefined && !$copilotInfo.exists_ai_resource) {
 							sendUserToast(
 								'For better input generation, enable Windmill AI in the workspace settings',
 								true
@@ -5,17 +5,36 @@
 	import Popup from './common/popup/Popup.svelte'
 	import { offset, flip, shift } from 'svelte-floating-ui/dom'
 	import ChangeInstanceUsernameInner from './ChangeInstanceUsernameInner.svelte'
+	import { UserService } from '$lib/gen'
+	import { sendUserToast } from '$lib/toast'

 	export let value: string | undefined
 	export let email: string
 	export let username: string | undefined = undefined
 	export let automateUsernameCreation: boolean = false
+	export let login_type: string
+
+	let password: string = ''

 	const dispatch = createEventDispatcher()

-	function save() {
+	function saveName() {
 		dispatch('save', value)
 	}
+	async function savePassword() {
+		if (password.length < 5) {
+			sendUserToast('Password must be at least 5 characters long', true)
+			return
+		}
+		await UserService.setPasswordForUser({user: email,  requestBody:{ password }})
+		sendUserToast(`Password updated for ${email}`)
+	}
+
+	async function saveLoginType() {
+		await UserService.setLoginTypeForUser({user: email,  requestBody:{ login_type }})
+		sendUserToast(`Login type updated for ${email}`)
+		dispatch('refresh')
+	}
 </script>

 <Popup
@@ -34,7 +53,7 @@
 			<ChangeInstanceUsernameInner {email} {username} on:renamed />
 		{/if}
 		<label class="block text-primary">
-			<div class="pb-1 text-xs text-secondary">Name</div>
+			<div class="pb-2 text-sm font-semibold text-primary">Name</div>
 			<div class="flex w-full">
 				<input
 					type="text"
@@ -44,7 +63,7 @@
 					on:keydown|stopPropagation
 					on:keypress|stopPropagation={({ key }) => {
 						if (key === 'Enter') {
-							save()
+							saveName()
 							close(null)
 						}
 					}}
@@ -57,12 +76,78 @@
 				btnClasses="mt-2 "
 				aria-label="Save ID"
 				on:click={() => {
-					save()
+					saveName()
 					close(null)
 				}}
 			>
 				Update name
 			</Button>
 		</label>
+		<label class="block text-primary">
+			<div class="pb-2 text-sm font-semibold text-primary">Password</div>
+			<div class="flex w-full">
+				<input
+					type="password"
+					bind:value={password}
+					class="!w-auto grow"
+					on:click|stopPropagation={() => {}}
+					on:keydown|stopPropagation
+					on:keypress|stopPropagation={({ key }) => {
+						if (key === 'Enter') {
+							savePassword()
+							close(null)
+						}
+					}}
+				/>
+			</div>
+			<Button
+				size="xs"
+				color="blue"
+				buttonType="button"
+				btnClasses="mt-2 "
+				aria-label="Save ID"
+				on:click={() => {
+					savePassword()
+					close(null)
+				}}
+			>
+				Update password
+			</Button>
+		</label>
+		<label class="block text-primary">
+			<div class="pb-2 text-sm font-semibold text-primary">Login type</div>
+			<div class="text-xs text-secondary mb-2">
+				Must match exact SSO name, "password" or "saml". Examples: password, google, saml, microsoft
+			</div>
+
+			<div class="flex w-full">
+				<input
+					type="text"
+					bind:value={login_type}
+					class="!w-auto grow"
+					on:click|stopPropagation={() => {}}
+					on:keydown|stopPropagation
+					on:keypress|stopPropagation={({ key }) => {
+						if (key === 'Enter') {
+							saveLoginType()
+							close(null)
+						}
+					}}
+				/>
+			</div>
+			<Button
+				size="xs"
+				color="blue"
+				buttonType="button"
+				btnClasses="mt-2 "
+				aria-label="Save login type"
+				on:click={() => {
+					saveLoginType()
+					close(null)
+				}}
+			>
+				Update login type
+			</Button>
+		</label>
 	</div>
 </Popup>
@@ -14,7 +14,7 @@
 	import Alert from './common/alert/Alert.svelte'
 	import { isCloudHosted } from '$lib/cloud'
 	import { capitalize, classNames, sleep } from '$lib/utils'
-	import { enterpriseLicense } from '$lib/stores'
+	import { enterpriseLicense, isCriticalAlertsUIOpen } from '$lib/stores'
 	import CustomOauth from './CustomOauth.svelte'
 	import {
 		AlertCircle,
@@ -43,6 +43,7 @@
 	export let tab: string = 'Core'
 	export let hideTabs: boolean = false
 	export let hideSave: boolean = false
+	export let closeDrawer: (() => void) | undefined = () => {}

 	let values: Record<string, any> = {}
 	let initialOauths: Record<string, any> = {}
@@ -814,6 +815,26 @@
 													placeholder={setting.placeholder}
 													bind:value={values[setting.key]}
 												/>
+											{:else if setting.key == 'critical_alert_mute_ui'}
+												<div class="flex flex-col gap-y-2 my-2 py-2">
+													<Toggle
+														disabled={!$enterpriseLicense}
+														bind:checked={values[setting.key]}
+														options={{ right: setting.description }}
+													/>
+													<div class="flex flex-row">
+														<Button
+															disabled={!$enterpriseLicense}
+															size="sm"
+															on:click={() => {
+																isCriticalAlertsUIOpen.set(true)
+																closeDrawer?.()
+															}}
+														>
+															Show Critical Alerts
+														</Button>
+													</div>
+												</div>
 											{:else if setting.fieldType == 'critical_error_channels'}
 												<div class="w-full flex gap-x-16 flex-wrap">
 													<div class="w-full max-w-lg">
@@ -276,9 +276,13 @@
 														<td>
 															<div class="flex flex-row gap-x-1 justify-end">
 																<InstanceNameEditor
+																	login_type={login_type}
 																	value={name}
 																	{username}
 																	{email}
+																	on:refresh={() => {
+																		listUsers(activeOnly)
+																	}}
 																	on:save={(e) => {
 																		updateName(e.detail, email)
 																	}}
@@ -321,7 +325,7 @@
 							</div>
 						</TabContent>
 						<TabContent value="" values={settingsKeys}>
-							<InstanceSettings bind:this={instanceSettings} hideTabs hideSave {tab} />
+							<InstanceSettings bind:this={instanceSettings} hideTabs hideSave {tab} {closeDrawer}/>
 						</TabContent>
 					</svelte:fragment>
 				</Tabs>
@@ -221,7 +221,7 @@
 		JOBS_DISAPPROVAL: 'jobs.disapproval',
 		JOBS_DELETE: 'jobs.delete',
 		ACCOUNT_DELETE: 'account.delete',
-		OPENAI_REQUEST: 'openai.request',
+		AI_REQUEST: 'ai.request',
 		RESOURCES_CREATE: 'resources.create',
 		RESOURCES_UPDATE: 'resources.update',
 		RESOURCES_DELETE: 'resources.delete',
@@ -44,7 +44,7 @@
 	<div
 		on:click={() => (open = false)}
 		transition:fadeFast|local
-		class={'absolute top-0 bottom-0 left-0 right-0 z-50'}
+		class={'absolute top-0 bottom-0 left-0 right-0 z-[9999]'}
 		role="dialog"
 	>
 		<div
@@ -12,7 +12,7 @@
 	}
 </script>

-{#if $copilotInfo.exists_openai_resource_path && $copilotInfo.code_completion_enabled}
+{#if $copilotInfo.exists_ai_resource && $copilotInfo.code_completion_enabled}
 	<Toggle
 		size="xs"
 		bind:checked={$codeCompletionSessionEnabled}
@@ -1,11 +1,13 @@
 <script lang="ts">
 	import { ExternalLink, Wand2 } from 'lucide-svelte'
 	import Button from '../common/button/Button.svelte'
-	import { getNonStreamingCompletion } from './lib'
+	import { getNonStreamingCompletion, type AiProviderTypes } from './lib'
 	import Popup from '../common/popup/Popup.svelte'
 	import { sendUserToast } from '$lib/toast'
 	import { copilotInfo } from '$lib/stores'
+
 	import { base } from '$lib/base'
+	import type { ChatCompletionMessageParam } from 'openai/resources/index.mjs'

 	export let schedule: string

@@ -14,31 +16,35 @@
 	let genLoading = false
 	let abortController = new AbortController()
 	$: instructionsField && setTimeout(() => instructionsField?.focus(), 100)
+
 	const SYSTEM =
-		"You are a helpful assitant for creating CRON schedules. The structure is 'second minute hour dayOfMonth month dayOfWeek'. Weekdays are Sunday (1), Monday (2), Tuesday (3), Wednesday (4), Thursday (5), Friday (6), Saturday (7). You only return the CRON string without any wrapping characters. If it is invalid, you will return an error message preceeded by 'ERROR:'."
+		"You are a helpful assistant for creating CRON schedules using both standard and extended Croner patterns. The structure is 'second minute hour dayOfMonth month dayOfWeek'. Supported modifiers: ? (wildcard), L (last day/weekday), # (nth occurrence of a weekday), and W (closest weekday). Weekdays are Sunday (0 or 7), Monday (1), Tuesday (2), Wednesday (3), Thursday (4), Friday (5), Saturday (6). Ensure syntax is valid, including optional seconds and special modifiers. You only return either the CRON string without any leading/closing quotes or an error message prefixed with 'ERROR:'."
 	const USER = 'CRON schedule instructions: {instructions}'
 	async function generateCron() {
 		genLoading = true
 		abortController = new AbortController()
+		const aiProvider = $copilotInfo.ai_provider
 		try {
+			const messages: ChatCompletionMessageParam[] = [
+				{
+					role: 'system',
+					content: SYSTEM
+				},
+				{
+					role: 'user',
+					content: USER.replace('{instructions}', instructions)
+				}
+			]
+
 			const response = await getNonStreamingCompletion(
-				[
-					{
-						role: 'system',
-						content: SYSTEM
-					},
-					{
-						role: 'user',
-						content: USER.replace('{instructions}', instructions)
-					}
-				],
-				abortController
+				messages,
+				abortController,
+				aiProvider as AiProviderTypes
 			)

 			if (response.startsWith('ERROR:')) {
 				throw response.replace('ERROR:', '').trim()
 			}
-
 			schedule = response
 		} catch (err) {
 			if (!abortController.signal.aborted) {
@@ -69,7 +75,7 @@
 			on:click={genLoading ? () => abortController?.abort() : undefined}
 		/>
 	</svelte:fragment>
-	{#if $copilotInfo.exists_openai_resource_path}
+	{#if $copilotInfo.exists_ai_resource}
 		<div class="flex w-96">
 			<input
 				bind:this={instructionsField}
@@ -104,7 +110,7 @@
 		<div class="block text-primary">
 			<p class="text-sm"
 				>Enable Windmill AI in the <a
-					href="{base}/workspace_settings?tab=openai"
+					href="{base}/workspace_settings?tab=ai"
 					target="_blank"
 					class="inline-flex flex-row items-center gap-1"
 					>workspace settings <ExternalLink size={16} /></a
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
dieriba dieriba Ruben Fiszel Ruben Fiszel	3162bc8ab0	feat: Support mistral anthropic for ai (#4692 ) * wip: openai proxy and other ai proxy integration * fixing migration script * wip: support different ai provider in front, fix proxy openai * wip: adding frontend ai provider * updated copilot types * wip: working on anthropic integration * done AI proxy front * adding new type and support for anthropic * updating gitignore * adding streaming response * added streaming prompt * push lib/gen * wip: fixing anthropic * anthropic fully supported * fix backend missing var error and fully support stream event for anthropic * remove gen directory * fixing openapi file * add support for mistral, and update create workspace components * remove deref.json * remove package-json * openapi * fix ui enable code * added utility function for init workspace ai provider * fix workspace switch bug * update anthropic property and fixed frontend error * fix workspace settings * update error message and fix typo migration file * chore: update openapi file * fix dev file * add .sqlx * all * update sqlx --------- Co-authored-by: dieriba <t.dieriba@gmail.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>	2024-11-15 12:01:20 -05:00
Alexander Petric	2859dba79e	updating ai system prompt	2024-11-15 12:00:38 -05:00
Henri Courdent	20b78eac72	Result node placeholder (#4721 )	2024-11-15 11:58:58 -05:00
Alexander Petric	3ef86d13d9	if CLOUD_HOSTED and workspace_id set, autoacknowledge for superadmin (#4719 )	2024-11-15 11:58:58 -05:00
Lucas Abel	959bf6d48e	benchmarks: reduce 'big' task job count (#4718 )	2024-11-15 11:58:58 -05:00
Ruben Fiszel	dc868a608d	improve variable and resource not visible error message	2024-11-15 11:58:58 -05:00
Alexander Petric Ruben Fiszel Alexander Petric Guilhem Ruben Fiszel	04c631a11a	feat(monitoring): workspace critical alerts (#4684 ) * critical alert ui * updating ui, backend logic * revert * type check fix npm * checking out cli files from main * moving alert icon * adding sqlx mock data * more sqlx changes * feat(frontend): nodes from flow can be connected directly in expr input through a plug icon (#4652) * Add flow prop picker # Conflicts: # frontend/src/lib/components/propertyPicker/PropPicker.svelte * fix unwanted copy * cleaning * Fix unset context * move button and always display input * fix unwanted proppicker display * update * update * clean all * clean all --------- Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com> * replace hide/show with toggle * adding mutable setting and navigation to settings to configure channels * merge fix * ee non ee changees * auto-acknowledge when muted * pr comments * fix bad log * user inner modal component * update unaknowledge alerts after acknowledging from modal * aknowledge -> acknowledge * format * adding backend support for workspace critical alerts * immediately check for alerts * immediately check for alerts * adding openapi changes * simplify loading of superadmin/ee * update modal logic * frontend logic update * sqlx prepare * adding mute functionality for workspace critical alerts + show alerts button for instance and workspace settings * npm / rust warnings & errors * reverting non-mac cargo toml * reverting non-mac cargo toml * mute toggles in critical alert ui * ui polish * adding acknowledged_workspace column * sqlx prepare * make sure we wait for all stores to be loaded * adding workspace mute logic to report_critical alert * auto ack recovered alerts * toggle workspace as superadmin and critical alerts in logs menu * adding critical alert button if width < 786px * refresh on change * Move notification to logs (#4698) * Move notification to logs * removing unused button --------- Co-authored-by: Alexander Petric <petric.al@gmail.com> * tailwind typo * auto ack on mute * don't change deref * sqlx prep * avoid renaming db column, keep acknowledged instead of acknowledged_global * hide critical alert menu when not ee * if workspace muted, also acknowledge global when workspace set * removing save button for mute setting and improve ux/responsiveness * fix: deployment callbacks have a concurrency limit of 1 on same path * sqlx prep * ee-repo ref * z-[9999] for modal --------- Co-authored-by: Guilhem <guilhemlemouel@gmail.com> Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev>	2024-11-15 11:58:58 -05:00
Ruben Fiszel	69f767e1ff	improve error messages on windows	2024-11-15 11:58:58 -05:00
pyranota	20422b30b4	Fix dirs in uv install (#4717 )	2024-11-15 11:58:58 -05:00
Ruben Fiszel	a6cbaa8e71	nits logs	2024-11-15 11:58:58 -05:00
Ruben Fiszel	99c71f01f2	delete venv folder if pip install didn't succeed	2024-11-15 11:58:58 -05:00
pyranota Ruben Fiszel Alexander Petric	5403aab203	feat: Handle `pip install` by `uv` (#4517 ) * feat: Handle `pip install` by `uv` Dirty and untested, but already something working * Integrate with NSJAIL and prepare fallbacks * Refactor fallback no_uv disable compile and install where no_uv_install and no_uv_compile are a bit more specific * Remove `--disable-pip-version-check` Reason: warning: pip's `--disable-pip-version-check` has no effect * Fix backend compilation error * Pip fallback overwrite UV's cache * Initially refactor cache (No S3) * Support S3 * Remove unused import * Handle flags for NSJAIL * Return deleted flag * Update Dockerfile * Update docker-image.yml * Update docker-image.yml * Add --link-mode=copy and remove -v * Fix NSJAIL INDEX_URL * Fix flock and windows * Update python_executor.rs * Remove line from Dockerfile We dont need it and to trigger build * fixing for windows * Dont pin python to specific version * Change TMP for windows * Revert docker-image.yml * Disable UV for ansible Will be enabled later. Needs proper testing and its better to split onto 2 PRs with first modifying python and second ansible --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: Alexander Petric <petric.al@gmail.com>	2024-11-15 11:58:58 -05:00
Ruben Fiszelandrubenfiszel	7cfc87d5f3	chore(main): release 1.424.0 (#4709 ) * chore(main): release 1.424.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <rubenfiszel@users.noreply.github.com>	2024-11-15 11:29:52 -05:00
Ruben Fiszel	53367c6a36	fix: deployment callbacks have a concurrency limit of 1 on same path	2024-11-15 11:29:52 -05:00
Ruben Fiszel	be9a7a0d0a	fix: add countCompletedJobs api	2024-11-15 11:29:52 -05:00
Ruben Fiszel	72532a3820	update openapi	2024-11-15 11:29:52 -05:00
Ruben Fiszel	b3e6fcbb8c	fix: autoscaling when count < min worker set to min_workers	2024-11-15 11:29:52 -05:00
dieriba dieriba Ruben Fiszel Ruben Fiszel	f8c9237f8a	feat: Support mistral anthropic for ai (#4692 ) * wip: openai proxy and other ai proxy integration * fixing migration script * wip: support different ai provider in front, fix proxy openai * wip: adding frontend ai provider * updated copilot types * wip: working on anthropic integration * done AI proxy front * adding new type and support for anthropic * updating gitignore * adding streaming response * added streaming prompt * push lib/gen * wip: fixing anthropic * anthropic fully supported * fix backend missing var error and fully support stream event for anthropic * remove gen directory * fixing openapi file * add support for mistral, and update create workspace components * remove deref.json * remove package-json * openapi * fix ui enable code * added utility function for init workspace ai provider * fix workspace switch bug * update anthropic property and fixed frontend error * fix workspace settings * update error message and fix typo migration file * chore: update openapi file * fix dev file * add .sqlx * all * update sqlx --------- Co-authored-by: dieriba <t.dieriba@gmail.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>	2024-11-15 11:29:52 -05:00
Ruben Fiszel	979baab7d3	feat: allow setting password and login type from superadmin UI	2024-11-15 11:29:52 -05:00
Lucas Abel	34fd21a26c	feat(backend): monitor minimal version of living workers (#4704 )	2024-11-15 11:29:52 -05:00
Lucas Abel	483cc6649b	api: cleanup job data structure (#4705 )	2024-11-15 11:29:52 -05:00
Lucas Abel	e3aa4e5359	nit: cleanup `raw_flow` usage (#4707 ) * nit: cleanup `raw_flow` usage * nit: refactor two queries into one	2024-11-15 11:29:52 -05:00
Ruben Fiszel	1d12f4e4c0	fix: add queue_couts api	2024-11-15 11:29:52 -05:00
Lucas Abel	800838964b	nix: nit fixes + bench script (#4703 )	2024-11-15 11:29:52 -05:00
Lucas Abel	4f70c4d90f	nit: generalize usage of `has_failure_module` (#4706 )	2024-11-15 11:29:52 -05:00
Alexander Petricandellipsis-dev[bot]	a91145aba4	remove redundant def of Sunday Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>	2024-11-13 22:06:11 -05:00
Alexander Petric	49e239d19a	making sure ai doesn't return quotes around the cron expression	2024-11-13 22:03:44 -05:00
Alexander Petric	8ffe1f990e	updating ai system prompt	2024-11-13 21:41:34 -05:00
Alexander Petric	188be5a3f8	Merge branch 'main' into alp/quartz_schedules	2024-11-13 21:02:10 -05:00
Alexander Petric	75db8e2e8b	queue	2024-11-13 20:27:13 -05:00
Alexander Petric	feaa7e2d0f	cron -> croner	2024-11-13 20:08:49 -05:00