Merge remote-tracking branch 'origin/main' into datatable-perms-3

# Conflicts:
#	backend/ee-repo-ref.txt

.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "1.796.0"
+  ".": "1.800.0"
 }

AGENTS.md

@@ -165,6 +165,15 @@ $NAV --root backend callees "X"                           # what does X call?
 - Search for existing code to reuse before writing new code
 - Follow established patterns in the codebase
 - Keep changes focused — don't refactor beyond what's asked
+- **A simpler design found late is still the design.** Work already spent is not an argument
+  for a shape, and neither is a clean review round, a passing suite, or a long PR thread. The
+  signal to stop and re-derive rather than patch again is a change that keeps growing to defend
+  its own structure: each review finding fixing an assumption the previous fix broke, the same
+  class of bug reappearing somewhere new, or most of the diff being consequences of one early
+  choice rather than the thing you set out to do. When that happens, say plainly what the
+  simpler design is and what switching costs — a migration, a review cycle restarted from zero,
+  work discarded — and let the user decide. Do not keep paying down the harder one because it
+  is nearly finished, and do not present the accumulated cost as a reason to continue.
 - **Ship only the tests the PR needs.** A committed test must pin behavior a future change could plausibly break, and be the smallest setup that exercises the new logic. While developing, write as many exhaustive tests and do as much manual testing as you need to convince yourself the change works — then remove that scaffolding before marking the PR ready, keeping only the essential regression guard(s). A test that merely re-exercises pre-existing behavior, or needs elaborate fixtures to assert something trivial, is scaffolding: delete it. If nothing meaningful is left to guard, ship no test rather than a ceremonial one.
 - **Comments record constraints, not narration.** Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's "simplified" away. State each invariant once, at the place where someone would break it, in ≤4 lines. Don't describe what the next line does, don't repeat the same rationale at multiple sites, and don't address the PR reviewer (justifying a change belongs in the PR description, not the code). Reference nothing ephemeral — no numbered steps from your dev flow, no "the poller / the test does X" scaffolding, no transient state that won't exist for the next reader; keep only the essential, durable rationale. Describe the code as it is, never its drafting history: "we no longer do X", "unchanged behavior", "instead of the previous approach" are meaningless to a reader who never saw the earlier iteration — before finishing, reread your comments as if the current state is the only state that ever existed.
 - **Never attribute work to a specific customer, account, or "requested by a customer" in repo-tracked content** (PR descriptions, commit messages, code comments, docs). Describe changes by their technical motivation instead.

CHANGELOG.md

@@ -1,5 +1,94 @@
 # Changelog
 
+## [1.800.0](https://github.com/windmill-labs/windmill/compare/v1.799.0...v1.800.0) (2026-08-31)
+
+
+### Features
+
+* add --keep-deleted flag to wmill sync pull and push ([#10878](https://github.com/windmill-labs/windmill/issues/10878)) ([66123f3](https://github.com/windmill-labs/windmill/commit/66123f3a9b8978c0084b02f50b44cffba125a13a))
+* day-partition the service log index and expire whole chunks ([#10893](https://github.com/windmill-labs/windmill/issues/10893)) ([d91ee46](https://github.com/windmill-labs/windmill/commit/d91ee4614a70f20a194f47e190327129f499ec63))
+* free AI tokens + home search/filter revamp ([#10020](https://github.com/windmill-labs/windmill/issues/10020)) ([716ce2e](https://github.com/windmill-labs/windmill/commit/716ce2ece00cd5cfb8641afc6432636bc4aa46e9))
+* make the service log retention period an instance setting ([#10889](https://github.com/windmill-labs/windmill/issues/10889)) ([815de49](https://github.com/windmill-labs/windmill/commit/815de49e2322f85ca92b1e41a2bcd22591ebe93f))
+* rework the evals dataset drawer and run navigation ([#10884](https://github.com/windmill-labs/windmill/issues/10884)) ([1462f17](https://github.com/windmill-labs/windmill/commit/1462f17643302127b4bd76bacfde80cc03f9d606))
+* serve service log context from parquet and retire the raw log files ([#10892](https://github.com/windmill-labs/windmill/issues/10892)) ([338d75c](https://github.com/windmill-labs/windmill/commit/338d75cc5227e352cb84828c99bfd3b984cf0fa5))
+* serve service log retrieval from a columnar parquet store ([#10886](https://github.com/windmill-labs/windmill/issues/10886)) ([7c1a785](https://github.com/windmill-labs/windmill/commit/7c1a785f756ed27e4425f6534709b19971a73a97))
+
+
+### Bug Fixes
+
+* correct the service log ingest flush boundary ([#10898](https://github.com/windmill-labs/windmill/issues/10898)) ([ac56586](https://github.com/windmill-labs/windmill/commit/ac56586c0e56d4022761d3c80306a03d57f8bfcb))
+* harden the service log indexer's recovery and read paths ([#10904](https://github.com/windmill-labs/windmill/issues/10904)) ([831370c](https://github.com/windmill-labs/windmill/commit/831370cdde8e06f4298b17baa1a0041bacdd98c7))
+* keep raw-app editor selection consistent across sidebar and tabs ([#10885](https://github.com/windmill-labs/windmill/issues/10885)) ([b57e231](https://github.com/windmill-labs/windmill/commit/b57e231c2bf5e5fe007f0aa7b958a51e32b47141))
+* register every rotated service log file exactly once ([#10891](https://github.com/windmill-labs/windmill/issues/10891)) ([c817248](https://github.com/windmill-labs/windmill/commit/c8172480b0b1be6c57210212afc71d6ec8711235))
+* show a loading indicator while the initial data table migration is generated ([#10900](https://github.com/windmill-labs/windmill/issues/10900)) ([b998267](https://github.com/windmill-labs/windmill/commit/b998267c91b9dcf02787768f6205cc5aeda494fb))
+* track outstanding service log files on the rows themselves ([#10894](https://github.com/windmill-labs/windmill/issues/10894)) ([aa4a6ff](https://github.com/windmill-labs/windmill/commit/aa4a6ffd66813010a79c07741b01a984ed4e7df6))
+
+
+### Performance Improvements
+
+* add service log documents to the index one batch at a time ([#10906](https://github.com/windmill-labs/windmill/issues/10906)) ([0c2eb0a](https://github.com/windmill-labs/windmill/commit/0c2eb0ae3d18f49c21370131d15011e8dd103746))
+
+## [1.799.0](https://github.com/windmill-labs/windmill/compare/v1.798.1...v1.799.0) (2026-08-28)
+
+
+### Features
+
+* enable Anthropic prompt caching on Vertex AI agent steps ([#10876](https://github.com/windmill-labs/windmill/issues/10876)) ([320f400](https://github.com/windmill-labs/windmill/commit/320f4005124202852e6e9c70b394e7f87231d278))
+* instrument AI fill/fix, evals, agents and the debugger ([#10853](https://github.com/windmill-labs/windmill/issues/10853)) ([0bbd559](https://github.com/windmill-labs/windmill/commit/0bbd559ac8a35dba04ba5e8d6f2fd8d1d1124891))
+
+
+### Bug Fixes
+
+* **datatables:** stop a fork's pg_dump restore from failing silently ([#10830](https://github.com/windmill-labs/windmill/issues/10830)) ([3ce9bbc](https://github.com/windmill-labs/windmill/commit/3ce9bbc7168b837cb2111aabd533bb67803502b8))
+* key build artifact caches on a runnable's inline modules ([#10819](https://github.com/windmill-labs/windmill/issues/10819)) ([b72ccc3](https://github.com/windmill-labs/windmill/commit/b72ccc35934165b4bad112b947ca5af064aab26f))
+* nested template literals in step inputs, and unresolvable $args tags ([#10856](https://github.com/windmill-labs/windmill/issues/10856)) ([8f349c0](https://github.com/windmill-labs/windmill/commit/8f349c032a0d75fc3350292075e5050a030f6166))
+* pre-fill the test panel JSON args editor and align its placeholder ([#10871](https://github.com/windmill-labs/windmill/issues/10871)) ([fb82f36](https://github.com/windmill-labs/windmill/commit/fb82f36e6d6492dd0740984d8d78ea4eaa30361e))
+* reject a prefixed error_handler_path on triggers ([#10847](https://github.com/windmill-labs/windmill/issues/10847)) ([d334831](https://github.com/windmill-labs/windmill/commit/d33483173526a3b352d2829ac8a2e1e229cc1127))
+* unify billable seat counting and prevent fork subscriptions ([#10818](https://github.com/windmill-labs/windmill/issues/10818)) ([7dd88c4](https://github.com/windmill-labs/windmill/commit/7dd88c470caee5f095dc240667aa7550c55696bc))
+
+## [1.798.1](https://github.com/windmill-labs/windmill/compare/v1.798.0...v1.798.1) (2026-08-27)
+
+
+### Bug Fixes
+
+* allow job tokens to read the automate_username_creation setting ([#10869](https://github.com/windmill-labs/windmill/issues/10869)) ([c2279db](https://github.com/windmill-labs/windmill/commit/c2279db8a96ac76382eafe254627dafd24d173fd))
+
+## [1.798.0](https://github.com/windmill-labs/windmill/compare/v1.797.0...v1.798.0) (2026-08-27)
+
+
+### Features
+
+* a wizard for importing a hub project, and finishing what the import cannot ([#10729](https://github.com/windmill-labs/windmill/issues/10729)) ([2913339](https://github.com/windmill-labs/windmill/commit/29133398f99cd2dd5b33057ee9df4492d82e067a))
+
+## [1.797.0](https://github.com/windmill-labs/windmill/compare/v1.796.0...v1.797.0) (2026-08-26)
+
+
+### Features
+
+* configurable expiry for presigned s3 public url signatures ([#10835](https://github.com/windmill-labs/windmill/issues/10835)) ([8a6dc27](https://github.com/windmill-labs/windmill/commit/8a6dc27236aca67f0efe941d9606b787c2305ea8))
+* **frontend:** flag the fork-compare datatable schema diff as legacy ([#10829](https://github.com/windmill-labs/windmill/issues/10829)) ([07c77ea](https://github.com/windmill-labs/windmill/commit/07c77ead7425f1877372d358d867445a4c525c96))
+* keep a Hub project live while an update is under review ([#10814](https://github.com/windmill-labs/windmill/issues/10814)) ([c04b570](https://github.com/windmill-labs/windmill/commit/c04b5705745c36ecbb3a551ac59459218d2e3807))
+
+
+### Bug Fixes
+
+* **cli:** keep svelte component styles in the raw-app bundle ([#10838](https://github.com/windmill-labs/windmill/issues/10838)) ([b8bf539](https://github.com/windmill-labs/windmill/commit/b8bf539c3fe2b4db9c74dd73f04b3029287acdc6))
+* **debugger:** parse bun 1.4's UUID inspector token ([#10828](https://github.com/windmill-labs/windmill/issues/10828)) ([4658224](https://github.com/windmill-labs/windmill/commit/46582245926a7f8ea961bcd125a58fbfba3530cf))
+* force HTTP router rebuild on trigger-change notification ([#10849](https://github.com/windmill-labs/windmill/issues/10849)) ([ffdf17e](https://github.com/windmill-labs/windmill/commit/ffdf17ef8dc5575dd92d62d0d0ba887c1e378576))
+* **frontend:** follow the operating workspace in step input forms ([#10834](https://github.com/windmill-labs/windmill/issues/10834)) ([6b73145](https://github.com/windmill-labs/windmill/commit/6b73145e7220232601538b801ebc9dc73fe79bbb))
+* **frontend:** key the GitHub App installation selector on installation_id ([#10831](https://github.com/windmill-labs/windmill/issues/10831)) ([78331fd](https://github.com/windmill-labs/windmill/commit/78331fda8b290a2d9a5dd92b8362ff32c8b39432))
+* **frontend:** operator menu opens on hover, pins on click ([#10824](https://github.com/windmill-labs/windmill/issues/10824)) ([665f83e](https://github.com/windmill-labs/windmill/commit/665f83e1f438e34d006429889d51a5fb6a6b6176))
+* keep connection string query parameters under token auth ([#10859](https://github.com/windmill-labs/windmill/issues/10859)) ([f131c39](https://github.com/windmill-labs/windmill/commit/f131c3920f50f9fa18cd637eac39609495999aef))
+* migrate slack resource-connect oauth to v2 ([#10836](https://github.com/windmill-labs/windmill/issues/10836)) ([9fa8159](https://github.com/windmill-labs/windmill/commit/9fa8159ad16204cab52fd18a34a48ebf13f800f6))
+* recover from unresolvable AI session links instead of a dead end ([#10854](https://github.com/windmill-labs/windmill/issues/10854)) ([e38c449](https://github.com/windmill-labs/windmill/commit/e38c449007f27b952808cba5aa812441f2ce5946))
+* require admin on workspace tarball settings export ([#10817](https://github.com/windmill-labs/windmill/issues/10817)) ([46c363f](https://github.com/windmill-labs/windmill/commit/46c363ffa4bc72bef6b367ece4bdbeef5e0eadc9))
+* restrict filesystem workspace storage to debug builds ([#10864](https://github.com/windmill-labs/windmill/issues/10864)) ([8b80b09](https://github.com/windmill-labs/windmill/commit/8b80b09f33d311f0881678577ca6004c12d97c22))
+
+
+### Performance Improvements
+
+* index the suspended-job resume test instead of filtering it ([#10863](https://github.com/windmill-labs/windmill/issues/10863)) ([69320b2](https://github.com/windmill-labs/windmill/commit/69320b28f615b897a92f580bd5961c41e5c29951))
+
 ## [1.796.0](https://github.com/windmill-labs/windmill/compare/v1.795.0...v1.796.0) (2026-08-24)
 
 

backend/.sqlx/query-0e03dc960c0a22e042e54af719ac90c4b8506acefc85ef5b2f92a7bc451b1c5e.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "DELETE FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval RETURNING file_path, hostname",
+  "query": "DELETE FROM log_file WHERE (hostname, log_ts) IN (\n                 SELECT hostname, log_ts FROM log_file\n                 WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval\n                 LIMIT $2\n             ) RETURNING file_path, hostname",
   "describe": {
     "columns": [
       {
@@ -16,6 +16,7 @@
     ],
     "parameters": {
       "Left": [
+        "Int8",
         "Int8"
       ]
     },
@@ -24,5 +25,5 @@
       false
     ]
   },
-  "hash": "94da1e7feb4f58cc7ebe99752736f956d47810a94cb052fdcffb5cfe440f8033"
+  "hash": "0e03dc960c0a22e042e54af719ac90c4b8506acefc85ef5b2f92a7bc451b1c5e"
 }

backend/.sqlx/query-247486558e023ec3adf0c1e8f5664fc5e65995ce81e622c174a89befc1a527e5.json

@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT cost_nanos FROM ai_free_token_usage WHERE email = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "cost_nanos",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "247486558e023ec3adf0c1e8f5664fc5e65995ce81e622c174a89befc1a527e5"
+}

backend/.sqlx/query-44b9400fed2082c0df9d57f940923f37b8d452ea8e199ccc207da238368f1996.json

@@ -0,0 +1,23 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO ai_free_token_daily_usage (day, cost_nanos, updated_at)\n         VALUES ($2::date, $1::bigint, now())\n         ON CONFLICT (day) DO UPDATE\n         SET cost_nanos = ai_free_token_daily_usage.cost_nanos + $1::bigint,\n             updated_at = now()\n         RETURNING cost_nanos",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "cost_nanos",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Date"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "44b9400fed2082c0df9d57f940923f37b8d452ea8e199ccc207da238368f1996"
+}

backend/.sqlx/query-624a7dbc6cc951a199b0e70d86c463a0e7b5248c226ee92d95df94c3099cc400.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE log_file SET indexed_at = now()\n         FROM unnest($1::text[], $2::text[]) AS c(hostname, file_path)\n         WHERE log_file.indexed_at IS NULL\n           AND log_file.hostname = c.hostname\n           AND log_file.file_path = c.file_path",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "TextArray",
+        "TextArray"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "624a7dbc6cc951a199b0e70d86c463a0e7b5248c226ee92d95df94c3099cc400"
+}

backend/.sqlx/query-67a83afb708c90b2132cba81a0701dfc8b5e7aedf2f26fc1fe5be8f685cf709c.json

@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT max(log_ts) FROM log_file\n         WHERE hostname = $1 AND log_ts < (SELECT max(log_ts) FROM log_file WHERE hostname = $1)",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "max",
+        "type_info": "Timestamp"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "67a83afb708c90b2132cba81a0701dfc8b5e7aedf2f26fc1fe5be8f685cf709c"
+}

backend/.sqlx/query-6bbcb27a3bb70302076c559c8394b14b842f595f68dd885248abaaeabd2d0bf1.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT\n                        hostname,\n                        mode::text,\n                        worker_group,\n                        log_ts,\n                        file_path,\n                        ok_lines,\n                        err_lines,\n                        json_fmt\n                    FROM log_file\n                    WHERE log_ts > $1\n                    ORDER BY log_ts ASC LIMIT $2",
+  "query": "SELECT\n                        hostname,\n                        mode::text,\n                        worker_group,\n                        log_ts,\n                        file_path,\n                        ok_lines,\n                        err_lines,\n                        json_fmt\n                    FROM log_file\n                    WHERE indexed_at IS NULL\n                    ORDER BY log_ts ASC, hostname ASC LIMIT $1",
   "describe": {
     "columns": [
       {
@@ -46,7 +46,6 @@
     ],
     "parameters": {
       "Left": [
-        "Timestamp",
         "Int8"
       ]
     },
@@ -61,5 +60,5 @@
       true
     ]
   },
-  "hash": "b5c839baab25c4dcdd503d380cf7a886242277cd50555f20b2e22e13942d2a3a"
+  "hash": "6bbcb27a3bb70302076c559c8394b14b842f595f68dd885248abaaeabd2d0bf1"
 }

backend/.sqlx/query-872be44352d4a27a2005f6bdc38f927ac4886755fd17ec755037763aa92f6c0e.json

@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "WITH moved AS (\n             UPDATE log_file SET indexed_at = CASE\n                 WHEN log_ts > NOW() - make_interval(secs => $1) THEN NULL\n                 ELSE now() END\n             WHERE indexed_at = 'epoch' RETURNING 1)\n         SELECT count(*) AS \"n!\" FROM moved",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "n!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Float8"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "872be44352d4a27a2005f6bdc38f927ac4886755fd17ec755037763aa92f6c0e"
+}

backend/.sqlx/query-87e8e034b1cf5ea7ce43870d77d33fa0fd05b067454f0d4686d8b155fe562ffe.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE log_file SET indexed_at = now()\n         FROM unnest($1::text[], $2::timestamp[]) AS c(hostname, log_ts)\n         WHERE log_file.hostname = c.hostname AND log_file.log_ts = c.log_ts",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "TextArray",
+        "TimestampArray"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "87e8e034b1cf5ea7ce43870d77d33fa0fd05b067454f0d4686d8b155fe562ffe"
+}

backend/.sqlx/query-8d207cc9ed101ff116b617d25a94633c1531170ded1fdf09114718b941f5e1db.json

@@ -1,65 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT\n                        hostname,\n                        mode::text,\n                        worker_group,\n                        log_ts,\n                        file_path,\n                        ok_lines,\n                        err_lines,\n                        json_fmt\n                    FROM log_file\n                    WHERE log_ts > NOW() - make_interval(secs => $1)\n                    ORDER BY log_ts ASC LIMIT $2",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "hostname",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 1,
-        "name": "mode",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 2,
-        "name": "worker_group",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 3,
-        "name": "log_ts",
-        "type_info": "Timestamp"
-      },
-      {
-        "ordinal": 4,
-        "name": "file_path",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 5,
-        "name": "ok_lines",
-        "type_info": "Int8"
-      },
-      {
-        "ordinal": 6,
-        "name": "err_lines",
-        "type_info": "Int8"
-      },
-      {
-        "ordinal": 7,
-        "name": "json_fmt",
-        "type_info": "Bool"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Float8",
-        "Int8"
-      ]
-    },
-    "nullable": [
-      false,
-      null,
-      true,
-      false,
-      false,
-      true,
-      true,
-      true
-    ]
-  },
-  "hash": "8d207cc9ed101ff116b617d25a94633c1531170ded1fdf09114718b941f5e1db"
-}

backend/.sqlx/query-a7d450e34084d561f69e588bd76fd56e616ee79d895b7dcc37ad9442789e1574.json

@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "WITH retired AS (\n                     UPDATE log_file SET indexed_at = now()\n                     WHERE indexed_at IS NULL\n                       AND log_ts <= NOW() - make_interval(secs => $1) RETURNING 1)\n                 SELECT count(*) AS \"n!\" FROM retired",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "n!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Float8"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "a7d450e34084d561f69e588bd76fd56e616ee79d895b7dcc37ad9442789e1574"
+}

backend/.sqlx/query-acf4a5f4b67ebd06157819677fee6be4cc3adc6aa859c3dbaa57fa50b63741cf.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO ai_free_token_daily_usage (day, cost_nanos, updated_at)\n         VALUES ($2::date, GREATEST(0, $1::bigint), now())\n         ON CONFLICT (day) DO UPDATE\n         SET cost_nanos = GREATEST(0, ai_free_token_daily_usage.cost_nanos + $1::bigint),\n             updated_at = now()",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Date"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "acf4a5f4b67ebd06157819677fee6be4cc3adc6aa859c3dbaa57fa50b63741cf"
+}

backend/.sqlx/query-b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384.json

@@ -98,12 +98,12 @@
       null,
       null,
       null,
-      true,
+      false,
       null,
       null,
       null,
-      true,
-      true
+      false,
+      false
     ]
   },
   "hash": "b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384"

backend/.sqlx/query-c29d815cc38493f17950f12e9e5fe34b6c4ed1b2a49ed2b61dc9fb494026e956.json

@@ -0,0 +1,23 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO ai_free_token_usage (email, cost_nanos, updated_at)\n         VALUES ($1, $2::bigint, now())\n         ON CONFLICT (email) DO UPDATE\n         SET cost_nanos = ai_free_token_usage.cost_nanos + $2::bigint,\n             updated_at = now()\n         RETURNING cost_nanos",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "cost_nanos",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Int8"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "c29d815cc38493f17950f12e9e5fe34b6c4ed1b2a49ed2b61dc9fb494026e956"
+}

backend/.sqlx/query-daa5b57290cd1f821a53eebe96434f1befe6b16eee363c843faa1f836d53ca8d.json

@@ -0,0 +1,29 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT mode::text AS \"mode!\", log_ts FROM log_file WHERE hostname = $1 AND file_path = $2 ORDER BY log_ts DESC LIMIT 1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "mode!",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 1,
+        "name": "log_ts",
+        "type_info": "Timestamp"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      null,
+      false
+    ]
+  },
+  "hash": "daa5b57290cd1f821a53eebe96434f1befe6b16eee363c843faa1f836d53ca8d"
+}

backend/.sqlx/query-f08ef43b4523c74fcbfc7879c147e1bf2c9a0d11e9841caafca62d85c9fb4c8f.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO ai_free_token_usage (email, cost_nanos, updated_at)\n         VALUES ($1, GREATEST(0, $2::bigint), now())\n         ON CONFLICT (email) DO UPDATE\n         SET cost_nanos = GREATEST(0, ai_free_token_usage.cost_nanos + $2::bigint),\n             updated_at = now()",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Int8"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "f08ef43b4523c74fcbfc7879c147e1bf2c9a0d11e9841caafca62d85c9fb4c8f"
+}

backend/.sqlx/query-f277db0459ff311d8a396aa4e03876dac75ba6df5f8eaa9f185300483e3ee36f.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)\n             VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)\n             ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
+  "query": "INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)\n     VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)\n     ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
   "describe": {
     "columns": [],
     "parameters": {
@@ -17,5 +17,5 @@
     },
     "nullable": []
   },
-  "hash": "92faee8714a45a403b623e04d789f7f99067a05e9dfe270223164db8a1df2e4b"
+  "hash": "f277db0459ff311d8a396aa4e03876dac75ba6df5f8eaa9f185300483e3ee36f"
 }

backend/.sqlx/query-f47cafb6e9a6ec58ccafb06cf5e806e3fe749119214863b9111b58fff0bb9583.json

@@ -0,0 +1,22 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT plan FROM workspace_settings WHERE workspace_id = $1",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "plan",
+        "type_info": "Varchar"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text"
+      ]
+    },
+    "nullable": [
+      true
+    ]
+  },
+  "hash": "f47cafb6e9a6ec58ccafb06cf5e806e3fe749119214863b9111b58fff0bb9583"
+}

backend/AGENTS.md

@@ -131,9 +131,11 @@ minimal explicit set for dev.
 ## Workspace object storage in dev — use the local filesystem
 
 For a dev workspace you don't need MinIO/S3: use the built-in **`FilesystemStorage`** large-file
-storage (a root path on local disk). It is intentionally hidden from the settings-UI storage
-dropdown (dev-only), so set it via the API. Requires the backend built with `parquet` (+ `private`
-for the real S3 helpers, + `enterprise` if you want advanced permission rules enforced):
+storage (a root path on local disk). It is a **debug-build affordance only** — every site that
+builds a filesystem object store calls `ensure_filesystem_storage_allowed`, so release builds
+refuse it, and the settings UI never offers it — so set it via the API on a `cargo run`/`cargo
+test` binary. Requires the backend built with `parquet` (+ `private` for the real S3 helpers,
++ `enterprise` if you want advanced permission rules enforced):
 
 ```bash
 curl -X POST "$BASE/api/w/<ws>/workspaces/edit_large_file_storage_config" \

backend/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "windmill"
-version = "1.796.0"
+version = "1.800.0"
 authors.workspace = true
 edition.workspace = true
 
@@ -88,7 +88,7 @@ members = [
 exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
 
 [workspace.package]
-version = "1.796.0"
+version = "1.800.0"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]
 edition = "2021"
 
@@ -477,7 +477,10 @@ rust-embed = { version = "^6", features = ["interpolate-folder-path"] }
 mime_guess = "^2"
 hex = "^0"
 sql-builder = "^3"
-argon2 = "^0"
+# Minor-pinned rather than the `^0` used elsewhere in this file: argon2's 0.x
+# minors are API-breaking (0.6 moved `SaltString` into `phc`, put `rand_core`
+# behind a feature and changed `hash_password`), so a float breaks the build.
+argon2 = "0.6"
 quick_cache = "^0"
 rand = "=0.9.0"
 rand_core = { version = "^0", features = ["std"] }
@@ -697,7 +700,7 @@ tikv-jemalloc-ctl = { version = "^0.5" }
 triomphe = "^0"
 pin-project-lite = "^0"
 
-tantivy = { git="https://github.com/windmill-labs/tantivy", rev="6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" }
+tantivy = { git="https://github.com/windmill-labs/tantivy", rev="ea3b818c7b93db0c2b5db4f5e7ffef38f339060a" }
 
 backon = "1.3.0"
 

backend/ee-repo-ref.txt

@@ -1 +1 @@
-d63dfdf143d743a57295a63284b61001a0a491f3
+56f5e6a82056fa63323d6d4e1ec44832e64512fc

backend/migrations/20260622072905_ai_free_token_usage.down.sql

@@ -0,0 +1,2 @@
+DROP TABLE ai_free_token_daily_usage;
+DROP TABLE ai_free_token_usage;

backend/migrations/20260622072905_ai_free_token_usage.up.sql

@@ -0,0 +1,19 @@
+-- One-time grant of the Windmill-provided free AI tier, measured as cost in nano-dollars
+-- (1e-9 USD) rather than raw tokens — a prompt-cache hit costs a fraction of a fresh input
+-- token, so a token count wildly overstates the real bill. The grant never resets: once
+-- spent, the user must bring their own API key. Keyed by normalized email so the allowance
+-- is shared across a user's workspaces (and is resistant to +tag / gmail-dot aliasing).
+CREATE TABLE ai_free_token_usage (
+    email VARCHAR(255) PRIMARY KEY,
+    cost_nanos BIGINT NOT NULL DEFAULT 0,
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+-- Instance-wide daily cost ceiling (nano-dollars) for the free tier — a kill-switch
+-- independent of the per-user grant, bounding the blast radius of a bad day. One row per
+-- UTC day.
+CREATE TABLE ai_free_token_daily_usage (
+    day DATE PRIMARY KEY,
+    cost_nanos BIGINT NOT NULL DEFAULT 0,
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);

backend/migrations/20260826202939_queue_suspended_resume_at_index.down.sql

@@ -0,0 +1 @@
+DROP INDEX IF EXISTS queue_suspended_v2;

backend/migrations/20260826202939_queue_suspended_resume_at_index.up.sql

@@ -0,0 +1,20 @@
+-- Serves the suspended-job pull in windmill-common/src/worker.rs, whose resume test is the
+-- indexed CASE expression. Two things about the shape are load-bearing:
+--   * (priority DESC NULLS LAST, created_at) leads, so the scan yields that query's ORDER BY
+--     and stops at the first match rather than sorting.
+--   * the index is dropped before it is built rather than relying on IF NOT EXISTS. The
+--     OVERRIDDEN_MIGRATIONS rewrite in windmill-api/src/db.rs runs these CONCURRENTLY, and an
+--     interrupted concurrent build leaves the index present but invalid, which IF NOT EXISTS
+--     would then skip rebuilding. Retiring the index this replaces is left to the migration
+--     that follows, so this one can only ever be replayed while that index is still there to
+--     cover the rebuild.
+DROP INDEX IF EXISTS queue_suspended_v2;
+
+CREATE INDEX IF NOT EXISTS queue_suspended_v2
+    ON v2_job_queue (
+        priority DESC NULLS LAST,
+        created_at,
+        (CASE WHEN suspend <= 0 THEN '-infinity'::timestamptz ELSE suspend_until END),
+        tag
+    )
+    WHERE suspend_until IS NOT NULL;

backend/migrations/20260826214706_queue_suspended_drop_legacy_index.down.sql

@@ -0,0 +1,3 @@
+CREATE INDEX IF NOT EXISTS queue_suspended
+    ON v2_job_queue (priority DESC NULLS LAST, created_at, suspend_until, suspend, tag)
+    WHERE suspend_until IS NOT NULL;

backend/migrations/20260826214706_queue_suspended_drop_legacy_index.up.sql

@@ -0,0 +1,6 @@
+-- Retires the index queue_suspended_v2 replaces. Separate from the migration that builds it
+-- so that one is only ever replayed while this index still exists: sqlx records a migration
+-- only after all its statements run, so a process that dies before the record is written
+-- replays the build, and its leading DROP would otherwise be destroying the sole usable
+-- index rather than an interrupted build.
+DROP INDEX IF EXISTS queue_suspended;

backend/migrations/20260830085453_log_file_indexed_at.down.sql

@@ -0,0 +1,4 @@
+DROP INDEX IF EXISTS index_log_file_premigration;
+DROP INDEX IF EXISTS index_log_file_pending_path;
+DROP INDEX IF EXISTS index_log_file_pending;
+ALTER TABLE log_file DROP COLUMN IF EXISTS indexed_at;

backend/migrations/20260830085453_log_file_indexed_at.up.sql

@@ -0,0 +1,35 @@
+-- The service log ingest walked `log_file` with a cursor over `log_ts`, which is when a line
+-- was written rather than when its row appeared. Rows do not arrive in that order — an upload
+-- retried after a failure, a host that has just started, a batch the row limit cut mid-minute —
+-- and a row that becomes visible behind the cursor is never read: it stays in `log_file` and its
+-- lines stay out of search until retention drops them.
+--
+-- No ordering fixes this. A cursor over arrival order fails the same way, because `nextval` is
+-- allocated before its INSERT commits: a row can be assigned a lower value and commit after a
+-- higher one has already moved the cursor past it. Which rows are outstanding is a property of
+-- the rows, so it is recorded on them.
+ALTER TABLE log_file ADD COLUMN indexed_at TIMESTAMPTZ;
+
+-- Rows that already existed are marked, not queued: on a 14-day window most were ingested long
+-- ago and their raw files are gone. A sentinel rather than a timestamp, because the indexer has to
+-- tell them apart from rows registered since — those start NULL — and it puts the window's worth of
+-- them back on the queue on its first pass, keeping only what the columnar store can vouch for.
+--
+-- Not split here on the cursor the old ingest had reached. Below that cursor sits every row it
+-- skipped, which is the loss this migration exists to stop; recording those as done would carry the
+-- bug into its own fix.
+UPDATE log_file SET indexed_at = 'epoch' WHERE indexed_at IS NULL;
+
+-- The work queue, and the only index the ingest query needs: outstanding rows are a small
+-- fraction of the table, so this stays proportional to what is left to do rather than to the
+-- retention window.
+CREATE INDEX index_log_file_pending ON log_file (log_ts) WHERE indexed_at IS NULL;
+
+-- A rebuild takes rows out of the queue by the file it just read out of the store, which is
+-- the one lookup that arrives without a `log_ts`: the primary key is `(hostname, log_ts)`, so
+-- nothing else covers it and each batch would scan every outstanding row instead.
+CREATE INDEX index_log_file_pending_path ON log_file (hostname, file_path) WHERE indexed_at IS NULL;
+
+-- Reached once per pass while pre-migration rows survive, and never again after the first
+-- conversion clears them.
+CREATE INDEX index_log_file_premigration ON log_file (log_ts) WHERE indexed_at = 'epoch';

backend/parsers/windmill-parser-wasm/Cargo.lock

@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
 
 [[package]]
 name = "windmill-common"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "aho-corasick",
  "anyhow",
@@ -6274,7 +6274,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-macros"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -6286,7 +6286,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "convert_case",
  "serde",
@@ -6295,7 +6295,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-bash"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6307,7 +6307,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-csharp"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -6319,7 +6319,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-go"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "gosyn",
@@ -6331,7 +6331,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-graphql"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6343,7 +6343,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-java"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -6355,7 +6355,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-nu"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "nu-parser",
@@ -6366,7 +6366,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-php"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "itertools 0.14.0",
@@ -6377,7 +6377,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "itertools 0.14.0",
@@ -6389,7 +6389,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py-asset"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "rustpython-ast",
@@ -6400,7 +6400,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py-imports"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "async-recursion",
@@ -6422,7 +6422,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-r"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -6434,7 +6434,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ruby"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6448,7 +6448,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-rust"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "convert_case",
@@ -6465,7 +6465,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-sql"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6478,7 +6478,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-sql-asset"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "serde",
@@ -6490,7 +6490,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ts"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6508,7 +6508,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ts-asset"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "serde-wasm-bindgen",
@@ -6524,7 +6524,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-wac"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "rustpython-ast",
@@ -6540,7 +6540,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-wasm"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "getrandom 0.2.17",
@@ -6572,7 +6572,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-yaml"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6586,7 +6586,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-types"
-version = "1.796.0"
+version = "1.800.0"
 dependencies = [
  "anyhow",
  "bitflags",

backend/parsers/windmill-parser-wasm/Cargo.toml

@@ -12,7 +12,7 @@ resolver = "2"
 members = ["."]
 
 [workspace.package]
-version = "1.796.0"
+version = "1.800.0"
 edition = "2021"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]
 

backend/src/main.rs

@@ -14,7 +14,7 @@ use monitor::{
     reload_nuget_config_setting, reload_powershell_repo_pat_setting,
     reload_powershell_repo_url_setting, reload_ruby_repos_setting,
     reload_timeout_wait_result_setting, reload_workspace_registries_setting,
-    send_current_log_file_to_object_store, send_logs_to_object_store, WORKERS_NAMES,
+    flush_pending_log_files_to_object_store, send_logs_to_object_store, WORKERS_NAMES,
 };
 use rand::Rng;
 use sqlx::{Pool, Postgres};
@@ -61,8 +61,9 @@ use windmill_common::{
         SAML_METADATA_SETTING, SANDBOX_IMAGE_CACHE_MAX_MB_SETTING,
         SANDBOX_IMAGE_DEFAULT_REGISTRY_SETTING, SANDBOX_IMAGE_MAX_SIZE_MB_SETTING,
         SANDBOX_IMAGE_PULL_POLICY_SETTING, SANDBOX_REGISTRY_AUTH_SETTING, SCIM_TOKEN_SETTING,
-        SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING, TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
-        UV_EXCLUDE_NEWER_SETTING, UV_INDEX_STRATEGY_SETTING, UV_PYTHON_INSTALL_MIRROR_SETTING,
+        SERVICE_LOG_RETENTION_SECS_SETTING, SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING,
+        TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING, UV_EXCLUDE_NEWER_SETTING,
+        UV_INDEX_STRATEGY_SETTING, UV_PYTHON_INSTALL_MIRROR_SETTING,
         WORKSPACE_FAIRNESS_DURATION_SECS_SETTING, WORKSPACE_FAIRNESS_ENABLED_SETTING,
         WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING,
         WORKSPACE_MAX_QUEUED_JOBS_SETTING, WORKSPACE_REGISTRIES_SETTING,
@@ -143,7 +144,8 @@ use crate::monitor::{
     reload_pip_index_url_setting, reload_retention_period_setting,
     reload_sandbox_image_cache_max_setting, reload_sandbox_image_default_registry_setting,
     reload_sandbox_image_max_size_setting, reload_sandbox_image_pull_policy_setting,
-    reload_sandbox_registry_auth_setting, reload_scim_token_setting, reload_smtp_config,
+    reload_sandbox_registry_auth_setting, reload_scim_token_setting,
+    reload_service_log_retention_secs_setting, reload_smtp_config,
     reload_store_audit_logs_s3_setting, reload_uv_exclude_newer_setting,
     reload_uv_index_strategy_setting, reload_uv_python_install_mirror_setting,
     reload_worker_config, MonitorIteration,
@@ -380,6 +382,7 @@ async fn cache_hub_scripts(file_path: Option<String>) -> anyhow::Result<()> {
                         "",
                         &mut None,
                         &None,
+                        None,
                     )
                     .await
                     {
@@ -1261,10 +1264,12 @@ Windmill Community Edition {GIT_VERSION}
         #[cfg(all(feature = "tantivy", feature = "parquet"))]
         let log_indexer_f = {
             let log_indexer_rx = killpill_rx.resubscribe();
-            let log_index_writer2 = log_index_writer.clone();
+            // Moved, not cloned: sealing a chunk takes sole ownership of its
+            // tantivy writer, which a second live handle would silently prevent.
+            let moved_log_index_writer = log_index_writer;
             async {
                 if let Some(db) = conn.as_sql() {
-                    if let Some(log_index_writer) = log_index_writer2 {
+                    if let Some(log_index_writer) = moved_log_index_writer {
                         windmill_indexer::service_logs_oss::run_indexer(
                             db.clone(),
                             log_index_writer,
@@ -1661,7 +1666,7 @@ Windmill Community Edition {GIT_VERSION}
     } else {
         tracing::info!("Nothing to do, exiting.");
     }
-    send_current_log_file_to_object_store(&conn, &hostname, &mode).await;
+    flush_pending_log_files_to_object_store(&conn, &hostname, &mode).await;
 
     if let Some(db) = conn.as_sql() {
         tracing::info!("Exiting connection pool");
@@ -1952,6 +1957,9 @@ async fn process_notify_event(
                 }
                 TIMEOUT_WAIT_RESULT_SETTING => reload_timeout_wait_result_setting(conn).await,
                 RETENTION_PERIOD_SECS_SETTING => reload_retention_period_setting(conn).await,
+                SERVICE_LOG_RETENTION_SECS_SETTING => {
+                    reload_service_log_retention_secs_setting(conn).await
+                }
                 RETENTION_PERIOD_SECS_OVERRIDES_SETTING => {
                     if let Err(e) = load_retention_period_overrides(db).await {
                         tracing::error!("Error loading per-workspace retention overrides: {e:#}");

backend/src/monitor.rs

@@ -70,18 +70,17 @@ use windmill_common::{
         RETENTION_PERIOD_SECS_SETTING, SAML_METADATA_SETTING, SANDBOX_IMAGE_CACHE_MAX_MB_SETTING,
         SANDBOX_IMAGE_DEFAULT_REGISTRY_SETTING, SANDBOX_IMAGE_MAX_SIZE_MB_SETTING,
         SANDBOX_IMAGE_PULL_POLICY_SETTING, SANDBOX_REGISTRY_AUTH_SETTING, SCIM_TOKEN_SETTING,
-        SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
-        UV_EXCLUDE_NEWER_SETTING, UV_INDEX_STRATEGY_SETTING, UV_PYTHON_INSTALL_MIRROR_SETTING,
-        WORKSPACE_FAIRNESS_DURATION_SECS_SETTING, WORKSPACE_FAIRNESS_ENABLED_SETTING,
-        WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING,
-        WORKSPACE_MAX_QUEUED_JOBS_SETTING,
+        SERVICE_LOG_RETENTION_SECS_SETTING, SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING,
+        TIMEOUT_WAIT_RESULT_SETTING, UV_EXCLUDE_NEWER_SETTING, UV_INDEX_STRATEGY_SETTING,
+        UV_PYTHON_INSTALL_MIRROR_SETTING, WORKSPACE_FAIRNESS_DURATION_SECS_SETTING,
+        WORKSPACE_FAIRNESS_ENABLED_SETTING, WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING,
+        WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING, WORKSPACE_MAX_QUEUED_JOBS_SETTING,
     },
     indexer::load_indexer_config,
     jobs::delete_jobs,
     jwt::JWT_SECRET,
     oauth2::REQUIRE_PREEXISTING_USER_FOR_OAUTH,
     server::load_smtp_config,
-    tracing_init::JSON_FMT,
     users::truncate_token,
     utils::{empty_as_none, now_from_db, report_critical_error, Mode, HUB_API_SECRET},
     worker::{
@@ -98,10 +97,10 @@ use windmill_common::{
     KillpillSender, AUDIT_LOG_RETENTION_DAYS, BASE_URL, CRITICAL_ALERTS_ON_DB_OVERSIZE,
     CRITICAL_ALERTS_ON_TOKEN_EXPIRY, CRITICAL_ALERT_MUTE_UI_ENABLED,
     CRITICAL_ALERT_MUTE_ZOMBIE_JOB_RESTART, CRITICAL_ERROR_CHANNELS, DB, DEFAULT_HUB_BASE_URL,
-    HUB_BASE_URL, JOB_RETENTION_SECS, JOB_RETENTION_SECS_OVERRIDES,
-    JOB_RETENTION_SECS_OVERRIDES_LOADED, METRICS_DEBUG_ENABLED, METRICS_ENABLED,
-    MONITOR_LOGS_ON_OBJECT_STORE, OTEL_LOGS_ENABLED, OTEL_METRICS_ENABLED, OTEL_TRACING_ENABLED,
-    SERVICE_LOG_RETENTION_SECS, STORE_AUDIT_LOGS_S3,
+    DEFAULT_SERVICE_LOG_RETENTION_SECS, HUB_BASE_URL, JOB_RETENTION_SECS,
+    JOB_RETENTION_SECS_OVERRIDES, JOB_RETENTION_SECS_OVERRIDES_LOADED, METRICS_DEBUG_ENABLED,
+    METRICS_ENABLED, MONITOR_LOGS_ON_OBJECT_STORE, OTEL_LOGS_ENABLED, OTEL_METRICS_ENABLED,
+    OTEL_TRACING_ENABLED, STORE_AUDIT_LOGS_S3,
 };
 use windmill_common::{
     client::AuthedClient,
@@ -476,6 +475,19 @@ pub async fn initial_load(
         |v: Option<String>| async move { HUB_API_SECRET.store(std::sync::Arc::new(v)) },
     );
 
+    // Outside the `server_mode` guard below: every mode reads this. A worker registers its
+    // rotated log files against the cutoff, and a dedicated indexer trims the search index to a
+    // window derived from it — neither is a server.
+    pass.setting(SERVICE_LOG_RETENTION_SECS_SETTING, true, |v| async move {
+        windmill_common::set_service_log_retention_secs(parse_setting_value::<i64>(
+            v,
+            SERVICE_LOG_RETENTION_SECS_SETTING,
+            "SERVICE_LOG_RETENTION_SECS",
+            DEFAULT_SERVICE_LOG_RETENTION_SECS,
+            |x| x,
+        ))
+    });
+
     if server_mode {
         pass.setting(RETENTION_PERIOD_SECS_SETTING, true, |v| async move {
             JOB_RETENTION_SECS.store(
@@ -1221,32 +1233,60 @@ async fn sleep_until_next_minute_start_plus_one_s() {
 }
 
 use windmill_common::tracing_init::TMP_WINDMILL_LOGS_SERVICE;
-async fn find_two_highest_files(hostname: &str) -> (Option<String>, Option<String>) {
+
+/// The minutely rolling appender names each file `<hostname>.log.<%Y-%m-%d-%H-%M>`;
+/// anything else in the directory is not a rotated log file.
+fn parse_log_file_ts(file_name: &str) -> Option<NaiveDateTime> {
+    NaiveDateTime::parse_from_str(
+        file_name.rsplit('.').next()?,
+        windmill_common::tracing_init::LOG_TIMESTAMP_FMT,
+    )
+    .ok()
+}
+
+/// Oldest first. Readdir order is filesystem-dependent — tmpfs hands back the
+/// newest entry first, ext4 hashes the names — so the listing has to be sorted
+/// before anything picks a file out of it.
+fn sorted_log_files(file_names: impl Iterator<Item = String>) -> Vec<(NaiveDateTime, String)> {
+    let mut files = file_names
+        .filter_map(|name| parse_log_file_ts(&name).map(|ts| (ts, name)))
+        .collect::<Vec<_>>();
+    files.sort();
+    files
+}
+
+/// Every log file but the newest one: that one is still being appended to, every
+/// older one is final.
+fn rotated_log_files(file_names: impl Iterator<Item = String>) -> Vec<(NaiveDateTime, String)> {
+    let mut files = sorted_log_files(file_names);
+    files.pop();
+    files
+}
+
+async fn read_log_file_names(hostname: &str) -> Vec<String> {
     let log_dir = format!("{}/{}/", *TMP_WINDMILL_LOGS_SERVICE, hostname);
-    let rd_dir = tokio::fs::read_dir(log_dir).await;
-    if let Ok(mut log_files) = rd_dir {
-        let mut highest_file: Option<String> = None;
-        let mut second_highest_file: Option<String> = None;
-        while let Ok(Some(file)) = log_files.next_entry().await {
-            let file_name = file
-                .file_name()
-                .to_str()
-                .map(|x| x.to_string())
-                .unwrap_or_default();
-            if file_name > highest_file.clone().unwrap_or_default() {
-                second_highest_file = highest_file;
-                highest_file = Some(file_name);
-            }
+    let mut rd_dir = match tokio::fs::read_dir(&log_dir).await {
+        Ok(rd_dir) => rd_dir,
+        Err(e) => {
+            tracing::error!("Error reading log files: {}, {:#?}", log_dir, e);
+            return vec![];
+        }
+    };
+    let mut file_names = vec![];
+    while let Ok(Some(file)) = rd_dir.next_entry().await {
+        if let Some(file_name) = file.file_name().to_str() {
+            file_names.push(file_name.to_string());
         }
-        (highest_file, second_highest_file)
-    } else {
-        tracing::error!(
-            "Error reading log files: {}, {:#?}",
-            *TMP_WINDMILL_LOGS_SERVICE,
-            rd_dir.unwrap_err()
-        );
-        (None, None)
     }
+    file_names
+}
+
+async fn list_log_files(hostname: &str) -> Vec<(NaiveDateTime, String)> {
+    sorted_log_files(read_log_file_names(hostname).await.into_iter())
+}
+
+async fn list_rotated_log_files(hostname: &str) -> Vec<(NaiveDateTime, String)> {
+    rotated_log_files(read_log_file_names(hostname).await.into_iter())
 }
 
 fn get_worker_group(mode: &Mode) -> Option<String> {
@@ -1266,133 +1306,188 @@ pub fn send_logs_to_object_store(conn: &Connection, hostname: &str, mode: &Mode)
     tokio::spawn(async move {
         let mut interval = tokio::time::interval(Duration::from_secs(10));
         interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
+        init_last_log_file_sent(&conn, &hostname).await;
         sleep_until_next_minute_start_plus_one_s().await;
         loop {
             interval.tick().await;
-            let (_, snd_highest_file) = find_two_highest_files(&hostname).await;
-            send_log_file_to_object_store(
-                &hostname,
-                &mode,
-                &worker_group,
-                &conn,
-                snd_highest_file,
-                false,
-            )
-            .await;
+            let files = list_rotated_log_files(&hostname).await;
+            send_log_files_to_object_store(&hostname, &mode, &worker_group, &conn, files).await;
         }
     });
 }
 
-pub async fn send_current_log_file_to_object_store(conn: &Connection, hostname: &str, mode: &Mode) {
-    tracing::info!("Sending current log file to object store");
-    let (highest_file, _) = find_two_highest_files(hostname).await;
+pub async fn flush_pending_log_files_to_object_store(
+    conn: &Connection,
+    hostname: &str,
+    mode: &Mode,
+) {
+    tracing::info!("Sending pending log files to object store");
     let worker_group = get_worker_group(&mode);
-    send_log_file_to_object_store(hostname, mode, &worker_group, conn, highest_file, true).await;
-}
-
-fn get_now_and_str() -> (NaiveDateTime, String) {
-    let ts = Utc::now().naive_utc();
-    (
-        ts,
-        ts.format(windmill_common::tracing_init::LOG_TIMESTAMP_FMT)
-            .to_string(),
-    )
+    // Nothing rotates after this, so the file still being appended to is registered
+    // here, along with any rotated one the loop had not reached yet. Bounded like the
+    // pool close that follows: a backlog against a slow object store would otherwise
+    // hold the process past its termination grace period. Whatever is left over is
+    // registered by the next run's catch-up.
+    let flush = async {
+        let files = list_log_files(hostname).await;
+        send_log_files_to_object_store(hostname, mode, &worker_group, conn, files).await;
+    };
+    if timeout(Duration::from_secs(15), flush).await.is_err() {
+        tracing::warn!("Could not send all pending log files in time (15s). Exiting anyway.");
+    }
 }
 
 lazy_static::lazy_static! {
     static ref LAST_LOG_FILE_SENT: Arc<Mutex<Option<NaiveDateTime>>> = Arc::new(Mutex::new(None));
+    /// Serializes the periodic uploader against the shutdown flush. The uploader is a
+    /// detached task that keeps ticking while the flush runs and both walk the same
+    /// files, so without this both can clear the watermark for one file and count its
+    /// lines twice through the additive upsert.
+    static ref SENDING_LOG_FILES: tokio::sync::Mutex<()> = tokio::sync::Mutex::new(());
 }
 
+fn last_log_file_sent() -> Option<NaiveDateTime> {
+    LAST_LOG_FILE_SENT.lock().ok().and_then(|ts| *ts)
+}
+
+/// Resume from what this host already registered, so a previous run's leftovers reach
+/// the object store rather than being dropped. Their line counts come out zero, this
+/// run having counted none of them, which only flattens their bars in the UI.
+///
+/// The newest registered minute is left out on purpose: the shutdown flush registers
+/// the file that was still open and the appender reopens that minute in append mode,
+/// so a restart inside it would otherwise strand everything written afterwards.
+///
+/// A row rewritten this way restores the object and sums the counters, but it keeps the
+/// `indexed_at` it already had, so one the indexers have taken is not offered again and
+/// the lines added by the rewrite stay out of search.
+async fn init_last_log_file_sent(conn: &Connection, hostname: &str) {
+    let Some(db) = conn.as_sql() else {
+        return;
+    };
+    match sqlx::query_scalar!(
+        "SELECT max(log_ts) FROM log_file
+         WHERE hostname = $1 AND log_ts < (SELECT max(log_ts) FROM log_file WHERE hostname = $1)",
+        hostname
+    )
+    .fetch_one(db)
+    .await
+    {
+        Ok(Some(ts)) => {
+            if let Err(e) = LAST_LOG_FILE_SENT.lock().map(|mut last_log_file_sent| {
+                last_log_file_sent.replace(ts);
+            }) {
+                tracing::error!("Error initializing last log file sent: {:?}", e);
+            }
+        }
+        Ok(None) => {}
+        Err(e) => tracing::error!("Error loading last log file sent: {:?}", e),
+    }
+}
+
+async fn send_log_files_to_object_store(
+    hostname: &str,
+    mode: &Mode,
+    worker_group: &Option<String>,
+    conn: &Connection,
+    files: Vec<(NaiveDateTime, String)>,
+) {
+    let _guard = SENDING_LOG_FILES.lock().await;
+    let retention_cutoff = Utc::now().naive_utc()
+        - chrono::Duration::seconds(windmill_common::service_log_retention_secs());
+    for (ts, file_name) in files {
+        if last_log_file_sent().is_some_and(|last| last >= ts) {
+            continue;
+        }
+        // A run coming back from a long outage still finds its predecessor's files on
+        // disk. Registering one past the retention cutoff inserts a row
+        // `delete_expired_items` drops on its next pass, once the indexers have already
+        // paid to parse it.
+        if ts < retention_cutoff {
+            continue;
+        }
+        // Stop at the first failure rather than moving on, so a file is never
+        // registered before an older one that has not made it to the store yet.
+        // The indexers do not depend on that ordering — every row is offered until
+        // it is marked — but a gap here would still be visible while it lasts.
+        if !send_log_file_to_object_store(hostname, mode, worker_group, conn, &file_name, ts).await
+        {
+            break;
+        }
+    }
+}
+
+/// Returns whether the file ended up registered in `log_file`.
 async fn send_log_file_to_object_store(
     hostname: &str,
     mode: &Mode,
     worker_group: &Option<String>,
     conn: &Connection,
-    snd_highest_file: Option<String>,
-    use_now: bool,
-) {
-    if let Some(highest_file) = snd_highest_file {
-        //parse datetime frome file xxxx.yyyy-MM-dd-HH-mm
-        let (ts, ts_str) = if use_now {
-            get_now_and_str()
-        } else {
-            highest_file
-                .split(".")
-                .last()
-                .and_then(|x| {
-                    NaiveDateTime::parse_from_str(
-                        x,
-                        windmill_common::tracing_init::LOG_TIMESTAMP_FMT,
-                    )
-                    .ok()
-                    .map(|y| (y, x.to_string()))
-                })
-                .unwrap_or_else(get_now_and_str)
-        };
+    file_name: &str,
+    ts: NaiveDateTime,
+) -> bool {
+    #[cfg(feature = "parquet")]
+    if let Some(s3_client) = windmill_object_store::get_object_store().await {
+        let path = std::path::Path::new(&*TMP_WINDMILL_LOGS_SERVICE)
+            .join(hostname)
+            .join(file_name);
 
-        let exists = LAST_LOG_FILE_SENT.lock().map(|last_log_file_sent| {
-            last_log_file_sent
-                .map(|last_log_file_sent| last_log_file_sent >= ts)
-                .unwrap_or(false)
-        });
-
-        if exists.unwrap_or(false) {
-            return;
-        }
-
-        #[cfg(feature = "parquet")]
-        let s3_client = windmill_object_store::get_object_store().await;
-        #[cfg(feature = "parquet")]
-        if let Some(s3_client) = s3_client {
-            let path = std::path::Path::new(&*TMP_WINDMILL_LOGS_SERVICE)
-                .join(hostname)
-                .join(&highest_file);
-
-            //read file as byte stream
-            let bytes = tokio::fs::read(&path).await;
-            if let Err(e) = bytes {
+        //read file as byte stream
+        let bytes = match tokio::fs::read(&path).await {
+            Ok(bytes) => bytes,
+            Err(e) => {
                 tracing::error!("Error reading log file: {:?}", e);
-                return;
+                return false;
             }
-            let path = windmill_object_store::object_store_reexports::Path::from_url_path(format!(
-                "{}{hostname}/{highest_file}",
-                windmill_common::tracing_init::LOGS_SERVICE
-            ));
-            if let Err(e) = path {
+        };
+        let path = windmill_object_store::object_store_reexports::Path::from_url_path(format!(
+            "{}{hostname}/{file_name}",
+            windmill_common::tracing_init::LOGS_SERVICE
+        ));
+        let path = match path {
+            Ok(path) => path,
+            Err(e) => {
                 tracing::error!("Error creating log file path: {:?}", e);
-                return;
-            }
-            if let Err(e) = s3_client.put(&path.unwrap(), bytes.unwrap().into()).await {
-                tracing::error!("Error sending logs to object store: {:?}", e);
+                return false;
             }
+        };
+        if let Err(e) = s3_client.put(&path, bytes.into()).await {
+            tracing::error!("Error sending logs to object store: {:?}", e);
+            return false;
         }
+    }
 
-        let (ok_lines, err_lines) = read_log_counters(ts_str);
+    let ts_str = ts
+        .format(windmill_common::tracing_init::LOG_TIMESTAMP_FMT)
+        .to_string();
+    let (ok_lines, err_lines) = read_log_counters(ts_str);
 
-        if let Some(db) = conn.as_sql() {
-            match timeout(Duration::from_secs(10), sqlx::query!("INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)
-             VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)
-             ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
-                hostname, mode.to_string(), worker_group.clone(), ts, highest_file, ok_lines as i64, err_lines as i64, *JSON_FMT)
-                .execute(db)).await {
-                Ok(Ok(_)) => {
-                    if let Err(e) = LAST_LOG_FILE_SENT.lock().map(|mut last_log_file_sent| {
-                        last_log_file_sent.replace(ts);
-                    }) {
-                        tracing::error!("Error updating last log file sent: {:?}", e);
-                    }
-                    tracing::info!("Log file sent: {}", highest_file);
-                }
-                Ok(Err(e)) => {
-                    tracing::error!("Error inserting log file: {:?}", e);
-                }
-                Err(e) => {
-                    tracing::error!("Error inserting log file, timeout elapsed: {:?}", e);
-                }
+    let Some(db) = conn.as_sql() else {
+        // not sending log file to object store in agent mode
+        return false;
+    };
+
+    match timeout(Duration::from_secs(10), sqlx::query!("INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)
+     VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)
+     ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
+        hostname, mode.to_string(), worker_group.clone(), ts, file_name, ok_lines as i64, err_lines as i64, true)
+        .execute(db)).await {
+        Ok(Ok(_)) => {
+            if let Err(e) = LAST_LOG_FILE_SENT.lock().map(|mut last_log_file_sent| {
+                last_log_file_sent.replace(ts);
+            }) {
+                tracing::error!("Error updating last log file sent: {:?}", e);
             }
-        } else {
-            // tracing::warn!("Not sending log file to object store in agent mode");
-            ()
+            tracing::info!("Log file sent: {}", file_name);
+            true
+        }
+        Ok(Err(e)) => {
+            tracing::error!("Error inserting log file: {:?}", e);
+            false
+        }
+        Err(e) => {
+            tracing::error!("Error inserting log file, timeout elapsed: {:?}", e);
+            false
         }
     }
 }
@@ -1580,6 +1675,13 @@ pub async fn trim_resource_versions(db: &DB) -> () {
     }
 }
 
+/// Matches the batch the settings-page cleanup uses for the same table.
+const SERVICE_LOG_DELETE_BATCH: i64 = 2_000;
+/// Batches per pass. `monitor_db` runs under a 600s timeout that cancels every maintenance
+/// future in the same `join!` and reports a critical error, so a large backlog has to drain
+/// across ticks rather than inside one, the way the neighbouring sweeps already do.
+const SERVICE_LOG_DELETE_MAX_BATCHES: usize = 10;
+
 pub async fn delete_expired_items(db: &DB) -> () {
     let expired_tokens_r = sqlx::query_as!(
         TokenRow,
@@ -1662,23 +1764,48 @@ pub async fn delete_expired_items(db: &DB) -> () {
         Err(e) => tracing::error!("Error deleting cache resource {}", e.to_string()),
     }
 
-    match sqlx::query_as!(
-        LogFile,
-        "DELETE FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval RETURNING file_path, hostname",
-        SERVICE_LOG_RETENTION_SECS,
-    )
-    .fetch_all(db)
-    .await
-    {
-        Ok(log_files_to_delete) => {
+    // Batched: every process rotates a log file a minute, so lowering the retention makes one
+    // ordinary setting change expire millions of rows at once. An unbounded `DELETE ...
+    // RETURNING` would materialize all of them, and their deletion futures, in this one tick.
+    for _ in 0..SERVICE_LOG_DELETE_MAX_BATCHES {
+        let batch = sqlx::query_as!(
+            LogFile,
+            "DELETE FROM log_file WHERE (hostname, log_ts) IN (
+                 SELECT hostname, log_ts FROM log_file
+                 WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval
+                 LIMIT $2
+             ) RETURNING file_path, hostname",
+            windmill_common::service_log_retention_secs(),
+            SERVICE_LOG_DELETE_BATCH,
+        )
+        .fetch_all(db)
+        .await;
+
+        match batch {
+            Ok(log_files_to_delete) => {
+                if log_files_to_delete.is_empty() {
+                    break;
+                }
+                let n = log_files_to_delete.len();
                 let paths = log_files_to_delete
                     .iter()
                     .map(|f| format!("{}/{}", f.hostname, f.file_path))
                     .collect();
-                delete_log_files_from_disk_and_store(paths, &*TMP_WINDMILL_LOGS_SERVICE, windmill_common::tracing_init::LOGS_SERVICE).await;
-
+                delete_log_files_from_disk_and_store(
+                    paths,
+                    &*TMP_WINDMILL_LOGS_SERVICE,
+                    windmill_common::tracing_init::LOGS_SERVICE,
+                )
+                .await;
+                if (n as i64) < SERVICE_LOG_DELETE_BATCH {
+                    break;
+                }
+            }
+            Err(e) => {
+                tracing::error!("Error deleting log file: {:?}", e);
+                break;
+            }
         }
-        Err(e) => tracing::error!("Error deleting log file: {:?}", e),
     }
 
     let audit_retention_days = audit_log_retention_days().await;
@@ -2785,6 +2912,21 @@ pub async fn reload_retention_period_setting(conn: &Connection) {
     }
 }
 
+pub async fn reload_service_log_retention_secs_setting(conn: &Connection) {
+    match load_setting_value::<i64>(
+        conn,
+        SERVICE_LOG_RETENTION_SECS_SETTING,
+        "SERVICE_LOG_RETENTION_SECS",
+        DEFAULT_SERVICE_LOG_RETENTION_SECS,
+        |x| x,
+    )
+    .await
+    {
+        Ok(v) => windmill_common::set_service_log_retention_secs(v),
+        Err(e) => tracing::error!("Error reloading service log retention period: {:?}", e),
+    }
+}
+
 pub async fn reload_audit_log_retention_days_setting(conn: &Connection) {
     match load_setting_value::<i64>(
         conn,
@@ -6832,3 +6974,54 @@ mod zombie_worker_memory_pct_tests {
         );
     }
 }
+
+#[cfg(test)]
+mod log_file_listing_tests {
+    use super::{rotated_log_files, sorted_log_files};
+
+    fn names(files: Vec<(chrono::NaiveDateTime, String)>) -> Vec<String> {
+        files.into_iter().map(|(_, n)| n).collect()
+    }
+
+    /// A directory read newest-entry-first is what tmpfs actually hands back.
+    #[test]
+    fn orders_by_minute_whatever_order_readdir_used() {
+        let newest_first = [
+            "h.log.2026-08-29-06-49",
+            "h.log.2026-08-29-06-46",
+            "h.log.2026-08-29-06-48",
+            "h.log.2026-08-29-06-47",
+        ];
+        assert_eq!(
+            names(sorted_log_files(newest_first.iter().map(|x| x.to_string()))),
+            vec![
+                "h.log.2026-08-29-06-46",
+                "h.log.2026-08-29-06-47",
+                "h.log.2026-08-29-06-48",
+                "h.log.2026-08-29-06-49",
+            ]
+        );
+        assert_eq!(
+            names(rotated_log_files(
+                newest_first.iter().map(|x| x.to_string())
+            )),
+            vec![
+                "h.log.2026-08-29-06-46",
+                "h.log.2026-08-29-06-47",
+                "h.log.2026-08-29-06-48",
+            ]
+        );
+    }
+
+    #[test]
+    fn drops_names_that_are_not_rotated_log_files() {
+        let files = sorted_log_files(
+            ["h.log", "not-a-log-file", "h.log.2026-08-29-06-46"]
+                .iter()
+                .map(|x| x.to_string()),
+        );
+        assert_eq!(files.len(), 1);
+        assert_eq!(files[0].1, "h.log.2026-08-29-06-46");
+        assert_eq!(files[0].0.to_string(), "2026-08-29 06:46:00");
+    }
+}

backend/summarized_schema.txt

@@ -38,6 +38,8 @@ account: workspace_id(char), id(int), expires_at(ts), refresh_token(char), clien
   FK: (workspace_id) -> workspace(id)
 agent_token_blacklist: token(char), expires_at(ts), blacklisted_at(ts), blacklisted_by(char)
 ai_agent_memory: workspace_id(char), conversation_id(uuid), step_id(char), messages(jsonb), created_at(ts), updated_at(ts)
+ai_free_token_daily_usage: day(date), cost_nanos(bigint), updated_at(ts)
+ai_free_token_usage: email(char), cost_nanos(bigint), updated_at(ts)
 ai_token_usage: workspace_id(char), day(date), email(char), provider(char), model(char), session_id(char), input_tokens(bigint), cache_read_tokens(bigint), cache_write_tokens(bigint), output_tokens(bigint), reported_cost_nano_usd(bigint), requests(bigint), updated_at(ts)
   FK: (workspace_id) -> workspace(id)
 alerts: id(int), alert_type(char), message(text), created_at(ts), acknowledged(bool), workspace_id(text), acknowledged_workspace(bool), resource(text)
@@ -128,7 +130,7 @@ job_stats: workspace_id(char), job_id(uuid), metric_id(char), metric_name(char),
 kafka_pending_commits: id(bigint), workspace_id(char), kafka_trigger_path(char), topic(char), partition(int), offset(bigint), created_at(ts)
   FK: (workspace_id, kafka_trigger_path) -> kafka_trigger(workspace_id, path)
 kafka_trigger: path(char), kafka_resource_path(char), topics(char), group_id(char), script_path(char), is_flow(bool), workspace_id(char), edited_by(char), email(char), edited_at(ts), extra_perms(jsonb), server_id(char), last_server_ping(ts), error(text), error_handler_path(char), error_handler_args(jsonb), retry(jsonb), mode(trigger_mode), filters(jsonb[]), auto_commit(bool), labels(text[])
-log_file: hostname(char), log_ts(ts), ok_lines(bigint), err_lines(bigint), mode(log_mode), worker_group(char), file_path(char), json_fmt(bool)
+log_file: hostname(char), log_ts(ts), ok_lines(bigint), err_lines(bigint), mode(log_mode), worker_group(char), file_path(char), json_fmt(bool), indexed_at(ts)
 macro_definition: workspace_id(char), name(char), provider_path(char), params(text), body(text), is_table_macro(bool), created_at(ts)
   FK: (workspace_id) -> workspace(id)
 macro_usage: workspace_id(char), consumer_path(char), macro_name(char)

backend/tests/suspended_pull_index.rs

@@ -0,0 +1,77 @@
+//! Pins the plan of the suspended-job pull. Its resume test degrades silently: once the
+//! query expression and `queue_suspended_v2` stop matching, Postgres still returns the right
+//! job, just by falling back to a heap filter and fetching one tuple per suspended row on
+//! every worker poll. No functional test can see that, so assert on the plan instead.
+
+use serde_json::Value;
+use sqlx::{Pool, Postgres};
+use windmill_common::worker::make_suspended_pull_query;
+
+/// Depth-first walk of an `EXPLAIN (FORMAT JSON)` plan tree.
+fn nodes(plan: &Value, out: &mut Vec<Value>) {
+    out.push(plan.clone());
+    for child in plan["Plans"].as_array().unwrap_or(&vec![]) {
+        nodes(child, out);
+    }
+}
+
+#[sqlx::test(fixtures("base"))]
+async fn suspended_pull_tests_resume_time_inside_the_index(
+    db: Pool<Postgres>,
+) -> anyhow::Result<()> {
+    sqlx::query(
+        "INSERT INTO v2_job_queue (id, workspace_id, created_at, scheduled_for, running, suspend, suspend_until, tag)
+         SELECT gen_random_uuid(), 'test-workspace', now() - make_interval(secs => i),
+                now(), true, 1 + (i % 3), now() + interval '7 day', 'flow'
+         FROM generate_series(1, 2000) i",
+    )
+    .execute(&db)
+    .await?;
+    sqlx::query("ANALYZE v2_job_queue").execute(&db).await?;
+
+    // Both plans are cheap on a 2000-row table, and which one wins there says nothing
+    // about a queue with a large suspended backlog. Force the index path, which is the
+    // one production takes, and assert on how it evaluates the resume test.
+    let mut conn = db.acquire().await?;
+    sqlx::query("SET enable_seqscan = off")
+        .execute(&mut *conn)
+        .await?;
+    let version: String = sqlx::query_scalar("SELECT version()")
+        .fetch_one(&mut *conn)
+        .await?;
+    // FORMAT JSON rather than the default: `Index Cond` and `Filter` are separate keys on the
+    // node, so this does not ride on EXPLAIN's line layout staying put across a major bump.
+    let explained: Value = sqlx::query_scalar(&format!(
+        "EXPLAIN (FORMAT JSON) {}",
+        make_suspended_pull_query(&["flow".to_string()])
+    ))
+    .bind("test-worker")
+    .fetch_one(&mut *conn)
+    .await?;
+
+    let mut all = vec![];
+    nodes(&explained[0]["Plan"], &mut all);
+    let pretty = serde_json::to_string_pretty(&explained)?;
+    let scan = all
+        .iter()
+        .find(|n| n["Index Name"] == "queue_suspended_v2")
+        .unwrap_or_else(|| {
+            panic!("suspended pull did not scan queue_suspended_v2 on {version}:\n{pretty}")
+        });
+    // Only `Index Cond` is checked against the index tuple, so that is where the resume test
+    // has to land — as a `Filter` it would cost a heap fetch per suspended row. The residual
+    // `suspend_until IS NOT NULL` filter is not that: it is always true for rows the partial
+    // index holds, and only ever runs on the row LIMIT 1 already fetched.
+    let cond = scan["Index Cond"].as_str().unwrap_or_else(|| {
+        panic!("no Index Cond on the suspended pull scan on {version}:\n{pretty}")
+    });
+    assert!(
+        cond.contains("CASE WHEN"),
+        "resume test is not an index condition on {version}:\n{pretty}"
+    );
+    assert!(
+        !scan["Filter"].as_str().unwrap_or("").contains("CASE WHEN"),
+        "resume test fell back to a heap filter on {version}:\n{pretty}"
+    );
+    Ok(())
+}

backend/tests/wm_token_confinement.rs

@@ -318,6 +318,33 @@ async fn test_wm_token_is_confined_to_its_workspace(db: Pool<Postgres>) -> anyho
             resp.text().await?
         );
     }
+    // ...and the one `settings/global` key on the allowlist, which the CLI reads before
+    // creating a user on a git-sync push. `ws_base_url` is the control: the handler leaves
+    // it as ungated as `automate_username_creation`, so only the allowlist stops it.
+    let resp = authed(
+        client().get(format!("{api}/settings/global/automate_username_creation")),
+        &user_wm,
+    )
+    .send()
+    .await?;
+    assert_eq!(
+        resp.status(),
+        200,
+        "WM_TOKEN must still read automate_username_creation: {}",
+        resp.text().await?
+    );
+    let resp = authed(
+        client().get(format!("{api}/settings/global/ws_base_url")),
+        &user_wm,
+    )
+    .send()
+    .await?;
+    assert_eq!(
+        resp.status(),
+        403,
+        "WM_TOKEN must not read any other global setting: {}",
+        resp.text().await?
+    );
     let resp = authed(client().post(format!("{api}/schedules/preview")), &user_wm)
         .json(&json!({ "schedule": "0 0 12 * * *", "timezone": "UTC" }))
         .send()

backend/windmill-ai/src/ai_providers.rs

@@ -24,6 +24,16 @@ lazy_static::lazy_static! {
         .ok()
         .map(|v| v == "true" || v == "1")
         .unwrap_or(false);
+    /// Drops the cache breakpoints from agent-step requests on every Anthropic platform,
+    /// not just the one that motivates it: a Google Cloud project can have explicit prompt
+    /// caching turned off (by request to Cloud support), and Vertex then rejects any request
+    /// carrying breakpoints. An instance that sets this to unblock such a project also gives
+    /// up caching on its direct-Anthropic and Foundry resources.
+    pub static ref DISABLE_ANTHROPIC_PROMPT_CACHING: bool =
+        std::env::var("DISABLE_ANTHROPIC_PROMPT_CACHING")
+            .ok()
+            .map(|v| v == "true" || v == "1")
+            .unwrap_or(false);
 }
 
 pub const OPENAI_BASE_URL: &str = "https://api.openai.com/v1";

backend/windmill-ai/src/providers/anthropic.rs

@@ -1,7 +1,7 @@
 use super::{anthropic_model_rejects_sampling_params, REASONING_OFF_SENTINEL};
 use crate::{
     ai_google::parse_data_url,
-    ai_providers::{AIPlatform, AIProvider},
+    ai_providers::{AIPlatform, AIProvider, DISABLE_ANTHROPIC_PROMPT_CACHING},
     image_handler::prepare_messages_for_api,
     proxy::{
         add_user_to_body, common_outbound_headers, credential_header, ProxyBuildArgs, ProxyRequest,
@@ -632,15 +632,13 @@ impl AnthropicQueryBuilder {
             }
         }
 
+        let caching = !*DISABLE_ANTHROPIC_PROMPT_CACHING;
+
         let system = collect_system_prompt(&prepared_messages, args.system_prompt).map(|text| {
             vec![AnthropicSystemContent {
                 r#type: "text".to_string(),
                 text,
-                cache_control: if self.is_vertex() {
-                    None
-                } else {
-                    Some(CacheControl::ephemeral())
-                },
+                cache_control: caching.then(CacheControl::ephemeral),
             }]
         });
 
@@ -665,7 +663,7 @@ impl AnthropicQueryBuilder {
         let max_tokens = Some(args.max_tokens.unwrap_or(64000));
 
         // Apply cache_control on the last custom tool
-        if !self.is_vertex() {
+        if caching {
             if let Some(ref mut tools_vec) = tools_option {
                 if let Some(AnthropicTool::Custom(ref mut custom)) = tools_vec.last_mut() {
                     custom.cache_control = Some(CacheControl::ephemeral());
@@ -674,7 +672,7 @@ impl AnthropicQueryBuilder {
         }
 
         // Apply cache_control on the last content block of the last message
-        if !self.is_vertex() {
+        if caching {
             if let Some(last_msg) = anthropic_messages.last_mut() {
                 if let Some(last_block) = last_msg.content.last_mut() {
                     match last_block {
@@ -882,10 +880,15 @@ mod tests {
         }
     }
 
-    async fn build_text_body(messages: &[OpenAIMessage], system_prompt: Option<&str>) -> String {
+    async fn build_text_body_on(
+        platform: AIPlatform,
+        messages: &[OpenAIMessage],
+        system_prompt: Option<&str>,
+        tools: Option<&[ToolDef]>,
+    ) -> String {
         let args = BuildRequestArgs {
             messages,
-            tools: None,
+            tools,
             model: "claude-sonnet-4",
             temperature: None,
             reasoning_effort: None,
@@ -899,12 +902,16 @@ mod tests {
             prompt_cache_key: None,
         };
 
-        AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::Standard)
+        AnthropicQueryBuilder::new(AIProvider::Anthropic, platform)
             .build_request(&args, &authed_client(), "test-workspace")
             .await
             .unwrap()
     }
 
+    async fn build_text_body(messages: &[OpenAIMessage], system_prompt: Option<&str>) -> String {
+        build_text_body_on(AIPlatform::Standard, messages, system_prompt, None).await
+    }
+
     /// The worker prepends the system prompt as a system message *and* passes it as
     /// `system_prompt`; the request must still carry it exactly once.
     #[tokio::test]
@@ -946,6 +953,35 @@ mod tests {
         assert!(request.get("system").is_none());
     }
 
+    /// Vertex serves the same Messages API and honours `cache_control` breakpoints, so
+    /// its requests must carry the same three the standard platform gets.
+    #[tokio::test]
+    async fn sets_cache_breakpoints_on_every_platform() {
+        let messages = vec![message("system", SYSTEM_PROMPT), message("user", "hi")];
+        let tools = vec![ToolDef {
+            r#type: "function".to_string(),
+            function: ToolDefFunction {
+                name: "get_weather".to_string(),
+                description: None,
+                parameters: RawValue::from_string("{}".to_string()).unwrap(),
+            },
+        }];
+        let ephemeral = serde_json::json!({ "type": "ephemeral" });
+
+        for platform in [AIPlatform::Standard, AIPlatform::GoogleVertexAi] {
+            let body =
+                build_text_body_on(platform, &messages, Some(SYSTEM_PROMPT), Some(&tools)).await;
+            let request: serde_json::Value = serde_json::from_str(&body).unwrap();
+
+            assert_eq!(request["system"][0]["cache_control"], ephemeral);
+            let sent_tools = request["tools"].as_array().unwrap();
+            assert_eq!(sent_tools.last().unwrap()["cache_control"], ephemeral);
+            let sent = request["messages"].as_array().unwrap();
+            let content = sent.last().unwrap()["content"].as_array().unwrap();
+            assert_eq!(content.last().unwrap()["cache_control"], ephemeral);
+        }
+    }
+
     fn has_header(headers: &[(String, String)], name: &str, value: &str) -> bool {
         headers
             .iter()

backend/windmill-api-auth/src/scopes.rs

@@ -996,6 +996,13 @@ fn scope_grants_access(
 /// the caller's own row; `email` and `allowed_domain_auto_invite` are derived from the
 /// token itself and touch no table.
 ///
+/// `settings/global/automate_username_creation` is the one instance setting on the list.
+/// `get_global_setting` exempts a handful of keys from its own super-admin gate, that one
+/// among them, so the boolean is already readable by every authenticated user; it is here
+/// because the CLI reads it before creating a user during a git-sync push, which runs as a
+/// job. The other ungated keys have no such caller, so they stay confined — being ungated
+/// earns a key nothing on its own.
+///
 /// Deliberately absent, as each crosses that line: `users/list_invites` (returns the
 /// workspace ids the identity was invited to), `users/tokens/list` (credential metadata
 /// of the borrowed identity), `users/exists/{email}` (an oracle over arbitrary
@@ -1011,6 +1018,7 @@ fn is_global_read_open_to_job_token(route_path: &str) -> bool {
             | "/api/users/usage"
             | "/api/users/tutorial_progress"
             | "/api/workspaces/allowed_domain_auto_invite"
+            | "/api/settings/global/automate_username_creation"
             | "/api/docs/search"
             | "/api/docs/page"
             | "/api/integrations/hub/list"

backend/windmill-api-debug/src/lib.rs

@@ -468,6 +468,9 @@ async fn sign_debug_request(
 
     // Parse the language
     let script_lang: ScriptLang = request.language.parse().unwrap_or(ScriptLang::Bun);
+    // Taken from the parsed language, not the request's string: the telemetry key vocabulary has
+    // to stay the closed set of languages rather than whatever a caller sent.
+    let lang_key = script_lang.as_str();
 
     // Hash the code (we don't include full code in JWT to keep it small)
     let mut hasher = Sha256::new();
@@ -578,6 +581,8 @@ async fn sign_debug_request(
 
     tx.commit().await?;
 
+    windmill_common::feature_usage::log_feature_usage("debugger", "session", lang_key);
+
     Ok(Json(SignedDebugPayload {
         token,
         code: request.code,

backend/windmill-api-integration-tests/tests/users.rs

@@ -308,14 +308,17 @@ async fn test_user_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
     let auth_base = format!("http://localhost:{port}/api/auth");
 
     // --- login (will fail: password hash in fixture is fake) ---
+    // An unparseable stored hash must read as a failed login, not as a server error
+    // relaying the hash parser's message to an unauthenticated caller.
     let resp = client()
         .post(format!("{auth_base}/login"))
         .json(&json!({"email": "test@windmill.dev", "password": "wrong-password"}))
         .send()
         .await
         .unwrap();
-    assert!(
-        resp.status() == 400 || resp.status() == 401 || resp.status() == 500,
+    assert_eq!(
+        resp.status(),
+        400,
         "login: unexpected status {}",
         resp.status()
     );
@@ -804,12 +807,16 @@ async fn test_change_user_email_leaves_group_identities(db: Pool<Postgres>) -> a
     let server = ApiServer::start(db.clone()).await?;
     let global_base = format!("http://localhost:{}/api/users", server.addr.port());
 
-    sqlx::query!("UPDATE password SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'")
-        .execute(&db)
-        .await?;
-    sqlx::query!("UPDATE usr SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'")
-        .execute(&db)
-        .await?;
+    sqlx::query!(
+        "UPDATE password SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'"
+    )
+    .execute(&db)
+    .await?;
+    sqlx::query!(
+        "UPDATE usr SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'"
+    )
+    .execute(&db)
+    .await?;
     sqlx::query!(
         "INSERT INTO group_(workspace_id, name, summary, extra_perms) VALUES ('test-workspace', 'ops', '', '{}')"
     )

backend/windmill-api-settings/src/log_cleanup.rs

@@ -32,7 +32,7 @@ use windmill_common::tracing_init::{LOGS_SERVICE, TMP_WINDMILL_LOGS_SERVICE};
 use windmill_common::worker::WINDMILL_DIR;
 use windmill_common::{
     DB, INSTANCE_NAME, JOB_RETENTION_SECS, JOB_RETENTION_SECS_OVERRIDES,
-    JOB_RETENTION_SECS_OVERRIDES_LOADED, SERVICE_LOG_RETENTION_SECS,
+    JOB_RETENTION_SECS_OVERRIDES_LOADED,
 };
 
 use windmill_object_store::object_store_reexports::{
@@ -249,7 +249,7 @@ async fn cleanup_service_logs(
     // Count candidates upfront for progress reporting.
     let total: i64 = sqlx::query_scalar!(
         "SELECT COUNT(*) FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval",
-        SERVICE_LOG_RETENTION_SECS,
+        windmill_common::service_log_retention_secs(),
     )
     .fetch_one(db)
     .await?
@@ -274,7 +274,7 @@ async fn cleanup_service_logs(
                  WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval
                  LIMIT $2
              ) RETURNING file_path, hostname",
-            SERVICE_LOG_RETENTION_SECS,
+            windmill_common::service_log_retention_secs(),
             SERVICE_LOG_BATCH,
         )
         .fetch_all(db)
@@ -680,9 +680,10 @@ async fn cleanup_s3_orphans(
 ) -> error::Result<()> {
     let job_retention_secs = JOB_RETENTION_SECS.load(std::sync::atomic::Ordering::Relaxed);
     let now = Utc::now();
-    // Service logs always have a retention (hardcoded SERVICE_LOG_RETENTION_SECS),
-    // so we scan for service-log orphans regardless of JOB_RETENTION_SECS.
-    let service_cutoff = now - chrono::Duration::seconds(SERVICE_LOG_RETENTION_SECS);
+    // Service logs always have a retention, so we scan for service-log orphans regardless of
+    // JOB_RETENTION_SECS.
+    let service_cutoff =
+        now - chrono::Duration::seconds(windmill_common::service_log_retention_secs());
 
     // Job-log orphans are only considered once past a job's effective retention window. That window
     // is the instance one OR, for an override workspace (EE), its own — and jobs orphan their logs as

backend/windmill-api-users/src/users.rs

@@ -19,7 +19,7 @@ use windmill_api_auth::ApiAuthed;
 
 pub use windmill_api_auth::Tokened;
 
-use argon2::{Argon2, PasswordHash, PasswordVerifier};
+use argon2::{Argon2, PasswordVerifier};
 use axum::{
     extract::{Extension, Path, Query},
     response::{IntoResponse, Response},
@@ -2680,10 +2680,8 @@ async fn login(
     .await?;
 
     if let Some((email, hash, super_admin)) = email_w_h {
-        let parsed_hash =
-            PasswordHash::new(&hash).map_err(|e| Error::internal_err(e.to_string()))?;
         if argon2
-            .verify_password(password.as_bytes(), &parsed_hash)
+            .verify_password(password.as_bytes(), hash.as_str())
             .is_err()
         {
             audit_log(
@@ -3710,3 +3708,23 @@ async fn request_password_reset(
 }
 
 // NOTE: reset_password is in windmill-api (depends on users_oss::hash_password EE dispatch)
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// Stored hashes outlive the hashing crate: every instance still holds hashes minted by
+    /// older argon2 releases, and an upgrade that stopped reading them locks their users out.
+    #[test]
+    fn verifies_a_hash_minted_by_an_older_argon2() {
+        // The seeded admin hash from migration 20220508150023, m=4096,t=3,p=1.
+        let seeded = "$argon2id$v=19$m=4096,t=3,p=1$oLJo/lPn/gezXCuFOEyaNw$i0T2tCkw3xUFsrBIKZwr8jVNHlIfoxQe+HfDnLtd12I";
+
+        assert!(Argon2::default()
+            .verify_password(b"changeme", seeded)
+            .is_ok());
+        assert!(Argon2::default()
+            .verify_password(b"not-the-password", seeded)
+            .is_err());
+    }
+}

backend/windmill-api-workspaces/src/datatable_migrations.rs

@@ -10,7 +10,10 @@
 //! management, and the workspace-merge diff helper. Split out of `workspaces.rs`
 //! to keep that file focused on core workspace configuration.
 
-use crate::workspaces::{pg_dump_database, ItemComparison};
+use crate::workspaces::{
+    is_instance_datatable, pg_dump_database, strip_unreplayable_dump_lines, ItemComparison,
+    PgDumpOptions,
+};
 
 use axum::{
     extract::{Extension, Path, Query},
@@ -1448,18 +1451,25 @@ async fn generate_initial_datatable_migration(
     let pg_db: PgDatabase = serde_json::from_value(db_resource)
         .map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))?;
 
-    // Snapshot the schema, excluding Windmill's own migration bookkeeping table.
-    let dump_file = pg_dump_database(&pg_db, true, &["_wm_migrations"]).await?;
+    // Snapshot the schema without `_wm_migrations`, Windmill's own bookkeeping table, and
+    // without what a replay elsewhere cannot run: the replaying user owns none of this
+    // database's objects, and the grants Windmill plants in an instance database (`ALTER
+    // DEFAULT PRIVILEGES FOR ROLE ...`) fail even replaying onto the same server.
+    let no_acl = is_instance_datatable(&db, &w_id, &datatable_name).await?;
+    let dump_file = pg_dump_database(
+        &pg_db,
+        PgDumpOptions {
+            schema_only: true,
+            exclude_tables: &["_wm_migrations"],
+            no_owner: true,
+            no_acl,
+        },
+    )
+    .await?;
     let raw_dump = tokio::fs::read_to_string(&dump_file.path)
         .await
         .map_err(|e| Error::internal_err(format!("Failed to read schema dump: {}", e)))?;
-    // pg_dump emits psql meta-commands (\restrict / \unrestrict) that aren't
-    // valid SQL; drop them so the migration body can run via a plain query.
-    let code_up: String = raw_dump
-        .lines()
-        .filter(|line| !line.trim_start().starts_with('\\'))
-        .collect::<Vec<_>>()
-        .join("\n");
+    let code_up = strip_unreplayable_dump_lines(&raw_dump);
 
     // Record the definition first, then mark it installed. If marking fails we
     // delete the definition, so a failure leaves no phantom "initial" (rather

backend/windmill-api-workspaces/src/workspaces.rs

@@ -119,6 +119,7 @@ pub fn workspaced_service() -> Router {
             get(get_secondary_storage_names),
         )
         .route("/is_premium", get(is_premium))
+        .route("/billable_seats", get(get_billable_seats))
         .route("/edit_error_handler", post(edit_error_handler))
         .route("/edit_success_handler", post(edit_success_handler))
         .route(
@@ -690,6 +691,48 @@ async fn is_premium(
     Ok(Json(premium))
 }
 
+#[derive(Serialize)]
+struct BillableSeatsResponse {
+    /// Both omitted when the seats counted are another workspace's: a fork member need not be a
+    /// member of the billing root, so the root's headcount is not theirs to read. The total is,
+    /// since it is the divisor of the quota their own executions draw on.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    developers: Option<i64>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    operators: Option<i64>,
+    seats: i64,
+}
+
+async fn get_billable_seats(
+    _authed: ApiAuthed,
+    Extension(db): Extension<DB>,
+    Path(w_id): Path<String>,
+) -> JsonResult<BillableSeatsResponse> {
+    // Readable by any workspace member, like `is_premium`: this is what the sidebar usage meter
+    // divides by, and that meter is shown to non-admin developers too.
+    //
+    // On cloud a fork draws its plan, quota and bill from the root, so the seats its usage is
+    // measured against are the root's. Resolved here rather than by the caller: a fork member need
+    // not be a member of that root, and so cannot count its seats from the member list. Off cloud
+    // a fork is not billed through a root at all, so the workspace answers for itself.
+    #[cfg(feature = "cloud")]
+    let billing_w_id = if *CLOUD_HOSTED {
+        windmill_common::workspaces::get_billing_workspace_id(&db, &w_id).await?
+    } else {
+        w_id.clone()
+    };
+    #[cfg(not(feature = "cloud"))]
+    let billing_w_id = w_id.clone();
+
+    let counted = windmill_common::workspaces::billable_seats(&db, &billing_w_id).await?;
+    let own = billing_w_id == w_id;
+    Ok(Json(BillableSeatsResponse {
+        developers: own.then_some(counted.developers),
+        operators: own.then_some(counted.operators),
+        seats: counted.seats,
+    }))
+}
+
 async fn exists_workspace(
     authed: ApiAuthed,
     Extension(user_db): Extension<UserDB>,
@@ -1975,6 +2018,23 @@ async fn edit_large_file_storage_config(
             )));
         }
 
+        if !windmill_common::workspaces::filesystem_storage_allowed() {
+            let named = std::iter::once(("primary storage", &lfs_config.large_file_storage)).chain(
+                lfs_config
+                    .secondary_storage
+                    .iter()
+                    .map(|(name, storage)| (name.as_str(), storage)),
+            );
+            for (name, storage) in named {
+                if matches!(storage, LargeFileStorage::FilesystemStorage(_)) {
+                    return Err(Error::BadRequest(format!(
+                        "{name}: {}",
+                        windmill_common::workspaces::FILESYSTEM_STORAGE_DEV_ONLY_MSG
+                    )));
+                }
+            }
+        }
+
         let serialized_lfs_config =
             serde_json::to_value::<LargeFileStorageWithSecondary>(lfs_config)
                 .map_err(|err| Error::internal_err(err.to_string()))?;
@@ -2757,6 +2817,66 @@ fn truncate_column_default(default: String) -> String {
 mod tests {
     use super::*;
 
+    /// The header of a pg_dump, followed by an object whose body also holds a `SET`.
+    const DUMP: &str = "--\n\
+        -- PostgreSQL database dump\n\
+        --\n\
+        \n\
+        \\restrict aBcD\n\
+        \n\
+        SET statement_timeout = 0;\n\
+        SET transaction_timeout = 0;\n\
+        SET client_encoding = 'UTF8';\n\
+        SELECT pg_catalog.set_config('search_path', '', false);\n\
+        \n\
+        SET default_table_access_method = heap;\n\
+        \n\
+        CREATE FUNCTION public.f() RETURNS void LANGUAGE plpgsql AS $$\n\
+        BEGIN\n\
+        SET transaction_timeout = 0;\n\
+        END;\n\
+        $$;\n";
+
+    #[test]
+    fn replayable_dump_keeps_everything_but_meta_commands_and_session_timeouts() {
+        let replayable = strip_unreplayable_dump_lines(DUMP);
+
+        assert!(!replayable.contains("\\restrict"));
+        assert!(!replayable.contains("SET statement_timeout"));
+        assert!(!replayable.contains("SET transaction_timeout = 0;\nSET client_encoding"));
+        assert!(replayable.contains("SET client_encoding = 'UTF8';"));
+        assert!(replayable.contains("SET default_table_access_method = heap;"));
+        // Past the preamble the dump is an object's own text: left exactly as it is.
+        assert!(replayable.contains("BEGIN\nSET transaction_timeout = 0;\nEND;"));
+    }
+
+    #[tokio::test]
+    async fn dump_preamble_only_drops_settings_the_server_lacks() {
+        let dump_file = DumpFile::new().unwrap();
+        tokio::fs::write(&dump_file.path, DUMP).await.unwrap();
+        let supported = [
+            "statement_timeout",
+            "client_encoding",
+            "default_table_access_method",
+        ]
+        .map(String::from)
+        .into_iter()
+        .collect();
+
+        comment_out_unsupported_settings(&dump_file, &supported)
+            .await
+            .unwrap();
+
+        let patched = tokio::fs::read_to_string(&dump_file.path).await.unwrap();
+        // Rewriting the header must not shift the rest of the dump.
+        assert_eq!(patched.len(), DUMP.len());
+        assert!(patched.contains("--  transaction_timeout = 0;"));
+        assert!(patched.contains("SET statement_timeout = 0;"));
+        assert!(patched.contains("SET default_table_access_method = heap;"));
+        // The `SET` inside the function body is past the preamble: never touched.
+        assert!(patched.contains("BEGIN\nSET transaction_timeout = 0;\nEND;"));
+    }
+
     #[test]
     fn compact_column_type_truncates_multibyte_defaults_safely() {
         let default = "é".repeat(31);
@@ -2859,6 +2979,35 @@ pub(crate) async fn resolve_pg_source_checked(
         .map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))
 }
 
+/// Whether the data table `name` is backed by the Windmill instance's own PostgreSQL
+/// rather than a user resource.
+pub(crate) async fn is_instance_datatable(db: &DB, w_id: &str, name: &str) -> Result<bool> {
+    let config = sqlx::query_scalar!(
+        "SELECT datatable->'datatables'->$2 FROM workspace_settings WHERE workspace_id = $1",
+        w_id,
+        name
+    )
+    .fetch_optional(db)
+    .await?
+    .flatten();
+    Ok(config
+        .and_then(|v| {
+            v.get("database")
+                .and_then(|d| d.get("resource_type"))
+                .and_then(|r| r.as_str())
+                .map(|s| s == "instance")
+        })
+        .unwrap_or(false))
+}
+
+/// Same, for the `datatable://<name>` / `$res:<path>` form the import endpoints take.
+async fn is_instance_datatable_source(db: &DB, w_id: &str, source: &str) -> Result<bool> {
+    match source.strip_prefix("datatable://") {
+        Some(name) => is_instance_datatable(db, w_id, name).await,
+        None => Ok(false),
+    }
+}
+
 /// A temporary file for pg_dump output that is automatically deleted when dropped.
 pub(crate) struct DumpFile {
     pub(crate) path: std::path::PathBuf,
@@ -2906,12 +3055,21 @@ impl Drop for DumpFile {
     }
 }
 
+#[derive(Default)]
+pub(crate) struct PgDumpOptions<'a> {
+    pub(crate) schema_only: bool,
+    pub(crate) exclude_tables: &'a [&'a str],
+    /// Leave out `ALTER ... OWNER TO`.
+    pub(crate) no_owner: bool,
+    /// Leave out `GRANT`, `REVOKE` and `ALTER DEFAULT PRIVILEGES`.
+    pub(crate) no_acl: bool,
+}
+
 /// Run pg_dump against a PgDatabase, writing output to a temp file on disk.
 /// Returns a DumpFile handle; the file is deleted when the handle is dropped.
 pub(crate) async fn pg_dump_database(
     pg_db: &PgDatabase,
-    schema_only: bool,
-    exclude_tables: &[&str],
+    opts: PgDumpOptions<'_>,
 ) -> Result<DumpFile> {
     let dump_file = DumpFile::new()?;
 
@@ -2922,10 +3080,16 @@ pub(crate) async fn pg_dump_database(
 
     let mut cmd = tokio::process::Command::new("pg_dump");
     cmd.arg("--format=plain").arg("--file").arg(&dump_file.path);
-    if schema_only {
+    if opts.schema_only {
         cmd.arg("--schema-only");
     }
-    for table in exclude_tables {
+    if opts.no_owner {
+        cmd.arg("--no-owner");
+    }
+    if opts.no_acl {
+        cmd.arg("--no-privileges");
+    }
+    for table in opts.exclude_tables {
         cmd.arg(format!("--exclude-table={table}"));
     }
     cmd.arg("--host")
@@ -2957,37 +3121,179 @@ pub(crate) async fn pg_dump_database(
     Ok(dump_file)
 }
 
-/// Import a pg_dump file into a target database using psql.
-async fn pg_import_dump(target_db: &PgDatabase, dump_file: &DumpFile) -> Result<()> {
-    let host = &target_db.host;
-    let port = target_db.port.unwrap_or(5432).to_string();
-    let user = target_db.login_name();
-    let dbname = &target_db.dbname;
+/// Whether `line` still belongs to the preamble pg_dump emits before the first
+/// dumped object: comments, blank lines, psql meta-commands and the session `SET`s.
+fn is_dump_preamble_line(line: &[u8]) -> bool {
+    let line = line.trim_ascii_start();
+    line.is_empty()
+        || line.starts_with(b"--")
+        || line.starts_with(b"\\")
+        || line.starts_with(b"SET ")
+        || line.starts_with(b"SELECT pg_catalog.set_config(")
+}
 
+/// The GUCs pg_dump's preamble sets only to keep the dumping session out of the way.
+/// They are also the ones that come and go across versions (`transaction_timeout` is
+/// PG 17+), so they are what a dump replayed on an older server trips over first.
+const DUMP_SESSION_TIMEOUTS: [&str; 4] = [
+    "statement_timeout",
+    "lock_timeout",
+    "idle_in_transaction_session_timeout",
+    "transaction_timeout",
+];
+
+/// Turn a dump into SQL that can be replayed on another database: drop pg_dump's psql
+/// meta-commands (`\restrict` / `\unrestrict`, not valid SQL) and the session timeouts
+/// its preamble sets, which the replaying server may not have as GUCs at all. Only the
+/// preamble is filtered, so an object's body keeps whatever it holds.
+pub(crate) fn strip_unreplayable_dump_lines(dump: &str) -> String {
+    let mut in_preamble = true;
+    dump.lines()
+        .filter(|line| {
+            in_preamble = in_preamble && is_dump_preamble_line(line.as_bytes());
+            if line.trim_start().starts_with('\\') {
+                return false;
+            }
+            !(in_preamble
+                && preamble_setting_name(line.as_bytes())
+                    .is_some_and(|name| DUMP_SESSION_TIMEOUTS.contains(&name)))
+        })
+        .collect::<Vec<_>>()
+        .join("\n")
+}
+
+/// The GUC a preamble `SET <name> = ...;` line assigns, if the line is one.
+fn preamble_setting_name(line: &[u8]) -> Option<&str> {
+    let name = line.strip_prefix(b"SET ")?.split(|c| *c == b' ').next()?;
+    std::str::from_utf8(name).ok()
+}
+
+/// The preamble Windmill's postgres client writes can set GUCs an older server does not
+/// have — harmless session tuning, but one failing statement aborts a restore that stops
+/// on the first error. Comment those out in place, three bytes each, so the data
+/// section's offsets stay put.
+async fn comment_out_unsupported_settings(
+    dump_file: &DumpFile,
+    supported_settings: &HashSet<String>,
+) -> Result<()> {
+    use tokio::io::{AsyncBufReadExt, AsyncWriteExt};
+
+    let file = tokio::fs::File::open(&dump_file.path)
+        .await
+        .map_err(|e| Error::internal_err(format!("Failed to open dump file: {}", e)))?;
+    let mut reader = tokio::io::BufReader::new(file);
+
+    let mut preamble: Vec<u8> = Vec::new();
+    let mut patched = false;
+    loop {
+        let start = preamble.len();
+        let read = reader
+            .read_until(b'\n', &mut preamble)
+            .await
+            .map_err(|e| Error::internal_err(format!("Failed to read dump file: {}", e)))?;
+        if read == 0 {
+            break;
+        }
+        let line = &preamble[start..];
+        if !is_dump_preamble_line(line) {
+            preamble.truncate(start);
+            break;
+        }
+        if preamble_setting_name(line).is_some_and(|name| !supported_settings.contains(name)) {
+            preamble[start..start + 3].copy_from_slice(b"-- ");
+            patched = true;
+        }
+    }
+    if !patched {
+        return Ok(());
+    }
+
+    let mut file = tokio::fs::OpenOptions::new()
+        .write(true)
+        .open(&dump_file.path)
+        .await
+        .map_err(|e| Error::internal_err(format!("Failed to open dump file: {}", e)))?;
+    file.write_all(&preamble)
+        .await
+        .map_err(|e| Error::internal_err(format!("Failed to rewrite dump preamble: {}", e)))?;
+    file.flush()
+        .await
+        .map_err(|e| Error::internal_err(format!("Failed to rewrite dump preamble: {}", e)))?;
+    Ok(())
+}
+
+/// A psql invocation against `pg_db`, carrying the connection settings the CLI reads
+/// from the environment.
+fn psql_command(pg_db: &PgDatabase) -> tokio::process::Command {
     let mut cmd = tokio::process::Command::new("psql");
     cmd.arg("--host")
-        .arg(host)
+        .arg(&pg_db.host)
         .arg("--port")
-        .arg(&port)
+        .arg(pg_db.port.unwrap_or(5432).to_string())
         .arg("--username")
-        .arg(user)
+        .arg(pg_db.login_name())
         .arg("--dbname")
-        .arg(dbname)
+        .arg(&pg_db.dbname)
         .arg("--no-psqlrc")
-        .arg("--file")
-        .arg(&dump_file.path)
         .stdout(std::process::Stdio::piped())
         .stderr(std::process::Stdio::piped());
 
-    if let Some(ref password) = target_db.password {
+    if let Some(ref password) = pg_db.password {
         cmd.env("PGPASSWORD", password);
     }
-
-    if let Some(ref sslmode) = target_db.sslmode {
+    if let Some(ref sslmode) = pg_db.sslmode {
         cmd.env("PGSSLMODE", sslmode);
     }
+    cmd
+}
 
-    let output = cmd
+/// GUC names the server backing `pg_db` knows about.
+///
+/// Asked through psql rather than a tokio-postgres connection so the lookup reaches
+/// exactly the servers the restore itself can: libpq negotiates TLS for `sslmode=prefer`
+/// and an unset mode, where `PgDatabase::connect` would hand a TLS-only server a
+/// plaintext socket and fail before the import ever starts.
+async fn server_setting_names(pg_db: &PgDatabase) -> Result<HashSet<String>> {
+    let output = psql_command(pg_db)
+        .arg("--tuples-only")
+        .arg("--no-align")
+        .arg("--command")
+        .arg("SELECT name FROM pg_settings")
+        .output()
+        .await
+        .map_err(|e| Error::internal_err(format!("Failed to execute psql: {}", e)))?;
+
+    if !output.status.success() {
+        let stderr = String::from_utf8_lossy(&output.stderr);
+        return Err(Error::internal_err(format!(
+            "Failed to list the settings of the target server: {}",
+            stderr
+        )));
+    }
+
+    Ok(String::from_utf8_lossy(&output.stdout)
+        .lines()
+        .map(|name| name.trim().to_string())
+        .filter(|name| !name.is_empty())
+        .collect())
+}
+
+/// Import a pg_dump file into a target database using psql.
+///
+/// Left to its defaults psql reports a failed statement, carries on and still exits 0,
+/// so a dump that breaks partway through imports partially and reads as a success.
+/// ON_ERROR_STOP surfaces the failure and --single-transaction makes the restore
+/// all-or-nothing, leaving the target as it was and the import retryable.
+async fn pg_import_dump(target_db: &PgDatabase, dump_file: &DumpFile) -> Result<()> {
+    let supported_settings = server_setting_names(target_db).await?;
+    comment_out_unsupported_settings(dump_file, &supported_settings).await?;
+
+    let output = psql_command(target_db)
+        .arg("--set")
+        .arg("ON_ERROR_STOP=1")
+        .arg("--single-transaction")
+        .arg("--file")
+        .arg(&dump_file.path)
         .output()
         .await
         .map_err(|e| Error::internal_err(format!("Failed to execute psql: {}", e)))?;
@@ -3032,29 +3338,7 @@ async fn create_pg_database(
         }
     }
 
-    // Determine if this is an instance or resource-backed datatable
-    let is_instance_datatable = if let Some(dt_name) = req.source.strip_prefix("datatable://") {
-        let config = sqlx::query_scalar!(
-            "SELECT datatable->'datatables'->$2 FROM workspace_settings WHERE workspace_id = $1",
-            &w_id,
-            dt_name
-        )
-        .fetch_optional(&db)
-        .await?
-        .flatten();
-        config
-            .and_then(|v| {
-                v.get("database")
-                    .and_then(|d| d.get("resource_type"))
-                    .and_then(|r| r.as_str())
-                    .map(|s| s == "instance")
-            })
-            .unwrap_or(false)
-    } else {
-        false
-    };
-
-    if is_instance_datatable {
+    if is_instance_datatable_source(&db, &w_id, &req.source).await? {
         windmill_common::create_custom_instance_database(&db, &req.target_dbname, "datatable")
             .await?;
     } else {
@@ -3152,7 +3436,18 @@ async fn import_pg_database(
     }
     windmill_common::validate_dbname(&target_pg.dbname)?;
 
-    let dump_file = pg_dump_database(&source_pg, schema_only, &[]).await?;
+    // Ownership never replays: the restore runs as the target's own connection user, and
+    // what it creates it owns. Grants do, except around an instance data table — Windmill
+    // plants `custom_instance_user` grants in one, which nothing else can replay. Elsewhere
+    // the ACLs are user intent (`REVOKE ... FROM PUBLIC`) and dropping them widens access.
+    let no_acl = is_instance_datatable_source(&db, &w_id, &req.target).await?
+        || is_instance_datatable_source(&db, &w_id, &req.source).await?;
+
+    let dump_file = pg_dump_database(
+        &source_pg,
+        PgDumpOptions { schema_only, no_owner: true, no_acl, ..Default::default() },
+    )
+    .await?;
     pg_import_dump(&target_pg, &dump_file).await?;
 
     Ok(format!(
@@ -3174,7 +3469,11 @@ async fn export_pg_schema(
     Json(req): Json<ExportPgSchemaRequest>,
 ) -> Result<String> {
     let pg = resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
-    let dump_file = pg_dump_database(&pg, true, &[]).await?;
+    let dump_file = pg_dump_database(
+        &pg,
+        PgDumpOptions { schema_only: true, ..Default::default() },
+    )
+    .await?;
     tokio::fs::read_to_string(&dump_file.path)
         .await
         .map_err(|e| Error::internal_err(format!("Failed to read dump file: {}", e)))
@@ -7296,6 +7595,76 @@ async fn enforce_cloud_fork_cap(db: &DB, parent_workspace_id: &str) -> Result<()
     enforce_cloud_fork_count(db, &root, 1).await
 }
 
+/// Cloud: refuse to attach a workspace that already has a paid plan of its own.
+///
+/// Once attached it draws the root's plan and meters its usage there, so a subscription of its own
+/// bills a second time for one plan. Only an attach can reach this state: a fork is created as a
+/// fresh workspace and never had a plan to keep.
+///
+/// Asked only of a candidate joining this family, never of one already under the same root: that
+/// one is already in the double-billed state, where the settings page surfaces the leftover
+/// subscription and the portal that cancels it, and refusing there would block re-designating a
+/// renamed dev workspace over a billing problem the attach did not cause.
+#[cfg(feature = "cloud")]
+async fn reject_attach_of_subscribed_workspace(db: &DB, dev_w_id: &str) -> Result<()> {
+    let plan = sqlx::query_scalar!(
+        "SELECT plan FROM workspace_settings WHERE workspace_id = $1",
+        dev_w_id
+    )
+    .fetch_optional(db)
+    .await?
+    .flatten();
+    // Any plan, not just `'team'`: the column is written by the subscription webhook, and a plan
+    // value it does not write yet would otherwise walk straight past this. An enterprise
+    // arrangement is deliberately not covered — it sets `premium` without a plan and has no
+    // self-serve portal, so refusing there would be a dead end rather than something to act on.
+    if plan.is_some() {
+        return Err(Error::BadRequest(format!(
+            "Workspace {dev_w_id} is on a paid plan of its own. A dev or fork workspace runs on its parent's plan and is never invoiced separately, so cancel that subscription from its own billing settings before attaching it."
+        )));
+    }
+    Ok(())
+}
+
+#[cfg(all(test, feature = "cloud"))]
+mod attach_billing_guard_tests {
+    use super::reject_attach_of_subscribed_workspace;
+    use sqlx::{Pool, Postgres};
+
+    async fn workspace_on_plan(db: &Pool<Postgres>, id: &str, plan: Option<&str>) {
+        sqlx::query("INSERT INTO workspace (id, name, owner) VALUES ($1, $1, 'test-user')")
+            .bind(id)
+            .execute(db)
+            .await
+            .expect("insert workspace");
+        sqlx::query("INSERT INTO workspace_settings (workspace_id, plan) VALUES ($1, $2)")
+            .bind(id)
+            .bind(plan)
+            .execute(db)
+            .await
+            .expect("insert workspace_settings");
+    }
+
+    #[sqlx::test(migrations = "../migrations")]
+    async fn refuses_a_candidate_that_still_pays_for_itself(db: Pool<Postgres>) {
+        workspace_on_plan(&db, "subscribed", Some("team")).await;
+        workspace_on_plan(&db, "cancelled", None).await;
+
+        let err = reject_attach_of_subscribed_workspace(&db, "subscribed")
+            .await
+            .expect_err("a workspace on a paid plan of its own must not be attachable");
+        assert!(err.to_string().contains("paid plan of its own"), "{err}");
+
+        // Cancelling clears `plan` but keeps `customer_id`, so the plan column is what decides.
+        reject_attach_of_subscribed_workspace(&db, "cancelled")
+            .await
+            .expect("a workspace with no plan is attachable");
+        reject_attach_of_subscribed_workspace(&db, "no-settings-row")
+            .await
+            .expect("a workspace with no settings row is attachable");
+    }
+}
+
 /// General guardrail (all builds): reject creating a fork/dev under `parent` when it would nest deeper
 /// than `MAX_FORK_DEPTH`. `added_subtree_height` is the height of the subtree grafted below the new
 /// node — 0 for a plain fork, or the candidate's own subtree height for an attach.
@@ -7864,6 +8233,17 @@ async fn attach_dev_workspace(
         )));
     }
 
+    // Deliberately below the admin-of-candidate check, unlike the cap enforcement above: the
+    // refusal names the candidate's plan, so running it earlier would tell any admin of any
+    // premium workspace whether an arbitrary workspace id is on a team plan.
+    #[cfg(feature = "cloud")]
+    if *CLOUD_HOSTED {
+        let root = windmill_common::workspaces::get_billing_workspace_id(&db, &prod_w_id).await?;
+        if windmill_common::workspaces::get_billing_workspace_id(&db, &dev_w_id).await? != root {
+            reject_attach_of_subscribed_workspace(&db, &dev_w_id).await?;
+        }
+    }
+
     let mut tx = db.begin().await?;
     // Everything above ran outside a transaction, so prod's eligibility and the chain's labels could
     // have changed under us: re-decide both here, under the pairing lock.

backend/windmill-api/Cargo.toml

@@ -10,7 +10,7 @@ path = "src/lib.rs"
 
 [features]
 default = []
-private = ["windmill-audit/private", "windmill-common/private", "windmill-api-auth/private", "windmill-store/private", "windmill-api-users/private", "windmill-api-workspaces/private", "windmill-api-groups/private", "windmill-api-configs/private", "windmill-api-settings/private", "windmill-api-assets/private", "windmill-api-agent-workers?/private", "windmill-trigger-kafka?/private", "windmill-trigger-postgres?/private", "windmill-trigger-mqtt?/private", "windmill-trigger-amqp?/private", "windmill-trigger-websocket?/private", "windmill-trigger-nats?/private", "windmill-trigger-sqs?/private", "windmill-trigger-gcp?/private", "windmill-trigger-azure?/private", "windmill-trigger-email?/private", "windmill-git-sync/private", "windmill-autoscaling?/private", "windmill-object-store/private", "windmill-api-npm-proxy/private"]
+private = ["windmill-audit/private", "windmill-common/private", "windmill-api-auth/private", "windmill-store/private", "windmill-api-users/private", "windmill-api-workspaces/private", "windmill-api-groups/private", "windmill-api-configs/private", "windmill-api-settings/private", "windmill-api-assets/private", "windmill-api-agent-workers?/private", "windmill-trigger-kafka?/private", "windmill-trigger-postgres?/private", "windmill-trigger-mqtt?/private", "windmill-trigger-amqp?/private", "windmill-trigger-websocket?/private", "windmill-trigger-nats?/private", "windmill-trigger-sqs?/private", "windmill-trigger-gcp?/private", "windmill-trigger-azure?/private", "windmill-trigger-email?/private", "windmill-git-sync/private", "windmill-autoscaling?/private", "windmill-object-store/private", "windmill-api-npm-proxy/private", "windmill-indexer?/private"]
 enterprise = ["windmill-queue/enterprise", "windmill-audit/enterprise", "windmill-git-sync/enterprise", "windmill-common/enterprise", "windmill-worker?/enterprise", "windmill-api-auth/enterprise", "windmill-store/enterprise", "windmill-api-jobs/enterprise", "windmill-api-scripts/enterprise", "windmill-api-flows/enterprise", "windmill-api-users/enterprise", "windmill-api-workspaces/enterprise", "windmill-api-groups/enterprise", "windmill-api-configs/enterprise", "windmill-api-settings/enterprise", "windmill-api-schedule/enterprise", "windmill-api-debug/enterprise", "windmill-api-agent-workers?/enterprise", "windmill-trigger/enterprise", "windmill-trigger-kafka?/enterprise", "windmill-trigger-postgres?/enterprise", "windmill-trigger-mqtt?/enterprise", "windmill-trigger-amqp?/enterprise", "windmill-trigger-websocket?/enterprise", "windmill-trigger-email?/enterprise", "windmill-trigger-nats?/enterprise", "windmill-trigger-sqs?/enterprise", "windmill-trigger-gcp?/enterprise", "windmill-trigger-azure?/enterprise", "windmill-trigger-http?/enterprise", "windmill-native-triggers?/enterprise", "dep:windmill-autoscaling", "windmill-autoscaling/enterprise", "windmill-api-npm-proxy/enterprise", "license"]
 stripe = []
 run_inline = ["dep:windmill-worker", "windmill-api-configs/run_inline"]
@@ -18,10 +18,12 @@ agent_worker_server = ["dep:windmill-worker", "dep:windmill-api-agent-workers"]
 enterprise_saml = ["dep:samael", "dep:libxml"]
 benchmark = []
 embedding = ["windmill-api-embeddings/embedding"]
-parquet = ["dep:datafusion", "windmill-common/parquet", "windmill-object-store/parquet", "windmill-worker?/parquet", "windmill-api-users/parquet", "windmill-api-settings/parquet", "windmill-api-workspaces/parquet", "windmill-api-npm-proxy/parquet", "dep:aws-sigv4", "dep:aws-sdk-config", "dep:quick-xml"]
+parquet = ["dep:datafusion", "windmill-common/parquet", "windmill-object-store/parquet", "windmill-worker?/parquet", "windmill-indexer?/parquet", "windmill-api-users/parquet", "windmill-api-settings/parquet", "windmill-api-workspaces/parquet", "windmill-api-npm-proxy/parquet", "dep:aws-sigv4", "dep:aws-sdk-config", "dep:quick-xml"]
 prometheus = ["windmill-common/prometheus", "windmill-queue/prometheus", "dep:prometheus", "windmill-worker?/prometheus", "windmill-api-scripts/prometheus"]
 openidconnect = ["dep:openidconnect", "windmill-common/openidconnect", "windmill-store/openidconnect"]
-tantivy = ["dep:windmill-indexer"]
+# The service log search handler reads the columnar store, so tantivy alone is
+# not enough for this crate to build on its own.
+tantivy = ["dep:windmill-indexer", "parquet"]
 kafka = ["dep:windmill-trigger-kafka", "windmill-store/kafka"]
 kafka-gssapi = ["kafka", "windmill-trigger-kafka/kafka-gssapi"]
 nats = ["dep:windmill-trigger-nats", "windmill-store/nats"]

backend/windmill-api/openapi.yaml

@@ -1,7 +1,7 @@
 openapi: "3.0.3"
 
 info:
-  version: 1.796.0
+  version: 1.800.0
   title: Windmill API
 
   contact:
@@ -3788,6 +3788,38 @@ paths:
               schema:
                 type: boolean
 
+  /w/{workspace}/workspaces/billable_seats:
+    get:
+      summary: get the billable seats of the workspace the plan is billed on
+      operationId: getBillableSeats
+      tags:
+        - workspace
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+
+      responses:
+        "200":
+          description: billable seats
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  developers:
+                    type: integer
+                    description: >-
+                      Omitted when the seats counted are another workspace's, as they are for a
+                      fork resolving to its billing root.
+                  operators:
+                    type: integer
+                    description: >-
+                      Omitted when the seats counted are another workspace's, as they are for a
+                      fork resolving to its billing root.
+                  seats:
+                    type: integer
+                required:
+                  - seats
+
   /w/{workspace}/workspaces/premium_info:
     get:
       summary: get premium info
@@ -24085,7 +24117,7 @@ paths:
                     items:
                       type: string
                   hits:
-                    description: log files that matched the query
+                    description: the log lines that matched the query, newest first
                     type: array
                     items:
                       $ref: "#/components/schemas/LogSearchHit"
@@ -25514,6 +25546,68 @@ paths:
               schema:
                 type: string
 
+  /w/{workspace}/hub/projects/{slug}/withdraw:
+    post:
+      summary: take a hub project submission back out of review
+      description: |
+        Requires the caller to be a workspace admin. Forwards the request to the
+        configured Hub scoped to the `{workspace}:{folder}` source and returns
+        the Hub's status code and raw response body. Everything pushed for the
+        submission is kept, so it can be fixed and submitted again.
+      operationId: withdrawHubProject
+      tags:
+        - hubPublish
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+        - name: slug
+          in: path
+          required: true
+          description: hub project slug (3-50 chars, lowercase alphanumeric and hyphens, no leading/trailing hyphen)
+          schema:
+            type: string
+            minLength: 3
+            maxLength: 50
+            pattern: "^[a-z0-9][a-z0-9-]*[a-z0-9]$"
+        - $ref: "#/components/parameters/HubPublishFolder"
+      responses:
+        "200":
+          description: raw Hub response body (status code is passed through from the Hub)
+          content:
+            text/plain:
+              schema:
+                type: string
+
+  /w/{workspace}/hub/projects/{slug}/discard_update:
+    post:
+      summary: discard the pending update to a published hub project
+      description: |
+        Requires the caller to be a workspace admin. Forwards the request to the
+        configured Hub scoped to the `{workspace}:{folder}` source and returns
+        the Hub's status code and raw response body. The published project is
+        left untouched.
+      operationId: discardHubProjectUpdate
+      tags:
+        - hubPublish
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+        - name: slug
+          in: path
+          required: true
+          description: hub project slug (3-50 chars, lowercase alphanumeric and hyphens, no leading/trailing hyphen)
+          schema:
+            type: string
+            minLength: 3
+            maxLength: 50
+            pattern: "^[a-z0-9][a-z0-9-]*[a-z0-9]$"
+        - $ref: "#/components/parameters/HubPublishFolder"
+      responses:
+        "200":
+          description: raw Hub response body (status code is passed through from the Hub)
+          content:
+            text/plain:
+              schema:
+                type: string
+
   /w/{workspace}/hub/project:
     get:
       summary: get the hub project linked to a workspace folder
@@ -27389,11 +27483,29 @@ components:
             type: integer
             minimum: 1
             maximum: 2000000
+        free_tier:
+          $ref: "#/components/schemas/FreeTierInfo"
         model_pricing:
           type: object
           additionalProperties:
             $ref: "#/components/schemas/ModelPriceOverride"
 
+    FreeTierInfo:
+      type: object
+      description: >-
+        Read-only. Present when the workspace has no AI provider of its own and is running
+        on Windmill's free tier. Ignored on write.
+      properties:
+        exhausted:
+          type: boolean
+          description: The one-time grant is spent; no provider is served and the user must add their own API key.
+        used_ratio:
+          type: number
+          description: Fraction of the grant consumed, 0 to 1.
+      required:
+        - exhausted
+        - used_ratio
+
     ModelPriceOverride:
       type: object
       description: negotiated rates in USD per million tokens, keyed `provider:model`
@@ -28155,7 +28267,10 @@ components:
           type: string
         args:
           $ref: "#/components/schemas/ScriptArgs"
-        result: {}
+        result:
+          description: |
+            For large results, this may be the placeholder string 'WINDMILL_TOO_BIG'.
+            Use the completed job result endpoint to retrieve the full result.
         logs:
           type: string
         deleted:
@@ -30429,7 +30544,9 @@ components:
           description: If true, passes the request body as a raw string instead of parsing as JSON
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -30518,7 +30635,9 @@ components:
           description: If true, passes the request body as a raw string instead of parsing as JSON
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -30614,7 +30733,9 @@ components:
           description: If true, passes the request body as a raw string instead of parsing as JSON
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -30806,7 +30927,9 @@ components:
           description: Optional periodic heartbeat message configuration
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           description: Arguments to pass to the error handler
           $ref: "#/components/schemas/ScriptArgs"
@@ -30871,7 +30994,9 @@ components:
           description: Optional periodic heartbeat message configuration
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           description: Arguments to pass to the error handler
           $ref: "#/components/schemas/ScriptArgs"
@@ -30947,7 +31072,9 @@ components:
           description: Optional periodic heartbeat message configuration
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           description: Arguments to pass to the error handler
           $ref: "#/components/schemas/ScriptArgs"
@@ -31075,7 +31202,9 @@ components:
           description: Last error message if the trigger failed
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -31126,7 +31255,9 @@ components:
           $ref: "#/components/schemas/TriggerMode"
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -31190,7 +31321,9 @@ components:
           $ref: "#/components/schemas/TriggerMode"
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -31273,7 +31406,9 @@ components:
           description: Last error message if the trigger failed
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -31314,7 +31449,9 @@ components:
           $ref: "#/components/schemas/TriggerMode"
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -31368,7 +31505,9 @@ components:
           $ref: "#/components/schemas/TriggerMode"
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -31766,7 +31905,9 @@ components:
           description: Last error message if the trigger failed
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -31868,7 +32009,9 @@ components:
           $ref: "#/components/schemas/TriggerMode"
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -31924,7 +32067,9 @@ components:
           $ref: "#/components/schemas/TriggerMode"
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -32052,7 +32197,9 @@ components:
           description: Timestamp of last server heartbeat (internal)
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -32092,7 +32239,9 @@ components:
           description: Configuration for creating/managing the publication (tables, operations)
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -32144,7 +32293,9 @@ components:
           description: Configuration for creating/managing the publication (tables, operations)
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -32221,7 +32372,9 @@ components:
           description: Last error message if the trigger failed
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -32285,7 +32438,9 @@ components:
           $ref: "#/components/schemas/TriggerMode"
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -32360,7 +32515,9 @@ components:
           description: True if script_path points to a flow, false if it points to a script
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -32423,7 +32580,9 @@ components:
           description: Last error message if the trigger failed
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -32471,7 +32630,9 @@ components:
           $ref: "#/components/schemas/TriggerMode"
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -32530,7 +32691,9 @@ components:
           description: True if script_path points to a flow, false if it points to a script
         error_handler_path:
           type: string
-          description: Path to a script or flow to run when the triggered job fails
+          description: Path to a script to run when the triggered job fails. A bare
+            path, without the script/ or flow/ prefix a schedule error handler takes;
+            it cannot be a flow.
         error_handler_args:
           $ref: "#/components/schemas/ScriptArgs"
           description: Arguments to pass to the error handler
@@ -34580,8 +34743,34 @@ components:
     LogSearchHit:
       type: object
       properties:
-        dancer:
+        ts:
+          description: timestamp of the log line itself, not of the file containing it
           type: string
+          format: date-time
+        host:
+          type: string
+        level:
+          type: string
+          enum: [TRACE, DEBUG, INFO, WARN, ERROR]
+        target:
+          description: the tracing target that emitted the line
+          type: string
+          nullable: true
+        message:
+          type: string
+        file_path:
+          description: the log file the line came from
+          type: string
+        line_no:
+          description: offset of the line within its file
+          type: integer
+      required:
+        - ts
+        - host
+        - level
+        - message
+        - file_path
+        - line_no
 
     AutoscalingEvent:
       type: object

backend/windmill-api/src/ai.rs

@@ -409,6 +409,19 @@ impl ExpiringProviderCredentials {
     }
 }
 
+/// Set on the copilot config when the workspace has no AI provider of its own and is
+/// running on Windmill's free tier, so the client can label the lent model as free, warn
+/// before the grant runs out, and tell the user to add their own key once it has — rather
+/// than showing the same "no provider configured" state a never-configured workspace gets.
+#[derive(Serialize, Deserialize, Debug, Default, Clone)]
+pub struct FreeTierInfo {
+    /// The grant is spent: no provider is served and the user must bring their own key.
+    pub exhausted: bool,
+    /// Fraction of the grant consumed, 0.0..=1.0. A ratio, not a dollar amount — the
+    /// pricing model stays server-side.
+    pub used_ratio: f64,
+}
+
 #[derive(Serialize, Deserialize, Debug, Default)]
 pub struct AIConfig {
     #[serde(skip_serializing_if = "Option::is_none")]
@@ -423,6 +436,11 @@ pub struct AIConfig {
     pub custom_prompts: Option<HashMap<String, String>>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub max_tokens_per_model: Option<HashMap<String, i32>>,
+    /// Response-only: this same struct is the request body for saving a workspace's AI
+    /// config, and `skip_deserializing` is what stops a client from storing a forged
+    /// free-tier marker. Only the server sets it, per-request.
+    #[serde(skip_serializing_if = "Option::is_none", skip_deserializing)]
+    pub free_tier: Option<FreeTierInfo>,
     /// Per-model price overrides, keyed `provider:model` like `max_tokens_per_model`.
     /// Only models whose rates differ from the built-in table are stored.
     #[serde(skip_serializing_if = "Option::is_none")]
@@ -1013,83 +1031,119 @@ async fn proxy(
         check_scopes(&authed, || format!("resources:read:{}", resource_path))?;
     }
 
-    let mut credentials = match workspace_cache {
-        Some(request_cache) if !request_cache.is_expired() && forced_resource_path.is_none() => {
-            request_cache.credentials
-        }
-        _ => {
-            let (resource_path, save_to_cache, resource_workspace, instance_ai_config_revision) =
-                if let Some(resource_path) = forced_resource_path {
-                    // forced resource path
-                    (resource_path, false, w_id.clone(), None)
-                } else {
-                    let workspace_ai_config = sqlx::query_scalar!(
-                        "SELECT ai_config FROM workspace_settings WHERE workspace_id = $1",
-                        &w_id
-                    )
-                    .fetch_one(&db)
-                    .await?;
+    // Set when serving the request through Windmill's free AI tier (the lent key). Holds
+    // the per-user concurrency lock and drives response metering.
+    let mut free_lease: Option<crate::ai_free_tier_oss::FreeTierLease> = None;
+    let mut credentials = 'cred: {
+        match workspace_cache {
+            Some(request_cache)
+                if !request_cache.is_expired() && forced_resource_path.is_none() =>
+            {
+                request_cache.credentials
+            }
+            _ => {
+                let (resource_path, save_to_cache, resource_workspace, instance_ai_config_revision) =
+                    if let Some(resource_path) = forced_resource_path {
+                        // forced resource path
+                        (resource_path, false, w_id.clone(), None)
+                    } else {
+                        let workspace_ai_config = sqlx::query_scalar!(
+                            "SELECT ai_config FROM workspace_settings WHERE workspace_id = $1",
+                            &w_id
+                        )
+                        .fetch_one(&db)
+                        .await?;
 
-                    let (ai_config_value, resource_workspace, instance_ai_config_revision) = {
-                        let ws_has_config = workspace_ai_config
-                            .as_ref()
-                            .and_then(|v| serde_json::from_value::<AIConfig>(v.clone()).ok())
-                            .is_some_and(|config| config.has_providers());
+                        let (ai_config_value, resource_workspace, instance_ai_config_revision) = {
+                            let ws_has_config = workspace_ai_config
+                                .as_ref()
+                                .and_then(|v| serde_json::from_value::<AIConfig>(v.clone()).ok())
+                                .is_some_and(|config| config.has_providers());
 
-                        if ws_has_config {
-                            (workspace_ai_config.unwrap(), w_id.clone(), None)
-                        } else {
-                            let instance_config = sqlx::query_scalar!(
-                                "SELECT value FROM global_settings WHERE name = 'ai_config'"
-                            )
-                            .fetch_optional(&db)
-                            .await?;
+                            if ws_has_config {
+                                (workspace_ai_config.unwrap(), w_id.clone(), None)
+                            } else {
+                                let instance_config = sqlx::query_scalar!(
+                                    "SELECT value FROM global_settings WHERE name = 'ai_config'"
+                                )
+                                .fetch_optional(&db)
+                                .await?;
 
-                            match instance_config {
-                                Some(config) => (
-                                    config,
-                                    "admins".to_string(),
-                                    Some(current_instance_ai_config_revision()),
-                                ),
-                                None => {
-                                    return Err(Error::internal_err(
-                                        "AI resource not configured".to_string(),
-                                    ));
+                                let instance_has_config =
+                                    instance_config.as_ref().is_some_and(|v| {
+                                        serde_json::from_value::<AIConfig>(v.clone())
+                                            .ok()
+                                            .is_some_and(|c| c.has_providers())
+                                    });
+                                match instance_config {
+                                    // An instance `ai_config` row with no usable provider (e.g. `{}`
+                                    // or `{"providers":{}}`) is treated as unconfigured, exactly as
+                                    // build_copilot_settings_state does — otherwise its mere presence
+                                    // would suppress the free-tier fallback below.
+                                    Some(config) if instance_has_config => (
+                                        config,
+                                        "admins".to_string(),
+                                        Some(current_instance_ai_config_revision()),
+                                    ),
+                                    _ => {
+                                        // Nothing configured: fall back to Windmill's free AI tier
+                                        // (EE-only) if a lent key is set and both the user's
+                                        // one-time grant and the instance's daily cap have room.
+                                        // Errors once the grant is spent, the day is capped, or the
+                                        // user already has a request in flight; None otherwise.
+                                        // Ineligible identities (e.g. service accounts) are refused
+                                        // inside the helper, so every path treats them alike.
+                                        let free =
+                                            crate::ai_free_tier_oss::resolve_free_tier_credentials(
+                                                &provider,
+                                                &db,
+                                                &ai_path,
+                                                &authed.email,
+                                                &body,
+                                            )
+                                            .await?;
+                                        if let Some((free_credentials, lease)) = free {
+                                            free_lease = Some(lease);
+                                            break 'cred free_credentials;
+                                        }
+                                        return Err(Error::internal_err(
+                                            "AI resource not configured".to_string(),
+                                        ));
+                                    }
                                 }
                             }
+                        };
+
+                        let mut ai_config = serde_json::from_value::<AIConfig>(ai_config_value)
+                            .map_err(|e| Error::BadRequest(e.to_string()))?;
+
+                        let provider_config = ai_config
+                            .providers
+                            .as_mut()
+                            .and_then(|providers| providers.remove(&provider))
+                            .ok_or_else(|| {
+                                Error::BadRequest(format!("Provider {:?} not configured", provider))
+                            })?;
+
+                        if provider_config.resource_path.is_empty() {
+                            return Err(Error::BadRequest("Resource path is empty".to_string()));
                         }
+
+                        (
+                            provider_config.resource_path,
+                            true,
+                            resource_workspace,
+                            instance_ai_config_revision,
+                        )
                     };
 
-                    let mut ai_config = serde_json::from_value::<AIConfig>(ai_config_value)
-                        .map_err(|e| Error::BadRequest(e.to_string()))?;
-
-                    let provider_config = ai_config
-                        .providers
-                        .as_mut()
-                        .and_then(|providers| providers.remove(&provider))
-                        .ok_or_else(|| {
-                            Error::BadRequest(format!("Provider {:?} not configured", provider))
-                        })?;
-
-                    if provider_config.resource_path.is_empty() {
-                        return Err(Error::BadRequest("Resource path is empty".to_string()));
-                    }
-
-                    (
-                        provider_config.resource_path,
-                        true,
-                        resource_workspace,
-                        instance_ai_config_revision,
-                    )
-                };
-
-            // For user-specified resources, fetch through an RLS-scoped
-            // connection so PostgreSQL row-level security enforces the same
-            // folder/group boundaries as the regular resource API. For the
-            // workspace/instance ai_config path, the resource_path was already
-            // validated by an admin/devops user when configuring the workspace,
-            // so the raw pool is used.
-            let resource = if is_user_specified_resource {
+                // For user-specified resources, fetch through an RLS-scoped
+                // connection so PostgreSQL row-level security enforces the same
+                // folder/group boundaries as the regular resource API. For the
+                // workspace/instance ai_config path, the resource_path was already
+                // validated by an admin/devops user when configuring the workspace,
+                // so the raw pool is used.
+                let resource = if is_user_specified_resource {
                 let mut tx = user_db.clone().begin(&authed).await?;
                 let res = sqlx::query_scalar::<_, Option<sqlx::types::Json<Box<RawValue>>>>(
                     "SELECT value FROM resource WHERE path = $1 AND workspace_id = $2",
@@ -1112,38 +1166,45 @@ async fn proxy(
             .ok_or_else(|| Error::NotFound(format!("Could not find the resource {}, update the resource path in the workspace settings", resource_path)))?
             .ok_or_else(|| Error::BadRequest(format!("Empty resource value for {}", resource_path)))?;
 
-            let resource = serde_json::from_str::<AIResource>(resource.0.get())
-                .map_err(|e| Error::BadRequest(e.to_string()))?;
+                let resource = serde_json::from_str::<AIResource>(resource.0.get())
+                    .map_err(|e| Error::BadRequest(e.to_string()))?;
 
-            // Enforce RLS on $var: resolution when the resource path was
-            // user-specified (X-Resource-Path header) so users can only read
-            // variables they have permission to access.
-            let enforce_authed = if is_user_specified_resource {
-                Some(&authed)
-            } else {
-                None
-            };
-            let credentials = resolve_provider_credentials(
-                &provider,
-                &db,
-                &resource_workspace,
-                resource,
-                enforce_authed,
-            )
-            .await?;
-            if save_to_cache {
-                AI_REQUEST_CACHE.insert(
-                    (w_id.clone(), provider.clone()),
-                    ExpiringProviderCredentials::new(
-                        credentials.clone(),
-                        instance_ai_config_revision,
-                    ),
-                );
+                // Enforce RLS on $var: resolution when the resource path was
+                // user-specified (X-Resource-Path header) so users can only read
+                // variables they have permission to access.
+                let enforce_authed = if is_user_specified_resource {
+                    Some(&authed)
+                } else {
+                    None
+                };
+                let credentials = resolve_provider_credentials(
+                    &provider,
+                    &db,
+                    &resource_workspace,
+                    resource,
+                    enforce_authed,
+                )
+                .await?;
+                if save_to_cache {
+                    AI_REQUEST_CACHE.insert(
+                        (w_id.clone(), provider.clone()),
+                        ExpiringProviderCredentials::new(
+                            credentials.clone(),
+                            instance_ai_config_revision,
+                        ),
+                    );
+                }
+                credentials
             }
-            credentials
         }
     };
 
+    // Free tier: pin the model and clamp max_tokens server-side before forwarding,
+    // since the request body is otherwise client-controlled.
+    if free_lease.is_some() {
+        body = crate::ai_free_tier_oss::enforce_free_tier_body(&body)?;
+    }
+
     if let Some(fim_transform) =
         maybe_transform_fim_request(&provider, &ai_path, &credentials.base_url, &body)?
     {
@@ -1291,8 +1352,32 @@ async fn proxy(
 
     let status_code = response.status();
     let headers = response.headers().clone();
+    let is_sse = is_sse_response(&headers);
+
+    // Free tier: reconcile the cost reserved up-front against what the response actually
+    // used, holding the per-user lock (via the lease) until it is recorded. The chat
+    // streams (SSE), where the usage report only arrives in the final chunk; the
+    // non-streaming JSON path is handled for completeness.
+    if let Some(lease) = free_lease {
+        let body = if is_sse {
+            axum::body::Body::from_stream(inject_keepalives(
+                Box::pin(crate::ai_free_tier_oss::meter_usage(
+                    response.bytes_stream(),
+                    db.clone(),
+                    lease,
+                )),
+                Duration::from_secs(KEEPALIVE_INTERVAL_SECS),
+            ))
+        } else {
+            let bytes = response.bytes().await.map_err(to_anyhow)?;
+            crate::ai_free_tier_oss::record_json_usage(db.clone(), lease, &bytes);
+            axum::body::Body::from(bytes)
+        };
+        return Ok((status_code, headers, body));
+    }
+
     let stream = response.bytes_stream();
-    let body = if is_sse_response(&headers) {
+    let body = if is_sse {
         axum::body::Body::from_stream(inject_keepalives(
             stream,
             Duration::from_secs(KEEPALIVE_INTERVAL_SECS),

backend/windmill-api/src/ai_evals/datasets.rs

@@ -190,6 +190,8 @@ pub async fn create_dataset(
     }
     tx.commit().await?;
 
+    windmill_common::feature_usage::log_feature_usage("ai_agent_eval", "dataset_created", "");
+
     Ok(format!("Created eval dataset {}", payload.path))
 }
 

backend/windmill-api/src/ai_evals/run.rs

@@ -707,6 +707,17 @@ pub async fn run_experiment(
             .await?;
         return Err(e);
     }
+    // Which state of the agent was measured is the whole key vocabulary: it is what separates
+    // running what is deployed from measuring edits or an older version.
+    windmill_common::feature_usage::log_feature_usage(
+        "ai_agent_eval",
+        "run",
+        match subject.kind {
+            EvalSubjectKind::Agent => "agent",
+            EvalSubjectKind::AgentDraft => "agent_draft",
+            EvalSubjectKind::AgentVersion => "agent_version",
+        },
+    );
     Ok(experiment_id.to_string())
 }
 

backend/windmill-api/src/ai_free_tier_oss.rs

@@ -0,0 +1,65 @@
+#[cfg(feature = "private")]
+#[allow(unused)]
+pub use crate::ai_free_tier_ee::*;
+
+// Open-source build: Windmill's free AI tier does not exist. These stubs make the
+// callers in `ai.rs` / `workspaces.rs` compile while disabling the feature entirely —
+// `resolve_free_tier_credentials` never opts in, so the proxy falls through to its
+// normal "AI resource not configured" path and the copilot stays hidden.
+//
+// Caller contract (enforced by the private impl, restated here for parity): the `email`
+// passed to `resolve_free_tier_credentials` / `free_tier_copilot_config` MUST be the
+// authenticated caller's own identity (an `ApiAuthed` email), never a client-supplied one —
+// it selects whose lent-key grant is spent and whose usage is read.
+
+#[cfg(not(feature = "private"))]
+use crate::ai::AIConfig;
+#[cfg(not(feature = "private"))]
+use crate::db::DB;
+#[cfg(not(feature = "private"))]
+use axum::body::Bytes;
+#[cfg(not(feature = "private"))]
+use windmill_ai::ai_providers::AIProvider;
+#[cfg(not(feature = "private"))]
+use windmill_ai::credentials::ProviderCredentials;
+#[cfg(not(feature = "private"))]
+use windmill_common::error::Result;
+
+#[cfg(not(feature = "private"))]
+pub struct FreeTierLease;
+
+#[cfg(not(feature = "private"))]
+pub async fn resolve_free_tier_credentials(
+    _provider: &AIProvider,
+    _db: &DB,
+    _ai_path: &str,
+    _email: &str,
+    _body: &Bytes,
+) -> Result<Option<(ProviderCredentials, FreeTierLease)>> {
+    Ok(None)
+}
+
+#[cfg(not(feature = "private"))]
+pub fn enforce_free_tier_body(body: &Bytes) -> Result<Bytes> {
+    Ok(body.clone())
+}
+
+#[cfg(not(feature = "private"))]
+pub async fn free_tier_copilot_config(_db: &DB, _email: &str) -> Result<Option<AIConfig>> {
+    Ok(None)
+}
+
+#[cfg(not(feature = "private"))]
+pub fn record_json_usage(_db: DB, _lease: FreeTierLease, _bytes: &[u8]) {}
+
+#[cfg(not(feature = "private"))]
+pub fn meter_usage<S>(
+    upstream: S,
+    _db: DB,
+    _lease: FreeTierLease,
+) -> impl futures::Stream<Item = std::result::Result<Bytes, reqwest::Error>>
+where
+    S: futures::Stream<Item = std::result::Result<Bytes, reqwest::Error>> + Unpin,
+{
+    upstream
+}

backend/windmill-api/src/db.rs

@@ -102,6 +102,12 @@ lazy_static::lazy_static! {
                     (20260727151319, include_str!(
                         "../../migrations/20260727151319_draft_only_listing_indexes.up.sql"
                     ).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY")),
+                    (20260826202939, include_str!(
+                        "../../migrations/20260826202939_queue_suspended_resume_at_index.up.sql"
+                    ).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY").replace("DROP INDEX", "DROP INDEX CONCURRENTLY")),
+                    (20260826214706, include_str!(
+                        "../../migrations/20260826214706_queue_suspended_drop_legacy_index.up.sql"
+                    ).replace("DROP INDEX", "DROP INDEX CONCURRENTLY")),
                     ].into_iter().collect();
 }
 
@@ -228,6 +234,8 @@ impl Migrate for CustomMigrator {
                     // CONCURRENTLY operations cannot run inside a transaction block
                     // or a multi-statement query (PostgreSQL requires top-level execution).
                     // Split into individual statements and execute each separately.
+                    // The split is naive, so a `;` anywhere in an overridden migration —
+                    // inside a comment or a string literal included — splits mid-statement.
                     for stmt in migration_sql.split(';') {
                         let stmt = stmt.trim();
                         if !stmt.is_empty()

backend/windmill-api/src/hub_publish.rs

@@ -42,6 +42,11 @@ pub fn workspaced_service() -> Router {
         .route("/migrations", post(publish_migrations))
         .route("/projects/{slug}/export", get(get_project_export))
         .route("/projects/{slug}/submit", post(submit_project))
+        .route("/projects/{slug}/withdraw", post(withdraw_project))
+        .route(
+            "/projects/{slug}/discard_update",
+            post(discard_project_update),
+        )
         .route("/project", get(get_project_by_source))
 }
 
@@ -554,9 +559,35 @@ async fn submit_project(
     .await
 }
 
-// The Hub has no auth of its own: it validates bearer tokens by calling this
-// instance's /api/users/whoami. Forwarding the caller's own token logs them in
-// on the Hub as themselves (account auto-created on first use).
+// Take a submission back out of review, keeping what was pushed for it.
+async fn withdraw_project(
+    ctx: HubPublishCtx,
+    Path((_workspace, slug)): Path<(String, ProjectSlug)>,
+) -> Result<impl IntoResponse, Error> {
+    ctx.post(
+        &format!("/projects/{}/withdraw", slug),
+        &serde_json::json!({}),
+    )
+    .await
+}
+
+// Throw away the pending update to an already-published project. The published
+// version is untouched — it never saw the update.
+async fn discard_project_update(
+    ctx: HubPublishCtx,
+    Path((_workspace, slug)): Path<(String, ProjectSlug)>,
+) -> Result<impl IntoResponse, Error> {
+    ctx.post(
+        &format!("/projects/{}/discard_update", slug),
+        &serde_json::json!({}),
+    )
+    .await
+}
+
+// The Hub has no auth of its own: it validates bearer tokens by calling
+// /api/users/whoami — on app.windmill.dev for the public Hub, on the paired
+// instance for a private one. Forwarding the caller's own token logs them in on
+// the Hub as themselves (account auto-created on first use).
 async fn get_from_hub(
     path: &str,
     source_id: &str,

backend/windmill-api/src/lib.rs

@@ -66,6 +66,9 @@ use crate::scim_oss::has_scim_token;
 use windmill_common::error::AppError;
 
 mod ai;
+#[cfg(feature = "private")]
+mod ai_free_tier_ee;
+mod ai_free_tier_oss;
 mod ai_skills;
 mod apps;
 mod apps_raw_bundle;

backend/windmill-api/src/service_logs.rs

@@ -88,6 +88,66 @@ async fn list_files(
     Ok(Json(rows))
 }
 
+/// Rebuild one source log file from the columnar store.
+///
+/// Not the original bytes: the store holds a line's fields rather than its text,
+/// so the JSON is re-serialized here and key order and whitespace are this
+/// writer's. Everything a reader can see survives — the drawer this feeds
+/// renders a prettified view of each line either way, and a line that was never
+/// JSON comes back exactly as it was written.
+#[cfg(all(feature = "tantivy", feature = "private"))]
+async fn get_log_file_from_store(
+    db: &DB,
+    store: &windmill_indexer::service_logs_store_ee::Store,
+    path: &str,
+) -> windmill_common::error::Result<Response> {
+    let (hostname, file_name) = path
+        .split_once('/')
+        .ok_or_else(|| Error::BadRequest("Invalid path".to_string()))?;
+
+    // The store is partitioned by day and mode, neither of which the path
+    // carries. `log_file` names both, and its primary key starts with hostname.
+    let file = sqlx::query!(
+        // `mode!` because the column is NOT NULL and only the cast makes sqlx
+        // think otherwise; a silent default would look up a `mode=` partition
+        // that matches nothing and read as a missing file.
+        "SELECT mode::text AS \"mode!\", log_ts FROM log_file WHERE hostname = $1 AND file_path = $2 ORDER BY log_ts DESC LIMIT 1",
+        hostname,
+        file_name
+    )
+    .fetch_optional(db)
+    .await?
+    .ok_or_else(|| Error::NotFound(format!("File {path} not found")))?;
+
+    // A row registered by this version carries the minute in the file's own name,
+    // so the two agree and the second is redundant. One written before the
+    // uploader derived `log_ts` from the name carries a wall clock instead, and
+    // those outlive an upgrade by the retention period — which is also what makes
+    // the `ORDER BY` above worth having. The name is authoritative, so both go.
+    let mut known_ts = vec![chrono::DateTime::from_naive_utc_and_offset(
+        file.log_ts,
+        chrono::Utc,
+    )];
+    if let Some(named) = file_name.rsplit('.').next().and_then(|s| {
+        chrono::NaiveDateTime::parse_from_str(s, windmill_common::tracing_init::LOG_TIMESTAMP_FMT)
+            .ok()
+    }) {
+        known_ts.push(chrono::DateTime::from_naive_utc_and_offset(
+            named,
+            chrono::Utc,
+        ));
+    }
+
+    let text = windmill_indexer::service_logs_store_ee::read_log_file(
+        store, &file.mode, hostname, file_name, &known_ts,
+    )
+    .await
+    .map_err(|e| Error::internal_err(format!("Error reading the service log store: {e}")))?
+    .ok_or_else(|| Error::NotFound(format!("File {path} not found")))?;
+
+    Ok(content_plain(Body::from(text)))
+}
+
 async fn get_log_file(
     authed: ApiAuthed,
     Extension(db): Extension<DB>,
@@ -104,27 +164,30 @@ async fn get_log_file(
     let s3_client = windmill_object_store::get_object_store().await;
     #[cfg(feature = "parquet")]
     if let Some(s3_client) = s3_client {
-        let path = format!("{}{}", windmill_common::tracing_init::LOGS_SERVICE, path);
-        let file = s3_client
+        use windmill_object_store::object_store_reexports::ObjectStoreError;
+
+        // The raw file, for as long as it is there. It outlives its ingestion by
+        // one indexer pass at most, so this covers the most recent minutes of a
+        // host's logs byte for byte; everything older is rebuilt from the store.
+        let object_path = format!("{}{}", windmill_common::tracing_init::LOGS_SERVICE, path);
+        match s3_client
             .get(&windmill_object_store::object_store_reexports::Path::from(
-                path,
+                object_path,
             ))
-            .await;
-        match file {
-            Ok(file) => {
-                let bytes = file.bytes().await;
-                match bytes {
-                    Ok(bytes) => {
-                        return Ok(content_plain(Body::from(bytes::Bytes::from(bytes))));
-                    }
-                    Err(e) => {
-                        return Err(Error::internal_err(format!(
-                            "Error pulling the bytes: {}",
-                            e
-                        )));
-                    }
+            .await
+        {
+            Ok(file) => match file.bytes().await {
+                Ok(bytes) => {
+                    return Ok(content_plain(Body::from(bytes::Bytes::from(bytes))));
                 }
-            }
+                Err(e) => {
+                    return Err(Error::internal_err(format!(
+                        "Error pulling the bytes: {}",
+                        e
+                    )));
+                }
+            },
+            Err(ObjectStoreError::NotFound { .. }) => {}
             Err(e) => {
                 return Err(Error::internal_err(format!(
                     "Error fetching the file: {}",
@@ -132,6 +195,11 @@ async fn get_log_file(
                 )));
             }
         }
+
+        #[cfg(all(feature = "tantivy", feature = "private"))]
+        return get_log_file_from_store(&db, &s3_client, &path).await;
+        #[cfg(not(all(feature = "tantivy", feature = "private")))]
+        return Err(Error::NotFound(format!("File {path} not found")));
     }
     let full_path = format!("{}{}", *TMP_WINDMILL_LOGS_SERVICE, path);
     // SECURITY (defense in depth): refuse to read through a symlink so a planted

backend/windmill-api/src/workspaces.rs

@@ -18,7 +18,6 @@ use crate::teams_oss::{
     connect_teams, edit_teams_command, run_teams_message_test_job,
     workspaces_list_available_teams_channels, workspaces_list_available_teams_ids,
 };
-
 use axum::{
     extract::{Extension, Path},
     routing::{get, post},
@@ -153,10 +152,23 @@ async fn edit_copilot_config(
             .await?;
     let settings_state =
         build_copilot_settings_state(workspace_has_config, instance_ai_config.as_ref());
+    // A provider-less instance config (e.g. `{}`) is unconfigured, same as build_copilot_settings_state
+    // treats it — so it must not shadow the free-tier fallback here either.
+    let instance_config_with_providers = instance_ai_config
+        .as_ref()
+        .and_then(|v| serde_json::from_value::<AIConfig>(v.clone()).ok())
+        .filter(|c| c.has_providers());
     let effective_ai_config = if workspace_has_config {
         ai_config
-    } else if let Some(instance_ai_config) = instance_ai_config {
-        serde_json::from_value::<AIConfig>(instance_ai_config).unwrap_or_default()
+    } else if let Some(instance_config) = instance_config_with_providers {
+        instance_config
+    } else if let Some(free_config) =
+        crate::ai_free_tier_oss::free_tier_copilot_config(&db, &authed.email).await?
+    {
+        // Same fallback as get_copilot_info: with nothing configured, surface Windmill's free
+        // tier (EE-only) so clearing a workspace provider activates it immediately, instead of
+        // returning an empty config that disables AI until the next page reload re-fetches it.
+        free_config
     } else {
         AIConfig::default()
     };
@@ -179,6 +191,7 @@ struct EditCopilotConfigResponse {
 }
 
 async fn get_copilot_info(
+    authed: ApiAuthed,
     Extension(db): Extension<DB>,
     Path(w_id): Path<String>,
 ) -> JsonResult<AIConfig> {
@@ -194,16 +207,25 @@ async fn get_copilot_info(
         ))
     })?;
 
-    if let Some(workspace_ai_config) = workspace_ai_config.filter(|c| c.0.has_providers()) {
-        Ok(Json(workspace_ai_config.0))
-    } else if let Some(instance_config) =
+    let instance_config =
         sqlx::query_scalar!("SELECT value FROM global_settings WHERE name = 'ai_config'")
             .fetch_optional(&db)
             .await?
+            .and_then(|v| serde_json::from_value::<AIConfig>(v).ok())
+            // A provider-less instance config (e.g. `{}`) is unconfigured; don't let it shadow the
+            // free-tier fallback, matching the proxy and edit_copilot_config paths.
+            .filter(|c| c.has_providers());
+    if let Some(workspace_ai_config) = workspace_ai_config.filter(|c| c.0.has_providers()) {
+        Ok(Json(workspace_ai_config.0))
+    } else if let Some(instance_config) = instance_config {
+        Ok(Json(instance_config))
+    } else if let Some(free_config) =
+        crate::ai_free_tier_oss::free_tier_copilot_config(&db, &authed.email).await?
     {
-        Ok(Json(
-            serde_json::from_value::<AIConfig>(instance_config).unwrap_or_default(),
-        ))
+        // Nothing configured: fall back to Windmill's free tier (EE-only). The config
+        // carries a `free_tier` marker even once the user's grant is spent — with no
+        // providers, but telling the client *why* AI is off.
+        Ok(Json(free_config))
     } else {
         Ok(Json(AIConfig::default()))
     }
@@ -216,7 +238,14 @@ pub async fn get_critical_alerts(
     authed: ApiAuthed,
     Query(params): Query<crate::utils::AlertQueryParams>,
 ) -> JsonResult<serde_json::Value> {
-    require_admin_or_devops(authed.is_admin, &authed.username, &authed.email, authed.job_id.is_some(), &db).await?;
+    require_admin_or_devops(
+        authed.is_admin,
+        &authed.username,
+        &authed.email,
+        authed.job_id.is_some(),
+        &db,
+    )
+    .await?;
 
     crate::utils::get_critical_alerts(db, params, Some(w_id)).await
 }
@@ -232,7 +261,14 @@ pub async fn acknowledge_critical_alert(
     Path((w_id, id)): Path<(String, i32)>,
     authed: ApiAuthed,
 ) -> Result<String> {
-    require_admin_or_devops(authed.is_admin, &authed.username, &authed.email, authed.job_id.is_some(), &db).await?;
+    require_admin_or_devops(
+        authed.is_admin,
+        &authed.username,
+        &authed.email,
+        authed.job_id.is_some(),
+        &db,
+    )
+    .await?;
     crate::utils::acknowledge_critical_alert(db, Some(w_id), id).await
 }
 

backend/windmill-common/src/db_params.rs

@@ -1,45 +0,0 @@
-use anyhow::Result;
-
-/// Parsed database connection parameters, shared across DB auth providers (IAM RDS, Entra ID, etc.)
-#[derive(Debug, Clone)]
-pub struct DatabaseParams {
-    pub hostname: String,
-    pub port: u64,
-    pub username: String,
-    pub database: String,
-}
-
-/// Extract database connection parameters from a PostgreSQL URL
-pub fn extract_database_params(database_url: &str) -> Result<DatabaseParams> {
-    let url = url::Url::parse(database_url)
-        .map_err(|e| anyhow::anyhow!("Failed to parse database URL: {}", e))?;
-
-    let hostname = url
-        .host_str()
-        .ok_or_else(|| anyhow::anyhow!("Database URL missing hostname"))?
-        .to_string();
-
-    let port = url.port().unwrap_or(5432) as u64;
-
-    let username = if url.username().is_empty() {
-        return Err(anyhow::anyhow!("Database URL missing username"));
-    } else {
-        urlencoding::decode(url.username())?.to_string()
-    };
-
-    let database = url
-        .path()
-        .trim_start_matches('/')
-        .split('/')
-        .next()
-        .filter(|s| !s.is_empty())
-        .ok_or_else(|| anyhow::anyhow!("Database URL missing database name"))?
-        .to_string();
-
-    Ok(DatabaseParams {
-        hostname,
-        port,
-        username,
-        database: urlencoding::decode(&database)?.to_string(),
-    })
-}

backend/windmill-common/src/global_settings.rs

@@ -15,6 +15,7 @@ pub const OAUTH_SETTING: &str = "oauths";
 pub const AI_CONFIG_SETTING: &str = "ai_config";
 pub const RETENTION_PERIOD_SECS_SETTING: &str = "retention_period_secs";
 pub const RETENTION_PERIOD_SECS_OVERRIDES_SETTING: &str = "retention_period_secs_overrides";
+pub const SERVICE_LOG_RETENTION_SECS_SETTING: &str = "service_log_retention_secs";
 /// Upper bound on how many per-workspace retention overrides may be configured. The periodic monitor
 /// sweeps each override workspace in its own transaction every pass, so this keeps a pass bounded
 /// (and the feature is a targeted escape hatch for a handful of special workspaces, not a bulk knob).

backend/windmill-common/src/indexer.rs

@@ -94,6 +94,21 @@ pub async fn load_indexer_config(db: &DB) -> error::Result<TantivyIndexerSetting
     })
 }
 
+/// How far back the service log index reaches, in seconds.
+///
+/// [`crate::service_log_retention_secs`] is the ceiling: past it a line's `log_file` row is
+/// deleted and can no longer be indexed. `max_index_time_window_secs` of `0` means "do not
+/// shrink below that ceiling", not "unbounded" — both sites that trim and populate the index
+/// derive the window here so the two cannot disagree about it.
+pub fn service_log_index_window_secs(max_index_time_window_secs: i64) -> i64 {
+    let retention = crate::service_log_retention_secs();
+    if max_index_time_window_secs > 0 {
+        std::cmp::min(max_index_time_window_secs, retention)
+    } else {
+        retention
+    }
+}
+
 pub fn get_env_var(env_var: &str) -> Option<u64> {
     match std::env::var(env_var).map(|x| x.parse()) {
         Ok(Ok(i)) => Some(i),
@@ -136,3 +151,60 @@ pub fn get_indexer_rates_from_env() -> TantivyIndexerSettings {
 
     settings
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    // One test rather than several: both halves share the process-wide retention, and the
+    // setter half writes it, which parallel tests would race.
+    #[test]
+    fn retention_rejects_unusable_values_and_the_index_window_clamps_to_it() {
+        use crate::{
+            service_log_retention_secs, set_service_log_retention_secs,
+            DEFAULT_SERVICE_LOG_RETENTION_SECS,
+        };
+
+        // See `set_service_log_retention_secs` for why the two unusable directions land apart:
+        // too large keeps the intent by capping, non-positive cannot and falls back.
+        let rejected: Vec<i64> = [0, -1, i64::MIN]
+            .iter()
+            .map(|v| {
+                set_service_log_retention_secs(*v);
+                service_log_retention_secs()
+            })
+            .collect();
+        let capped: Vec<i64> = [i64::MAX, 60 * 60 * 24 * 365 * 101]
+            .iter()
+            .map(|v| {
+                set_service_log_retention_secs(*v);
+                service_log_retention_secs()
+            })
+            .collect();
+
+        set_service_log_retention_secs(60 * 60 * 24 * 3);
+        let retention = service_log_retention_secs();
+        let windows = [
+            // `0` disables the extra shrinking rather than lifting the ceiling — the trap that
+            // makes an unset setting look unbounded.
+            service_log_index_window_secs(0),
+            // Retention is the ceiling: the index cannot reach lines whose `log_file` row is gone.
+            service_log_index_window_secs(retention * 2),
+            service_log_index_window_secs(60),
+        ];
+        set_service_log_retention_secs(DEFAULT_SERVICE_LOG_RETENTION_SECS);
+
+        assert_eq!(
+            rejected,
+            vec![DEFAULT_SERVICE_LOG_RETENTION_SECS; 3],
+            "a value that would expire everything must fall back to the default"
+        );
+        assert_eq!(
+            capped,
+            vec![60 * 60 * 24 * 365 * 100; 2],
+            "an oversized value must cap, not shorten retention to the default"
+        );
+        assert_eq!(retention, 60 * 60 * 24 * 3);
+        assert_eq!(windows, [retention, retention, 60]);
+    }
+}

backend/windmill-common/src/lib.rs

@@ -42,7 +42,6 @@ pub mod db;
 mod db_entra_ee;
 #[cfg(all(feature = "enterprise", feature = "private"))]
 mod db_iam_ee;
-pub mod db_params;
 pub mod dbt_manifest;
 pub mod deploy_origin;
 #[cfg(feature = "private")]
@@ -148,9 +147,53 @@ pub const DEFAULT_MAX_CONNECTIONS_INDEXER: u32 = 5;
 
 pub const DEFAULT_HUB_BASE_URL: &str = "https://hub.windmill.dev";
 pub const PRIVATE_HUB_MIN_VERSION: i32 = 10_000_000;
-pub const SERVICE_LOG_RETENTION_SECS: i64 = 60 * 60 * 24 * 14; // 2 weeks retention period for logs
+pub const DEFAULT_SERVICE_LOG_RETENTION_SECS: i64 = 60 * 60 * 24 * 14; // 2 weeks retention period for logs
 pub const WM_DEPLOYERS_GROUP: &str = "wm_deployers";
 
+/// A century. Every consumer has to survive `now - retention`, and the ceilings are much lower
+/// than an `i64`: `DateTime` subtraction panics past year 262143, and the `(<n> s)::interval`
+/// the cleanup queries build overflows Postgres' microsecond field.
+const MAX_SERVICE_LOG_RETENTION_SECS: i64 = 60 * 60 * 24 * 365 * 100;
+
+/// Apply a configured service log retention, in seconds.
+///
+/// The only way into [`SERVICE_LOG_RETENTION_SECS`], so an unusable value can never reach a
+/// cutoff. The two unusable directions are not the same mistake and must not share a landing
+/// point: too large still says "keep these for a very long time", so it is capped and the
+/// intent survives, whereas falling back would delete logs the operator meant to keep. A
+/// non-positive value has no such reading — every cutoff is `now - retention`, so it lands at
+/// or after `now` and the next sweep expires the entire history, rows and object-storage files
+/// alike. Unlike job retention there is no "keep forever" spelling here, so `0` — what an
+/// operator types by analogy with it, and what the settings UI writes into a field that was
+/// merely focused — falls back to the default.
+pub fn set_service_log_retention_secs(configured: i64) {
+    let effective = if configured > MAX_SERVICE_LOG_RETENTION_SECS {
+        tracing::warn!(
+            "service log retention of {configured}s exceeds the maximum of \
+             {MAX_SERVICE_LOG_RETENTION_SECS}s, capping it there"
+        );
+        MAX_SERVICE_LOG_RETENTION_SECS
+    } else if configured >= 1 {
+        configured
+    } else {
+        tracing::warn!(
+            "service log retention of {configured}s would expire every service log, \
+             falling back to the default of {DEFAULT_SERVICE_LOG_RETENTION_SECS}s"
+        );
+        DEFAULT_SERVICE_LOG_RETENTION_SECS
+    };
+    SERVICE_LOG_RETENTION_SECS.store(effective, std::sync::atomic::Ordering::Relaxed);
+}
+
+/// How long a service log line stays retrievable, in seconds.
+///
+/// The outer bound on everything service-log: the `log_file` rows, the raw files in object
+/// storage, the columnar store queried by retrieval, and — through
+/// [`indexer::service_log_index_window_secs`] — the search index.
+pub fn service_log_retention_secs() -> i64 {
+    SERVICE_LOG_RETENTION_SECS.load(std::sync::atomic::Ordering::Relaxed)
+}
+
 /// Canonical form of a base URL, used as one of the inputs to the offline-license
 /// instance hash (`compute_instance_hash`).
 ///
@@ -376,6 +419,10 @@ lazy_static::lazy_static! {
     /// workspace configured before its override could be read.
     pub static ref JOB_RETENTION_SECS_OVERRIDES_LOADED: AtomicBool = AtomicBool::new(false);
     pub static ref AUDIT_LOG_RETENTION_DAYS: AtomicI64 = AtomicI64::new(0);
+    /// Private on purpose: [`set_service_log_retention_secs`] is the only writer, so a value that
+    /// would expire every service log cannot reach a cutoff. Read it with
+    /// [`service_log_retention_secs`].
+    static ref SERVICE_LOG_RETENTION_SECS: AtomicI64 = AtomicI64::new(DEFAULT_SERVICE_LOG_RETENTION_SECS);
 
     pub static ref MONITOR_LOGS_ON_OBJECT_STORE: AtomicBool = AtomicBool::new(false);
 
@@ -1519,6 +1566,17 @@ pub async fn create_custom_instance_database(
     Ok(())
 }
 
+/// Connection options parsed from a database URL.
+///
+/// The only place a database URL becomes `PgConnectOptions`. Providers that mint the password
+/// themselves override it on these and keep the rest: options assembled field by field instead
+/// would drop every query parameter, `sslmode` and `sslrootcert` above all, leaving the
+/// connection on sqlx's default TLS policy rather than the operator's.
+pub fn base_connect_options(database_url: &str) -> Result<sqlx::postgres::PgConnectOptions, Error> {
+    sqlx::postgres::PgConnectOptions::from_str(database_url)
+        .map_err(|e| Error::InternalErr(format!("Failed to parse database URL: {}", e)))
+}
+
 #[derive(Clone)]
 pub enum DatabaseUrl {
     #[cfg(all(feature = "enterprise", feature = "private"))]
@@ -1549,8 +1607,8 @@ impl DatabaseUrl {
     }
 
     /// Get PgConnectOptions for this database URL.
-    /// For token-based auth (IAM RDS, Entra ID), this returns options built directly from the
-    /// token to avoid double-encoding issues with temporary credentials.
+    /// For token-based auth (IAM RDS, Entra ID), this returns options carrying the current
+    /// token, set on the builder to avoid double-encoding temporary credentials.
     /// For static URLs, this parses the URL string.
     pub async fn connect_options(&self) -> Result<sqlx::postgres::PgConnectOptions, Error> {
         match self {
@@ -1564,8 +1622,7 @@ impl DatabaseUrl {
                 let guard = entra_url.read().await;
                 Ok(guard.connect_options())
             }
-            DatabaseUrl::Static(url) => sqlx::postgres::PgConnectOptions::from_str(url)
-                .map_err(|e| Error::InternalErr(format!("Failed to parse database URL: {}", e))),
+            DatabaseUrl::Static(url) => base_connect_options(url),
         }
     }
 

backend/windmill-common/src/tracing_init.rs

@@ -178,56 +178,40 @@ pub fn initialize_tracing(
         .with(logs_bridge.with_filter(otel_logs_filter))
         .with(opentelemetry_filtered);
 
-    match *JSON_FMT {
-        true => {
-            // Stdout layer with its own filter
-            let stdout_layer = json_layer()
-                .with_writer(std::io::stdout)
-                .flatten_event(true)
-                .with_filter(stdout_env_filter)
-                .with_filter(create_targets_filter(default_env_filter));
+    // The service log files are written to be indexed, not tailed, so they always carry the
+    // JSON format: it is what preserves level, target and the current span as fields rather
+    // than as text the index would have to recover by regex. JSON_FMT governs stdout only.
+    let file_layer = json_layer()
+        .with_writer(log_file_writer)
+        .flatten_event(true)
+        .with_filter(file_env_filter)
+        .with_filter(create_targets_filter(default_env_filter));
 
-            // File layer with its own filter
-            let file_layer = json_layer()
-                .with_writer(log_file_writer)
-                .flatten_event(true)
-                .with_filter(file_env_filter)
-                .with_filter(create_targets_filter(default_env_filter));
+    // Boxed so both arms have one type: the file layer is a single value and could not
+    // otherwise be typed against two different subscriber stacks.
+    let stdout_layer = match *JSON_FMT {
+        true => json_layer()
+            .with_writer(std::io::stdout)
+            .flatten_event(true)
+            .with_filter(stdout_env_filter)
+            .with_filter(create_targets_filter(default_env_filter))
+            .boxed(),
+        false => compact_layer()
+            .with_writer(std::io::stdout)
+            .with_ansi(style.to_lowercase() != "never")
+            .with_file(true)
+            .with_line_number(true)
+            .with_target(false)
+            .with_filter(stdout_env_filter)
+            .with_filter(create_targets_filter(default_env_filter))
+            .boxed(),
+    };
 
-            base_layer
-                .with(stdout_layer)
-                .with(file_layer)
-                .with(CountingLayer::new())
-                .init()
-        }
-        false => {
-            // Stdout layer with its own filter
-            let stdout_layer = compact_layer()
-                .with_writer(std::io::stdout)
-                .with_ansi(style.to_lowercase() != "never")
-                .with_file(true)
-                .with_line_number(true)
-                .with_target(false)
-                .with_filter(stdout_env_filter)
-                .with_filter(create_targets_filter(default_env_filter));
-
-            // File layer with its own filter
-            let file_layer = compact_layer()
-                .with_writer(log_file_writer)
-                .with_ansi(false) // No ANSI codes in log files
-                .with_file(true)
-                .with_line_number(true)
-                .with_target(false)
-                .with_filter(file_env_filter)
-                .with_filter(create_targets_filter(default_env_filter));
-
-            base_layer
-                .with(stdout_layer)
-                .with(file_layer)
-                .with(CountingLayer::new())
-                .init()
-        }
-    }
+    base_layer
+        .with(stdout_layer)
+        .with(file_layer)
+        .with(CountingLayer::new())
+        .init();
     (_guard, meter_provider)
 }
 

backend/windmill-common/src/worker.rs

@@ -734,11 +734,17 @@ fn format_pull_query(peek: String) -> String {
     r
 }
 
+// The `CASE` is `suspend <= 0 OR suspend_until <= now()` written as one indexable
+// expression, equivalent only under the `suspend_until IS NOT NULL` guard. It must stay in
+// sync with `queue_suspended_v2` (migration 20260826202939): if it no longer matches, the
+// test silently reverts to a heap filter over every suspended row on every worker poll.
 pub fn make_suspended_pull_query(tags: &[String]) -> String {
     format_pull_query(format!(
         "SELECT id
         FROM v2_job_queue
-        WHERE suspend_until IS NOT NULL AND (suspend <= 0 OR suspend_until <= now()) AND tag IN ({})
+        WHERE suspend_until IS NOT NULL
+          AND (CASE WHEN suspend <= 0 THEN '-infinity'::timestamptz ELSE suspend_until END) <= now()
+          AND tag IN ({})
         ORDER BY priority DESC NULLS LAST, created_at
         FOR UPDATE SKIP LOCKED
         LIMIT 1",

backend/windmill-common/src/workspaces.rs

@@ -759,15 +759,24 @@ pub async fn count_workspace_forks(db: &crate::DB, root: &str) -> Result<i64> {
     Ok(count)
 }
 
-/// Approximate paid seats of a workspace as `ceil(developers + operators/2)`, excluding disabled and
-/// service-account members. Reuses billing's author/operator weighting, but counts provisioned
-/// members rather than the active-user population billing meters, so it only ever loosens the fork
-/// cap (never blocks a paid seat) — good enough for a soft guardrail.
+/// The billable members of a workspace and the seats they add up to.
+#[derive(Clone, Debug, Serialize)]
+pub struct BillableSeats {
+    pub developers: i64,
+    pub operators: i64,
+    pub seats: i64,
+}
+
+/// Billable members of `w_id` and the seats they cost, as `ceil(developers + operators/2)`. Service
+/// accounts cannot log in and do not take a seat; a disabled member is not billed either.
+///
+/// The workspace is invoiced by a job outside this codebase that counts the same rows with its own
+/// SQL. The two must be changed together: this rule disagreeing with that one is what bills a
+/// workspace for seats the product never credits it for.
 ///
 /// Unauthenticated metering helper: reads member counts for any `w_id`, so callers must already be
 /// authorized for that workspace (or run in trusted server-side code).
-#[cfg(feature = "cloud")]
-pub async fn count_paid_seats(db: &crate::DB, w_id: &str) -> Result<i64> {
+pub async fn billable_seats(db: &crate::DB, w_id: &str) -> Result<BillableSeats> {
     let row = sqlx::query!(
         r#"SELECT
             COUNT(*) FILTER (WHERE NOT operator AND NOT disabled AND NOT is_service_account) AS "developers!",
@@ -777,8 +786,18 @@ pub async fn count_paid_seats(db: &crate::DB, w_id: &str) -> Result<i64> {
     )
     .fetch_one(db)
     .await
-    .map_err(|e| Error::internal_err(format!("counting paid seats of {w_id}: {e:#}")))?;
-    Ok(((row.developers as f64) + 0.5 * (row.operators as f64)).ceil() as i64)
+    .map_err(|e| Error::internal_err(format!("counting billable seats of {w_id}: {e:#}")))?;
+    Ok(BillableSeats {
+        developers: row.developers,
+        operators: row.operators,
+        seats: ((row.developers as f64) + 0.5 * (row.operators as f64)).ceil() as i64,
+    })
+}
+
+/// Seats only, for the fork cap. See [`billable_seats`].
+#[cfg(feature = "cloud")]
+pub async fn count_paid_seats(db: &crate::DB, w_id: &str) -> Result<i64> {
+    Ok(billable_seats(db, w_id).await?.seats)
 }
 
 #[cfg(feature = "cloud")]
@@ -2531,6 +2550,32 @@ pub fn lfs_entry_storage_ref(entry: &serde_json::Value) -> Option<String> {
     Some(format!("{typ}:{path}"))
 }
 
+pub const FILESYSTEM_STORAGE_DEV_ONLY_MSG: &str =
+    "Filesystem storage is only available in development builds of Windmill: it points the \
+     workspace at a directory on the server's own disk rather than at a resource. Use an S3, \
+     Azure Blob or Google Cloud Storage backend instead.";
+
+/// A filesystem workspace storage names a directory on the server's own disk, so it hands whoever
+/// configures it — a workspace admin, or any member who can write a `filesystem` resource —
+/// whatever the server process can reach, and it only resolves when server and workers share that
+/// disk. It is there so local development can skip MinIO, hence debug builds only. Instance object
+/// storage on local disk is a separate, superadmin-only setting and stays allowed everywhere.
+pub fn filesystem_storage_allowed() -> bool {
+    cfg!(debug_assertions)
+}
+
+/// Guards every site that builds an `ObjectStoreResource::Filesystem`, so nothing downstream can
+/// reach a local-disk store: a stored config outlives the build that accepted it, and the resource
+/// route never passes through the workspace-storage settings at all.
+pub fn ensure_filesystem_storage_allowed() -> Result<()> {
+    if !filesystem_storage_allowed() {
+        return Err(Error::BadRequest(
+            FILESYSTEM_STORAGE_DEV_ONLY_MSG.to_string(),
+        ));
+    }
+    Ok(())
+}
+
 /// Resolve a `$res:`/`$var:` reference tree to its concrete value (recursively, secrets
 /// decrypted). No permission checks — trusted server-side callers only; never echo the result
 /// to a user.

backend/windmill-common/tests/billing_workspace.rs

@@ -3,7 +3,7 @@
 
 use sqlx::{Pool, Postgres};
 use windmill_common::workspaces::{
-    count_paid_seats, count_workspace_forks, fork_chain_depth, fork_subtree_height,
+    billable_seats, count_paid_seats, count_workspace_forks, fork_chain_depth, fork_subtree_height,
     get_billing_workspace_id, invalidate_billing_workspace_cache, list_fork_descendants,
 };
 
@@ -106,11 +106,17 @@ async fn paid_seats_and_fork_count(db: Pool<Postgres>) {
     insert_member(&db, "seat-root", "dev2@w.dev", false, false, false).await;
     insert_member(&db, "seat-root", "op1@w.dev", true, false, false).await;
     insert_member(&db, "seat-root", "op2@w.dev", true, false, false).await;
-    // These must NOT count towards seats.
+    // These must NOT count towards seats. The service account is a non-operator, so counting it
+    // would inflate the developer tally the invoice line is written from, not the operator one.
     insert_member(&db, "seat-root", "disabled@w.dev", false, true, false).await;
     insert_member(&db, "seat-root", "svc@w.dev", false, false, true).await;
 
     assert_eq!(count_paid_seats(&db, "seat-root").await.unwrap(), 3);
+    let breakdown = billable_seats(&db, "seat-root").await.unwrap();
+    assert_eq!(
+        (breakdown.developers, breakdown.operators, breakdown.seats),
+        (2, 2, 3)
+    );
 
     insert_ws(&db, "seat-fork1", Some("seat-root"), false).await;
     insert_ws(&db, "seat-fork2", Some("seat-root"), false).await;

backend/windmill-indexer/Cargo.toml

@@ -10,7 +10,13 @@ path = "src/lib.rs"
 
 [features]
 default = []
-parquet = ["windmill-common/parquet", "windmill-object-store/parquet"]
+parquet = [
+    "windmill-common/parquet",
+    "windmill-object-store/parquet",
+    "dep:datafusion",
+    "dep:object_store",
+    "dep:url",
+]
 private = ["windmill-common/private"]
 enterprise = ["windmill-common/enterprise", "windmill-object-store/enterprise"]
 
@@ -33,3 +39,6 @@ astral-tokio-tar.workspace = true
 lazy_static.workspace = true
 const_format.workspace = true
 flume.workspace = true
+datafusion = { workspace = true, optional = true }
+object_store = { workspace = true, optional = true }
+url = { workspace = true, optional = true }

backend/windmill-indexer/src/lib.rs

@@ -7,3 +7,5 @@ pub mod indexer_oss;
 #[cfg(feature = "private")]
 pub mod service_logs_ee;
 pub mod service_logs_oss;
+#[cfg(all(feature = "private", feature = "parquet"))]
+pub mod service_logs_store_ee;

backend/windmill-object-store/src/lib.rs

@@ -1171,6 +1171,7 @@ pub fn lfs_to_object_store_resource(
             Ok(ObjectStoreResource::Gcs(gcs_resource))
         }
         LargeFileStorage::FilesystemStorage(fs) => {
+            windmill_common::workspaces::ensure_filesystem_storage_allowed()?;
             Ok(ObjectStoreResource::Filesystem(FilesystemSettings {
                 root_path: fs.root_path.clone(),
             }))

backend/windmill-queue/src/jobs.rs

@@ -4671,6 +4671,12 @@ pub async fn check_debouncing_within_limits(
     }
 }
 
+/// Whether the tag's queue name is computed from the job's arguments, so that a caller holding
+/// arguments it could not build knows the tag cannot be built either.
+pub fn tag_reads_args(tag: &str) -> bool {
+    RE_ARG_TAG.is_match(tag)
+}
+
 pub fn interpolate_args(x: String, args: &PushArgs, workspace_id: &str) -> String {
     // Save this value to avoid parsing twice
     let workspaced = x.as_str().replace("$workspace", workspace_id).to_string();

backend/windmill-trigger-http/src/handler.rs

@@ -268,6 +268,15 @@ pub async fn create_many_http_triggers(
             format!("http_triggers:write:{}", &new_http_trigger.base.path)
         })?;
 
+        // This route inserts directly, bypassing the shared create handler.
+        // `error_wrapper` would turn the rejection into a 500.
+        new_http_trigger.error_handling.validate().map_err(|err| {
+            Error::BadRequest(format!(
+                "Error occurred for HTTP route at route path: {}, error: {}",
+                new_http_trigger.config.route_path, err
+            ))
+        })?;
+
         handler
             .validate_new(&db, &w_id, &new_http_trigger.config)
             .await
@@ -570,8 +579,8 @@ impl TriggerCrud for HttpTrigger {
                 route_path,
                 &route_path_key,
                 Some(effective_workspaced),
-                trigger.config.wrap_body,
-                trigger.config.raw_string,
+                trigger.config.wrap_body.unwrap_or(false),
+                trigger.config.raw_string.unwrap_or(false),
                 trigger.config.authentication_resource_path,
                 trigger.base.script_path,
                 trigger.base.path,
@@ -626,8 +635,8 @@ impl TriggerCrud for HttpTrigger {
                 workspace_id = $20 AND
                 path = $21
             "#,
-                trigger.config.wrap_body,
-                trigger.config.raw_string,
+                trigger.config.wrap_body.unwrap_or(false),
+                trigger.config.raw_string.unwrap_or(false),
                 trigger.config.authentication_resource_path,
                 trigger.base.script_path,
                 trigger.base.path,

backend/windmill-trigger/src/handler.rs

@@ -554,6 +554,8 @@ async fn create_trigger<T: TriggerCrud>(
         )));
     }
 
+    new_trigger.error_handling.validate()?;
+
     handler
         .validate_new(&db, &workspace_id, &new_trigger.config)
         .await?;
@@ -815,6 +817,8 @@ async fn update_trigger<T: TriggerCrud>(
         )
     })?;
 
+    edit_trigger.error_handling.validate()?;
+
     handler
         .validate_edit(&db, &workspace_id, &edit_trigger.config, path)
         .await?;

backend/windmill-trigger/src/types.rs

@@ -84,6 +84,31 @@ pub struct TriggerErrorHandling {
     pub retry: Option<sqlx::types::Json<windmill_common::flows::Retry>>,
 }
 
+impl TriggerErrorHandling {
+    /// Schedule and workspace error handlers encode script-vs-flow as a
+    /// `script/`/`flow/` prefix; a trigger's handler is always a script, so a
+    /// prefixed path here would be looked up verbatim as a script name and fail
+    /// only once the trigger errors, which is when the handler is needed.
+    pub fn validate(&self) -> windmill_common::error::Result<()> {
+        let Some(path) = self.error_handler_path.as_deref() else {
+            return Ok(());
+        };
+        if let Some(bare) = path.strip_prefix("script/") {
+            return Err(windmill_common::error::Error::BadRequest(format!(
+                "error_handler_path is a plain script path, not the prefixed form a schedule \
+                 error handler takes: got '{path}', use '{bare}'"
+            )));
+        }
+        if path.starts_with("flow/") {
+            return Err(windmill_common::error::Error::BadRequest(format!(
+                "error_handler_path must be a script: a trigger error handler cannot be a flow \
+                 (got '{path}')"
+            )));
+        }
+        Ok(())
+    }
+}
+
 #[derive(Serialize, Deserialize, Clone)]
 pub struct Trigger<T>
 where
@@ -253,6 +278,43 @@ mod tests {
     use super::*;
     use serde_json::json;
 
+    // --- TriggerErrorHandling::validate ---
+
+    fn error_handling(path: Option<&str>) -> TriggerErrorHandling {
+        TriggerErrorHandling {
+            error_handler_path: path.map(str::to_string),
+            error_handler_args: None,
+            retry: None,
+        }
+    }
+
+    #[test]
+    fn test_error_handler_path_accepts_bare_and_hub_paths() {
+        for path in [None, Some("f/team/handler"), Some("u/admin/handler")] {
+            assert!(error_handling(path).validate().is_ok(), "{path:?}");
+        }
+        assert!(error_handling(Some("hub/13953/windmill/handler"))
+            .validate()
+            .is_ok());
+    }
+
+    #[test]
+    fn test_error_handler_path_rejects_prefixed_paths() {
+        // The rejection names the bare path to use, so the caller can fix it
+        // without knowing which of the two conventions a trigger follows.
+        let err = error_handling(Some("script/f/team/handler"))
+            .validate()
+            .unwrap_err()
+            .to_string();
+        assert!(err.contains("f/team/handler"), "{err}");
+
+        let err = error_handling(Some("flow/f/team/handler"))
+            .validate()
+            .unwrap_err()
+            .to_string();
+        assert!(err.contains("cannot be a flow"), "{err}");
+    }
+
     // --- TriggerMode serde ---
 
     #[test]

backend/windmill-types/src/jobs.rs

@@ -632,6 +632,28 @@ pub enum JobPayload {
     },
 }
 
+impl JobPayload {
+    /// Whether the payload itself declares a dedicated worker, in which case `push` replaces
+    /// whatever tag it is handed and the caller's tag never reaches the queue.
+    ///
+    /// This reads what the payload carries, not what `push` will conclude: a `SingleStepFlow`
+    /// loads the flag from the script row at push time and reports `false` here. That only
+    /// matters to a caller reasoning about the tag, and for those the answer is the same either
+    /// way, since `push` replaces the tag in exactly the case this misses.
+    pub fn is_dedicated_worker(&self) -> bool {
+        let dedicated_worker = match self {
+            JobPayload::ScriptHash { dedicated_worker, .. }
+            | JobPayload::FlowScript { dedicated_worker, .. }
+            | JobPayload::Dependencies { dedicated_worker, .. }
+            | JobPayload::FlowDependencies { dedicated_worker, .. }
+            | JobPayload::Flow { dedicated_worker, .. } => dedicated_worker,
+            JobPayload::Code(raw) => &raw.dedicated_worker,
+            _ => &None,
+        };
+        dedicated_worker.is_some_and(|x| x)
+    }
+}
+
 #[derive(Clone, Serialize, Deserialize, Debug)]
 pub struct SkipHandler {
     pub path: String,

backend/windmill-worker/src/bun_executor.rs

@@ -1256,6 +1256,7 @@ pub async fn prebundle_bun_script(
     token: &str,
     occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
     temp_script_refs: &Option<HashMap<String, String>>,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
 ) -> Result<()> {
     let (local_path, remote_path) = compute_bundle_local_and_remote_path(
         inner_content,
@@ -1264,6 +1265,7 @@ pub async fn prebundle_bun_script(
         db,
         w_id,
         temp_script_refs,
+        modules,
     )
     .await;
     if exists_in_cache(&local_path, &remote_path).await {
@@ -1442,6 +1444,7 @@ pub async fn compute_bundle_local_and_remote_path(
     db: Option<&DB>,
     w_id: &str,
     temp_script_refs: &Option<HashMap<String, String>>,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
 ) -> (String, String) {
     let mut input_src = format!("{inner_content}{lock}",);
 
@@ -1470,7 +1473,10 @@ pub async fn compute_bundle_local_and_remote_path(
 
     let ws_suffix = crate::workspace_registry_cache_suffix(w_id).await;
     input_src.push_str(&ws_suffix);
-    let hash = windmill_common::utils::calculate_hash(&input_src);
+
+    // The loader resolves relative imports against the module files in the job dir, so
+    // their content is inlined into the bundle this name covers.
+    let hash = crate::worker::artifact_cache_name(input_src, modules);
     let local_path = format!("{}/{hash}", *BUN_BUNDLE_CACHE_DIR);
 
     #[cfg(windows)]
@@ -1569,6 +1575,7 @@ pub async fn handle_bun_job(
                     Some(db),
                     &job.workspace_id,
                     &temp_script_refs,
+                    modules.as_ref(),
                 )
                 .await
             }
@@ -2962,9 +2969,8 @@ pub async fn handle_wac_v2_output(
                                     version: flow_info.version,
                                     labels: flow_info.labels.clone(),
                                 };
-                                let on_behalf_of = flow_info
-                                    .on_behalf_of(&job.workspace_id, db)
-                                    .await?;
+                                let on_behalf_of =
+                                    flow_info.on_behalf_of(&job.workspace_id, db).await?;
                                 let step_args: HashMap<String, Box<RawValue>> = step
                                     .args
                                     .iter()
@@ -4382,4 +4388,49 @@ export function main(x: number) { return x; }"#;
         assert!(wrapper.contains(r#"line.startsWith("exec_preprocess:")"#));
         assert!(wrapper.contains(r#"line.startsWith("exec:")"#));
     }
+
+    /// The bundle cache is global and content-keyed, so a key that ignores the inline
+    /// modules hands one workspace's bundle — attacker helper code and all — to the next
+    /// job whose main content and lockfile happen to match.
+    #[tokio::test]
+    async fn bundle_cache_key_separates_inline_module_content() {
+        use windmill_common::scripts::ScriptModule;
+
+        async fn key_for(modules: Option<&HashMap<String, ScriptModule>>) -> String {
+            compute_bundle_local_and_remote_path(
+                "import { h } from './helper.ts';\nexport async function main() { return h(); }",
+                "{}\n//bun.lock\n<empty>",
+                "u/alice/script",
+                None,
+                "w1",
+                &None,
+                modules,
+            )
+            .await
+            .1
+        }
+        fn modules(content: &str) -> HashMap<String, ScriptModule> {
+            HashMap::from([(
+                "helper.ts".to_string(),
+                ScriptModule {
+                    content: content.to_string(),
+                    language: ScriptLang::Bun,
+                    lock: None,
+                },
+            )])
+        }
+
+        let attacker = key_for(Some(&modules("export const h = () => 'attacker'"))).await;
+        let victim = key_for(Some(&modules("export const h = () => 'victim'"))).await;
+        assert_ne!(attacker, victim);
+        assert_eq!(
+            attacker,
+            key_for(Some(&modules("export const h = () => 'attacker'"))).await,
+            "same modules must still share a cache slot"
+        );
+
+        // An absent map and an empty one are the same script, so they share a slot.
+        assert_eq!(key_for(None).await, key_for(Some(&HashMap::new())).await);
+        assert_ne!(key_for(None).await, attacker);
+    }
 }

backend/windmill-worker/src/common.rs

@@ -1640,6 +1640,7 @@ pub(crate) async fn get_workspace_s3_resource_path(
             )
         }
         Some(LargeFileStorage::FilesystemStorage(fs)) => {
+            windmill_common::workspaces::ensure_filesystem_storage_allowed()?;
             return Ok(Some(
                 windmill_object_store::ObjectStoreResource::Filesystem(
                     windmill_object_store::FilesystemSettings { root_path: fs.root_path.clone() },

backend/windmill-worker/src/csharp_executor.rs

@@ -13,7 +13,7 @@ use itertools::Itertools;
 #[cfg(feature = "csharp")]
 use tokio::{fs::File, io::AsyncReadExt, process::Command};
 #[cfg(feature = "csharp")]
-use windmill_common::{utils::calculate_hash, worker::write_file};
+use windmill_common::worker::write_file;
 
 #[cfg(feature = "csharp")]
 use crate::global_cache::save_cache;
@@ -72,13 +72,21 @@ const CSHARP_OBJECT_STORE_PREFIX: &str =
 /// Cache key of a C# build. The run path and the deploy-time prebuild must derive it the
 /// same way or the prebuilt binary is never found and gets rebuilt on first run.
 #[cfg(feature = "csharp")]
-async fn csharp_cache_key(code: &str, requirements_o: Option<&str>, w_id: &str) -> String {
-    let mut hash = calculate_hash(&format!(
+async fn csharp_cache_key(
+    code: &str,
+    requirements_o: Option<&str>,
+    w_id: &str,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
+) -> String {
+    // The SDK project globs every `.cs` under the job dir, so companion modules are
+    // compiled into the binary this key names and have to be part of it.
+    let base = format!(
         "{}{}{}",
         code,
         requirements_o.unwrap_or(""),
         DOTNET_TARGET_FRAMEWORK.as_str()
-    ));
+    );
+    let mut hash = crate::worker::artifact_cache_name(base, modules);
     hash.push_str(&crate::workspace_registry_cache_suffix(w_id).await);
     hash
 }
@@ -487,10 +495,11 @@ pub async fn prebuild_csharp_binary(
     worker_name: &str,
     base_internal_url: &str,
     occupancy_metrics: &mut OccupancyMetrics,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
 ) -> error::Result<Option<String>> {
     check_executor_binary_exists("dotnet", DOTNET_PATH.as_str(), "C#")?;
 
-    let hash = csharp_cache_key(code, Some(lock), &job.workspace_id).await;
+    let hash = csharp_cache_key(code, Some(lock), &job.workspace_id, modules).await;
     let remote_path = format!("{CSHARP_OBJECT_STORE_PREFIX}{hash}");
     if crate::global_cache::exists_in_object_store(&remote_path).await {
         return Ok(None);
@@ -549,6 +558,7 @@ pub async fn handle_csharp_job(
     _worker_name: &str,
     _envs: HashMap<String, String>,
     _occupancy_metrics: &mut OccupancyMetrics,
+    _modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
 ) -> Result<Box<RawValue>, Error> {
     Err(anyhow!("C# is not available because the feature is not enabled").into())
 }
@@ -568,6 +578,7 @@ pub async fn handle_csharp_job(
     worker_name: &str,
     envs: HashMap<String, String>,
     occupancy_metrics: &mut OccupancyMetrics,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
 ) -> Result<Box<RawValue>, Error> {
     check_executor_binary_exists("dotnet", DOTNET_PATH.as_str(), "C#")?;
 
@@ -575,6 +586,7 @@ pub async fn handle_csharp_job(
         inner_content,
         requirements_o.map(|x| x.as_str()),
         &job.workspace_id,
+        modules,
     )
     .await;
     let bin_path = format!("{}/{hash}", *CSHARP_CACHE_DIR);

backend/windmill-worker/src/go_executor.rs

@@ -230,8 +230,15 @@ fn go_runtime_int32(v: &str) -> Option<i32> {
 
 /// Cache key of a Go build. The run path and the deploy-time prebuild must derive it the
 /// same way or the prebuilt binary is never found and gets rebuilt on first run.
-fn go_cache_key(code: &str, maybe_lock: &MaybeLock) -> String {
-    calculate_hash(&format!("{}{:?}v2", code, maybe_lock))
+fn go_cache_key(
+    code: &str,
+    maybe_lock: &MaybeLock,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
+) -> String {
+    // A module whose path starts with `go/` lands inside the module dir this builds, so
+    // its content ends up in the binary the key names.
+    let base = format!("{}{:?}v2", code, maybe_lock);
+    crate::worker::artifact_cache_name(base, modules)
 }
 
 /// Install the deps, generate the entrypoint wrapper, `go build`, and push the binary to
@@ -459,9 +466,10 @@ pub async fn prebuild_go_binary(
     worker_name: &str,
     base_internal_url: &str,
     occupancy_metrics: &mut OccupancyMetrics,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
 ) -> Result<Option<String>, Error> {
     let maybe_lock = MaybeLock::Resolved { lock: lock.to_string() };
-    let hash = go_cache_key(code, &maybe_lock);
+    let hash = go_cache_key(code, &maybe_lock, modules);
     let remote_path = format!("{GO_OBJECT_STORE_PREFIX}{hash}");
     if crate::global_cache::exists_in_object_store(&remote_path).await {
         return Ok(None);
@@ -512,6 +520,7 @@ pub async fn handle_go_job(
     envs: HashMap<String, String>,
     occupation_metrics: &mut OccupancyMetrics,
     maybe_lock: MaybeLock,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
 ) -> Result<Box<RawValue>, Error> {
     //go does not like executing modules at temp root
     let job_dir = &format!("{job_dir}/go");
@@ -520,7 +529,7 @@ pub async fn handle_go_job(
         .create(&job_dir)
         .expect("could not create go job dir");
 
-    let hash = go_cache_key(inner_content, &maybe_lock);
+    let hash = go_cache_key(inner_content, &maybe_lock, modules);
     let bin_path = format!("{}/{hash}", *GO_BIN_CACHE_DIR);
     let remote_path = format!("{GO_OBJECT_STORE_PREFIX}{hash}");
     let (cache, cache_logs) = crate::global_cache::load_cache(&bin_path, &remote_path, false).await;

backend/windmill-worker/src/java_executor.rs

@@ -66,6 +66,7 @@ pub(crate) struct JobHandlerInput<'a> {
     pub requirements_o: Option<&'a String>,
     pub shared_mount: &'a str,
     pub worker_name: &'a str,
+    pub modules: Option<&'a HashMap<String, windmill_common::scripts::ScriptModule>>,
 }
 
 pub async fn handle_java_job<'a>(mut args: JobHandlerInput<'a>) -> Result<Box<RawValue>, Error> {
@@ -612,25 +613,33 @@ async fn compile<'a>(
         inner_content,
         requirements_o,
         parent_runnable_path,
+        modules,
         ..
     }: &mut JobHandlerInput<'a>,
     classpath: &'a str,
     // plugins: Vec<&'a str>,
 ) -> Result<(), Error> {
-    fn compute_hash(code: &str, requirements_o: Option<&String>) -> String {
-        calculate_hash(&format!(
+    // The cached artifact is the whole `target/` dir, and companion modules are written
+    // into the job dir before this runs, so their content can land in it.
+    fn compute_hash(
+        code: &str,
+        requirements_o: Option<&String>,
+        modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
+    ) -> String {
+        let base = format!(
             "{}{}",
             code,
             requirements_o
                 .as_ref()
                 .map(|x| x.to_string())
                 .unwrap_or_default()
-        ))
+        );
+        crate::worker::artifact_cache_name(base, modules)
     }
     let reserved_variables =
         get_reserved_variables(job, &client.token, conn, parent_runnable_path.clone()).await?;
     let ws_suffix = crate::workspace_registry_cache_suffix(&job.workspace_id).await;
-    let mut hash = compute_hash(inner_content, *requirements_o);
+    let mut hash = compute_hash(inner_content, *requirements_o, *modules);
     hash.push_str(&ws_suffix);
     let bin_path = format!("{}/{hash}", *JAVA_CACHE_DIR);
     let remote_path = format!("java_jar/{hash}");

backend/windmill-worker/src/rust_executor.rs

@@ -14,7 +14,6 @@ use tokio::{
 };
 use windmill_common::{
     error::{self, Error},
-    utils::calculate_hash,
     worker::{write_file, Connection},
 };
 use windmill_queue::MiniPulledJob;
@@ -602,8 +601,13 @@ pub async fn build_rust_crate(
 
 /// Cache key of a Rust build. The run path and the deploy-time prebuild must derive it
 /// the same way or the prebuilt binary is never found and gets rebuilt on first run.
-async fn rust_cache_key(code: &str, requirements_o: Option<&String>, w_id: &str) -> String {
-    let mut hash = compute_rust_hash(code, requirements_o);
+async fn rust_cache_key(
+    code: &str,
+    requirements_o: Option<&String>,
+    w_id: &str,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
+) -> String {
+    let mut hash = compute_rust_hash(code, requirements_o, modules);
     hash.push_str(&crate::workspace_registry_cache_suffix(w_id).await);
     hash
 }
@@ -621,11 +625,12 @@ pub async fn prebuild_rust_binary(
     worker_name: &str,
     base_internal_url: &str,
     occupancy_metrics: &mut OccupancyMetrics,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
 ) -> error::Result<Option<String>> {
     ensure_rust_runtime_dirs();
     check_executor_binary_exists("cargo", CARGO_PATH.as_str(), "rust")?;
 
-    let hash = rust_cache_key(code, Some(&lock.to_string()), &job.workspace_id).await;
+    let hash = rust_cache_key(code, Some(&lock.to_string()), &job.workspace_id, modules).await;
     let remote_path = format!("{RUST_OBJECT_STORE_PREFIX}{hash}");
     if crate::global_cache::exists_in_object_store(&remote_path).await {
         return Ok(None);
@@ -652,15 +657,22 @@ pub async fn prebuild_rust_binary(
     Ok(Some(logs))
 }
 
-pub fn compute_rust_hash(code: &str, requirements_o: Option<&String>) -> String {
-    calculate_hash(&format!(
+pub fn compute_rust_hash(
+    code: &str,
+    requirements_o: Option<&String>,
+    // Companion modules are written into the crate dir and compiled into the binary this
+    // key names, so leaving them out shares one script's binary with another.
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
+) -> String {
+    let base = format!(
         "{}{}",
         code,
         requirements_o
             .as_ref()
             .map(|x| x.to_string())
             .unwrap_or_default()
-    ))
+    );
+    crate::worker::artifact_cache_name(base, modules)
 }
 
 #[tracing::instrument(level = "trace", skip_all)]
@@ -679,11 +691,12 @@ pub async fn handle_rust_job(
     worker_name: &str,
     envs: HashMap<String, String>,
     occupancy_metrics: &mut OccupancyMetrics,
+    modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
 ) -> Result<Box<RawValue>, Error> {
     ensure_rust_runtime_dirs();
     check_executor_binary_exists("cargo", CARGO_PATH.as_str(), "rust")?;
 
-    let hash = rust_cache_key(inner_content, requirements_o, &job.workspace_id).await;
+    let hash = rust_cache_key(inner_content, requirements_o, &job.workspace_id, modules).await;
     let bin_path = format!("{}/{hash}", *RUST_CACHE_DIR);
     let remote_path = format!("{RUST_OBJECT_STORE_PREFIX}{hash}");
 

backend/windmill-worker/src/worker.rs

@@ -950,6 +950,51 @@ pub async fn workspace_registry_cache_suffix(w_id: &str) -> String {
     }
 }
 
+/// The name a build artifact is cached under, derived from `base` — the runnable's own
+/// cache-key input — and its inline modules.
+///
+/// `write_module_files` puts module content in the job dir where the build inlines it into
+/// the artifact, so a name without it serves one runnable's modules to another whose main
+/// content and lockfile match — across workspaces, the cache being global.
+///
+/// Only the path and content may name the artifact, because they are all the build reads.
+/// `ScriptModule::lock` especially must stay out: deploy regenerates it *after* the parent
+/// has prebuilt, so naming it would strand every prebuilt artifact.
+pub(crate) fn artifact_cache_name(
+    base: String,
+    modules: Option<&std::collections::HashMap<String, ScriptModule>>,
+) -> String {
+    let Some(modules) = modules.filter(|m| !m.is_empty()) else {
+        // Byte-identical to the name a module-free runnable had before modules entered this,
+        // so its cached artifacts stay reachable. A pre-fix multi-file runnable also stored
+        // here, so one of those stays reachable too — accepted over invalidating every cache,
+        // and once this ships nothing can be stored here with module content in it again.
+        return windmill_common::utils::calculate_hash(&base);
+    };
+    let mut entries: Vec<(&String, &ScriptModule)> = modules.iter().collect();
+    entries.sort_by(|a, b| a.0.cmp(b.0));
+    // `base` ends in caller-supplied bytes (a preview brings its own lockfile), so it is
+    // sealed to a fixed width before the module block is appended — raw, a crafted lockfile
+    // could spell out another runnable's block and reach its slot.
+    let mut keyed = format!(
+        "{}:modules:{}",
+        windmill_common::utils::calculate_hash(&base),
+        entries.len()
+    );
+    for (path, module) in entries {
+        // Both length-prefixed, else `{"a": "bc"}` and `{"ab": "c"}` encode alike.
+        keyed.push_str(&format!(
+            ":{}:{path}:{}:{}",
+            path.len(),
+            module.content.len(),
+            module.content,
+        ));
+    }
+    // Its own namespace: `calculate_hash` emits hex, so however a module-free runnable
+    // crafts its content and lockfile it can never land on a module-bearing name.
+    format!("mod-{}", windmill_common::utils::calculate_hash(&keyed))
+}
+
 pub fn is_sandboxing_enabled() -> bool {
     if !*DISABLE_NSJAIL {
         return true;
@@ -3532,7 +3577,13 @@ pub async fn run_worker(
                     job.kind,
                     JobKind::Script | JobKind::Preview | JobKind::FlowScript
                 ) {
-                    if !dedicated_workers.is_empty() {
+                    // A job carrying a pre-run error never runs its code: it only has to be
+                    // pulled so `handle_queued_job` can fail it. Both hand-off paths below
+                    // dispatch by path and return before that check, so a job sent down them
+                    // would run with whatever arguments survived the failure.
+                    let fails_before_running = job.pre_run_error.is_some();
+
+                    if !dedicated_workers.is_empty() && !fails_before_running {
                         let dedicated_worker_tx = job.runnable_path.as_ref().and_then(|path| {
                             // For flow steps inside branches/loops, runnable_path includes
                             // nesting segments (e.g. f/flow/branchone-0/a) but the dedicated
@@ -3577,7 +3628,7 @@ pub async fn run_worker(
                         NextJob::Http(_) => None,
                     };
 
-                    if let Some(flow_runners) = flow_runners {
+                    if let Some(flow_runners) = flow_runners.filter(|_| !fails_before_running) {
                         let key_o = job.flow_step_id.as_ref().map(|x| x.to_string());
                         if let Some(key) = key_o {
                             if let Some(flow_runner_tx) = flow_runners.runners.get(&key) {
@@ -5453,7 +5504,9 @@ async fn handle_code_execution_job(
         None => job,
     };
 
-    // For preview jobs, extract modules from args._MODULES if not already set
+    // Any job kind, not just previews: whatever is here is what gets written to the job dir
+    // and built in, so the agent-worker server precomputing a cache name has to resolve
+    // modules the same way (`windmill-api-agent-workers`, `get_code_and_lock`).
     let modules = modules_from_data.clone().or_else(|| {
         job.args.as_ref().and_then(|args| {
             args.get("_MODULES").and_then(|raw| {
@@ -5646,11 +5699,106 @@ mod write_module_files_tests {
     use super::*;
     use std::collections::HashMap;
     use windmill_common::scripts::ScriptLang;
+    use windmill_common::utils::calculate_hash;
 
     fn module(content: &str) -> ScriptModule {
         ScriptModule { content: content.to_string(), language: ScriptLang::Python3, lock: None }
     }
 
+    /// Every language's artifact cache name funnels module content through this, so an
+    /// ambiguous encoding puts two different runnables back on one name.
+    #[test]
+    fn artifact_name_cannot_be_re_cut_into_another_module_map() {
+        fn name(entries: &[(&str, &str)]) -> String {
+            let map: HashMap<String, ScriptModule> = entries
+                .iter()
+                .map(|(p, c)| (p.to_string(), module(c)))
+                .collect();
+            artifact_cache_name("base".to_string(), Some(&map))
+        }
+
+        // Naive `path + content` concatenation renders both of these as "abc".
+        assert_ne!(name(&[("a", "bc")]), name(&[("ab", "c")]));
+        // Splitting one module into two must not read back as the joined one.
+        assert_ne!(name(&[("a", "b"), ("c", "d")]), name(&[("ac", "bd")]));
+        // Iteration order of the map must not move the name.
+        assert_eq!(
+            name(&[("a", "1"), ("b", "2")]),
+            name(&[("b", "2"), ("a", "1")])
+        );
+    }
+
+    /// A module-free runnable must keep the exact name it had before modules entered the
+    /// derivation, or upgrading strands every artifact already in the cache.
+    #[test]
+    fn artifact_name_is_unchanged_without_modules() {
+        assert_eq!(
+            artifact_cache_name("code+lock".to_string(), None),
+            calculate_hash("code+lock")
+        );
+        assert_eq!(
+            artifact_cache_name("code+lock".to_string(), Some(&HashMap::new())),
+            calculate_hash("code+lock")
+        );
+    }
+
+    /// A preview brings its own source and lockfile, so a module-free runnable picks its
+    /// whole `base`. Module-bearing names live in their own namespace precisely so that no
+    /// crafted `base` can be made to land on one.
+    #[test]
+    fn a_module_free_runnable_cannot_forge_a_module_bearing_name() {
+        let modules = HashMap::from([("h.ts".to_string(), module("evil"))]);
+        let victim = artifact_cache_name("code+lock".to_string(), Some(&modules));
+
+        // `calculate_hash` emits hex, so the namespace is unreachable however `base` is
+        // chosen — including by feeding it the victim's own name.
+        assert!(victim.starts_with("mod-"));
+        assert_ne!(artifact_cache_name(victim.clone(), None), victim);
+        assert!(!artifact_cache_name("anything".to_string(), None).starts_with("mod-"));
+    }
+
+    /// The `mod-` namespace separates module-free from module-bearing, and nothing separates
+    /// two module-bearing runnables — only the seal does. Unsealed, `base` is variable-width,
+    /// so the split between it and the module block is ambiguous and a preview (which brings
+    /// its own source *and* lockfile) can absorb part of another runnable's block.
+    #[test]
+    fn a_module_bearing_runnable_cannot_absorb_another_ones_block() {
+        let victim = artifact_cache_name(
+            "V".to_string(),
+            Some(&HashMap::from([(
+                "h.ts".to_string(),
+                module(":modules:1:1:a:1:b"),
+            )])),
+        );
+        // Byte-identical to the victim's without the seal: the forger's `base` spells out the
+        // victim's leading block, leaving its own single module to supply the tail.
+        let forged = artifact_cache_name(
+            "V:modules:1:4:h.ts:18:".to_string(),
+            Some(&HashMap::from([("a".to_string(), module("b"))])),
+        );
+
+        assert_ne!(victim, forged);
+    }
+
+    /// Deploy fills a module's lock in after the parent has prebuilt, so a name that moved
+    /// with it would leave every prebuilt artifact unreachable by the runs it was built for.
+    #[test]
+    fn artifact_name_ignores_the_lock_deploy_fills_in_later() {
+        let prebuild = artifact_cache_name(
+            "base".to_string(),
+            Some(&HashMap::from([("h.ts".to_string(), module("x"))])),
+        );
+
+        let mut locked = module("x");
+        locked.lock = Some("{}\n//bun.lock\n<empty>".to_string());
+        let after_deploy = artifact_cache_name(
+            "base".to_string(),
+            Some(&HashMap::from([("h.ts".to_string(), locked)])),
+        );
+
+        assert_eq!(prebuild, after_deploy);
+    }
+
     #[test]
     fn contained_relative_path_rejects_traversal_and_absolute() {
         assert!(is_contained_relative_path("u/admin/pkg"));
@@ -6380,6 +6528,7 @@ mount {{
                 envs,
                 occupancy_metrics,
                 maybe_lock,
+                modules.as_ref(),
             ))
             .await
         }
@@ -6509,6 +6658,7 @@ mount {{
                     worker_name,
                     envs,
                     occupancy_metrics,
+                    modules.as_ref(),
                 ))
                 .await
             }
@@ -6567,6 +6717,7 @@ mount {{
                 worker_name,
                 envs,
                 occupancy_metrics,
+                modules.as_ref(),
             ))
             .await
         }
@@ -6631,6 +6782,7 @@ mount {{
                     worker_name,
                     envs,
                     occupancy_metrics,
+                    modules: modules.as_ref(),
                 }))
                 .await
             }

backend/windmill-worker/src/worker_flow.rs

@@ -70,9 +70,9 @@ use windmill_queue::schedule::get_schedule_opt;
 use windmill_queue::{
     add_completed_job, add_completed_job_error, append_logs, get_mini_pulled_job,
     insert_concurrency_key_capped, interpolate_args,
-    report_error_to_workspace_handler_or_critical_side_channel, try_schedule_next_job, CanceledBy,
-    FlowRunners, MiniCompletedJob, MiniPulledJob, PushArgs, PushIsolationLevel, SameWorkerPayload,
-    WrappedError,
+    report_error_to_workspace_handler_or_critical_side_channel, tag_reads_args,
+    try_schedule_next_job, CanceledBy, FlowRunners, MiniCompletedJob, MiniPulledJob, PushArgs,
+    PushIsolationLevel, SameWorkerPayload, WrappedError,
 };
 
 use windmill_audit::audit_oss::audit_log;
@@ -4403,6 +4403,23 @@ async fn push_next_flow_job(
             payload_tag.tag.as_deref(),
         );
 
+        // `push_args` is empty once the input transforms failed, so a tag reading `$args[...]`
+        // interpolates to a queue nobody serves and the step sits there instead of reporting
+        // the error. Send it to the flow's tag, which a worker is provably serving right now.
+        //
+        // A step handed over by id, or one whose tag `push` replaces, never reaches a worker
+        // through its tag, so rewriting theirs would be noise.
+        let step_is_pulled_by_tag = !continue_on_same_worker
+            && !continue_with_runners
+            && !payload_tag.payload.is_dedicated_worker();
+        let reroute_to_flow_tag =
+            err.is_some() && step_is_pulled_by_tag && tag.as_deref().is_some_and(tag_reads_args);
+        let tag = if reroute_to_flow_tag {
+            Some(flow_job.tag.clone())
+        } else {
+            tag
+        };
+
         let (email, permissioned_as) = if let Some(on_behalf_of) = payload_tag.on_behalf_of.as_ref()
         {
             (&on_behalf_of.email, on_behalf_of.permissioned_as.clone())
@@ -4421,8 +4438,7 @@ async fn push_next_flow_job(
             .as_deref()
             .filter(|t| !t.is_empty() && *t != flow_job.tag.as_str())
         {
-            let is_super_admin =
-                windmill_common::auth::is_super_admin_email(db, email).await?;
+            let is_super_admin = windmill_common::auth::is_super_admin_email(db, email).await?;
             check_tag_available_for_workspace_internal(
                 db,
                 &flow_job.workspace_id,
@@ -6155,9 +6171,7 @@ pub async fn script_to_payload(
                 .await?
                 .prefetch_cached(&db)
                 .await?;
-            let on_behalf_of = script_info
-                .on_behalf_of(&flow_job.workspace_id, db)
-                .await?;
+            let on_behalf_of = script_info.on_behalf_of(&flow_job.workspace_id, db).await?;
             let ScriptHashInfo {
                 tag,
                 cache_ttl,

backend/windmill-worker/src/worker_lockfiles.rs

@@ -199,6 +199,7 @@ async fn handle_build_binary_job(
                 worker_name,
                 base_internal_url,
                 occupancy_metrics,
+                script_data.modules.as_ref(),
             )
             .await?
         }
@@ -214,6 +215,7 @@ async fn handle_build_binary_job(
                 worker_name,
                 base_internal_url,
                 occupancy_metrics,
+                script_data.modules.as_ref(),
             )
             .await?
         }
@@ -235,6 +237,7 @@ async fn handle_build_binary_job(
                 worker_name,
                 base_internal_url,
                 occupancy_metrics,
+                script_data.modules.as_ref(),
             )
             .await?
         }
@@ -3155,7 +3158,11 @@ async fn capture_dependency_job(
             )
             .await?
             {
-                if !wd_exist {
+                // Nothing here writes the module files, so a bundle built now resolves a
+                // multi-file script's relative imports remotely rather than from its
+                // modules; caching that under a key naming them would serve the wrong
+                // code to every run. Leave it to the first run instead.
+                if !wd_exist && modules.map_or(true, |m| m.is_empty()) {
                     crate::bun_executor::prebundle_bun_script(
                         job_raw_code,
                         &lock,
@@ -3169,6 +3176,7 @@ async fn capture_dependency_job(
                         &token,
                         &mut Some(occupancy_metrics),
                         temp_script_refs,
+                        modules,
                     )
                     .await?;
                 }

benchmarks/lib.ts

@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
 import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
 import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
 
-export const VERSION = "v1.796.0";
+export const VERSION = "v1.800.0";
 
 export async function login(email: string, password: string): Promise<string> {
   return await windmill.UserService.login({

cli/src/commands/app/bundle.ts

@@ -191,7 +191,14 @@ function createSveltePlugin(appDir: string): any {
 
         // Convert Svelte syntax to JavaScript
         try {
-          const { js, warnings } = svelte.compile(source, { filename });
+          // The raw-app editor's in-browser bundler compiles with
+          // `css: "injected"`, so this must too, or the same app renders
+          // styled there and unstyled once the CLI builds it: Svelte's default
+          // ("external") hands the <style> back on a `css` field nothing emits.
+          const { js, warnings } = svelte.compile(source, {
+            filename,
+            css: "injected",
+          });
           const contents = js.code + `//# sourceMappingURL=` + js.map.toUrl();
           return { contents, warnings: warnings.map(convertMessage) };
         } catch (e: any) {

cli/src/commands/shared_ui.ts

@@ -26,37 +26,33 @@ async function readDirRecursive(
   return out;
 }
 
+/** The workspace's shared UI store, or undefined when it cannot be read. */
+async function fetchSharedUi(
+  workspace: string,
+): Promise<Record<string, string> | undefined> {
+  try {
+    const got = await wmill.getSharedUi({ workspace });
+    return got.files ?? {};
+  } catch {
+    return undefined;
+  }
+}
+
 export type SharedUiChange =
   | { type: "added"; path: string }
   | { type: "edited"; path: string; before: string; after: string }
   | { type: "deleted"; path: string };
 
 /**
- * Diff the local <cwd>/ui/ folder against the workspace's shared UI store in
- * the push direction (local -> remote), returning entries whose `path` is
- * prefixed with `ui/`. This is the same comparison pushSharedUi applies, so the
- * dry-run preview and the real push never diverge.
- *
- * Mirrors pushSharedUi's no-op: with no local ui/ folder there is nothing to
- * push, so the apply is a no-op and the preview must be empty (even when the
- * remote store is non-empty) to avoid phantom diffs the apply won't perform.
+ * The push-direction diff (local -> remote) of two already-read file maps.
+ * Split out so the preview and the push itself compare identically off one
+ * read of each side.
  */
-export async function diffSharedUi(workspace: string): Promise<SharedUiChange[]> {
-  const localDir = path.join(process.cwd(), SHARED_UI_DIR);
-  if (!fs.existsSync(localDir)) {
-    return [];
-  }
-  const files = await readDirRecursive(localDir);
-
-  let remote: Record<string, string> = {};
-  try {
-    const got = await wmill.getSharedUi({ workspace });
-    remote = got.files ?? {};
-  } catch {
-    // If endpoint missing or unauthorized, treat remote as empty (the push
-    // would attempt the PUT anyway).
-  }
-
+function sharedUiChanges(
+  files: Record<string, string>,
+  remote: Record<string, string>,
+  keepDeleted?: boolean,
+): SharedUiChange[] {
   // Use Object.hasOwn, not `in`: a file named after an Object.prototype member
   // (e.g. ui/toString) would otherwise register as always-present and be
   // misdiffed.
@@ -69,34 +65,97 @@ export async function diffSharedUi(workspace: string): Promise<SharedUiChange[]>
       changes.push({ type: "edited", path: p, before: remote[rel], after: content });
     }
   }
-  for (const rel of Object.keys(remote)) {
-    if (!Object.hasOwn(files, rel)) {
-      changes.push({ type: "deleted", path: `${SHARED_UI_DIR}/${rel}` });
+  if (!keepDeleted) {
+    for (const rel of Object.keys(remote)) {
+      if (!Object.hasOwn(files, rel)) {
+        changes.push({ type: "deleted", path: `${SHARED_UI_DIR}/${rel}` });
+      }
     }
   }
   return changes;
 }
 
+/**
+ * Diff the local <cwd>/ui/ folder against the workspace's shared UI store in
+ * the push direction (local -> remote), returning entries whose `path` is
+ * prefixed with `ui/`. This is the same comparison pushSharedUi applies, so the
+ * dry-run preview and the real push never diverge.
+ *
+ * Mirrors pushSharedUi's no-ops: with no local ui/ folder, or under
+ * `keepDeleted` with an unreadable store, the apply does nothing, so the
+ * preview must be empty (even when the remote store is non-empty) to avoid
+ * phantom diffs the apply won't perform.
+ */
+export async function diffSharedUi(
+  workspace: string,
+  keepDeleted?: boolean,
+): Promise<SharedUiChange[]> {
+  const localDir = path.join(process.cwd(), SHARED_UI_DIR);
+  if (!fs.existsSync(localDir)) {
+    return [];
+  }
+  const files = await readDirRecursive(localDir);
+  const remote = await fetchSharedUi(workspace);
+  if (remote === undefined && keepDeleted) {
+    return [];
+  }
+  // If endpoint missing or unauthorized, treat remote as empty (the push
+  // would attempt the PUT anyway).
+  return sharedUiChanges(files, remote ?? {}, keepDeleted);
+}
+
 /**
  * Push the local <cwd>/ui/ folder to the workspace's shared UI store.
  * Returns true if a push was performed, false if the folder is missing or
  * already matches the remote store. Note an empty-but-existing folder still
- * pushes an empty map (clearing the remote store) if the remote is non-empty.
+ * pushes an empty map (clearing the remote store) if the remote is non-empty —
+ * unless `keepDeleted`, which folds remote-only files back into the map.
  */
-export async function pushSharedUi(workspace: string): Promise<boolean> {
+export async function pushSharedUi(
+  workspace: string,
+  keepDeleted?: boolean,
+): Promise<boolean> {
   const localDir = path.join(process.cwd(), SHARED_UI_DIR);
   if (!fs.existsSync(localDir)) {
     return false;
   }
 
-  // Skip if no change — reuse diffSharedUi so preview and push never diverge.
-  const diff = await diffSharedUi(workspace);
-  if (diff.length === 0) {
+  const files = await readDirRecursive(localDir);
+  const remote = await fetchSharedUi(workspace);
+  // The store is written whole, so a remote-only file is pruned by omission
+  // alone. An unreadable store leaves no way to carry those files over, so
+  // the shared UI is left untouched rather than cleared.
+  if (remote === undefined && keepDeleted) {
+    log.warn(
+      colors.yellow(
+        "Could not read the shared UI folder from the remote; skipping its push so --keep-deleted does not clear it.",
+      ),
+    );
+    return false;
+  }
+
+  // Same comparison as diffSharedUi, off the same two maps, so preview and
+  // push never diverge.
+  if (sharedUiChanges(files, remote ?? {}, keepDeleted).length === 0) {
     log.info(colors.gray("Shared UI folder up to date"));
     return false;
   }
 
-  const files = await readDirRecursive(localDir);
+  if (keepDeleted) {
+    for (const [rel, content] of Object.entries(remote!)) {
+      // defineProperty, not assignment: `files.__proto__ = "..."` runs the
+      // inherited setter and creates no own property, so a remote `ui/__proto__`
+      // would be dropped from the whole-map PUT — i.e. deleted despite the flag.
+      if (!Object.hasOwn(files, rel)) {
+        Object.defineProperty(files, rel, {
+          value: content,
+          writable: true,
+          enumerable: true,
+          configurable: true,
+        });
+      }
+    }
+  }
   await wmill.updateSharedUi({
     workspace,
     requestBody: { files },
@@ -111,9 +170,12 @@ export async function pushSharedUi(workspace: string): Promise<boolean> {
 
 /**
  * Pull the workspace's shared UI store into <cwd>/ui/.
- * Files removed remotely are also removed locally.
+ * Files removed remotely are also removed locally, unless `keepDeleted`.
  */
-export async function pullSharedUi(workspace: string): Promise<boolean> {
+export async function pullSharedUi(
+  workspace: string,
+  keepDeleted?: boolean,
+): Promise<boolean> {
   const localDir = path.join(process.cwd(), SHARED_UI_DIR);
   let got;
   try {
@@ -145,15 +207,17 @@ export async function pullSharedUi(workspace: string): Promise<boolean> {
   }
 
   // Delete locally-orphaned files
-  const known = new Set(Object.keys(files));
-  const local = await readDirRecursive(localDir);
-  for (const rel of Object.keys(local)) {
-    if (!known.has(rel)) {
-      const full = path.join(localDir, rel);
-      try {
-        fs.unlinkSync(full);
-      } catch {
-        // ignore
+  if (!keepDeleted) {
+    const known = new Set(Object.keys(files));
+    const local = await readDirRecursive(localDir);
+    for (const rel of Object.keys(local)) {
+      if (!known.has(rel)) {
+        const full = path.join(localDir, rel);
+        try {
+          fs.unlinkSync(full);
+        } catch {
+          // ignore
+        }
       }
     }
   }

cli/src/commands/sync/sync.ts

@@ -3174,6 +3174,28 @@ export function untrackedDatatableMigrationDeletions<
   );
 }
 
+/**
+ * `--keep-deleted`: strip every deletion from the changeset, in place, so the
+ * sync only adds and updates. A path missing on one side is not on its own
+ * evidence that it should go from the other — a partial clone, a scoped
+ * checkout or an item authored in the UI all read as deletions here.
+ */
+function dropDeletions(changes: Change[], keptOn: "local" | "remote"): void {
+  const deletions = changes.filter((c) => c.name === "deleted");
+  if (deletions.length === 0) return;
+  const kept = changes.filter((c) => c.name !== "deleted");
+  changes.length = 0;
+  changes.push(...kept);
+  log.info(
+    colors.yellow(
+      `--keep-deleted: keeping ${deletions.length} item(s) that exist only ` +
+        (keptOn === "local"
+          ? `on disk instead of deleting them locally`
+          : `on the remote instead of deleting them from the workspace`),
+    ),
+  );
+}
+
 interface ChangeTracker {
   scripts: string[];
   flows: string[];
@@ -3433,6 +3455,7 @@ export async function pull(
       repository?: string;
       promotion?: string;
       branch?: string;
+      keepDeleted?: boolean;
       useIndividualBranch?: boolean;
       groupByFolder?: boolean;
       gitDeployItems?: string;
@@ -3684,6 +3707,10 @@ export async function pull(
     await isCaseInsensitiveFilesystem(process.cwd()),
   );
 
+  if (opts.keepDeleted) {
+    dropDeletions(changes, "local");
+  }
+
   log.info(
     `remote (${workspace.name}) -> local: ${changes.length} changes to apply`,
   );
@@ -4117,10 +4144,14 @@ export async function pull(
     }
   }
 
-  try {
-    await pullSharedUi(workspace.workspaceId);
-  } catch (e) {
-    log.warn(`Failed to pull shared UI folder: ${e}`);
+  // Skipped under --dry-run since pullSharedUi writes to the local ui/ folder.
+  // An empty changeset falls through the return above and reaches here.
+  if (!opts.dryRun) {
+    try {
+      await pullSharedUi(workspace.workspaceId, opts.keepDeleted);
+    } catch (e) {
+      log.warn(`Failed to pull shared UI folder: ${e}`);
+    }
   }
 
   // Datatable migrations are part of the workspace export now, so they flow
@@ -4425,12 +4456,19 @@ function removeSuffix(str: string, suffix: string) {
 }
 
 // Shown after a `wmill sync push --dry-run` preview that has changes. `sync push`
-// deploys to the remote workspace and is destructive (it overwrites and prunes
-// remote items that differ from or are absent locally), so the preview reminds
-// the caller — especially an AI agent that ran the dry-run to inspect changes —
-// to get explicit user confirmation before applying it for real.
-const SYNC_PUSH_DESTRUCTIVE_WARNING =
-  "`wmill sync push` is destructive: applying it deploys these changes to the remote workspace and overwrites or deletes remote items that differ from or are absent locally — this is not automatically reversible. If you are an AI agent, do NOT run `wmill sync push` (without --dry-run) until the user has explicitly confirmed this deploy, unless your custom instructions explicitly allow bypassing that confirmation.";
+// deploys to the remote workspace and is destructive (it overwrites remote items
+// that differ from local, and prunes those absent locally unless --keep-deleted),
+// so the preview reminds the caller — especially an AI agent that ran the dry-run
+// to inspect changes — to get explicit user confirmation before applying it for real.
+function syncPushDestructiveWarning(keepDeleted?: boolean): string {
+  return (
+    "`wmill sync push` is destructive: applying it deploys these changes to the remote workspace and overwrites " +
+    (keepDeleted
+      ? "remote items that differ from local"
+      : "or deletes remote items that differ from or are absent locally") +
+    " — this is not automatically reversible. If you are an AI agent, do NOT run `wmill sync push` (without --dry-run) until the user has explicitly confirmed this deploy, unless your custom instructions explicitly allow bypassing that confirmation."
+  );
+}
 
 // A script pushed without a local lock queues a server-side dependency job; if
 // that job fails the script deploys broken (no lock/assets) with no CLI signal.
@@ -4490,6 +4528,7 @@ export async function push(
     SyncOptions & {
       repository?: string;
       branch?: string;
+      keepDeleted?: boolean;
       acceptOverridingPermissionedAsWithSelf?: boolean;
     },
 ) {
@@ -4791,6 +4830,13 @@ export async function push(
     );
   }
 
+  // After the shared-lock pass, which reads a shared lockfile's deletion as the
+  // signal that this checkout is not deduplicated — an advisory about the local
+  // tree that holds whether or not remote items are being kept.
+  if (opts.keepDeleted) {
+    dropDeletions(changes, "remote");
+  }
+
   const autoRegenerate = !!(opts as any).autoMetadata;
   const staleScripts: string[] = [];
   const staleFlows: string[] = [];
@@ -5103,7 +5149,10 @@ export async function push(
   // unchanged (pushSharedUi still runs) and the summary count includes ui/.
   if (opts.dryRun) {
     try {
-      for (const c of await diffSharedUi(workspace.workspaceId)) {
+      for (const c of await diffSharedUi(
+        workspace.workspaceId,
+        opts.keepDeleted,
+      )) {
         if (c.type === "added") {
           changes.push({ name: "added", path: c.path, content: "" });
         } else if (c.type === "deleted") {
@@ -5275,7 +5324,9 @@ export async function push(
           : {}),
       })),
       total: changes.length,
-      ...(changes.length > 0 ? { warning: SYNC_PUSH_DESTRUCTIVE_WARNING } : {}),
+      ...(changes.length > 0
+        ? { warning: syncPushDestructiveWarning(opts.keepDeleted) }
+        : {}),
     };
     console.log(JSON.stringify(result, null, 2));
     return;
@@ -5339,7 +5390,9 @@ export async function push(
 
     if (opts.dryRun) {
       log.info(colors.gray(`Dry run complete.`));
-      log.warn(colors.yellow(`\n⚠ ${SYNC_PUSH_DESTRUCTIVE_WARNING}`));
+      log.warn(
+        colors.yellow(`\n⚠ ${syncPushDestructiveWarning(opts.keepDeleted)}`),
+      );
       return;
     }
 
@@ -6313,7 +6366,7 @@ export async function push(
       }
     }
     try {
-      await pushSharedUi(workspace.workspaceId);
+      await pushSharedUi(workspace.workspaceId, opts.keepDeleted);
     } catch (e) {
       log.warn(`Failed to push shared UI folder: ${e}`);
     }
@@ -6415,7 +6468,10 @@ export async function push(
     let sharedUiPushed = false;
     if (!opts.dryRun) {
       try {
-        sharedUiPushed = await pushSharedUi(workspace.workspaceId);
+        sharedUiPushed = await pushSharedUi(
+          workspace.workspaceId,
+          opts.keepDeleted,
+        );
       } catch (e) {
         log.warn(`Failed to push shared UI folder: ${e}`);
       }
@@ -6480,6 +6536,10 @@ const command = new Command()
   .option("--include-groups", "Include syncing groups")
   .option("--include-settings", "Include syncing workspace settings")
   .option("--include-key", "Include workspace encryption key")
+  .option(
+    "--keep-deleted",
+    "Do not delete local files for items that no longer exist on the remote workspace. Only adds and updates.",
+  )
   .option("--skip-branch-validation", "Skip git branch validation and prompts")
   .option("--json-output", "Output results in JSON format")
   .option(
@@ -6543,6 +6603,10 @@ const command = new Command()
     "--skip-reencrypt-on-key-change",
     "When the pushed encryption key differs from the remote, do NOT re-encrypt existing remote secrets. Only safe if they are already encrypted with the new key (e.g. workspace/instance migration). Default is to re-encrypt.",
   )
+  .option(
+    "--keep-deleted",
+    "Do not delete remote items that no longer exist locally. Only adds and updates.",
+  )
   .option("--skip-branch-validation", "Skip git branch validation and prompts")
   .option("--json-output", "Output results in JSON format")
   .option(

cli/src/core/constants.ts

@@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork";
 // (e.g. utils.ts) can read it without importing main.ts and creating a circular
 // dependency (main → workspace → utils → main) that triggers a TDZ.
 // Re-exported from main.ts for backwards compatibility.
-export const VERSION = "1.796.0";
+export const VERSION = "1.800.0";

cli/src/guidance/skills.gen.ts

@@ -7533,6 +7533,7 @@ sync local with a remote workspaces or the opposite (push or pull)
   - \`--include-groups\` - Include syncing groups
   - \`--include-settings\` - Include syncing workspace settings
   - \`--include-key\` - Include workspace encryption key
+  - \`--keep-deleted\` - Do not delete local files for items that no longer exist on the remote workspace. Only adds and updates.
   - \`--skip-branch-validation\` - Skip git branch validation and prompts
   - \`--json-output\` - Output results in JSON format
   - \`-i --includes <patterns:file[]>\` - Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string). Overrides wmill.yaml includes
@@ -7564,6 +7565,7 @@ sync local with a remote workspaces or the opposite (push or pull)
   - \`--include-settings\` - Include syncing workspace settings
   - \`--include-key\` - Include workspace encryption key
   - \`--skip-reencrypt-on-key-change\` - When the pushed encryption key differs from the remote, do NOT re-encrypt existing remote secrets. Only safe if they are already encrypted with the new key (e.g. workspace/instance migration). Default is to re-encrypt.
+  - \`--keep-deleted\` - Do not delete remote items that no longer exist locally. Only adds and updates.
   - \`--skip-branch-validation\` - Skip git branch validation and prompts
   - \`--json-output\` - Output results in JSON format
   - \`-i --includes <patterns:file[]>\` - Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string)
@@ -7949,7 +7951,9 @@ properties:
           at once (1-65535)
   error_handler_path:
     type: string
-    description: Path to a script or flow to run when the triggered job fails
+    description: Path to a script to run when the triggered job fails. A bare path,
+      without the script/ or flow/ prefix a schedule error handler takes; it cannot
+      be a flow.
   error_handler_args:
     type: object
     description: The arguments to pass to the script or flow
@@ -8399,7 +8403,9 @@ properties:
       as JSON
   error_handler_path:
     type: string
-    description: Path to a script or flow to run when the triggered job fails
+    description: Path to a script to run when the triggered job fails. A bare path,
+      without the script/ or flow/ prefix a schedule error handler takes; it cannot
+      be a flow.
   error_handler_args:
     type: object
     description: The arguments to pass to the script or flow
@@ -8566,7 +8572,9 @@ properties:
       endpoint.
   error_handler_path:
     type: string
-    description: Path to a script or flow to run when the triggered job fails
+    description: Path to a script to run when the triggered job fails. A bare path,
+      without the script/ or flow/ prefix a schedule error handler takes; it cannot
+      be a flow.
   error_handler_args:
     type: object
     description: The arguments to pass to the script or flow
@@ -8690,7 +8698,9 @@ properties:
     - v5
   error_handler_path:
     type: string
-    description: Path to a script or flow to run when the triggered job fails
+    description: Path to a script to run when the triggered job fails. A bare path,
+      without the script/ or flow/ prefix a schedule error handler takes; it cannot
+      be a flow.
   error_handler_args:
     type: object
     description: The arguments to pass to the script or flow
@@ -8786,7 +8796,9 @@ properties:
     description: Array of NATS subjects to subscribe to
   error_handler_path:
     type: string
-    description: Path to a script or flow to run when the triggered job fails
+    description: Path to a script to run when the triggered job fails. A bare path,
+      without the script/ or flow/ prefix a schedule error handler takes; it cannot
+      be a flow.
   error_handler_args:
     type: object
     description: The arguments to pass to the script or flow
@@ -8876,7 +8888,9 @@ properties:
     description: Name of the PostgreSQL logical replication slot to use
   error_handler_path:
     type: string
-    description: Path to a script or flow to run when the triggered job fails
+    description: Path to a script to run when the triggered job fails. A bare path,
+      without the script/ or flow/ prefix a schedule error handler takes; it cannot
+      be a flow.
   error_handler_args:
     type: object
     description: The arguments to pass to the script or flow
@@ -9123,7 +9137,9 @@ properties:
     description: Array of SQS message attribute names to include with each message
   error_handler_path:
     type: string
-    description: Path to a script or flow to run when the triggered job fails
+    description: Path to a script to run when the triggered job fails. A bare path,
+      without the script/ or flow/ prefix a schedule error handler takes; it cannot
+      be a flow.
   error_handler_args:
     type: object
     description: The arguments to pass to the script or flow
@@ -9322,7 +9338,9 @@ properties:
           The extracted value replaces {{state}} in the heartbeat message.
   error_handler_path:
     type: string
-    description: Path to a script or flow to run when the triggered job fails
+    description: Path to a script to run when the triggered job fails. A bare path,
+      without the script/ or flow/ prefix a schedule error handler takes; it cannot
+      be a flow.
   error_handler_args:
     type: object
     description: The arguments to pass to the script or flow

cli/test/raw_app_svelte_plugin_unit.test.ts

@@ -1,8 +1,5 @@
 /**
- * `lib.svelte.ts` / `lib.svelte.js` modules are plain modules that may use
- * runes. They need `svelte.compileModule`; without it esbuild happily bundles
- * `$state(...)` as an ordinary call and the app dies at runtime with
- * "ReferenceError: $state is not defined".
+ * The svelte esbuild plugin, driven through `createBundle`.
  */
 
 import { afterAll, beforeAll, describe, expect, test } from "bun:test";
@@ -59,6 +56,12 @@ afterAll(() => {
   fs.rmSync(tempDir, { recursive: true, force: true });
 });
 
+/**
+ * `lib.svelte.ts` / `lib.svelte.js` modules are plain modules that may use
+ * runes. They need `svelte.compileModule`; without it esbuild happily bundles
+ * `$state(...)` as an ordinary call and the app dies at runtime with
+ * "ReferenceError: $state is not defined".
+ */
 describe("svelte plugin: .svelte.ts modules", () => {
   test("compiles runes in a TypeScript rune module and the bundle runs", async () => {
     writeApp({
@@ -116,3 +119,36 @@ bump();
     expect((globalThis as any).__counterResult).toBe(2);
   });
 });
+
+/**
+ * Svelte's default `css: "external"` hands a component's <style> back on a
+ * field the plugin never emits, so the markup keeps its `svelte-<hash>` class
+ * while the rule matching it disappears — no build error, just an app that
+ * renders unstyled from the CLI and styled in the editor.
+ */
+describe("svelte plugin: component styles", () => {
+  test("a <style> block reaches the bundle under the class its markup carries", async () => {
+    writeApp({
+      "Styled.svelte": `<main>
+  <h1>Hello</h1>
+</main>
+
+<style>
+  h1 {
+    font-size: 1.5rem;
+  }
+</style>
+`,
+      "styles_entry.ts": `import Styled from './Styled.svelte';
+export default Styled;
+`,
+    });
+
+    const js = await bundle("styles_entry.ts");
+
+    const scopeClass = js.match(/<h1 class="(svelte-[a-z0-9]+)"/)?.[1];
+    expect(scopeClass).toBeDefined();
+    expect(js).toContain(`h1.${scopeClass}`);
+    expect(js).toContain("font-size");
+  });
+});

cli/test/setup.ts

@@ -10,6 +10,12 @@
  *    bear the startup cost inside their per-test timeout window.
  */
 
+// esbuild's node API pins the cwd it spawns its service with to process.cwd() at
+// module import, and test files chdir into temp dirs they later delete. Import it
+// here, from a cwd that outlives the run, or the first file to bundle pins the
+// service to a directory that stops existing and every later build fails ENOENT.
+import "esbuild";
+
 if (process.env["UNIT_ONLY"]) {
   // Nothing to do — unit tests don't need backend setup
 } else {

cli/test/shared_ui_diff_unit.test.ts

@@ -11,12 +11,25 @@ import * as os from "node:os";
 import * as path from "node:path";
 
 let remoteFiles: Record<string, string> = {};
+let remoteUnreadable = false;
+let pushedFiles: Record<string, string> | undefined;
 
 mock.module("../gen/services.gen.ts", () => ({
-  getSharedUi: async (_args: { workspace: string }) => ({ files: remoteFiles }),
+  getSharedUi: async (_args: { workspace: string }) => {
+    if (remoteUnreadable) throw new Error("shared UI store unreadable");
+    return { files: remoteFiles };
+  },
+  updateSharedUi: async (args: {
+    workspace: string;
+    requestBody: { files: Record<string, string> };
+  }) => {
+    pushedFiles = args.requestBody.files;
+  },
 }));
 
-const { diffSharedUi } = await import("../src/commands/shared_ui.ts");
+const { diffSharedUi, pushSharedUi } = await import(
+  "../src/commands/shared_ui.ts"
+);
 
 describe("diffSharedUi", () => {
   const ws = "test-workspace";
@@ -25,6 +38,8 @@ describe("diffSharedUi", () => {
 
   beforeEach(() => {
     remoteFiles = {};
+    remoteUnreadable = false;
+    pushedFiles = undefined;
     prevCwd = process.cwd();
     tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "wm-shared-ui-"));
     process.chdir(tmpDir);
@@ -89,4 +104,46 @@ describe("diffSharedUi", () => {
     const changes = await diffSharedUi(ws);
     expect(changes).toEqual([]);
   });
+
+  test("emits nothing under keepDeleted when the remote store is unreadable", async () => {
+    // pushSharedUi skips the push rather than clearing a store it can't read,
+    // so the preview must show that same nothing.
+    remoteUnreadable = true;
+    writeUi("theme.json", "{}");
+    expect(await diffSharedUi(ws, true)).toEqual([]);
+  });
+});
+
+describe("pushSharedUi with keepDeleted", () => {
+  const ws = "test-workspace";
+  let tmpDir: string;
+  let prevCwd: string;
+
+  beforeEach(() => {
+    remoteFiles = {};
+    remoteUnreadable = false;
+    pushedFiles = undefined;
+    prevCwd = process.cwd();
+    tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "wm-shared-ui-push-"));
+    process.chdir(tmpDir);
+  });
+
+  afterEach(() => {
+    process.chdir(prevCwd);
+    fs.rmSync(tmpDir, { recursive: true, force: true });
+  });
+
+  test("carries remote-only files, including ui/__proto__, into the pushed map", async () => {
+    // JSON.parse, not a literal: `{__proto__: …}` sets the prototype instead of
+    // creating the own property the API response really has.
+    remoteFiles = JSON.parse('{"__proto__":"keep me","extra.json":"1"}');
+    fs.mkdirSync(path.join(tmpDir, "ui"), { recursive: true });
+    fs.writeFileSync(path.join(tmpDir, "ui", "theme.json"), "{}", "utf-8");
+
+    expect(await pushSharedUi(ws, true)).toBe(true);
+    // The store is written whole, so anything missing here is deleted.
+    expect(pushedFiles!["extra.json"]).toEqual("1");
+    expect(Object.getOwnPropertyDescriptor(pushedFiles!, "__proto__")?.value)
+      .toEqual("keep me");
+  });
 });

cli/test/sync_pull_push.test.ts

@@ -2774,3 +2774,102 @@ kind: script
     });
   });
 });
+
+describe("keep deleted", () => {
+  test("Integration: --keep-deleted keeps items absent from the other side", async () => {
+    await withTestBackend(async (backend, tempDir) => {
+      const uniqueId = Date.now();
+      const scriptPath = `f/test/keep_deleted_${uniqueId}`;
+
+      const resp = await backend.apiRequest!(
+        `/api/w/${backend.workspace}/scripts/create`,
+        {
+          method: "POST",
+          headers: { "Content-Type": "application/json" },
+          body: JSON.stringify({
+            path: scriptPath,
+            content: 'export async function main() { return "keep me"; }',
+            language: "bun",
+            summary: "Kept by --keep-deleted",
+            schema: {
+              $schema: "https://json-schema.org/draft/2020-12/schema",
+              type: "object",
+              properties: {},
+              required: [],
+            },
+          }),
+        }
+      );
+      expect(resp.status).toBeLessThan(300);
+      await resp.text();
+
+      await writeWmillYaml(tempDir);
+      expect(
+        (await backend.runCLICommand(["sync", "pull", "--yes"], tempDir)).code
+      ).toEqual(0);
+
+      const contentFile = `${scriptPath}.ts`;
+      const metadataFile = `${scriptPath}.script.yaml`;
+      expect(await listFilesRecursive(tempDir)).toContain(contentFile);
+
+      // Push direction: the remote script survives losing its local files.
+      await rm(join(tempDir, contentFile));
+      await rm(join(tempDir, metadataFile));
+      expect(
+        (
+          await backend.runCLICommand(
+            ["sync", "push", "--yes", "--keep-deleted"],
+            tempDir
+          )
+        ).code
+      ).toEqual(0);
+      // A push deletion archives the script rather than removing the row, so
+      // `archived` — not the status code — is what says it survived.
+      const remote = await backend.apiRequest!(
+        `/api/w/${backend.workspace}/scripts/get/p/${scriptPath}`
+      );
+      expect(remote.status).toEqual(200);
+      expect((await remote.json()).archived).not.toEqual(true);
+
+      // Pull direction: a file with no remote counterpart survives the pull.
+      const localOnly = `f/test/local_only_${uniqueId}.ts`;
+      await writeFile(
+        join(tempDir, localOnly),
+        'export async function main() { return "local only"; }',
+        "utf-8"
+      );
+      expect(
+        (
+          await backend.runCLICommand(
+            ["sync", "pull", "--yes", "--keep-deleted"],
+            tempDir
+          )
+        ).code
+      ).toEqual(0);
+      const afterPull = await listFilesRecursive(tempDir);
+      expect(afterPull).toContain(localOnly);
+      // Adds still apply: the script deleted above is written back.
+      expect(afterPull).toContain(contentFile);
+
+      // An empty changeset falls past the dry-run return, on to the shared-UI
+      // step — which writes to disk, so a dry run must skip it.
+      // Including the metadata and lock the pull's auto-fill generated for it.
+      for (const ext of [".ts", ".script.yaml", ".script.lock"]) {
+        await rm(join(tempDir, `f/test/local_only_${uniqueId}${ext}`), {
+          force: true,
+        });
+      }
+      await mkdir(join(tempDir, "ui"), { recursive: true });
+      await writeFile(join(tempDir, "ui", "custom.css"), "body{}", "utf-8");
+      const dryRun = await backend.runCLICommand(
+        ["sync", "pull", "--dry-run"],
+        tempDir
+      );
+      expect(dryRun.code).toEqual(0);
+      // Guards against a vacuous pass: a non-empty changeset would return at
+      // the dry-run check above and never reach the shared-UI step.
+      expect(dryRun.stdout + dryRun.stderr).toContain("0 changes to apply");
+      expect(await listFilesRecursive(tempDir)).toContain("ui/custom.css");
+    });
+  });
+});

docker-compose.yml

@@ -8,15 +8,28 @@ x-logging: &default-logging
     compress: "true"
 
 services:
+  ## UPGRADING FROM POSTGRES 16: db_data holds a cluster 18 cannot read, so the
+  ## container exits with an explanatory error rather than coming up blank. Migrating
+  ## means dumping the WHOLE cluster (pg_dumpall), never just the windmill database:
+  ## Windmill keeps datatable, DuckLake and wm_fork_* databases beside it and grants
+  ## its RLS policies to cluster-level roles, and a single-database dump loses both
+  ## silently. Full procedure, and why 16 is still a valid choice until Nov 2028:
+  ## https://www.windmill.dev/docs/advanced/self_host#upgrade-postgresql-to-18
   db:
     deploy:
       # To use an external database, set replicas to 0 and set DATABASE_URL to the external database url in the .env file
       replicas: 1
-    image: postgres:16
+    image: postgres:18
     shm_size: 1g
     restart: unless-stopped
     volumes:
-      - db_data:/var/lib/postgresql/data
+      # From 18 on the official image keeps the cluster in a major-version
+      # subdirectory (/var/lib/postgresql/18/docker), so the mount has to be the
+      # parent directory: that is what lets pg_upgrade see an old and a new
+      # cluster inside a single mount point. Mounting the pre-18 .../data path
+      # instead makes the image exit rather than start, which is what turns a
+      # stale 16 cluster into a loud failure instead of an empty instance.
+      - db_data:/var/lib/postgresql
     expose:
       - 5432
     environment:

docs/feature-telemetry.md

@@ -4,10 +4,10 @@
 anonymous usage-stats payload. It answers "does anyone use this, and which variant do they pick"
 without any identifying data leaving the instance.
 
-It currently carries 21 registered actions across nine features (`ai_session`, `ai_chat`,
-`flow_editor`, `flow_run`, `flow_step`, `trigger`, `command_script`, `hub_script`,
-`usage_meter`). Nearly all of the product is uninstrumented, so new user-facing work is the
-opportunity to change that.
+It currently carries 28 registered actions across fourteen features (`ai_session`, `ai_chat`,
+`ai_fix`, `ai_agent`, `ai_agent_eval`, `flow_editor`, `flow_run`, `flow_step`, `run_form`,
+`debugger`, `trigger`, `command_script`, `hub_script`, `usage_meter`). Nearly all of the
+product is uninstrumented, so new user-facing work is the opportunity to change that.
 
 ## When to instrument
 

frontend/package-lock.json

@@ -1,12 +1,12 @@
 {
 	"name": "@windmill-labs/components",
-	"version": "1.796.0",
+	"version": "1.800.0",
 	"lockfileVersion": 3,
 	"requires": true,
 	"packages": {
 		"": {
 			"name": "@windmill-labs/components",
-			"version": "1.796.0",
+			"version": "1.800.0",
 			"hasInstallScript": true,
 			"license": "AGPL-3.0",
 			"dependencies": {
@@ -29,6 +29,7 @@
 				"@windmill-labs/svelte-dnd-action": "^0.9.44",
 				"@xterm/addon-fit": "^0.10.0",
 				"@xyflow/svelte": "^1.0.0",
+				"acorn": "^8.15.0",
 				"ag-charts-community": "^9.0.1",
 				"ag-charts-enterprise": "^9.0.1",
 				"ag-grid-community": "^31.3.4",
@@ -13195,9 +13196,9 @@
 			}
 		},
 		"node_modules/svelte-check/node_modules/picomatch": {
-			"version": "4.0.5",
-			"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.5.tgz",
-			"integrity": "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A==",
+			"version": "4.0.7",
+			"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.7.tgz",
+			"integrity": "sha512-qcJu88Q2IWqJsDD529JKMdwGm/dvInW4HvQnRwiH9JtihJvzGOscDtHE3x1pBKeUOTysQ8kVmLnJ2kJu7yhcGA==",
 			"dev": true,
 			"license": "MIT",
 			"optional": true,

frontend/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@windmill-labs/components",
-	"version": "1.796.0",
+	"version": "1.800.0",
 	"scripts": {
 		"dev": "vite dev",
 		"dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev",
@@ -105,6 +105,7 @@
 		"@windmill-labs/svelte-dnd-action": "^0.9.44",
 		"@xterm/addon-fit": "^0.10.0",
 		"@xyflow/svelte": "^1.0.0",
+		"acorn": "^8.15.0",
 		"ag-charts-community": "^9.0.1",
 		"ag-charts-enterprise": "^9.0.1",
 		"ag-grid-community": "^31.3.4",

frontend/src/lib/aiStore.ts

@@ -7,6 +7,7 @@ import {
 	type AIProviderModel,
 	type AIProvider,
 	type AIConfig,
+	type FreeTierInfo,
 	type ModelPriceOverride
 } from './gen'
 import {
@@ -49,6 +50,10 @@ export const copilotInfo = writable<{
 	/** Negotiated rates per `provider:model`, overriding the built-in price table. */
 	modelPricing?: Record<string, ModelPriceOverride>
 	webSearchEnabledProviders?: Partial<Record<AIProvider, boolean>>
+	// Set only when the workspace has no AI provider of its own and is running on
+	// Windmill's free tier. `exhausted` means the grant is spent: there is no model, but
+	// that is a different state from "never configured" and the UI must say so.
+	freeTier?: FreeTierInfo
 }>({
 	enabled: false,
 	codeCompletionModel: undefined,
@@ -132,8 +137,9 @@ export function setCopilotInfo(aiConfig: AIConfig) {
 			aiModels: aiModels,
 			customPrompts: aiConfig.custom_prompts ?? {},
 			maxTokensPerModel: aiConfig.max_tokens_per_model ?? {},
+			webSearchEnabledProviders,
 			modelPricing: aiConfig.model_pricing ?? {},
-			webSearchEnabledProviders
+			freeTier: aiConfig.free_tier
 		})
 	} else {
 		copilotSessionModel.set(undefined)
@@ -146,8 +152,11 @@ export function setCopilotInfo(aiConfig: AIConfig) {
 			aiModels: [],
 			customPrompts: {},
 			maxTokensPerModel: {},
+			webSearchEnabledProviders: {},
 			modelPricing: {},
-			webSearchEnabledProviders: {}
+			// An exhausted free grant lands here — no providers, but the reason AI is off
+			// is "you used it up", not "you never set it up".
+			freeTier: aiConfig.free_tier
 		})
 	}
 }

frontend/src/lib/components/AppConnectDrawer.svelte

@@ -6,7 +6,7 @@
 	import DrawerContent from './common/drawer/DrawerContent.svelte'
 
 	import AppConnectInner from './AppConnectInner.svelte'
-	import DarkModeObserver from './DarkModeObserver.svelte'
+	import GoogleSigninButton from './GoogleSigninButton.svelte'
 	import IconedResourceType from './IconedResourceType.svelte'
 	import { addResourceTitle } from './resourceTypeDisplay'
 
@@ -22,6 +22,10 @@
 		disableChatOffset = false
 	}: Props = $props()
 
+	/** Set by `open(rt, fillPath)`, not by the parent: which resource this run fills is a
+	 *  property of the click, and a prop would go stale between two different rows. */
+	let fillPath: string | undefined = $state(undefined)
+
 	let drawer: Drawer | undefined = $state()
 	let resourceType = $state('')
 	let step = $state(1)
@@ -32,29 +36,46 @@
 	let appConnectInner: AppConnectInner | undefined = $state(undefined)
 
 	let rtToLoad: string | undefined = $state('')
-	export async function open(rt?: string) {
+	/** `fill` connects into a resource that already exists, instead of creating one. */
+	export async function open(rt?: string, fill?: string) {
+		fillPath = fill
+		handedOff = false
 		rtToLoad = rt
 		drawer?.openDrawer?.()
 	}
 
+	/**
+	 * Hand off to the inner component exactly once per opening. The reactive statement below
+	 * re-runs both when `rtToLoad` changes and when `appConnectInner` binds — and it binds
+	 * afresh on every opening, since the drawer destroys its content on close. A second
+	 * `open()` runs `next()` a second time, which walks a drawer opened on a resource type
+	 * straight past the Connect button and into `window.open`; a popup opened from a reactive
+	 * effect rather than from the click is blocked, leaving "Finish connection in popup
+	 * window" with no popup behind it.
+	 *
+	 * A flag rather than the last resource type: `open()` with no argument leaves `rtToLoad`
+	 * undefined, which compares equal to the initial state and would skip the hand-off
+	 * entirely — the resources page opens it that way.
+	 */
+	let handedOff = false
 	function onRtToLoadChange(rtToLoad: string | undefined) {
+		if (handedOff) return
+		handedOff = true
 		appConnectInner?.open(rtToLoad)
 	}
 
 	const dispatch = createEventDispatcher()
 
-	let darkMode: boolean = $state(false)
 	run(() => {
 		appConnectInner && onRtToLoadChange(rtToLoad)
 	})
 </script>
 
-<DarkModeObserver bind:darkMode />
-
 <Drawer
 	bind:this={drawer}
 	on:close={() => {
 		step = 1
+		handedOff = false
 		dispatch('close')
 	}}
 	size="700px"
@@ -83,22 +104,26 @@
 			on:refresh
 			express={expressOAuthSetup}
 			{workspace}
+			{fillPath}
 		/>
 		{#snippet actions()}
 			<div class="flex gap-1">
-				{#if step > 1}
-					<Button variant="default" on:click={appConnectInner?.back ?? (() => {})}>Back</Button>
+				<!-- Only when the user came through the type picker: opening the drawer for one
+				     resource type skips step 1, so Back would land on a list they never chose from. -->
+				{#if step > 1 && !rtToLoad}
+					<Button variant="default" unifiedSize="md" onClick={() => appConnectInner?.back()}>
+						Back
+					</Button>
 				{/if}
 				{#if isGoogleSignin}
-					<button {disabled} onclick={appConnectInner?.next}>
-						<img
-							class="h-10 w-auto object-contain"
-							src={darkMode ? '/google_signin_dark.png' : '/google_signin_light.png'}
-							alt="Google sign-in"
-						/>
-					</button>
+					<GoogleSigninButton {disabled} onClick={() => appConnectInner?.next()} />
 				{:else}
-					<Button variant="accent" {disabled} on:click={appConnectInner?.next ?? (() => {})}>
+					<Button
+						variant="accent"
+						unifiedSize="md"
+						{disabled}
+						onClick={() => appConnectInner?.next()}
+					>
 						{#if step == 2 && !manual}
 							Connect
 						{:else if step == 1}