starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-10-19 08:03:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into ruben/win-2455-fix-pass-correct-worker_dir-to-handle_python_deps-in

Browse Source 
# Conflicts:
#	backend/ee-repo-ref.txt
This commit is contained in:
Ruben Fiszel
2026-09-01 13:14:43 +00:00
parent f77ba72a08 4808f21b6b
commit 09365c6740
290 changed files with 14359 additions and 4981 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/backend-test-windows.yml
-1
View File
@@ -101,7 +101,6 @@ jobs:
- name: Install OpenSSL via vcpkg
run: |
vcpkg.exe install openssl-windows:x64-windows
vcpkg.exe install openssl:x64-windows-static
vcpkg.exe integrate install
.github/workflows/build_windows_worker_.yml
-1
View File
@@ -62,7 +62,6 @@ jobs:
- name: Cargo build binary windows
timeout-minutes: 180
run: |
vcpkg.exe install openssl-windows:x64-windows
vcpkg.exe install openssl:x64-windows-static
vcpkg.exe integrate install
$env:VCPKGRS_DYNAMIC=1
.github/workflows/publish_windows_worker.yml
-1
View File
@@ -51,7 +51,6 @@ jobs:
- name: Cargo build windows
timeout-minutes: 180
run: |
vcpkg.exe install openssl-windows:x64-windows
vcpkg.exe install openssl:x64-windows-static
vcpkg.exe integrate install
$env:VCPKGRS_DYNAMIC=1
.release-please-manifest.json
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "1.796.0"
".": "1.800.1"
}
AGENTS.md
+9
View File
@@ -165,6 +165,15 @@ $NAV --root backend callees "X" # what does X call?
- Search for existing code to reuse before writing new code
- Follow established patterns in the codebase
- Keep changes focused — don't refactor beyond what's asked
- **A simpler design found late is still the design.** Work already spent is not an argument
for a shape, and neither is a clean review round, a passing suite, or a long PR thread. The
signal to stop and re-derive rather than patch again is a change that keeps growing to defend
its own structure: each review finding fixing an assumption the previous fix broke, the same
class of bug reappearing somewhere new, or most of the diff being consequences of one early
choice rather than the thing you set out to do. When that happens, say plainly what the
simpler design is and what switching costs — a migration, a review cycle restarted from zero,
work discarded — and let the user decide. Do not keep paying down the harder one because it
is nearly finished, and do not present the accumulated cost as a reason to continue.
- **Ship only the tests the PR needs.** A committed test must pin behavior a future change could plausibly break, and be the smallest setup that exercises the new logic. While developing, write as many exhaustive tests and do as much manual testing as you need to convince yourself the change works — then remove that scaffolding before marking the PR ready, keeping only the essential regression guard(s). A test that merely re-exercises pre-existing behavior, or needs elaborate fixtures to assert something trivial, is scaffolding: delete it. If nothing meaningful is left to guard, ship no test rather than a ceremonial one.
- **Comments record constraints, not narration.** Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's "simplified" away. State each invariant once, at the place where someone would break it, in ≤4 lines. Don't describe what the next line does, don't repeat the same rationale at multiple sites, and don't address the PR reviewer (justifying a change belongs in the PR description, not the code). Reference nothing ephemeral — no numbered steps from your dev flow, no "the poller / the test does X" scaffolding, no transient state that won't exist for the next reader; keep only the essential, durable rationale. Describe the code as it is, never its drafting history: "we no longer do X", "unchanged behavior", "instead of the previous approach" are meaningless to a reader who never saw the earlier iteration — before finishing, reread your comments as if the current state is the only state that ever existed.
- **Never attribute work to a specific customer, account, or "requested by a customer" in repo-tracked content** (PR descriptions, commit messages, code comments, docs). Describe changes by their technical motivation instead.
CHANGELOG.md
+99
View File
@@ -1,5 +1,104 @@
# Changelog
## [1.800.1](https://github.com/windmill-labs/windmill/compare/v1.800.0...v1.800.1) (2026-09-01)
### Bug Fixes
* add top margin to the home Build with AI section ([#10909](https://github.com/windmill-labs/windmill/issues/10909)) ([bedf5ae](https://github.com/windmill-labs/windmill/commit/bedf5ae57445025729e94e16f1b5f13f6ff38ffa))
* **ai-chat:** consume an @ mention with the message that carried it ([#10907](https://github.com/windmill-labs/windmill/issues/10907)) ([c512110](https://github.com/windmill-labs/windmill/commit/c512110a1f8d0d3437c20048f6446ef62b10222c))
* keep a local dbt descriptor under sync pull --keep-deleted ([#10911](https://github.com/windmill-labs/windmill/issues/10911)) ([4b5be38](https://github.com/windmill-labs/windmill/commit/4b5be386ce0f851a087f43c0b0ac6e4b1b055a47))
* keep windmill-indexer out of builds without tantivy ([#10908](https://github.com/windmill-labs/windmill/issues/10908)) ([db0f004](https://github.com/windmill-labs/windmill/commit/db0f004613e3f90428fea4c824cc53f1b2fc03b0))
## [1.800.0](https://github.com/windmill-labs/windmill/compare/v1.799.0...v1.800.0) (2026-08-31)
### Features
* add --keep-deleted flag to wmill sync pull and push ([#10878](https://github.com/windmill-labs/windmill/issues/10878)) ([66123f3](https://github.com/windmill-labs/windmill/commit/66123f3a9b8978c0084b02f50b44cffba125a13a))
* day-partition the service log index and expire whole chunks ([#10893](https://github.com/windmill-labs/windmill/issues/10893)) ([d91ee46](https://github.com/windmill-labs/windmill/commit/d91ee4614a70f20a194f47e190327129f499ec63))
* free AI tokens + home search/filter revamp ([#10020](https://github.com/windmill-labs/windmill/issues/10020)) ([716ce2e](https://github.com/windmill-labs/windmill/commit/716ce2ece00cd5cfb8641afc6432636bc4aa46e9))
* make the service log retention period an instance setting ([#10889](https://github.com/windmill-labs/windmill/issues/10889)) ([815de49](https://github.com/windmill-labs/windmill/commit/815de49e2322f85ca92b1e41a2bcd22591ebe93f))
* rework the evals dataset drawer and run navigation ([#10884](https://github.com/windmill-labs/windmill/issues/10884)) ([1462f17](https://github.com/windmill-labs/windmill/commit/1462f17643302127b4bd76bacfde80cc03f9d606))
* serve service log context from parquet and retire the raw log files ([#10892](https://github.com/windmill-labs/windmill/issues/10892)) ([338d75c](https://github.com/windmill-labs/windmill/commit/338d75cc5227e352cb84828c99bfd3b984cf0fa5))
* serve service log retrieval from a columnar parquet store ([#10886](https://github.com/windmill-labs/windmill/issues/10886)) ([7c1a785](https://github.com/windmill-labs/windmill/commit/7c1a785f756ed27e4425f6534709b19971a73a97))
### Bug Fixes
* correct the service log ingest flush boundary ([#10898](https://github.com/windmill-labs/windmill/issues/10898)) ([ac56586](https://github.com/windmill-labs/windmill/commit/ac56586c0e56d4022761d3c80306a03d57f8bfcb))
* harden the service log indexer's recovery and read paths ([#10904](https://github.com/windmill-labs/windmill/issues/10904)) ([831370c](https://github.com/windmill-labs/windmill/commit/831370cdde8e06f4298b17baa1a0041bacdd98c7))
* keep raw-app editor selection consistent across sidebar and tabs ([#10885](https://github.com/windmill-labs/windmill/issues/10885)) ([b57e231](https://github.com/windmill-labs/windmill/commit/b57e231c2bf5e5fe007f0aa7b958a51e32b47141))
* register every rotated service log file exactly once ([#10891](https://github.com/windmill-labs/windmill/issues/10891)) ([c817248](https://github.com/windmill-labs/windmill/commit/c8172480b0b1be6c57210212afc71d6ec8711235))
* show a loading indicator while the initial data table migration is generated ([#10900](https://github.com/windmill-labs/windmill/issues/10900)) ([b998267](https://github.com/windmill-labs/windmill/commit/b998267c91b9dcf02787768f6205cc5aeda494fb))
* track outstanding service log files on the rows themselves ([#10894](https://github.com/windmill-labs/windmill/issues/10894)) ([aa4a6ff](https://github.com/windmill-labs/windmill/commit/aa4a6ffd66813010a79c07741b01a984ed4e7df6))
### Performance Improvements
* add service log documents to the index one batch at a time ([#10906](https://github.com/windmill-labs/windmill/issues/10906)) ([0c2eb0a](https://github.com/windmill-labs/windmill/commit/0c2eb0ae3d18f49c21370131d15011e8dd103746))
## [1.799.0](https://github.com/windmill-labs/windmill/compare/v1.798.1...v1.799.0) (2026-08-28)
### Features
* enable Anthropic prompt caching on Vertex AI agent steps ([#10876](https://github.com/windmill-labs/windmill/issues/10876)) ([320f400](https://github.com/windmill-labs/windmill/commit/320f4005124202852e6e9c70b394e7f87231d278))
* instrument AI fill/fix, evals, agents and the debugger ([#10853](https://github.com/windmill-labs/windmill/issues/10853)) ([0bbd559](https://github.com/windmill-labs/windmill/commit/0bbd559ac8a35dba04ba5e8d6f2fd8d1d1124891))
### Bug Fixes
* **datatables:** stop a fork's pg_dump restore from failing silently ([#10830](https://github.com/windmill-labs/windmill/issues/10830)) ([3ce9bbc](https://github.com/windmill-labs/windmill/commit/3ce9bbc7168b837cb2111aabd533bb67803502b8))
* key build artifact caches on a runnable's inline modules ([#10819](https://github.com/windmill-labs/windmill/issues/10819)) ([b72ccc3](https://github.com/windmill-labs/windmill/commit/b72ccc35934165b4bad112b947ca5af064aab26f))
* nested template literals in step inputs, and unresolvable $args tags ([#10856](https://github.com/windmill-labs/windmill/issues/10856)) ([8f349c0](https://github.com/windmill-labs/windmill/commit/8f349c032a0d75fc3350292075e5050a030f6166))
* pre-fill the test panel JSON args editor and align its placeholder ([#10871](https://github.com/windmill-labs/windmill/issues/10871)) ([fb82f36](https://github.com/windmill-labs/windmill/commit/fb82f36e6d6492dd0740984d8d78ea4eaa30361e))
* reject a prefixed error_handler_path on triggers ([#10847](https://github.com/windmill-labs/windmill/issues/10847)) ([d334831](https://github.com/windmill-labs/windmill/commit/d33483173526a3b352d2829ac8a2e1e229cc1127))
* unify billable seat counting and prevent fork subscriptions ([#10818](https://github.com/windmill-labs/windmill/issues/10818)) ([7dd88c4](https://github.com/windmill-labs/windmill/commit/7dd88c470caee5f095dc240667aa7550c55696bc))
## [1.798.1](https://github.com/windmill-labs/windmill/compare/v1.798.0...v1.798.1) (2026-08-27)
### Bug Fixes
* allow job tokens to read the automate_username_creation setting ([#10869](https://github.com/windmill-labs/windmill/issues/10869)) ([c2279db](https://github.com/windmill-labs/windmill/commit/c2279db8a96ac76382eafe254627dafd24d173fd))
## [1.798.0](https://github.com/windmill-labs/windmill/compare/v1.797.0...v1.798.0) (2026-08-27)
### Features
* a wizard for importing a hub project, and finishing what the import cannot ([#10729](https://github.com/windmill-labs/windmill/issues/10729)) ([2913339](https://github.com/windmill-labs/windmill/commit/29133398f99cd2dd5b33057ee9df4492d82e067a))
## [1.797.0](https://github.com/windmill-labs/windmill/compare/v1.796.0...v1.797.0) (2026-08-26)
### Features
* configurable expiry for presigned s3 public url signatures ([#10835](https://github.com/windmill-labs/windmill/issues/10835)) ([8a6dc27](https://github.com/windmill-labs/windmill/commit/8a6dc27236aca67f0efe941d9606b787c2305ea8))
* **frontend:** flag the fork-compare datatable schema diff as legacy ([#10829](https://github.com/windmill-labs/windmill/issues/10829)) ([07c77ea](https://github.com/windmill-labs/windmill/commit/07c77ead7425f1877372d358d867445a4c525c96))
* keep a Hub project live while an update is under review ([#10814](https://github.com/windmill-labs/windmill/issues/10814)) ([c04b570](https://github.com/windmill-labs/windmill/commit/c04b5705745c36ecbb3a551ac59459218d2e3807))
### Bug Fixes
* **cli:** keep svelte component styles in the raw-app bundle ([#10838](https://github.com/windmill-labs/windmill/issues/10838)) ([b8bf539](https://github.com/windmill-labs/windmill/commit/b8bf539c3fe2b4db9c74dd73f04b3029287acdc6))
* **debugger:** parse bun 1.4's UUID inspector token ([#10828](https://github.com/windmill-labs/windmill/issues/10828)) ([4658224](https://github.com/windmill-labs/windmill/commit/46582245926a7f8ea961bcd125a58fbfba3530cf))
* force HTTP router rebuild on trigger-change notification ([#10849](https://github.com/windmill-labs/windmill/issues/10849)) ([ffdf17e](https://github.com/windmill-labs/windmill/commit/ffdf17ef8dc5575dd92d62d0d0ba887c1e378576))
* **frontend:** follow the operating workspace in step input forms ([#10834](https://github.com/windmill-labs/windmill/issues/10834)) ([6b73145](https://github.com/windmill-labs/windmill/commit/6b73145e7220232601538b801ebc9dc73fe79bbb))
* **frontend:** key the GitHub App installation selector on installation_id ([#10831](https://github.com/windmill-labs/windmill/issues/10831)) ([78331fd](https://github.com/windmill-labs/windmill/commit/78331fda8b290a2d9a5dd92b8362ff32c8b39432))
* **frontend:** operator menu opens on hover, pins on click ([#10824](https://github.com/windmill-labs/windmill/issues/10824)) ([665f83e](https://github.com/windmill-labs/windmill/commit/665f83e1f438e34d006429889d51a5fb6a6b6176))
* keep connection string query parameters under token auth ([#10859](https://github.com/windmill-labs/windmill/issues/10859)) ([f131c39](https://github.com/windmill-labs/windmill/commit/f131c3920f50f9fa18cd637eac39609495999aef))
* migrate slack resource-connect oauth to v2 ([#10836](https://github.com/windmill-labs/windmill/issues/10836)) ([9fa8159](https://github.com/windmill-labs/windmill/commit/9fa8159ad16204cab52fd18a34a48ebf13f800f6))
* recover from unresolvable AI session links instead of a dead end ([#10854](https://github.com/windmill-labs/windmill/issues/10854)) ([e38c449](https://github.com/windmill-labs/windmill/commit/e38c449007f27b952808cba5aa812441f2ce5946))
* require admin on workspace tarball settings export ([#10817](https://github.com/windmill-labs/windmill/issues/10817)) ([46c363f](https://github.com/windmill-labs/windmill/commit/46c363ffa4bc72bef6b367ece4bdbeef5e0eadc9))
* restrict filesystem workspace storage to debug builds ([#10864](https://github.com/windmill-labs/windmill/issues/10864)) ([8b80b09](https://github.com/windmill-labs/windmill/commit/8b80b09f33d311f0881678577ca6004c12d97c22))
### Performance Improvements
* index the suspended-job resume test instead of filtering it ([#10863](https://github.com/windmill-labs/windmill/issues/10863)) ([69320b2](https://github.com/windmill-labs/windmill/commit/69320b28f615b897a92f580bd5961c41e5c29951))
## [1.796.0](https://github.com/windmill-labs/windmill/compare/v1.795.0...v1.796.0) (2026-08-24)
backend/.sqlx/query-002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7.json
Generated
-29
View File
@@ -1,29 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n COUNT(*)::bigint AS \"total!\",\n COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS \"replacing!\"\n FROM ai_skill\n WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "total!",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "replacing!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": [
null,
null
]
},
"hash": "002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7"
}
backend/.sqlx/query-94da1e7feb4f58cc7ebe99752736f956d47810a94cb052fdcffb5cfe440f8033.json → backend/.sqlx/query-0e03dc960c0a22e042e54af719ac90c4b8506acefc85ef5b2f92a7bc451b1c5e.json
Generated
+3 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval RETURNING file_path, hostname",
"query": "DELETE FROM log_file WHERE (hostname, log_ts) IN (\n SELECT hostname, log_ts FROM log_file\n WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval\n LIMIT $2\n ) RETURNING file_path, hostname",
"describe": {
"columns": [
{
@@ -16,6 +16,7 @@
],
"parameters": {
"Left": [
"Int8",
"Int8"
]
},
@@ -24,5 +25,5 @@
false
]
},
"hash": "94da1e7feb4f58cc7ebe99752736f956d47810a94cb052fdcffb5cfe440f8033"
"hash": "0e03dc960c0a22e042e54af719ac90c4b8506acefc85ef5b2f92a7bc451b1c5e"
}
backend/.sqlx/query-247486558e023ec3adf0c1e8f5664fc5e65995ce81e622c174a89befc1a527e5.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT cost_nanos FROM ai_free_token_usage WHERE email = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "cost_nanos",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false
]
},
"hash": "247486558e023ec3adf0c1e8f5664fc5e65995ce81e622c174a89befc1a527e5"
}
backend/.sqlx/query-44b9400fed2082c0df9d57f940923f37b8d452ea8e199ccc207da238368f1996.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_free_token_daily_usage (day, cost_nanos, updated_at)\n VALUES ($2::date, $1::bigint, now())\n ON CONFLICT (day) DO UPDATE\n SET cost_nanos = ai_free_token_daily_usage.cost_nanos + $1::bigint,\n updated_at = now()\n RETURNING cost_nanos",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "cost_nanos",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Int8",
"Date"
]
},
"nullable": [
false
]
},
"hash": "44b9400fed2082c0df9d57f940923f37b8d452ea8e199ccc207da238368f1996"
}
backend/.sqlx/query-4b93550c7836fd3643180ade3548faa875e471d3f9ca37fc669f359e7a1818bb.json
Generated
-16
View File
@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO resource_type (workspace_id, name, schema, description, edited_at)\n VALUES ('admins', $1, $2, $3, now())\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET schema = EXCLUDED.schema, description = EXCLUDED.description, edited_at = now()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Jsonb",
"Text"
]
},
"nullable": []
},
"hash": "4b93550c7836fd3643180ade3548faa875e471d3f9ca37fc669f359e7a1818bb"
}
backend/.sqlx/query-624a7dbc6cc951a199b0e70d86c463a0e7b5248c226ee92d95df94c3099cc400.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE log_file SET indexed_at = now()\n FROM unnest($1::text[], $2::text[]) AS c(hostname, file_path)\n WHERE log_file.indexed_at IS NULL\n AND log_file.hostname = c.hostname\n AND log_file.file_path = c.file_path",
"describe": {
"columns": [],
"parameters": {
"Left": [
"TextArray",
"TextArray"
]
},
"nullable": []
},
"hash": "624a7dbc6cc951a199b0e70d86c463a0e7b5248c226ee92d95df94c3099cc400"
}
backend/.sqlx/query-67a83afb708c90b2132cba81a0701dfc8b5e7aedf2f26fc1fe5be8f685cf709c.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT max(log_ts) FROM log_file\n WHERE hostname = $1 AND log_ts < (SELECT max(log_ts) FROM log_file WHERE hostname = $1)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "max",
"type_info": "Timestamp"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "67a83afb708c90b2132cba81a0701dfc8b5e7aedf2f26fc1fe5be8f685cf709c"
}
backend/.sqlx/query-b5c839baab25c4dcdd503d380cf7a886242277cd50555f20b2e22e13942d2a3a.json → backend/.sqlx/query-6bbcb27a3bb70302076c559c8394b14b842f595f68dd885248abaaeabd2d0bf1.json
Generated
+2 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n hostname,\n mode::text,\n worker_group,\n log_ts,\n file_path,\n ok_lines,\n err_lines,\n json_fmt\n FROM log_file\n WHERE log_ts > $1\n ORDER BY log_ts ASC LIMIT $2",
"query": "SELECT\n hostname,\n mode::text,\n worker_group,\n log_ts,\n file_path,\n ok_lines,\n err_lines,\n json_fmt\n FROM log_file\n WHERE indexed_at IS NULL\n ORDER BY log_ts ASC, hostname ASC LIMIT $1",
"describe": {
"columns": [
{
@@ -46,7 +46,6 @@
],
"parameters": {
"Left": [
"Timestamp",
"Int8"
]
},
@@ -61,5 +60,5 @@
true
]
},
"hash": "b5c839baab25c4dcdd503d380cf7a886242277cd50555f20b2e22e13942d2a3a"
"hash": "6bbcb27a3bb70302076c559c8394b14b842f595f68dd885248abaaeabd2d0bf1"
}
backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json
Generated
-18
View File
@@ -1,18 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_skill (workspace_id, name, description, instructions, edited_at, edited_by)\n VALUES ($1, $2, $3, $4, now(), $5)\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET description = EXCLUDED.description,\n instructions = EXCLUDED.instructions,\n edited_at = now(),\n edited_by = EXCLUDED.edited_by",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Text",
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8"
}
backend/.sqlx/query-872be44352d4a27a2005f6bdc38f927ac4886755fd17ec755037763aa92f6c0e.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "WITH moved AS (\n UPDATE log_file SET indexed_at = CASE\n WHEN log_ts > NOW() - make_interval(secs => $1) THEN NULL\n ELSE now() END\n WHERE indexed_at = 'epoch' RETURNING 1)\n SELECT count(*) AS \"n!\" FROM moved",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "n!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Float8"
]
},
"nullable": [
null
]
},
"hash": "872be44352d4a27a2005f6bdc38f927ac4886755fd17ec755037763aa92f6c0e"
}
backend/.sqlx/query-87e8e034b1cf5ea7ce43870d77d33fa0fd05b067454f0d4686d8b155fe562ffe.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE log_file SET indexed_at = now()\n FROM unnest($1::text[], $2::timestamp[]) AS c(hostname, log_ts)\n WHERE log_file.hostname = c.hostname AND log_file.log_ts = c.log_ts",
"describe": {
"columns": [],
"parameters": {
"Left": [
"TextArray",
"TimestampArray"
]
},
"nullable": []
},
"hash": "87e8e034b1cf5ea7ce43870d77d33fa0fd05b067454f0d4686d8b155fe562ffe"
}
backend/.sqlx/query-1ea97f9085ec018f779e77e0fdbda3d4ecd67b3fbee9a58228ef577f846607ae.json → backend/.sqlx/query-8ad79b80033b38ebddf6c8cd4d8cb160d41bac4c45a0fc74d9c9e96d3ef4486a.json
Generated
+5 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT EXISTS(SELECT 1 FROM resource_type WHERE workspace_id = 'admins' AND name = $1 AND schema IS NOT DISTINCT FROM $2 AND description IS NOT DISTINCT FROM $3)",
"query": "SELECT EXISTS(SELECT 1 FROM resource_type WHERE workspace_id = 'admins' AND name = $1 AND schema IS NOT DISTINCT FROM $2 AND description IS NOT DISTINCT FROM $3 AND ($5 IS NOT TRUE OR format_extension IS NOT DISTINCT FROM $4))",
"describe": {
"columns": [
{
@@ -13,12 +13,14 @@
"Left": [
"Text",
"Jsonb",
"Text"
"Text",
"Text",
"Bool"
]
},
"nullable": [
null
]
},
"hash": "1ea97f9085ec018f779e77e0fdbda3d4ecd67b3fbee9a58228ef577f846607ae"
"hash": "8ad79b80033b38ebddf6c8cd4d8cb160d41bac4c45a0fc74d9c9e96d3ef4486a"
}
backend/.sqlx/query-8d207cc9ed101ff116b617d25a94633c1531170ded1fdf09114718b941f5e1db.json
Generated
-65
View File
@@ -1,65 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n hostname,\n mode::text,\n worker_group,\n log_ts,\n file_path,\n ok_lines,\n err_lines,\n json_fmt\n FROM log_file\n WHERE log_ts > NOW() - make_interval(secs => $1)\n ORDER BY log_ts ASC LIMIT $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "hostname",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "mode",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "worker_group",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "log_ts",
"type_info": "Timestamp"
},
{
"ordinal": 4,
"name": "file_path",
"type_info": "Varchar"
},
{
"ordinal": 5,
"name": "ok_lines",
"type_info": "Int8"
},
{
"ordinal": 6,
"name": "err_lines",
"type_info": "Int8"
},
{
"ordinal": 7,
"name": "json_fmt",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Float8",
"Int8"
]
},
"nullable": [
false,
null,
true,
false,
false,
true,
true,
true
]
},
"hash": "8d207cc9ed101ff116b617d25a94633c1531170ded1fdf09114718b941f5e1db"
}
backend/.sqlx/query-972df41db505fbbd20a558b200a2e2e8bc43633707d8365f73130c5bca3923b9.json
Generated
+18
View File
@@ -0,0 +1,18 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO resource_type (workspace_id, name, schema, description, format_extension, edited_at)\n VALUES ('admins', $1, $2, $3, $4, now())\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET schema = EXCLUDED.schema, description = EXCLUDED.description,\n -- A fileset is a set of files, so it cannot also be one file.\n -- Create and update reject the pair; this writer bypasses both, so\n -- it declines the extension rather than persisting the forbidden\n -- combination onto a same-named local fileset.\n format_extension = CASE\n WHEN resource_type.is_fileset THEN NULL\n WHEN $5 THEN EXCLUDED.format_extension\n ELSE resource_type.format_extension END,\n edited_at = now()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Jsonb",
"Text",
"Varchar",
"Bool"
]
},
"nullable": []
},
"hash": "972df41db505fbbd20a558b200a2e2e8bc43633707d8365f73130c5bca3923b9"
}
backend/.sqlx/query-a7d450e34084d561f69e588bd76fd56e616ee79d895b7dcc37ad9442789e1574.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "WITH retired AS (\n UPDATE log_file SET indexed_at = now()\n WHERE indexed_at IS NULL\n AND log_ts <= NOW() - make_interval(secs => $1) RETURNING 1)\n SELECT count(*) AS \"n!\" FROM retired",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "n!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Float8"
]
},
"nullable": [
null
]
},
"hash": "a7d450e34084d561f69e588bd76fd56e616ee79d895b7dcc37ad9442789e1574"
}
backend/.sqlx/query-acf4a5f4b67ebd06157819677fee6be4cc3adc6aa859c3dbaa57fa50b63741cf.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_free_token_daily_usage (day, cost_nanos, updated_at)\n VALUES ($2::date, GREATEST(0, $1::bigint), now())\n ON CONFLICT (day) DO UPDATE\n SET cost_nanos = GREATEST(0, ai_free_token_daily_usage.cost_nanos + $1::bigint),\n updated_at = now()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int8",
"Date"
]
},
"nullable": []
},
"hash": "acf4a5f4b67ebd06157819677fee6be4cc3adc6aa859c3dbaa57fa50b63741cf"
}
backend/.sqlx/query-b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384.json
Generated
+3 -3
View File
@@ -98,12 +98,12 @@
null,
null,
null,
true,
false,
null,
null,
null,
true,
true
false,
false
]
},
"hash": "b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384"
backend/.sqlx/query-c29d815cc38493f17950f12e9e5fe34b6c4ed1b2a49ed2b61dc9fb494026e956.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_free_token_usage (email, cost_nanos, updated_at)\n VALUES ($1, $2::bigint, now())\n ON CONFLICT (email) DO UPDATE\n SET cost_nanos = ai_free_token_usage.cost_nanos + $2::bigint,\n updated_at = now()\n RETURNING cost_nanos",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "cost_nanos",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Varchar",
"Int8"
]
},
"nullable": [
false
]
},
"hash": "c29d815cc38493f17950f12e9e5fe34b6c4ed1b2a49ed2b61dc9fb494026e956"
}
backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json
Generated
-28
View File
@@ -1,28 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT name, description FROM ai_skill WHERE workspace_id = $1 ORDER BY name",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "description",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
false
]
},
"hash": "c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b"
}
backend/.sqlx/query-daa5b57290cd1f821a53eebe96434f1befe6b16eee363c843faa1f836d53ca8d.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT mode::text AS \"mode!\", log_ts FROM log_file WHERE hostname = $1 AND file_path = $2 ORDER BY log_ts DESC LIMIT 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "mode!",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "log_ts",
"type_info": "Timestamp"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null,
false
]
},
"hash": "daa5b57290cd1f821a53eebe96434f1befe6b16eee363c843faa1f836d53ca8d"
}
backend/.sqlx/query-df13e7bb9c14aa19604c40754509f66af26042464ba199586838e073c318c53a.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT is_fileset, format_extension FROM resource_type\n WHERE name = $1 AND workspace_id = $2 FOR UPDATE",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "is_fileset",
"type_info": "Bool"
},
{
"ordinal": 1,
"name": "format_extension",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
true
]
},
"hash": "df13e7bb9c14aa19604c40754509f66af26042464ba199586838e073c318c53a"
}
backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json
Generated
-35
View File
@@ -1,35 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT name, description, instructions FROM ai_skill WHERE workspace_id = $1 AND name = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "description",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "instructions",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
false,
false
]
},
"hash": "e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d"
}
backend/.sqlx/query-f08ef43b4523c74fcbfc7879c147e1bf2c9a0d11e9841caafca62d85c9fb4c8f.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_free_token_usage (email, cost_nanos, updated_at)\n VALUES ($1, GREATEST(0, $2::bigint), now())\n ON CONFLICT (email) DO UPDATE\n SET cost_nanos = GREATEST(0, ai_free_token_usage.cost_nanos + $2::bigint),\n updated_at = now()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Int8"
]
},
"nullable": []
},
"hash": "f08ef43b4523c74fcbfc7879c147e1bf2c9a0d11e9841caafca62d85c9fb4c8f"
}
backend/.sqlx/query-92faee8714a45a403b623e04d789f7f99067a05e9dfe270223164db8a1df2e4b.json → backend/.sqlx/query-f277db0459ff311d8a396aa4e03876dac75ba6df5f8eaa9f185300483e3ee36f.json
Generated
+2 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)\n VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)\n ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
"query": "INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)\n VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)\n ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
"describe": {
"columns": [],
"parameters": {
@@ -17,5 +17,5 @@
},
"nullable": []
},
"hash": "92faee8714a45a403b623e04d789f7f99067a05e9dfe270223164db8a1df2e4b"
"hash": "f277db0459ff311d8a396aa4e03876dac75ba6df5f8eaa9f185300483e3ee36f"
}
backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json → backend/.sqlx/query-f47cafb6e9a6ec58ccafb06cf5e806e3fe749119214863b9111b58fff0bb9583.json
Generated
+4 -5
View File
@@ -1,23 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM ai_skill WHERE workspace_id = $1 AND name = $2 RETURNING name",
"query": "SELECT plan FROM workspace_settings WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"name": "plan",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false
true
]
},
"hash": "e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61"
"hash": "f47cafb6e9a6ec58ccafb06cf5e806e3fe749119214863b9111b58fff0bb9583"
}
backend/AGENTS.md
+5 -3
View File
@@ -131,9 +131,11 @@ minimal explicit set for dev.
## Workspace object storage in dev — use the local filesystem
For a dev workspace you don't need MinIO/S3: use the built-in **`FilesystemStorage`** large-file
storage (a root path on local disk). It is intentionally hidden from the settings-UI storage
dropdown (dev-only), so set it via the API. Requires the backend built with `parquet` (+ `private`
for the real S3 helpers, + `enterprise` if you want advanced permission rules enforced):
storage (a root path on local disk). It is a **debug-build affordance only** — every site that
builds a filesystem object store calls `ensure_filesystem_storage_allowed`, so release builds
refuse it, and the settings UI never offers it — so set it via the API on a `cargo run`/`cargo
test` binary. Requires the backend built with `parquet` (+ `private` for the real S3 helpers,
+ `enterprise` if you want advanced permission rules enforced):
```bash
curl -X POST "$BASE/api/w/<ws>/workspaces/edit_large_file_storage_config" \
backend/Cargo.lock
Generated
+346 -260
View File
File diff suppressed because it is too large Load Diff
backend/Cargo.toml
+12 -5
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.796.0"
version = "1.800.1"
authors.workspace = true
edition.workspace = true
@@ -88,7 +88,7 @@ members = [
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
[workspace.package]
version = "1.796.0"
version = "1.800.1"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
@@ -115,7 +115,7 @@ strip = "none"
[features]
default = []
private = ["windmill-api/private", "windmill-api-agent-workers?/private", "windmill-autoscaling/private", "windmill-common/private", "windmill-dep-map/private", "windmill-object-store/private", "windmill-git-sync/private", "windmill-indexer/private", "windmill-operator?/private", "windmill-queue/private", "windmill-worker/private", "windmill-test-utils/private"]
private = ["windmill-api/private", "windmill-api-agent-workers?/private", "windmill-autoscaling/private", "windmill-common/private", "windmill-dep-map/private", "windmill-object-store/private", "windmill-git-sync/private", "windmill-indexer?/private", "windmill-operator?/private", "windmill-queue/private", "windmill-worker/private", "windmill-test-utils/private"]
agent_worker_server = ["windmill-api/agent_worker_server", "dep:windmill-api-agent-workers", "windmill-test-utils/agent_worker_server"]
enterprise = ["windmill-worker/enterprise", "windmill-queue/enterprise", "windmill-api/enterprise", "windmill-api-agent-workers?/enterprise", "dep:windmill-autoscaling", "windmill-autoscaling/enterprise", "windmill-git-sync/enterprise", "windmill-common/prometheus", "windmill-common/enterprise", "windmill-object-store/enterprise", "license"]
local_reports = ["windmill-common/local_reports"]
@@ -131,6 +131,10 @@ quickjs = ["windmill-worker/quickjs", "windmill-api/quickjs"]
openidconnect = ["windmill-api/openidconnect", "windmill-common/openidconnect", "windmill-object-store/openidconnect"]
cloud = ["windmill-queue/cloud", "windmill-worker/cloud", "windmill-common/cloud", "windmill-api/cloud"]
jemalloc = ["windmill-common/jemalloc", "dep:tikv-jemallocator", "dep:tikv-jemalloc-sys", "dep:tikv-jemalloc-ctl"]
# `tantivy` must stay the only feature that enables windmill-indexer: it is the only one that
# also gives it `enterprise` + `parquet`, and its EE sources gate nearly everything on that
# pair, so a bare windmill-indexer is a crate of dead code that `-D warnings` rejects. Any
# other feature wanting one of its features has to use the optional `windmill-indexer?/` form.
tantivy = ["dep:windmill-indexer", "windmill-api/tantivy", "windmill-indexer/enterprise", "windmill-indexer/parquet", "windmill-common/tantivy", "enterprise", "parquet"]
sqlx = ["windmill-worker/sqlx"]
deno_core = ["windmill-worker/deno_core", "dep:windmill-runtime-nativets", "windmill-test-utils/deno_core"]
@@ -477,7 +481,10 @@ rust-embed = { version = "^6", features = ["interpolate-folder-path"] }
mime_guess = "^2"
hex = "^0"
sql-builder = "^3"
argon2 = "^0"
# Minor-pinned rather than the `^0` used elsewhere in this file: argon2's 0.x
# minors are API-breaking (0.6 moved `SaltString` into `phc`, put `rand_core`
# behind a feature and changed `hash_password`), so a float breaks the build.
argon2 = "0.6"
quick_cache = "^0"
rand = "=0.9.0"
rand_core = { version = "^0", features = ["std"] }
@@ -697,7 +704,7 @@ tikv-jemalloc-ctl = { version = "^0.5" }
triomphe = "^0"
pin-project-lite = "^0"
tantivy = { git="https://github.com/windmill-labs/tantivy", rev="6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" }
tantivy = { git="https://github.com/windmill-labs/tantivy", rev="ea3b818c7b93db0c2b5db4f5e7ffef38f339060a" }
backon = "1.3.0"
backend/ee-repo-ref.txt
+1 -1
View File
@@ -1 +1 @@
6d9a1870b8747fb0746554a586a56de2a7645e9b
96dc9b7b41bf217ecc80b9b341f74042ba7cdd8b
backend/migrations/20260622072905_ai_free_token_usage.down.sql
+2
View File
@@ -0,0 +1,2 @@
DROP TABLE ai_free_token_daily_usage;
DROP TABLE ai_free_token_usage;
backend/migrations/20260622072905_ai_free_token_usage.up.sql
+19
View File
@@ -0,0 +1,19 @@
-- One-time grant of the Windmill-provided free AI tier, measured as cost in nano-dollars
-- (1e-9 USD) rather than raw tokens — a prompt-cache hit costs a fraction of a fresh input
-- token, so a token count wildly overstates the real bill. The grant never resets: once
-- spent, the user must bring their own API key. Keyed by normalized email so the allowance
-- is shared across a user's workspaces (and is resistant to +tag / gmail-dot aliasing).
CREATE TABLE ai_free_token_usage (
email VARCHAR(255) PRIMARY KEY,
cost_nanos BIGINT NOT NULL DEFAULT 0,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Instance-wide daily cost ceiling (nano-dollars) for the free tier — a kill-switch
-- independent of the per-user grant, bounding the blast radius of a bad day. One row per
-- UTC day.
CREATE TABLE ai_free_token_daily_usage (
day DATE PRIMARY KEY,
cost_nanos BIGINT NOT NULL DEFAULT 0,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
backend/migrations/20260826202939_queue_suspended_resume_at_index.down.sql
+1
View File
@@ -0,0 +1 @@
DROP INDEX IF EXISTS queue_suspended_v2;
backend/migrations/20260826202939_queue_suspended_resume_at_index.up.sql
+20
View File
@@ -0,0 +1,20 @@
-- Serves the suspended-job pull in windmill-common/src/worker.rs, whose resume test is the
-- indexed CASE expression. Two things about the shape are load-bearing:
-- * (priority DESC NULLS LAST, created_at) leads, so the scan yields that query's ORDER BY
-- and stops at the first match rather than sorting.
-- * the index is dropped before it is built rather than relying on IF NOT EXISTS. The
-- OVERRIDDEN_MIGRATIONS rewrite in windmill-api/src/db.rs runs these CONCURRENTLY, and an
-- interrupted concurrent build leaves the index present but invalid, which IF NOT EXISTS
-- would then skip rebuilding. Retiring the index this replaces is left to the migration
-- that follows, so this one can only ever be replayed while that index is still there to
-- cover the rebuild.
DROP INDEX IF EXISTS queue_suspended_v2;
CREATE INDEX IF NOT EXISTS queue_suspended_v2
ON v2_job_queue (
priority DESC NULLS LAST,
created_at,
(CASE WHEN suspend <= 0 THEN '-infinity'::timestamptz ELSE suspend_until END),
tag
)
WHERE suspend_until IS NOT NULL;
backend/migrations/20260826214706_queue_suspended_drop_legacy_index.down.sql
+3
View File
@@ -0,0 +1,3 @@
CREATE INDEX IF NOT EXISTS queue_suspended
ON v2_job_queue (priority DESC NULLS LAST, created_at, suspend_until, suspend, tag)
WHERE suspend_until IS NOT NULL;
backend/migrations/20260826214706_queue_suspended_drop_legacy_index.up.sql
+6
View File
@@ -0,0 +1,6 @@
-- Retires the index queue_suspended_v2 replaces. Separate from the migration that builds it
-- so that one is only ever replayed while this index still exists: sqlx records a migration
-- only after all its statements run, so a process that dies before the record is written
-- replays the build, and its leading DROP would otherwise be destroying the sole usable
-- index rather than an interrupted build.
DROP INDEX IF EXISTS queue_suspended;
backend/migrations/20260830085453_log_file_indexed_at.down.sql
+4
View File
@@ -0,0 +1,4 @@
DROP INDEX IF EXISTS index_log_file_premigration;
DROP INDEX IF EXISTS index_log_file_pending_path;
DROP INDEX IF EXISTS index_log_file_pending;
ALTER TABLE log_file DROP COLUMN IF EXISTS indexed_at;
backend/migrations/20260830085453_log_file_indexed_at.up.sql
+35
View File
@@ -0,0 +1,35 @@
-- The service log ingest walked `log_file` with a cursor over `log_ts`, which is when a line
-- was written rather than when its row appeared. Rows do not arrive in that order — an upload
-- retried after a failure, a host that has just started, a batch the row limit cut mid-minute —
-- and a row that becomes visible behind the cursor is never read: it stays in `log_file` and its
-- lines stay out of search until retention drops them.
--
-- No ordering fixes this. A cursor over arrival order fails the same way, because `nextval` is
-- allocated before its INSERT commits: a row can be assigned a lower value and commit after a
-- higher one has already moved the cursor past it. Which rows are outstanding is a property of
-- the rows, so it is recorded on them.
ALTER TABLE log_file ADD COLUMN indexed_at TIMESTAMPTZ;
-- Rows that already existed are marked, not queued: on a 14-day window most were ingested long
-- ago and their raw files are gone. A sentinel rather than a timestamp, because the indexer has to
-- tell them apart from rows registered since — those start NULL — and it puts the window's worth of
-- them back on the queue on its first pass, keeping only what the columnar store can vouch for.
--
-- Not split here on the cursor the old ingest had reached. Below that cursor sits every row it
-- skipped, which is the loss this migration exists to stop; recording those as done would carry the
-- bug into its own fix.
UPDATE log_file SET indexed_at = 'epoch' WHERE indexed_at IS NULL;
-- The work queue, and the only index the ingest query needs: outstanding rows are a small
-- fraction of the table, so this stays proportional to what is left to do rather than to the
-- retention window.
CREATE INDEX index_log_file_pending ON log_file (log_ts) WHERE indexed_at IS NULL;
-- A rebuild takes rows out of the queue by the file it just read out of the store, which is
-- the one lookup that arrives without a `log_ts`: the primary key is `(hostname, log_ts)`, so
-- nothing else covers it and each batch would scan every outstanding row instead.
CREATE INDEX index_log_file_pending_path ON log_file (hostname, file_path) WHERE indexed_at IS NULL;
-- Reached once per pass while pre-migration rows survive, and never again after the first
-- conversion clears them.
CREATE INDEX index_log_file_premigration ON log_file (log_ts) WHERE indexed_at = 'epoch';
backend/migrations/20260831120907_skills_resource_type.down.sql
+13
View File
@@ -0,0 +1,13 @@
-- The up migration only ever added: `ai_skill` still holds every skill it copied,
-- so there is nothing to restore and nothing to delete. Removing the resources
-- would destroy any a user has since edited or created, and removing a folder
-- would take whatever else was put in it.
--
-- The seeded resource type goes. `created_by` only distinguishes this migration's
-- row from one a user created by hand: a hub sync updates the schema in place and
-- leaves `created_by` alone, so a synced-over row is still removed here and the
-- next sync puts it back.
DELETE FROM resource_type
WHERE workspace_id = 'admins'
AND name = 'ai_skill'
AND created_by = 'system';
backend/migrations/20260831120907_skills_resource_type.up.sql
+88
View File
@@ -0,0 +1,88 @@
-- AI chat skills move from the `ai_skill` table onto ordinary resources, so they
-- gain folder ACLs, version history, workspace export and git-sync. An `ai_skill`
-- resource holds the SKILL.md body in `value.content`; its description lives in
-- the resource's own `description` column and its name is the path basename.
--
-- Nothing here is destructive. `ai_skill` is left in place, unread, for a later
-- release to drop once operators have confirmed the copy. That is what lets every
-- step below skip on conflict rather than resolve one: a skipped row is still in
-- the table, so it is not lost, and the migration needs no record of what it did
-- in order to be reversible.
-- `format_extension` makes the resource editor render `value.content` as a plain
-- .md file. Seeded under 'admins' so every workspace sees it.
INSERT INTO resource_type (workspace_id, name, schema, description, created_by, format_extension, edited_at)
VALUES (
'admins',
'ai_skill',
'{"type": "object", "properties": {"content": {"type": "string"}}}',
'A reusable instruction set for the AI chat, in the SKILL.md format. The resource description is what the assistant sees when deciding whether the skill applies; the file body is the instructions it follows.',
'system',
'md',
now()
)
ON CONFLICT (workspace_id, name) DO NOTHING;
-- Shared home matching the admin-only upload these skills had. A workspace that
-- already has a `skills` folder keeps it untouched, ACL and all: adopting one
-- would hand its own grants — possibly write for everyone — over a set of
-- instructions the assistant follows.
INSERT INTO folder (workspace_id, name, display_name, owners, extra_perms)
SELECT DISTINCT workspace_id, 'skills', 'Skills', ARRAY[]::TEXT[], '{"g/all": false}'::jsonb
FROM ai_skill
ON CONFLICT (workspace_id, name) DO NOTHING;
-- Copied only where the destination is free and the folder matches the one above
-- exactly, owners included: a pre-existing folder carrying the same ACL but an
-- owner would hand that owner update and delete over skills the removed API let
-- only workspace admins touch. Anything else stays in `ai_skill` for an operator
-- to place deliberately.
--
-- What was actually inserted is recorded rather than inferred. Inferring it from
-- "is there an ai_skill resource at the destination" reports nothing when the
-- blocker is itself an ai_skill with different instructions — the one case where
-- the skipped skill is least likely to be noticed.
CREATE TEMP TABLE ai_skill_copied AS
WITH inserted AS (
INSERT INTO resource (workspace_id, path, value, description, resource_type, created_by, edited_at)
SELECT
s.workspace_id,
'f/skills/' || s.name,
jsonb_build_object('content', s.instructions),
s.description,
'ai_skill',
s.edited_by,
s.edited_at
FROM ai_skill s
JOIN folder f
ON f.workspace_id = s.workspace_id
AND f.name = 'skills'
AND f.extra_perms = '{"g/all": false}'::jsonb
AND cardinality(f.owners) = 0
ON CONFLICT (workspace_id, path) DO NOTHING
RETURNING workspace_id, path
)
SELECT workspace_id, path FROM inserted;
-- Anything not copied is still in `ai_skill`, but nothing reads that table any
-- more, so from the app's side the skill is missing until an operator places it.
-- Name them rather than leaving that to be discovered.
DO $$
DECLARE
leftover RECORD;
BEGIN
FOR leftover IN
SELECT s.workspace_id, s.name
FROM ai_skill s
WHERE NOT EXISTS (
SELECT 1 FROM ai_skill_copied c
WHERE c.workspace_id = s.workspace_id
AND c.path = 'f/skills/' || s.name
)
LOOP
RAISE WARNING 'ai_skill %/% was not copied to a resource (its destination or the f/skills folder is already taken); it remains in the ai_skill table',
leftover.workspace_id, leftover.name;
END LOOP;
END $$;
DROP TABLE ai_skill_copied;
backend/parsers/windmill-parser-wasm/Cargo.lock
Generated
+24 -24
View File
@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windmill-common"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"aho-corasick",
"anyhow",
@@ -6274,7 +6274,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"proc-macro2",
"quote",
@@ -6286,7 +6286,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"convert_case",
"serde",
@@ -6295,7 +6295,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"lazy_static",
@@ -6307,7 +6307,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"serde_json",
@@ -6319,7 +6319,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"gosyn",
@@ -6331,7 +6331,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"lazy_static",
@@ -6343,7 +6343,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"serde_json",
@@ -6355,7 +6355,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"nu-parser",
@@ -6366,7 +6366,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6377,7 +6377,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6389,7 +6389,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6400,7 +6400,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"async-recursion",
@@ -6422,7 +6422,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"serde_json",
@@ -6434,7 +6434,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"lazy_static",
@@ -6448,7 +6448,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"convert_case",
@@ -6465,7 +6465,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"lazy_static",
@@ -6478,7 +6478,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"serde",
@@ -6490,7 +6490,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"lazy_static",
@@ -6508,7 +6508,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -6524,7 +6524,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6540,7 +6540,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wasm"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"getrandom 0.2.17",
@@ -6572,7 +6572,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"lazy_static",
@@ -6586,7 +6586,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.796.0"
version = "1.800.1"
dependencies = [
"anyhow",
"bitflags",
backend/parsers/windmill-parser-wasm/Cargo.toml
+1 -1
View File
@@ -12,7 +12,7 @@ resolver = "2"
members = ["."]
[workspace.package]
version = "1.796.0"
version = "1.800.1"
edition = "2021"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
backend/src/main.rs
+82 -21
View File
@@ -7,14 +7,14 @@
*/
use anyhow::Context;
use monitor::{
load_base_url, load_otel, reload_critical_alerts_on_db_oversize,
reload_delete_logs_periodically_setting, reload_indexer_config,
reload_instance_python_version_setting, reload_maven_repos_setting,
flush_pending_log_files_to_object_store, load_base_url, load_otel,
reload_critical_alerts_on_db_oversize, reload_delete_logs_periodically_setting,
reload_indexer_config, reload_instance_python_version_setting, reload_maven_repos_setting,
reload_maven_settings_xml_setting, reload_no_default_maven_setting,
reload_nuget_config_setting, reload_powershell_repo_pat_setting,
reload_powershell_repo_url_setting, reload_ruby_repos_setting,
reload_timeout_wait_result_setting, reload_workspace_registries_setting,
send_current_log_file_to_object_store, send_logs_to_object_store, WORKERS_NAMES,
send_logs_to_object_store, WORKERS_NAMES,
};
use rand::Rng;
use sqlx::{Pool, Postgres};
@@ -61,8 +61,9 @@ use windmill_common::{
SAML_METADATA_SETTING, SANDBOX_IMAGE_CACHE_MAX_MB_SETTING,
SANDBOX_IMAGE_DEFAULT_REGISTRY_SETTING, SANDBOX_IMAGE_MAX_SIZE_MB_SETTING,
SANDBOX_IMAGE_PULL_POLICY_SETTING, SANDBOX_REGISTRY_AUTH_SETTING, SCIM_TOKEN_SETTING,
SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING, TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
UV_EXCLUDE_NEWER_SETTING, UV_INDEX_STRATEGY_SETTING, UV_PYTHON_INSTALL_MIRROR_SETTING,
SERVICE_LOG_RETENTION_SECS_SETTING, SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING,
TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING, UV_EXCLUDE_NEWER_SETTING,
UV_INDEX_STRATEGY_SETTING, UV_PYTHON_INSTALL_MIRROR_SETTING,
WORKSPACE_FAIRNESS_DURATION_SECS_SETTING, WORKSPACE_FAIRNESS_ENABLED_SETTING,
WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING,
WORKSPACE_MAX_QUEUED_JOBS_SETTING, WORKSPACE_REGISTRIES_SETTING,
@@ -143,7 +144,8 @@ use crate::monitor::{
reload_pip_index_url_setting, reload_retention_period_setting,
reload_sandbox_image_cache_max_setting, reload_sandbox_image_default_registry_setting,
reload_sandbox_image_max_size_setting, reload_sandbox_image_pull_policy_setting,
reload_sandbox_registry_auth_setting, reload_scim_token_setting, reload_smtp_config,
reload_sandbox_registry_auth_setting, reload_scim_token_setting,
reload_service_log_retention_secs_setting, reload_smtp_config,
reload_store_audit_logs_s3_setting, reload_uv_exclude_newer_setting,
reload_uv_index_strategy_setting, reload_uv_python_install_mirror_setting,
reload_worker_config, MonitorIteration,
@@ -380,6 +382,7 @@ async fn cache_hub_scripts(file_path: Option<String>) -> anyhow::Result<()> {
"",
&mut None,
&None,
None,
)
.await
{
@@ -403,8 +406,12 @@ struct HubResourceTypeRaw {
pub schema: Option<String>,
pub app: String,
pub description: Option<String>,
/// Absent from hubs predating the column, and from caches written before it.
#[serde(default)]
pub format_extension: Option<String>,
}
/// Processed resource type with parsed schema
#[derive(serde::Deserialize, serde::Serialize, Clone)]
pub struct HubResourceType {
@@ -413,6 +420,18 @@ pub struct HubResourceType {
pub schema: Option<serde_json::Value>,
pub app: String,
pub description: Option<String>,
/// Doubly optional on purpose. A cache written before this column has no key at
/// all (`None`) and must leave the stored extension alone; one written since
/// always writes the key, so an explicit null (`Some(None)`) is the hub genuinely
/// dropping it and must clear. A single `Option` conflates the two, and picking
/// either meaning breaks the other — as does plain serde, which folds `null`
/// into the outer `None`, hence the wrapping deserializer.
#[serde(
default,
deserialize_with = "windmill_common::more_serde::double_option",
skip_serializing_if = "Option::is_none"
)]
pub format_extension: Option<Option<String>>,
}
const HUB_RT_CACHE_FILE: &str = "resource_types.json";
@@ -459,6 +478,7 @@ async fn cache_hub_resource_types() -> anyhow::Result<()> {
schema,
app: rt.app,
description: rt.description,
format_extension: Some(rt.format_extension),
})
})
.collect();
@@ -500,9 +520,17 @@ pub async fn sync_cached_resource_types(db: &sqlx::Pool<sqlx::Postgres>) -> anyh
tracing::info!("Found {} cached resource types", cached_types.len());
// Get existing resource types in admins workspace
let existing_types: Vec<(String, Option<serde_json::Value>, Option<String>)> = sqlx::query_as(
"SELECT name, schema, description FROM resource_type WHERE workspace_id = 'admins'",
// Get existing resource types in admins workspace. `format_extension` is part of
// the comparison below, so a type whose only change is gaining or losing it is
// not mistaken for unchanged; `is_fileset` decides whether it may take one.
let existing_types: Vec<(
String,
Option<serde_json::Value>,
Option<String>,
Option<String>,
bool,
)> = sqlx::query_as(
"SELECT name, schema, description, format_extension, is_fileset FROM resource_type WHERE workspace_id = 'admins'",
)
.fetch_all(db)
.await
@@ -510,19 +538,42 @@ pub async fn sync_cached_resource_types(db: &sqlx::Pool<sqlx::Postgres>) -> anyh
let existing_map: std::collections::HashMap<
String,
(Option<serde_json::Value>, Option<String>),
(Option<serde_json::Value>, Option<String>, Option<String>, bool),
> = existing_types
.into_iter()
.map(|(name, schema, desc)| (name, (schema, desc)))
.map(|(name, schema, desc, format_extension, is_fileset)| {
(name, (schema, desc, format_extension, is_fileset))
})
.collect();
let mut synced_count = 0;
let mut skipped_count = 0;
for rt in cached_types {
// Check if resource type already exists with same schema and description
if let Some((existing_schema, existing_desc)) = existing_map.get(&rt.name) {
if existing_schema == &rt.schema && existing_desc == &rt.description {
let existing = existing_map.get(&rt.name);
let is_fileset = existing.map(|(_, _, _, f)| *f).unwrap_or(false);
let stored_extension = existing.and_then(|(_, _, e, _)| e.clone());
// A fileset is a set of files, so it cannot also be one file. Create, update
// and the manual sync all reject the pair; this writer would otherwise
// persist it onto a same-named local fileset.
//
// A cache with no key at all leaves the stored value alone, so the target is
// what is already there — which is also what makes the comparison below
// agree with the write instead of re-upserting the row on every boot.
let format_extension = if is_fileset {
None
} else {
match &rt.format_extension {
Some(from_cache) => from_cache.clone(),
None => stored_extension.clone(),
}
};
if let Some((existing_schema, existing_desc, _, _)) = existing {
if existing_schema == &rt.schema
&& existing_desc == &rt.description
&& stored_extension == format_extension
{
skipped_count += 1;
continue;
}
@@ -530,14 +581,19 @@ pub async fn sync_cached_resource_types(db: &sqlx::Pool<sqlx::Postgres>) -> anyh
// Insert or update resource type
sqlx::query(
"INSERT INTO resource_type (workspace_id, name, schema, description, edited_at)
VALUES ('admins', $1, $2, $3, now())
// `format_extension` is resolved above rather than coalesced here: a
// COALESCE could never clear one, so a hub that dropped an extension
// would leave the stale value behind forever.
"INSERT INTO resource_type (workspace_id, name, schema, description, format_extension, edited_at)
VALUES ('admins', $1, $2, $3, $4, now())
ON CONFLICT (workspace_id, name) DO UPDATE
SET schema = EXCLUDED.schema, description = EXCLUDED.description, edited_at = now()",
SET schema = EXCLUDED.schema, description = EXCLUDED.description,
format_extension = EXCLUDED.format_extension, edited_at = now()",
)
.bind(&rt.name)
.bind(&rt.schema)
.bind(&rt.description)
.bind(&format_extension)
.execute(db)
.await
.with_context(|| format!("Failed to upsert resource type {}", rt.name))?;
@@ -1261,10 +1317,12 @@ Windmill Community Edition {GIT_VERSION}
#[cfg(all(feature = "tantivy", feature = "parquet"))]
let log_indexer_f = {
let log_indexer_rx = killpill_rx.resubscribe();
let log_index_writer2 = log_index_writer.clone();
// Moved, not cloned: sealing a chunk takes sole ownership of its
// tantivy writer, which a second live handle would silently prevent.
let moved_log_index_writer = log_index_writer;
async {
if let Some(db) = conn.as_sql() {
if let Some(log_index_writer) = log_index_writer2 {
if let Some(log_index_writer) = moved_log_index_writer {
windmill_indexer::service_logs_oss::run_indexer(
db.clone(),
log_index_writer,
@@ -1661,7 +1719,7 @@ Windmill Community Edition {GIT_VERSION}
} else {
tracing::info!("Nothing to do, exiting.");
}
send_current_log_file_to_object_store(&conn, &hostname, &mode).await;
flush_pending_log_files_to_object_store(&conn, &hostname, &mode).await;
if let Some(db) = conn.as_sql() {
tracing::info!("Exiting connection pool");
@@ -1952,6 +2010,9 @@ async fn process_notify_event(
}
TIMEOUT_WAIT_RESULT_SETTING => reload_timeout_wait_result_setting(conn).await,
RETENTION_PERIOD_SECS_SETTING => reload_retention_period_setting(conn).await,
SERVICE_LOG_RETENTION_SECS_SETTING => {
reload_service_log_retention_secs_setting(conn).await
}
RETENTION_PERIOD_SECS_OVERRIDES_SETTING => {
if let Err(e) = load_retention_period_overrides(db).await {
tracing::error!("Error loading per-workspace retention overrides: {e:#}");
backend/src/monitor.rs
+337 -144
View File
@@ -70,18 +70,17 @@ use windmill_common::{
RETENTION_PERIOD_SECS_SETTING, SAML_METADATA_SETTING, SANDBOX_IMAGE_CACHE_MAX_MB_SETTING,
SANDBOX_IMAGE_DEFAULT_REGISTRY_SETTING, SANDBOX_IMAGE_MAX_SIZE_MB_SETTING,
SANDBOX_IMAGE_PULL_POLICY_SETTING, SANDBOX_REGISTRY_AUTH_SETTING, SCIM_TOKEN_SETTING,
SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
UV_EXCLUDE_NEWER_SETTING, UV_INDEX_STRATEGY_SETTING, UV_PYTHON_INSTALL_MIRROR_SETTING,
WORKSPACE_FAIRNESS_DURATION_SECS_SETTING, WORKSPACE_FAIRNESS_ENABLED_SETTING,
WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING,
WORKSPACE_MAX_QUEUED_JOBS_SETTING,
SERVICE_LOG_RETENTION_SECS_SETTING, SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING,
TIMEOUT_WAIT_RESULT_SETTING, UV_EXCLUDE_NEWER_SETTING, UV_INDEX_STRATEGY_SETTING,
UV_PYTHON_INSTALL_MIRROR_SETTING, WORKSPACE_FAIRNESS_DURATION_SECS_SETTING,
WORKSPACE_FAIRNESS_ENABLED_SETTING, WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING,
WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING, WORKSPACE_MAX_QUEUED_JOBS_SETTING,
},
indexer::load_indexer_config,
jobs::delete_jobs,
jwt::JWT_SECRET,
oauth2::REQUIRE_PREEXISTING_USER_FOR_OAUTH,
server::load_smtp_config,
tracing_init::JSON_FMT,
users::truncate_token,
utils::{empty_as_none, now_from_db, report_critical_error, Mode, HUB_API_SECRET},
worker::{
@@ -98,10 +97,10 @@ use windmill_common::{
KillpillSender, AUDIT_LOG_RETENTION_DAYS, BASE_URL, CRITICAL_ALERTS_ON_DB_OVERSIZE,
CRITICAL_ALERTS_ON_TOKEN_EXPIRY, CRITICAL_ALERT_MUTE_UI_ENABLED,
CRITICAL_ALERT_MUTE_ZOMBIE_JOB_RESTART, CRITICAL_ERROR_CHANNELS, DB, DEFAULT_HUB_BASE_URL,
HUB_BASE_URL, JOB_RETENTION_SECS, JOB_RETENTION_SECS_OVERRIDES,
JOB_RETENTION_SECS_OVERRIDES_LOADED, METRICS_DEBUG_ENABLED, METRICS_ENABLED,
MONITOR_LOGS_ON_OBJECT_STORE, OTEL_LOGS_ENABLED, OTEL_METRICS_ENABLED, OTEL_TRACING_ENABLED,
SERVICE_LOG_RETENTION_SECS, STORE_AUDIT_LOGS_S3,
DEFAULT_SERVICE_LOG_RETENTION_SECS, HUB_BASE_URL, JOB_RETENTION_SECS,
JOB_RETENTION_SECS_OVERRIDES, JOB_RETENTION_SECS_OVERRIDES_LOADED, METRICS_DEBUG_ENABLED,
METRICS_ENABLED, MONITOR_LOGS_ON_OBJECT_STORE, OTEL_LOGS_ENABLED, OTEL_METRICS_ENABLED,
OTEL_TRACING_ENABLED, STORE_AUDIT_LOGS_S3,
};
use windmill_common::{
client::AuthedClient,
@@ -476,6 +475,19 @@ pub async fn initial_load(
|v: Option<String>| async move { HUB_API_SECRET.store(std::sync::Arc::new(v)) },
);
// Outside the `server_mode` guard below: every mode reads this. A worker registers its
// rotated log files against the cutoff, and a dedicated indexer trims the search index to a
// window derived from it — neither is a server.
pass.setting(SERVICE_LOG_RETENTION_SECS_SETTING, true, |v| async move {
windmill_common::set_service_log_retention_secs(parse_setting_value::<i64>(
v,
SERVICE_LOG_RETENTION_SECS_SETTING,
"SERVICE_LOG_RETENTION_SECS",
DEFAULT_SERVICE_LOG_RETENTION_SECS,
|x| x,
))
});
if server_mode {
pass.setting(RETENTION_PERIOD_SECS_SETTING, true, |v| async move {
JOB_RETENTION_SECS.store(
@@ -1221,32 +1233,60 @@ async fn sleep_until_next_minute_start_plus_one_s() {
}
use windmill_common::tracing_init::TMP_WINDMILL_LOGS_SERVICE;
async fn find_two_highest_files(hostname: &str) -> (Option<String>, Option<String>) {
/// The minutely rolling appender names each file `<hostname>.log.<%Y-%m-%d-%H-%M>`;
/// anything else in the directory is not a rotated log file.
fn parse_log_file_ts(file_name: &str) -> Option<NaiveDateTime> {
NaiveDateTime::parse_from_str(
file_name.rsplit('.').next()?,
windmill_common::tracing_init::LOG_TIMESTAMP_FMT,
)
.ok()
}
/// Oldest first. Readdir order is filesystem-dependent — tmpfs hands back the
/// newest entry first, ext4 hashes the names — so the listing has to be sorted
/// before anything picks a file out of it.
fn sorted_log_files(file_names: impl Iterator<Item = String>) -> Vec<(NaiveDateTime, String)> {
let mut files = file_names
.filter_map(|name| parse_log_file_ts(&name).map(|ts| (ts, name)))
.collect::<Vec<_>>();
files.sort();
files
}
/// Every log file but the newest one: that one is still being appended to, every
/// older one is final.
fn rotated_log_files(file_names: impl Iterator<Item = String>) -> Vec<(NaiveDateTime, String)> {
let mut files = sorted_log_files(file_names);
files.pop();
files
}
async fn read_log_file_names(hostname: &str) -> Vec<String> {
let log_dir = format!("{}/{}/", *TMP_WINDMILL_LOGS_SERVICE, hostname);
let rd_dir = tokio::fs::read_dir(log_dir).await;
if let Ok(mut log_files) = rd_dir {
let mut highest_file: Option<String> = None;
let mut second_highest_file: Option<String> = None;
while let Ok(Some(file)) = log_files.next_entry().await {
let file_name = file
.file_name()
.to_str()
.map(|x| x.to_string())
.unwrap_or_default();
if file_name > highest_file.clone().unwrap_or_default() {
second_highest_file = highest_file;
highest_file = Some(file_name);
}
let mut rd_dir = match tokio::fs::read_dir(&log_dir).await {
Ok(rd_dir) => rd_dir,
Err(e) => {
tracing::error!("Error reading log files: {}, {:#?}", log_dir, e);
return vec![];
}
};
let mut file_names = vec![];
while let Ok(Some(file)) = rd_dir.next_entry().await {
if let Some(file_name) = file.file_name().to_str() {
file_names.push(file_name.to_string());
}
(highest_file, second_highest_file)
} else {
tracing::error!(
"Error reading log files: {}, {:#?}",
*TMP_WINDMILL_LOGS_SERVICE,
rd_dir.unwrap_err()
);
(None, None)
}
file_names
}
async fn list_log_files(hostname: &str) -> Vec<(NaiveDateTime, String)> {
sorted_log_files(read_log_file_names(hostname).await.into_iter())
}
async fn list_rotated_log_files(hostname: &str) -> Vec<(NaiveDateTime, String)> {
rotated_log_files(read_log_file_names(hostname).await.into_iter())
}
fn get_worker_group(mode: &Mode) -> Option<String> {
@@ -1266,133 +1306,188 @@ pub fn send_logs_to_object_store(conn: &Connection, hostname: &str, mode: &Mode)
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(10));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
init_last_log_file_sent(&conn, &hostname).await;
sleep_until_next_minute_start_plus_one_s().await;
loop {
interval.tick().await;
let (_, snd_highest_file) = find_two_highest_files(&hostname).await;
send_log_file_to_object_store(
&hostname,
&mode,
&worker_group,
&conn,
snd_highest_file,
false,
)
.await;
let files = list_rotated_log_files(&hostname).await;
send_log_files_to_object_store(&hostname, &mode, &worker_group, &conn, files).await;
}
});
}
pub async fn send_current_log_file_to_object_store(conn: &Connection, hostname: &str, mode: &Mode) {
tracing::info!("Sending current log file to object store");
let (highest_file, _) = find_two_highest_files(hostname).await;
pub async fn flush_pending_log_files_to_object_store(
conn: &Connection,
hostname: &str,
mode: &Mode,
) {
tracing::info!("Sending pending log files to object store");
let worker_group = get_worker_group(&mode);
send_log_file_to_object_store(hostname, mode, &worker_group, conn, highest_file, true).await;
}
fn get_now_and_str() -> (NaiveDateTime, String) {
let ts = Utc::now().naive_utc();
(
ts,
ts.format(windmill_common::tracing_init::LOG_TIMESTAMP_FMT)
.to_string(),
)
// Nothing rotates after this, so the file still being appended to is registered
// here, along with any rotated one the loop had not reached yet. Bounded like the
// pool close that follows: a backlog against a slow object store would otherwise
// hold the process past its termination grace period. Whatever is left over is
// registered by the next run's catch-up.
let flush = async {
let files = list_log_files(hostname).await;
send_log_files_to_object_store(hostname, mode, &worker_group, conn, files).await;
};
if timeout(Duration::from_secs(15), flush).await.is_err() {
tracing::warn!("Could not send all pending log files in time (15s). Exiting anyway.");
}
}
lazy_static::lazy_static! {
static ref LAST_LOG_FILE_SENT: Arc<Mutex<Option<NaiveDateTime>>> = Arc::new(Mutex::new(None));
/// Serializes the periodic uploader against the shutdown flush. The uploader is a
/// detached task that keeps ticking while the flush runs and both walk the same
/// files, so without this both can clear the watermark for one file and count its
/// lines twice through the additive upsert.
static ref SENDING_LOG_FILES: tokio::sync::Mutex<()> = tokio::sync::Mutex::new(());
}
fn last_log_file_sent() -> Option<NaiveDateTime> {
LAST_LOG_FILE_SENT.lock().ok().and_then(|ts| *ts)
}
/// Resume from what this host already registered, so a previous run's leftovers reach
/// the object store rather than being dropped. Their line counts come out zero, this
/// run having counted none of them, which only flattens their bars in the UI.
///
/// The newest registered minute is left out on purpose: the shutdown flush registers
/// the file that was still open and the appender reopens that minute in append mode,
/// so a restart inside it would otherwise strand everything written afterwards.
///
/// A row rewritten this way restores the object and sums the counters, but it keeps the
/// `indexed_at` it already had, so one the indexers have taken is not offered again and
/// the lines added by the rewrite stay out of search.
async fn init_last_log_file_sent(conn: &Connection, hostname: &str) {
let Some(db) = conn.as_sql() else {
return;
};
match sqlx::query_scalar!(
"SELECT max(log_ts) FROM log_file
WHERE hostname = $1 AND log_ts < (SELECT max(log_ts) FROM log_file WHERE hostname = $1)",
hostname
)
.fetch_one(db)
.await
{
Ok(Some(ts)) => {
if let Err(e) = LAST_LOG_FILE_SENT.lock().map(|mut last_log_file_sent| {
last_log_file_sent.replace(ts);
}) {
tracing::error!("Error initializing last log file sent: {:?}", e);
}
}
Ok(None) => {}
Err(e) => tracing::error!("Error loading last log file sent: {:?}", e),
}
}
async fn send_log_files_to_object_store(
hostname: &str,
mode: &Mode,
worker_group: &Option<String>,
conn: &Connection,
files: Vec<(NaiveDateTime, String)>,
) {
let _guard = SENDING_LOG_FILES.lock().await;
let retention_cutoff = Utc::now().naive_utc()
- chrono::Duration::seconds(windmill_common::service_log_retention_secs());
for (ts, file_name) in files {
if last_log_file_sent().is_some_and(|last| last >= ts) {
continue;
}
// A run coming back from a long outage still finds its predecessor's files on
// disk. Registering one past the retention cutoff inserts a row
// `delete_expired_items` drops on its next pass, once the indexers have already
// paid to parse it.
if ts < retention_cutoff {
continue;
}
// Stop at the first failure rather than moving on, so a file is never
// registered before an older one that has not made it to the store yet.
// The indexers do not depend on that ordering — every row is offered until
// it is marked — but a gap here would still be visible while it lasts.
if !send_log_file_to_object_store(hostname, mode, worker_group, conn, &file_name, ts).await
{
break;
}
}
}
/// Returns whether the file ended up registered in `log_file`.
async fn send_log_file_to_object_store(
hostname: &str,
mode: &Mode,
worker_group: &Option<String>,
conn: &Connection,
snd_highest_file: Option<String>,
use_now: bool,
) {
if let Some(highest_file) = snd_highest_file {
//parse datetime frome file xxxx.yyyy-MM-dd-HH-mm
let (ts, ts_str) = if use_now {
get_now_and_str()
} else {
highest_file
.split(".")
.last()
.and_then(|x| {
NaiveDateTime::parse_from_str(
x,
windmill_common::tracing_init::LOG_TIMESTAMP_FMT,
)
.ok()
.map(|y| (y, x.to_string()))
})
.unwrap_or_else(get_now_and_str)
};
file_name: &str,
ts: NaiveDateTime,
) -> bool {
#[cfg(feature = "parquet")]
if let Some(s3_client) = windmill_object_store::get_object_store().await {
let path = std::path::Path::new(&*TMP_WINDMILL_LOGS_SERVICE)
.join(hostname)
.join(file_name);
let exists = LAST_LOG_FILE_SENT.lock().map(|last_log_file_sent| {
last_log_file_sent
.map(|last_log_file_sent| last_log_file_sent >= ts)
.unwrap_or(false)
});
if exists.unwrap_or(false) {
return;
}
#[cfg(feature = "parquet")]
let s3_client = windmill_object_store::get_object_store().await;
#[cfg(feature = "parquet")]
if let Some(s3_client) = s3_client {
let path = std::path::Path::new(&*TMP_WINDMILL_LOGS_SERVICE)
.join(hostname)
.join(&highest_file);
//read file as byte stream
let bytes = tokio::fs::read(&path).await;
if let Err(e) = bytes {
//read file as byte stream
let bytes = match tokio::fs::read(&path).await {
Ok(bytes) => bytes,
Err(e) => {
tracing::error!("Error reading log file: {:?}", e);
return;
return false;
}
let path = windmill_object_store::object_store_reexports::Path::from_url_path(format!(
"{}{hostname}/{highest_file}",
windmill_common::tracing_init::LOGS_SERVICE
));
if let Err(e) = path {
};
let path = windmill_object_store::object_store_reexports::Path::from_url_path(format!(
"{}{hostname}/{file_name}",
windmill_common::tracing_init::LOGS_SERVICE
));
let path = match path {
Ok(path) => path,
Err(e) => {
tracing::error!("Error creating log file path: {:?}", e);
return;
}
if let Err(e) = s3_client.put(&path.unwrap(), bytes.unwrap().into()).await {
tracing::error!("Error sending logs to object store: {:?}", e);
return false;
}
};
if let Err(e) = s3_client.put(&path, bytes.into()).await {
tracing::error!("Error sending logs to object store: {:?}", e);
return false;
}
}
let (ok_lines, err_lines) = read_log_counters(ts_str);
let ts_str = ts
.format(windmill_common::tracing_init::LOG_TIMESTAMP_FMT)
.to_string();
let (ok_lines, err_lines) = read_log_counters(ts_str);
if let Some(db) = conn.as_sql() {
match timeout(Duration::from_secs(10), sqlx::query!("INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)
VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)
ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
hostname, mode.to_string(), worker_group.clone(), ts, highest_file, ok_lines as i64, err_lines as i64, *JSON_FMT)
.execute(db)).await {
Ok(Ok(_)) => {
if let Err(e) = LAST_LOG_FILE_SENT.lock().map(|mut last_log_file_sent| {
last_log_file_sent.replace(ts);
}) {
tracing::error!("Error updating last log file sent: {:?}", e);
}
tracing::info!("Log file sent: {}", highest_file);
}
Ok(Err(e)) => {
tracing::error!("Error inserting log file: {:?}", e);
}
Err(e) => {
tracing::error!("Error inserting log file, timeout elapsed: {:?}", e);
}
let Some(db) = conn.as_sql() else {
// not sending log file to object store in agent mode
return false;
};
match timeout(Duration::from_secs(10), sqlx::query!("INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)
VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)
ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
hostname, mode.to_string(), worker_group.clone(), ts, file_name, ok_lines as i64, err_lines as i64, true)
.execute(db)).await {
Ok(Ok(_)) => {
if let Err(e) = LAST_LOG_FILE_SENT.lock().map(|mut last_log_file_sent| {
last_log_file_sent.replace(ts);
}) {
tracing::error!("Error updating last log file sent: {:?}", e);
}
} else {
// tracing::warn!("Not sending log file to object store in agent mode");
()
tracing::info!("Log file sent: {}", file_name);
true
}
Ok(Err(e)) => {
tracing::error!("Error inserting log file: {:?}", e);
false
}
Err(e) => {
tracing::error!("Error inserting log file, timeout elapsed: {:?}", e);
false
}
}
}
@@ -1580,6 +1675,13 @@ pub async fn trim_resource_versions(db: &DB) -> () {
}
}
/// Matches the batch the settings-page cleanup uses for the same table.
const SERVICE_LOG_DELETE_BATCH: i64 = 2_000;
/// Batches per pass. `monitor_db` runs under a 600s timeout that cancels every maintenance
/// future in the same `join!` and reports a critical error, so a large backlog has to drain
/// across ticks rather than inside one, the way the neighbouring sweeps already do.
const SERVICE_LOG_DELETE_MAX_BATCHES: usize = 10;
pub async fn delete_expired_items(db: &DB) -> () {
let expired_tokens_r = sqlx::query_as!(
TokenRow,
@@ -1662,23 +1764,48 @@ pub async fn delete_expired_items(db: &DB) -> () {
Err(e) => tracing::error!("Error deleting cache resource {}", e.to_string()),
}
match sqlx::query_as!(
LogFile,
"DELETE FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval RETURNING file_path, hostname",
SERVICE_LOG_RETENTION_SECS,
)
.fetch_all(db)
.await
{
Ok(log_files_to_delete) => {
// Batched: every process rotates a log file a minute, so lowering the retention makes one
// ordinary setting change expire millions of rows at once. An unbounded `DELETE ...
// RETURNING` would materialize all of them, and their deletion futures, in this one tick.
for _ in 0..SERVICE_LOG_DELETE_MAX_BATCHES {
let batch = sqlx::query_as!(
LogFile,
"DELETE FROM log_file WHERE (hostname, log_ts) IN (
SELECT hostname, log_ts FROM log_file
WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval
LIMIT $2
) RETURNING file_path, hostname",
windmill_common::service_log_retention_secs(),
SERVICE_LOG_DELETE_BATCH,
)
.fetch_all(db)
.await;
match batch {
Ok(log_files_to_delete) => {
if log_files_to_delete.is_empty() {
break;
}
let n = log_files_to_delete.len();
let paths = log_files_to_delete
.iter()
.map(|f| format!("{}/{}", f.hostname, f.file_path))
.collect();
delete_log_files_from_disk_and_store(paths, &*TMP_WINDMILL_LOGS_SERVICE, windmill_common::tracing_init::LOGS_SERVICE).await;
delete_log_files_from_disk_and_store(
paths,
&*TMP_WINDMILL_LOGS_SERVICE,
windmill_common::tracing_init::LOGS_SERVICE,
)
.await;
if (n as i64) < SERVICE_LOG_DELETE_BATCH {
break;
}
}
Err(e) => {
tracing::error!("Error deleting log file: {:?}", e);
break;
}
}
Err(e) => tracing::error!("Error deleting log file: {:?}", e),
}
let audit_retention_days = audit_log_retention_days().await;
@@ -2785,6 +2912,21 @@ pub async fn reload_retention_period_setting(conn: &Connection) {
}
}
pub async fn reload_service_log_retention_secs_setting(conn: &Connection) {
match load_setting_value::<i64>(
conn,
SERVICE_LOG_RETENTION_SECS_SETTING,
"SERVICE_LOG_RETENTION_SECS",
DEFAULT_SERVICE_LOG_RETENTION_SECS,
|x| x,
)
.await
{
Ok(v) => windmill_common::set_service_log_retention_secs(v),
Err(e) => tracing::error!("Error reloading service log retention period: {:?}", e),
}
}
pub async fn reload_audit_log_retention_days_setting(conn: &Connection) {
match load_setting_value::<i64>(
conn,
@@ -6832,3 +6974,54 @@ mod zombie_worker_memory_pct_tests {
);
}
}
#[cfg(test)]
mod log_file_listing_tests {
use super::{rotated_log_files, sorted_log_files};
fn names(files: Vec<(chrono::NaiveDateTime, String)>) -> Vec<String> {
files.into_iter().map(|(_, n)| n).collect()
}
/// A directory read newest-entry-first is what tmpfs actually hands back.
#[test]
fn orders_by_minute_whatever_order_readdir_used() {
let newest_first = [
"h.log.2026-08-29-06-49",
"h.log.2026-08-29-06-46",
"h.log.2026-08-29-06-48",
"h.log.2026-08-29-06-47",
];
assert_eq!(
names(sorted_log_files(newest_first.iter().map(|x| x.to_string()))),
vec![
"h.log.2026-08-29-06-46",
"h.log.2026-08-29-06-47",
"h.log.2026-08-29-06-48",
"h.log.2026-08-29-06-49",
]
);
assert_eq!(
names(rotated_log_files(
newest_first.iter().map(|x| x.to_string())
)),
vec![
"h.log.2026-08-29-06-46",
"h.log.2026-08-29-06-47",
"h.log.2026-08-29-06-48",
]
);
}
#[test]
fn drops_names_that_are_not_rotated_log_files() {
let files = sorted_log_files(
["h.log", "not-a-log-file", "h.log.2026-08-29-06-46"]
.iter()
.map(|x| x.to_string()),
);
assert_eq!(files.len(), 1);
assert_eq!(files[0].1, "h.log.2026-08-29-06-46");
assert_eq!(files[0].0.to_string(), "2026-08-29 06:46:00");
}
}
backend/summarized_schema.txt
+3 -1
View File
@@ -38,6 +38,8 @@ account: workspace_id(char), id(int), expires_at(ts), refresh_token(char), clien
FK: (workspace_id) -> workspace(id)
agent_token_blacklist: token(char), expires_at(ts), blacklisted_at(ts), blacklisted_by(char)
ai_agent_memory: workspace_id(char), conversation_id(uuid), step_id(char), messages(jsonb), created_at(ts), updated_at(ts)
ai_free_token_daily_usage: day(date), cost_nanos(bigint), updated_at(ts)
ai_free_token_usage: email(char), cost_nanos(bigint), updated_at(ts)
ai_token_usage: workspace_id(char), day(date), email(char), provider(char), model(char), session_id(char), input_tokens(bigint), cache_read_tokens(bigint), cache_write_tokens(bigint), output_tokens(bigint), reported_cost_nano_usd(bigint), requests(bigint), updated_at(ts)
FK: (workspace_id) -> workspace(id)
alerts: id(int), alert_type(char), message(text), created_at(ts), acknowledged(bool), workspace_id(text), acknowledged_workspace(bool), resource(text)
@@ -128,7 +130,7 @@ job_stats: workspace_id(char), job_id(uuid), metric_id(char), metric_name(char),
kafka_pending_commits: id(bigint), workspace_id(char), kafka_trigger_path(char), topic(char), partition(int), offset(bigint), created_at(ts)
FK: (workspace_id, kafka_trigger_path) -> kafka_trigger(workspace_id, path)
kafka_trigger: path(char), kafka_resource_path(char), topics(char), group_id(char), script_path(char), is_flow(bool), workspace_id(char), edited_by(char), email(char), edited_at(ts), extra_perms(jsonb), server_id(char), last_server_ping(ts), error(text), error_handler_path(char), error_handler_args(jsonb), retry(jsonb), mode(trigger_mode), filters(jsonb[]), auto_commit(bool), labels(text[])
log_file: hostname(char), log_ts(ts), ok_lines(bigint), err_lines(bigint), mode(log_mode), worker_group(char), file_path(char), json_fmt(bool)
log_file: hostname(char), log_ts(ts), ok_lines(bigint), err_lines(bigint), mode(log_mode), worker_group(char), file_path(char), json_fmt(bool), indexed_at(ts)
macro_definition: workspace_id(char), name(char), provider_path(char), params(text), body(text), is_table_macro(bool), created_at(ts)
FK: (workspace_id) -> workspace(id)
macro_usage: workspace_id(char), consumer_path(char), macro_name(char)
backend/tests/suspended_pull_index.rs
+77
View File
@@ -0,0 +1,77 @@
//! Pins the plan of the suspended-job pull. Its resume test degrades silently: once the
//! query expression and `queue_suspended_v2` stop matching, Postgres still returns the right
//! job, just by falling back to a heap filter and fetching one tuple per suspended row on
//! every worker poll. No functional test can see that, so assert on the plan instead.
use serde_json::Value;
use sqlx::{Pool, Postgres};
use windmill_common::worker::make_suspended_pull_query;
/// Depth-first walk of an `EXPLAIN (FORMAT JSON)` plan tree.
fn nodes(plan: &Value, out: &mut Vec<Value>) {
out.push(plan.clone());
for child in plan["Plans"].as_array().unwrap_or(&vec![]) {
nodes(child, out);
}
}
#[sqlx::test(fixtures("base"))]
async fn suspended_pull_tests_resume_time_inside_the_index(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
sqlx::query(
"INSERT INTO v2_job_queue (id, workspace_id, created_at, scheduled_for, running, suspend, suspend_until, tag)
SELECT gen_random_uuid(), 'test-workspace', now() - make_interval(secs => i),
now(), true, 1 + (i % 3), now() + interval '7 day', 'flow'
FROM generate_series(1, 2000) i",
)
.execute(&db)
.await?;
sqlx::query("ANALYZE v2_job_queue").execute(&db).await?;
// Both plans are cheap on a 2000-row table, and which one wins there says nothing
// about a queue with a large suspended backlog. Force the index path, which is the
// one production takes, and assert on how it evaluates the resume test.
let mut conn = db.acquire().await?;
sqlx::query("SET enable_seqscan = off")
.execute(&mut *conn)
.await?;
let version: String = sqlx::query_scalar("SELECT version()")
.fetch_one(&mut *conn)
.await?;
// FORMAT JSON rather than the default: `Index Cond` and `Filter` are separate keys on the
// node, so this does not ride on EXPLAIN's line layout staying put across a major bump.
let explained: Value = sqlx::query_scalar(&format!(
"EXPLAIN (FORMAT JSON) {}",
make_suspended_pull_query(&["flow".to_string()])
))
.bind("test-worker")
.fetch_one(&mut *conn)
.await?;
let mut all = vec![];
nodes(&explained[0]["Plan"], &mut all);
let pretty = serde_json::to_string_pretty(&explained)?;
let scan = all
.iter()
.find(|n| n["Index Name"] == "queue_suspended_v2")
.unwrap_or_else(|| {
panic!("suspended pull did not scan queue_suspended_v2 on {version}:\n{pretty}")
});
// Only `Index Cond` is checked against the index tuple, so that is where the resume test
// has to land — as a `Filter` it would cost a heap fetch per suspended row. The residual
// `suspend_until IS NOT NULL` filter is not that: it is always true for rows the partial
// index holds, and only ever runs on the row LIMIT 1 already fetched.
let cond = scan["Index Cond"].as_str().unwrap_or_else(|| {
panic!("no Index Cond on the suspended pull scan on {version}:\n{pretty}")
});
assert!(
cond.contains("CASE WHEN"),
"resume test is not an index condition on {version}:\n{pretty}"
);
assert!(
!scan["Filter"].as_str().unwrap_or("").contains("CASE WHEN"),
"resume test fell back to a heap filter on {version}:\n{pretty}"
);
Ok(())
}
backend/tests/wm_token_confinement.rs
+27
View File
@@ -318,6 +318,33 @@ async fn test_wm_token_is_confined_to_its_workspace(db: Pool<Postgres>) -> anyho
resp.text().await?
);
}
// ...and the one `settings/global` key on the allowlist, which the CLI reads before
// creating a user on a git-sync push. `ws_base_url` is the control: the handler leaves
// it as ungated as `automate_username_creation`, so only the allowlist stops it.
let resp = authed(
client().get(format!("{api}/settings/global/automate_username_creation")),
&user_wm,
)
.send()
.await?;
assert_eq!(
resp.status(),
200,
"WM_TOKEN must still read automate_username_creation: {}",
resp.text().await?
);
let resp = authed(
client().get(format!("{api}/settings/global/ws_base_url")),
&user_wm,
)
.send()
.await?;
assert_eq!(
resp.status(),
403,
"WM_TOKEN must not read any other global setting: {}",
resp.text().await?
);
let resp = authed(client().post(format!("{api}/schedules/preview")), &user_wm)
.json(&json!({ "schedule": "0 0 12 * * *", "timezone": "UTC" }))
.send()
backend/windmill-ai/src/ai_providers.rs
+10
View File
@@ -24,6 +24,16 @@ lazy_static::lazy_static! {
.ok()
.map(|v| v == "true" || v == "1")
.unwrap_or(false);
/// Drops the cache breakpoints from agent-step requests on every Anthropic platform,
/// not just the one that motivates it: a Google Cloud project can have explicit prompt
/// caching turned off (by request to Cloud support), and Vertex then rejects any request
/// carrying breakpoints. An instance that sets this to unblock such a project also gives
/// up caching on its direct-Anthropic and Foundry resources.
pub static ref DISABLE_ANTHROPIC_PROMPT_CACHING: bool =
std::env::var("DISABLE_ANTHROPIC_PROMPT_CACHING")
.ok()
.map(|v| v == "true" || v == "1")
.unwrap_or(false);
}
pub const OPENAI_BASE_URL: &str = "https://api.openai.com/v1";
backend/windmill-ai/src/providers/anthropic.rs
+47 -11
View File
@@ -1,7 +1,7 @@
use super::{anthropic_model_rejects_sampling_params, REASONING_OFF_SENTINEL};
use crate::{
ai_google::parse_data_url,
ai_providers::{AIPlatform, AIProvider},
ai_providers::{AIPlatform, AIProvider, DISABLE_ANTHROPIC_PROMPT_CACHING},
image_handler::prepare_messages_for_api,
proxy::{
add_user_to_body, common_outbound_headers, credential_header, ProxyBuildArgs, ProxyRequest,
@@ -632,15 +632,13 @@ impl AnthropicQueryBuilder {
}
}
let caching = !*DISABLE_ANTHROPIC_PROMPT_CACHING;
let system = collect_system_prompt(&prepared_messages, args.system_prompt).map(|text| {
vec![AnthropicSystemContent {
r#type: "text".to_string(),
text,
cache_control: if self.is_vertex() {
None
} else {
Some(CacheControl::ephemeral())
},
cache_control: caching.then(CacheControl::ephemeral),
}]
});
@@ -665,7 +663,7 @@ impl AnthropicQueryBuilder {
let max_tokens = Some(args.max_tokens.unwrap_or(64000));
// Apply cache_control on the last custom tool
if !self.is_vertex() {
if caching {
if let Some(ref mut tools_vec) = tools_option {
if let Some(AnthropicTool::Custom(ref mut custom)) = tools_vec.last_mut() {
custom.cache_control = Some(CacheControl::ephemeral());
@@ -674,7 +672,7 @@ impl AnthropicQueryBuilder {
}
// Apply cache_control on the last content block of the last message
if !self.is_vertex() {
if caching {
if let Some(last_msg) = anthropic_messages.last_mut() {
if let Some(last_block) = last_msg.content.last_mut() {
match last_block {
@@ -882,10 +880,15 @@ mod tests {
}
}
async fn build_text_body(messages: &[OpenAIMessage], system_prompt: Option<&str>) -> String {
async fn build_text_body_on(
platform: AIPlatform,
messages: &[OpenAIMessage],
system_prompt: Option<&str>,
tools: Option<&[ToolDef]>,
) -> String {
let args = BuildRequestArgs {
messages,
tools: None,
tools,
model: "claude-sonnet-4",
temperature: None,
reasoning_effort: None,
@@ -899,12 +902,16 @@ mod tests {
prompt_cache_key: None,
};
AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::Standard)
AnthropicQueryBuilder::new(AIProvider::Anthropic, platform)
.build_request(&args, &authed_client(), "test-workspace")
.await
.unwrap()
}
async fn build_text_body(messages: &[OpenAIMessage], system_prompt: Option<&str>) -> String {
build_text_body_on(AIPlatform::Standard, messages, system_prompt, None).await
}
/// The worker prepends the system prompt as a system message *and* passes it as
/// `system_prompt`; the request must still carry it exactly once.
#[tokio::test]
@@ -946,6 +953,35 @@ mod tests {
assert!(request.get("system").is_none());
}
/// Vertex serves the same Messages API and honours `cache_control` breakpoints, so
/// its requests must carry the same three the standard platform gets.
#[tokio::test]
async fn sets_cache_breakpoints_on_every_platform() {
let messages = vec![message("system", SYSTEM_PROMPT), message("user", "hi")];
let tools = vec![ToolDef {
r#type: "function".to_string(),
function: ToolDefFunction {
name: "get_weather".to_string(),
description: None,
parameters: RawValue::from_string("{}".to_string()).unwrap(),
},
}];
let ephemeral = serde_json::json!({ "type": "ephemeral" });
for platform in [AIPlatform::Standard, AIPlatform::GoogleVertexAi] {
let body =
build_text_body_on(platform, &messages, Some(SYSTEM_PROMPT), Some(&tools)).await;
let request: serde_json::Value = serde_json::from_str(&body).unwrap();
assert_eq!(request["system"][0]["cache_control"], ephemeral);
let sent_tools = request["tools"].as_array().unwrap();
assert_eq!(sent_tools.last().unwrap()["cache_control"], ephemeral);
let sent = request["messages"].as_array().unwrap();
let content = sent.last().unwrap()["content"].as_array().unwrap();
assert_eq!(content.last().unwrap()["cache_control"], ephemeral);
}
}
fn has_header(headers: &[(String, String)], name: &str, value: &str) -> bool {
headers
.iter()
backend/windmill-api-auth/src/scopes.rs
+8 -49
View File
@@ -288,7 +288,6 @@ pub enum ScopeDomain {
Configs,
OAuth,
AI,
AiSkills,
AiEvals, // AI agent eval datasets
Indexer,
@@ -349,7 +348,6 @@ impl ScopeDomain {
Self::Configs => "configs",
Self::OAuth => "oauth",
Self::AI => "ai",
Self::AiSkills => "ai_skills",
Self::AiEvals => "ai_evals",
Self::Capture => "capture",
Self::Drafts => "drafts",
@@ -405,7 +403,6 @@ impl ScopeDomain {
"configs" => Some(Self::Configs),
"oauth" => Some(Self::OAuth),
"ai" => Some(Self::AI),
"ai_skills" => Some(Self::AiSkills),
"ai_evals" => Some(Self::AiEvals),
"indexer" | "srch" => Some(Self::Indexer),
"teams" => Some(Self::Teams),
@@ -996,6 +993,13 @@ fn scope_grants_access(
/// the caller's own row; `email` and `allowed_domain_auto_invite` are derived from the
/// token itself and touch no table.
///
/// `settings/global/automate_username_creation` is the one instance setting on the list.
/// `get_global_setting` exempts a handful of keys from its own super-admin gate, that one
/// among them, so the boolean is already readable by every authenticated user; it is here
/// because the CLI reads it before creating a user during a git-sync push, which runs as a
/// job. The other ungated keys have no such caller, so they stay confined — being ungated
/// earns a key nothing on its own.
///
/// Deliberately absent, as each crosses that line: `users/list_invites` (returns the
/// workspace ids the identity was invited to), `users/tokens/list` (credential metadata
/// of the borrowed identity), `users/exists/{email}` (an oracle over arbitrary
@@ -1011,6 +1015,7 @@ fn is_global_read_open_to_job_token(route_path: &str) -> bool {
| "/api/users/usage"
| "/api/users/tutorial_progress"
| "/api/workspaces/allowed_domain_auto_invite"
| "/api/settings/global/automate_username_creation"
| "/api/docs/search"
| "/api/docs/page"
| "/api/integrations/hub/list"
@@ -1194,12 +1199,6 @@ mod tests {
assert_eq!(domain, ScopeDomain::FlowConversations);
assert_eq!(kind, None);
assert_eq!(route_suffix, Some("flow_conversations/list".to_string()));
let (domain, kind, route_suffix) =
extract_domain_from_route("/api/w/test_workspace/ai_skills/list").unwrap();
assert_eq!(domain, ScopeDomain::AiSkills);
assert_eq!(kind, None);
assert_eq!(route_suffix, Some("ai_skills/list".to_string()));
}
#[test]
@@ -1360,11 +1359,6 @@ mod tests {
ScopeDomain::from_str("flow_conversations"),
Some(ScopeDomain::FlowConversations)
);
assert_eq!(
ScopeDomain::from_str("ai_skills"),
Some(ScopeDomain::AiSkills)
);
// Test canonical string conversion
assert_eq!(ScopeDomain::Acls.as_str(), "acls");
assert_eq!(ScopeDomain::RawApps.as_str(), "raw_apps");
@@ -1373,41 +1367,6 @@ mod tests {
ScopeDomain::FlowConversations.as_str(),
"flow_conversations"
);
assert_eq!(ScopeDomain::AiSkills.as_str(), "ai_skills");
}
#[test]
fn test_ai_skills_scope_access() {
let read_scopes = vec!["ai_skills:read".to_string()];
assert!(
check_route_access(&read_scopes, "/api/w/test_workspace/ai_skills/list", "GET").is_ok()
);
assert!(check_route_access(
&read_scopes,
"/api/w/test_workspace/ai_skills/get/foo",
"GET"
)
.is_ok());
assert!(check_route_access(
&read_scopes,
"/api/w/test_workspace/ai_skills/upload",
"POST"
)
.is_err());
let write_scopes = vec!["ai_skills:write".to_string()];
assert!(check_route_access(
&write_scopes,
"/api/w/test_workspace/ai_skills/upload",
"POST"
)
.is_ok());
assert!(check_route_access(
&write_scopes,
"/api/w/test_workspace/ai_skills/delete/foo",
"DELETE"
)
.is_ok());
}
#[test]
backend/windmill-api-debug/src/lib.rs
+5
View File
@@ -468,6 +468,9 @@ async fn sign_debug_request(
// Parse the language
let script_lang: ScriptLang = request.language.parse().unwrap_or(ScriptLang::Bun);
// Taken from the parsed language, not the request's string: the telemetry key vocabulary has
// to stay the closed set of languages rather than whatever a caller sent.
let lang_key = script_lang.as_str();
// Hash the code (we don't include full code in JWT to keep it small)
let mut hasher = Sha256::new();
@@ -578,6 +581,8 @@ async fn sign_debug_request(
tx.commit().await?;
windmill_common::feature_usage::log_feature_usage("debugger", "session", lang_key);
Ok(Json(SignedDebugPayload {
token,
code: request.code,
backend/windmill-api-integration-tests/tests/users.rs
+15 -8
View File
@@ -308,14 +308,17 @@ async fn test_user_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
let auth_base = format!("http://localhost:{port}/api/auth");
// --- login (will fail: password hash in fixture is fake) ---
// An unparseable stored hash must read as a failed login, not as a server error
// relaying the hash parser's message to an unauthenticated caller.
let resp = client()
.post(format!("{auth_base}/login"))
.json(&json!({"email": "test@windmill.dev", "password": "wrong-password"}))
.send()
.await
.unwrap();
assert!(
resp.status() == 400 || resp.status() == 401 || resp.status() == 500,
assert_eq!(
resp.status(),
400,
"login: unexpected status {}",
resp.status()
);
@@ -804,12 +807,16 @@ async fn test_change_user_email_leaves_group_identities(db: Pool<Postgres>) -> a
let server = ApiServer::start(db.clone()).await?;
let global_base = format!("http://localhost:{}/api/users", server.addr.port());
sqlx::query!("UPDATE password SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'")
.execute(&db)
.await?;
sqlx::query!("UPDATE usr SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'")
.execute(&db)
.await?;
sqlx::query!(
"UPDATE password SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'"
)
.execute(&db)
.await?;
sqlx::query!(
"UPDATE usr SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'"
)
.execute(&db)
.await?;
sqlx::query!(
"INSERT INTO group_(workspace_id, name, summary, extra_perms) VALUES ('test-workspace', 'ops', '', '{}')"
)
backend/windmill-api-settings/src/lib.rs
+29 -4
View File
@@ -2004,6 +2004,12 @@ struct CachedResourceType {
#[allow(dead_code)]
app: String,
description: Option<String>,
/// Doubly optional, and read through a wrapping deserializer: this struct also
/// decodes the on-disk cache, where an absent key means "written before the
/// column, leave the stored extension alone" and an explicit null means the hub
/// dropped it. Plain serde folds both into `None`.
#[serde(default, deserialize_with = "windmill_common::more_serde::double_option")]
format_extension: Option<Option<String>>,
}
#[derive(serde::Deserialize)]
@@ -2013,6 +2019,8 @@ struct HubResourceTypeRaw {
schema: Option<String>,
app: String,
description: Option<String>,
#[serde(default)]
format_extension: Option<String>,
}
async fn fetch_resource_types_from_hub() -> error::Result<Vec<CachedResourceType>> {
@@ -2054,6 +2062,7 @@ async fn fetch_resource_types_from_hub() -> error::Result<Vec<CachedResourceType
schema,
app: rt.app,
description: rt.description,
format_extension: Some(rt.format_extension),
})
})
.collect())
@@ -2107,10 +2116,12 @@ async fn sync_cached_resource_types(
for rt in &resource_types {
let exists: Option<bool> = sqlx::query_scalar!(
"SELECT EXISTS(SELECT 1 FROM resource_type WHERE workspace_id = 'admins' AND name = $1 AND schema IS NOT DISTINCT FROM $2 AND description IS NOT DISTINCT FROM $3)",
"SELECT EXISTS(SELECT 1 FROM resource_type WHERE workspace_id = 'admins' AND name = $1 AND schema IS NOT DISTINCT FROM $2 AND description IS NOT DISTINCT FROM $3 AND ($5 IS NOT TRUE OR format_extension IS NOT DISTINCT FROM $4))",
&rt.name,
rt.schema.as_ref(),
rt.description.as_deref(),
rt.format_extension.clone().flatten(),
rt.format_extension.is_some(),
)
.fetch_one(&db)
.await?;
@@ -2120,13 +2131,27 @@ async fn sync_cached_resource_types(
}
sqlx::query!(
"INSERT INTO resource_type (workspace_id, name, schema, description, edited_at)
VALUES ('admins', $1, $2, $3, now())
// Whether the payload carried the key at all is what decides: present
// (even as null) is authoritative and may clear, absent means a cache
// written before the column and must leave the stored value alone.
"INSERT INTO resource_type (workspace_id, name, schema, description, format_extension, edited_at)
VALUES ('admins', $1, $2, $3, $4, now())
ON CONFLICT (workspace_id, name) DO UPDATE
SET schema = EXCLUDED.schema, description = EXCLUDED.description, edited_at = now()",
SET schema = EXCLUDED.schema, description = EXCLUDED.description,
-- A fileset is a set of files, so it cannot also be one file.
-- Create and update reject the pair; this writer bypasses both, so
-- it declines the extension rather than persisting the forbidden
-- combination onto a same-named local fileset.
format_extension = CASE
WHEN resource_type.is_fileset THEN NULL
WHEN $5 THEN EXCLUDED.format_extension
ELSE resource_type.format_extension END,
edited_at = now()",
&rt.name,
rt.schema.as_ref(),
rt.description.as_deref(),
rt.format_extension.clone().flatten(),
rt.format_extension.is_some(),
)
.execute(&db)
.await?;
backend/windmill-api-settings/src/log_cleanup.rs
+7 -6
View File
@@ -32,7 +32,7 @@ use windmill_common::tracing_init::{LOGS_SERVICE, TMP_WINDMILL_LOGS_SERVICE};
use windmill_common::worker::WINDMILL_DIR;
use windmill_common::{
DB, INSTANCE_NAME, JOB_RETENTION_SECS, JOB_RETENTION_SECS_OVERRIDES,
JOB_RETENTION_SECS_OVERRIDES_LOADED, SERVICE_LOG_RETENTION_SECS,
JOB_RETENTION_SECS_OVERRIDES_LOADED,
};
use windmill_object_store::object_store_reexports::{
@@ -249,7 +249,7 @@ async fn cleanup_service_logs(
// Count candidates upfront for progress reporting.
let total: i64 = sqlx::query_scalar!(
"SELECT COUNT(*) FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval",
SERVICE_LOG_RETENTION_SECS,
windmill_common::service_log_retention_secs(),
)
.fetch_one(db)
.await?
@@ -274,7 +274,7 @@ async fn cleanup_service_logs(
WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval
LIMIT $2
) RETURNING file_path, hostname",
SERVICE_LOG_RETENTION_SECS,
windmill_common::service_log_retention_secs(),
SERVICE_LOG_BATCH,
)
.fetch_all(db)
@@ -680,9 +680,10 @@ async fn cleanup_s3_orphans(
) -> error::Result<()> {
let job_retention_secs = JOB_RETENTION_SECS.load(std::sync::atomic::Ordering::Relaxed);
let now = Utc::now();
// Service logs always have a retention (hardcoded SERVICE_LOG_RETENTION_SECS),
// so we scan for service-log orphans regardless of JOB_RETENTION_SECS.
let service_cutoff = now - chrono::Duration::seconds(SERVICE_LOG_RETENTION_SECS);
// Service logs always have a retention, so we scan for service-log orphans regardless of
// JOB_RETENTION_SECS.
let service_cutoff =
now - chrono::Duration::seconds(windmill_common::service_log_retention_secs());
// Job-log orphans are only considered once past a job's effective retention window. That window
// is the instance one OR, for an override workspace (EE), its own — and jobs orphan their logs as
backend/windmill-api-users/src/users.rs
+22 -4
View File
@@ -19,7 +19,7 @@ use windmill_api_auth::ApiAuthed;
pub use windmill_api_auth::Tokened;
use argon2::{Argon2, PasswordHash, PasswordVerifier};
use argon2::{Argon2, PasswordVerifier};
use axum::{
extract::{Extension, Path, Query},
response::{IntoResponse, Response},
@@ -2680,10 +2680,8 @@ async fn login(
.await?;
if let Some((email, hash, super_admin)) = email_w_h {
let parsed_hash =
PasswordHash::new(&hash).map_err(|e| Error::internal_err(e.to_string()))?;
if argon2
.verify_password(password.as_bytes(), &parsed_hash)
.verify_password(password.as_bytes(), hash.as_str())
.is_err()
{
audit_log(
@@ -3710,3 +3708,23 @@ async fn request_password_reset(
}
// NOTE: reset_password is in windmill-api (depends on users_oss::hash_password EE dispatch)
#[cfg(test)]
mod tests {
use super::*;
/// Stored hashes outlive the hashing crate: every instance still holds hashes minted by
/// older argon2 releases, and an upgrade that stopped reading them locks their users out.
#[test]
fn verifies_a_hash_minted_by_an_older_argon2() {
// The seeded admin hash from migration 20220508150023, m=4096,t=3,p=1.
let seeded = "$argon2id$v=19$m=4096,t=3,p=1$oLJo/lPn/gezXCuFOEyaNw$i0T2tCkw3xUFsrBIKZwr8jVNHlIfoxQe+HfDnLtd12I";
assert!(Argon2::default()
.verify_password(b"changeme", seeded)
.is_ok());
assert!(Argon2::default()
.verify_password(b"not-the-password", seeded)
.is_err());
}
}
backend/windmill-api-workspaces/src/datatable_migrations.rs
+20 -10
View File
@@ -10,7 +10,10 @@
//! management, and the workspace-merge diff helper. Split out of `workspaces.rs`
//! to keep that file focused on core workspace configuration.
use crate::workspaces::{pg_dump_database, ItemComparison};
use crate::workspaces::{
is_instance_datatable, pg_dump_database, strip_unreplayable_dump_lines, ItemComparison,
PgDumpOptions,
};
use axum::{
extract::{Extension, Path, Query},
@@ -1408,18 +1411,25 @@ async fn generate_initial_datatable_migration(
let pg_db: PgDatabase = serde_json::from_value(db_resource)
.map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))?;
// Snapshot the schema, excluding Windmill's own migration bookkeeping table.
let dump_file = pg_dump_database(&pg_db, true, &["_wm_migrations"]).await?;
// Snapshot the schema without `_wm_migrations`, Windmill's own bookkeeping table, and
// without what a replay elsewhere cannot run: the replaying user owns none of this
// database's objects, and the grants Windmill plants in an instance database (`ALTER
// DEFAULT PRIVILEGES FOR ROLE ...`) fail even replaying onto the same server.
let no_acl = is_instance_datatable(&db, &w_id, &datatable_name).await?;
let dump_file = pg_dump_database(
&pg_db,
PgDumpOptions {
schema_only: true,
exclude_tables: &["_wm_migrations"],
no_owner: true,
no_acl,
},
)
.await?;
let raw_dump = tokio::fs::read_to_string(&dump_file.path)
.await
.map_err(|e| Error::internal_err(format!("Failed to read schema dump: {}", e)))?;
// pg_dump emits psql meta-commands (\restrict / \unrestrict) that aren't
// valid SQL; drop them so the migration body can run via a plain query.
let code_up: String = raw_dump
.lines()
.filter(|line| !line.trim_start().starts_with('\\'))
.collect::<Vec<_>>()
.join("\n");
let code_up = strip_unreplayable_dump_lines(&raw_dump);
// Record the definition first, then mark it installed. If marking fails we
// delete the definition, so a failure leaves no phantom "initial" (rather
backend/windmill-api-workspaces/src/workspaces.rs
+425 -45
View File
@@ -117,6 +117,7 @@ pub fn workspaced_service() -> Router {
get(get_secondary_storage_names),
)
.route("/is_premium", get(is_premium))
.route("/billable_seats", get(get_billable_seats))
.route("/edit_error_handler", post(edit_error_handler))
.route("/edit_success_handler", post(edit_success_handler))
.route(
@@ -686,6 +687,48 @@ async fn is_premium(
Ok(Json(premium))
}
#[derive(Serialize)]
struct BillableSeatsResponse {
/// Both omitted when the seats counted are another workspace's: a fork member need not be a
/// member of the billing root, so the root's headcount is not theirs to read. The total is,
/// since it is the divisor of the quota their own executions draw on.
#[serde(skip_serializing_if = "Option::is_none")]
developers: Option<i64>,
#[serde(skip_serializing_if = "Option::is_none")]
operators: Option<i64>,
seats: i64,
}
async fn get_billable_seats(
_authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
) -> JsonResult<BillableSeatsResponse> {
// Readable by any workspace member, like `is_premium`: this is what the sidebar usage meter
// divides by, and that meter is shown to non-admin developers too.
//
// On cloud a fork draws its plan, quota and bill from the root, so the seats its usage is
// measured against are the root's. Resolved here rather than by the caller: a fork member need
// not be a member of that root, and so cannot count its seats from the member list. Off cloud
// a fork is not billed through a root at all, so the workspace answers for itself.
#[cfg(feature = "cloud")]
let billing_w_id = if *CLOUD_HOSTED {
windmill_common::workspaces::get_billing_workspace_id(&db, &w_id).await?
} else {
w_id.clone()
};
#[cfg(not(feature = "cloud"))]
let billing_w_id = w_id.clone();
let counted = windmill_common::workspaces::billable_seats(&db, &billing_w_id).await?;
let own = billing_w_id == w_id;
Ok(Json(BillableSeatsResponse {
developers: own.then_some(counted.developers),
operators: own.then_some(counted.operators),
seats: counted.seats,
}))
}
async fn exists_workspace(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
@@ -1970,6 +2013,23 @@ async fn edit_large_file_storage_config(
)));
}
if !windmill_common::workspaces::filesystem_storage_allowed() {
let named = std::iter::once(("primary storage", &lfs_config.large_file_storage)).chain(
lfs_config
.secondary_storage
.iter()
.map(|(name, storage)| (name.as_str(), storage)),
);
for (name, storage) in named {
if matches!(storage, LargeFileStorage::FilesystemStorage(_)) {
return Err(Error::BadRequest(format!(
"{name}: {}",
windmill_common::workspaces::FILESYSTEM_STORAGE_DEV_ONLY_MSG
)));
}
}
}
let serialized_lfs_config =
serde_json::to_value::<LargeFileStorageWithSecondary>(lfs_config)
.map_err(|err| Error::internal_err(err.to_string()))?;
@@ -2594,6 +2654,66 @@ fn truncate_column_default(default: String) -> String {
mod tests {
use super::*;
/// The header of a pg_dump, followed by an object whose body also holds a `SET`.
const DUMP: &str = "--\n\
-- PostgreSQL database dump\n\
--\n\
\n\
\\restrict aBcD\n\
\n\
SET statement_timeout = 0;\n\
SET transaction_timeout = 0;\n\
SET client_encoding = 'UTF8';\n\
SELECT pg_catalog.set_config('search_path', '', false);\n\
\n\
SET default_table_access_method = heap;\n\
\n\
CREATE FUNCTION public.f() RETURNS void LANGUAGE plpgsql AS $$\n\
BEGIN\n\
SET transaction_timeout = 0;\n\
END;\n\
$$;\n";
#[test]
fn replayable_dump_keeps_everything_but_meta_commands_and_session_timeouts() {
let replayable = strip_unreplayable_dump_lines(DUMP);
assert!(!replayable.contains("\\restrict"));
assert!(!replayable.contains("SET statement_timeout"));
assert!(!replayable.contains("SET transaction_timeout = 0;\nSET client_encoding"));
assert!(replayable.contains("SET client_encoding = 'UTF8';"));
assert!(replayable.contains("SET default_table_access_method = heap;"));
// Past the preamble the dump is an object's own text: left exactly as it is.
assert!(replayable.contains("BEGIN\nSET transaction_timeout = 0;\nEND;"));
}
#[tokio::test]
async fn dump_preamble_only_drops_settings_the_server_lacks() {
let dump_file = DumpFile::new().unwrap();
tokio::fs::write(&dump_file.path, DUMP).await.unwrap();
let supported = [
"statement_timeout",
"client_encoding",
"default_table_access_method",
]
.map(String::from)
.into_iter()
.collect();
comment_out_unsupported_settings(&dump_file, &supported)
.await
.unwrap();
let patched = tokio::fs::read_to_string(&dump_file.path).await.unwrap();
// Rewriting the header must not shift the rest of the dump.
assert_eq!(patched.len(), DUMP.len());
assert!(patched.contains("-- transaction_timeout = 0;"));
assert!(patched.contains("SET statement_timeout = 0;"));
assert!(patched.contains("SET default_table_access_method = heap;"));
// The `SET` inside the function body is past the preamble: never touched.
assert!(patched.contains("BEGIN\nSET transaction_timeout = 0;\nEND;"));
}
#[test]
fn compact_column_type_truncates_multibyte_defaults_safely() {
let default = "é".repeat(31);
@@ -2696,6 +2816,35 @@ pub(crate) async fn resolve_pg_source_checked(
.map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))
}
/// Whether the data table `name` is backed by the Windmill instance's own PostgreSQL
/// rather than a user resource.
pub(crate) async fn is_instance_datatable(db: &DB, w_id: &str, name: &str) -> Result<bool> {
let config = sqlx::query_scalar!(
"SELECT datatable->'datatables'->$2 FROM workspace_settings WHERE workspace_id = $1",
w_id,
name
)
.fetch_optional(db)
.await?
.flatten();
Ok(config
.and_then(|v| {
v.get("database")
.and_then(|d| d.get("resource_type"))
.and_then(|r| r.as_str())
.map(|s| s == "instance")
})
.unwrap_or(false))
}
/// Same, for the `datatable://<name>` / `$res:<path>` form the import endpoints take.
async fn is_instance_datatable_source(db: &DB, w_id: &str, source: &str) -> Result<bool> {
match source.strip_prefix("datatable://") {
Some(name) => is_instance_datatable(db, w_id, name).await,
None => Ok(false),
}
}
/// A temporary file for pg_dump output that is automatically deleted when dropped.
pub(crate) struct DumpFile {
pub(crate) path: std::path::PathBuf,
@@ -2743,12 +2892,21 @@ impl Drop for DumpFile {
}
}
#[derive(Default)]
pub(crate) struct PgDumpOptions<'a> {
pub(crate) schema_only: bool,
pub(crate) exclude_tables: &'a [&'a str],
/// Leave out `ALTER ... OWNER TO`.
pub(crate) no_owner: bool,
/// Leave out `GRANT`, `REVOKE` and `ALTER DEFAULT PRIVILEGES`.
pub(crate) no_acl: bool,
}
/// Run pg_dump against a PgDatabase, writing output to a temp file on disk.
/// Returns a DumpFile handle; the file is deleted when the handle is dropped.
pub(crate) async fn pg_dump_database(
pg_db: &PgDatabase,
schema_only: bool,
exclude_tables: &[&str],
opts: PgDumpOptions<'_>,
) -> Result<DumpFile> {
let dump_file = DumpFile::new()?;
@@ -2759,10 +2917,16 @@ pub(crate) async fn pg_dump_database(
let mut cmd = tokio::process::Command::new("pg_dump");
cmd.arg("--format=plain").arg("--file").arg(&dump_file.path);
if schema_only {
if opts.schema_only {
cmd.arg("--schema-only");
}
for table in exclude_tables {
if opts.no_owner {
cmd.arg("--no-owner");
}
if opts.no_acl {
cmd.arg("--no-privileges");
}
for table in opts.exclude_tables {
cmd.arg(format!("--exclude-table={table}"));
}
cmd.arg("--host")
@@ -2794,37 +2958,179 @@ pub(crate) async fn pg_dump_database(
Ok(dump_file)
}
/// Import a pg_dump file into a target database using psql.
async fn pg_import_dump(target_db: &PgDatabase, dump_file: &DumpFile) -> Result<()> {
let host = &target_db.host;
let port = target_db.port.unwrap_or(5432).to_string();
let user = target_db.login_name();
let dbname = &target_db.dbname;
/// Whether `line` still belongs to the preamble pg_dump emits before the first
/// dumped object: comments, blank lines, psql meta-commands and the session `SET`s.
fn is_dump_preamble_line(line: &[u8]) -> bool {
let line = line.trim_ascii_start();
line.is_empty()
|| line.starts_with(b"--")
|| line.starts_with(b"\\")
|| line.starts_with(b"SET ")
|| line.starts_with(b"SELECT pg_catalog.set_config(")
}
/// The GUCs pg_dump's preamble sets only to keep the dumping session out of the way.
/// They are also the ones that come and go across versions (`transaction_timeout` is
/// PG 17+), so they are what a dump replayed on an older server trips over first.
const DUMP_SESSION_TIMEOUTS: [&str; 4] = [
"statement_timeout",
"lock_timeout",
"idle_in_transaction_session_timeout",
"transaction_timeout",
];
/// Turn a dump into SQL that can be replayed on another database: drop pg_dump's psql
/// meta-commands (`\restrict` / `\unrestrict`, not valid SQL) and the session timeouts
/// its preamble sets, which the replaying server may not have as GUCs at all. Only the
/// preamble is filtered, so an object's body keeps whatever it holds.
pub(crate) fn strip_unreplayable_dump_lines(dump: &str) -> String {
let mut in_preamble = true;
dump.lines()
.filter(|line| {
in_preamble = in_preamble && is_dump_preamble_line(line.as_bytes());
if line.trim_start().starts_with('\\') {
return false;
}
!(in_preamble
&& preamble_setting_name(line.as_bytes())
.is_some_and(|name| DUMP_SESSION_TIMEOUTS.contains(&name)))
})
.collect::<Vec<_>>()
.join("\n")
}
/// The GUC a preamble `SET <name> = ...;` line assigns, if the line is one.
fn preamble_setting_name(line: &[u8]) -> Option<&str> {
let name = line.strip_prefix(b"SET ")?.split(|c| *c == b' ').next()?;
std::str::from_utf8(name).ok()
}
/// The preamble Windmill's postgres client writes can set GUCs an older server does not
/// have — harmless session tuning, but one failing statement aborts a restore that stops
/// on the first error. Comment those out in place, three bytes each, so the data
/// section's offsets stay put.
async fn comment_out_unsupported_settings(
dump_file: &DumpFile,
supported_settings: &HashSet<String>,
) -> Result<()> {
use tokio::io::{AsyncBufReadExt, AsyncWriteExt};
let file = tokio::fs::File::open(&dump_file.path)
.await
.map_err(|e| Error::internal_err(format!("Failed to open dump file: {}", e)))?;
let mut reader = tokio::io::BufReader::new(file);
let mut preamble: Vec<u8> = Vec::new();
let mut patched = false;
loop {
let start = preamble.len();
let read = reader
.read_until(b'\n', &mut preamble)
.await
.map_err(|e| Error::internal_err(format!("Failed to read dump file: {}", e)))?;
if read == 0 {
break;
}
let line = &preamble[start..];
if !is_dump_preamble_line(line) {
preamble.truncate(start);
break;
}
if preamble_setting_name(line).is_some_and(|name| !supported_settings.contains(name)) {
preamble[start..start + 3].copy_from_slice(b"-- ");
patched = true;
}
}
if !patched {
return Ok(());
}
let mut file = tokio::fs::OpenOptions::new()
.write(true)
.open(&dump_file.path)
.await
.map_err(|e| Error::internal_err(format!("Failed to open dump file: {}", e)))?;
file.write_all(&preamble)
.await
.map_err(|e| Error::internal_err(format!("Failed to rewrite dump preamble: {}", e)))?;
file.flush()
.await
.map_err(|e| Error::internal_err(format!("Failed to rewrite dump preamble: {}", e)))?;
Ok(())
}
/// A psql invocation against `pg_db`, carrying the connection settings the CLI reads
/// from the environment.
fn psql_command(pg_db: &PgDatabase) -> tokio::process::Command {
let mut cmd = tokio::process::Command::new("psql");
cmd.arg("--host")
.arg(host)
.arg(&pg_db.host)
.arg("--port")
.arg(&port)
.arg(pg_db.port.unwrap_or(5432).to_string())
.arg("--username")
.arg(user)
.arg(pg_db.login_name())
.arg("--dbname")
.arg(dbname)
.arg(&pg_db.dbname)
.arg("--no-psqlrc")
.arg("--file")
.arg(&dump_file.path)
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped());
if let Some(ref password) = target_db.password {
if let Some(ref password) = pg_db.password {
cmd.env("PGPASSWORD", password);
}
if let Some(ref sslmode) = target_db.sslmode {
if let Some(ref sslmode) = pg_db.sslmode {
cmd.env("PGSSLMODE", sslmode);
}
cmd
}
let output = cmd
/// GUC names the server backing `pg_db` knows about.
///
/// Asked through psql rather than a tokio-postgres connection so the lookup reaches
/// exactly the servers the restore itself can: libpq negotiates TLS for `sslmode=prefer`
/// and an unset mode, where `PgDatabase::connect` would hand a TLS-only server a
/// plaintext socket and fail before the import ever starts.
async fn server_setting_names(pg_db: &PgDatabase) -> Result<HashSet<String>> {
let output = psql_command(pg_db)
.arg("--tuples-only")
.arg("--no-align")
.arg("--command")
.arg("SELECT name FROM pg_settings")
.output()
.await
.map_err(|e| Error::internal_err(format!("Failed to execute psql: {}", e)))?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(Error::internal_err(format!(
"Failed to list the settings of the target server: {}",
stderr
)));
}
Ok(String::from_utf8_lossy(&output.stdout)
.lines()
.map(|name| name.trim().to_string())
.filter(|name| !name.is_empty())
.collect())
}
/// Import a pg_dump file into a target database using psql.
///
/// Left to its defaults psql reports a failed statement, carries on and still exits 0,
/// so a dump that breaks partway through imports partially and reads as a success.
/// ON_ERROR_STOP surfaces the failure and --single-transaction makes the restore
/// all-or-nothing, leaving the target as it was and the import retryable.
async fn pg_import_dump(target_db: &PgDatabase, dump_file: &DumpFile) -> Result<()> {
let supported_settings = server_setting_names(target_db).await?;
comment_out_unsupported_settings(dump_file, &supported_settings).await?;
let output = psql_command(target_db)
.arg("--set")
.arg("ON_ERROR_STOP=1")
.arg("--single-transaction")
.arg("--file")
.arg(&dump_file.path)
.output()
.await
.map_err(|e| Error::internal_err(format!("Failed to execute psql: {}", e)))?;
@@ -2869,29 +3175,7 @@ async fn create_pg_database(
}
}
// Determine if this is an instance or resource-backed datatable
let is_instance_datatable = if let Some(dt_name) = req.source.strip_prefix("datatable://") {
let config = sqlx::query_scalar!(
"SELECT datatable->'datatables'->$2 FROM workspace_settings WHERE workspace_id = $1",
&w_id,
dt_name
)
.fetch_optional(&db)
.await?
.flatten();
config
.and_then(|v| {
v.get("database")
.and_then(|d| d.get("resource_type"))
.and_then(|r| r.as_str())
.map(|s| s == "instance")
})
.unwrap_or(false)
} else {
false
};
if is_instance_datatable {
if is_instance_datatable_source(&db, &w_id, &req.source).await? {
windmill_common::create_custom_instance_database(&db, &req.target_dbname, "datatable")
.await?;
} else {
@@ -2989,7 +3273,18 @@ async fn import_pg_database(
}
windmill_common::validate_dbname(&target_pg.dbname)?;
let dump_file = pg_dump_database(&source_pg, schema_only, &[]).await?;
// Ownership never replays: the restore runs as the target's own connection user, and
// what it creates it owns. Grants do, except around an instance data table — Windmill
// plants `custom_instance_user` grants in one, which nothing else can replay. Elsewhere
// the ACLs are user intent (`REVOKE ... FROM PUBLIC`) and dropping them widens access.
let no_acl = is_instance_datatable_source(&db, &w_id, &req.target).await?
|| is_instance_datatable_source(&db, &w_id, &req.source).await?;
let dump_file = pg_dump_database(
&source_pg,
PgDumpOptions { schema_only, no_owner: true, no_acl, ..Default::default() },
)
.await?;
pg_import_dump(&target_pg, &dump_file).await?;
Ok(format!(
@@ -3011,7 +3306,11 @@ async fn export_pg_schema(
Json(req): Json<ExportPgSchemaRequest>,
) -> Result<String> {
let pg = resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
let dump_file = pg_dump_database(&pg, true, &[]).await?;
let dump_file = pg_dump_database(
&pg,
PgDumpOptions { schema_only: true, ..Default::default() },
)
.await?;
tokio::fs::read_to_string(&dump_file.path)
.await
.map_err(|e| Error::internal_err(format!("Failed to read dump file: {}", e)))
@@ -7118,6 +7417,76 @@ async fn enforce_cloud_fork_cap(db: &DB, parent_workspace_id: &str) -> Result<()
enforce_cloud_fork_count(db, &root, 1).await
}
/// Cloud: refuse to attach a workspace that already has a paid plan of its own.
///
/// Once attached it draws the root's plan and meters its usage there, so a subscription of its own
/// bills a second time for one plan. Only an attach can reach this state: a fork is created as a
/// fresh workspace and never had a plan to keep.
///
/// Asked only of a candidate joining this family, never of one already under the same root: that
/// one is already in the double-billed state, where the settings page surfaces the leftover
/// subscription and the portal that cancels it, and refusing there would block re-designating a
/// renamed dev workspace over a billing problem the attach did not cause.
#[cfg(feature = "cloud")]
async fn reject_attach_of_subscribed_workspace(db: &DB, dev_w_id: &str) -> Result<()> {
let plan = sqlx::query_scalar!(
"SELECT plan FROM workspace_settings WHERE workspace_id = $1",
dev_w_id
)
.fetch_optional(db)
.await?
.flatten();
// Any plan, not just `'team'`: the column is written by the subscription webhook, and a plan
// value it does not write yet would otherwise walk straight past this. An enterprise
// arrangement is deliberately not covered — it sets `premium` without a plan and has no
// self-serve portal, so refusing there would be a dead end rather than something to act on.
if plan.is_some() {
return Err(Error::BadRequest(format!(
"Workspace {dev_w_id} is on a paid plan of its own. A dev or fork workspace runs on its parent's plan and is never invoiced separately, so cancel that subscription from its own billing settings before attaching it."
)));
}
Ok(())
}
#[cfg(all(test, feature = "cloud"))]
mod attach_billing_guard_tests {
use super::reject_attach_of_subscribed_workspace;
use sqlx::{Pool, Postgres};
async fn workspace_on_plan(db: &Pool<Postgres>, id: &str, plan: Option<&str>) {
sqlx::query("INSERT INTO workspace (id, name, owner) VALUES ($1, $1, 'test-user')")
.bind(id)
.execute(db)
.await
.expect("insert workspace");
sqlx::query("INSERT INTO workspace_settings (workspace_id, plan) VALUES ($1, $2)")
.bind(id)
.bind(plan)
.execute(db)
.await
.expect("insert workspace_settings");
}
#[sqlx::test(migrations = "../migrations")]
async fn refuses_a_candidate_that_still_pays_for_itself(db: Pool<Postgres>) {
workspace_on_plan(&db, "subscribed", Some("team")).await;
workspace_on_plan(&db, "cancelled", None).await;
let err = reject_attach_of_subscribed_workspace(&db, "subscribed")
.await
.expect_err("a workspace on a paid plan of its own must not be attachable");
assert!(err.to_string().contains("paid plan of its own"), "{err}");
// Cancelling clears `plan` but keeps `customer_id`, so the plan column is what decides.
reject_attach_of_subscribed_workspace(&db, "cancelled")
.await
.expect("a workspace with no plan is attachable");
reject_attach_of_subscribed_workspace(&db, "no-settings-row")
.await
.expect("a workspace with no settings row is attachable");
}
}
/// General guardrail (all builds): reject creating a fork/dev under `parent` when it would nest deeper
/// than `MAX_FORK_DEPTH`. `added_subtree_height` is the height of the subtree grafted below the new
/// node — 0 for a plain fork, or the candidate's own subtree height for an attach.
@@ -7660,6 +8029,17 @@ async fn attach_dev_workspace(
)));
}
// Deliberately below the admin-of-candidate check, unlike the cap enforcement above: the
// refusal names the candidate's plan, so running it earlier would tell any admin of any
// premium workspace whether an arbitrary workspace id is on a team plan.
#[cfg(feature = "cloud")]
if *CLOUD_HOSTED {
let root = windmill_common::workspaces::get_billing_workspace_id(&db, &prod_w_id).await?;
if windmill_common::workspaces::get_billing_workspace_id(&db, &dev_w_id).await? != root {
reject_attach_of_subscribed_workspace(&db, &dev_w_id).await?;
}
}
let mut tx = db.begin().await?;
// Everything above ran outside a transaction, so prod's eligibility and the chain's labels could
// have changed under us: re-decide both here, under the pairing lock.
backend/windmill-api/Cargo.toml
+5 -3
View File
@@ -10,7 +10,7 @@ path = "src/lib.rs"
[features]
default = []
private = ["windmill-audit/private", "windmill-common/private", "windmill-api-auth/private", "windmill-store/private", "windmill-api-users/private", "windmill-api-workspaces/private", "windmill-api-groups/private", "windmill-api-configs/private", "windmill-api-settings/private", "windmill-api-assets/private", "windmill-api-agent-workers?/private", "windmill-trigger-kafka?/private", "windmill-trigger-postgres?/private", "windmill-trigger-mqtt?/private", "windmill-trigger-amqp?/private", "windmill-trigger-websocket?/private", "windmill-trigger-nats?/private", "windmill-trigger-sqs?/private", "windmill-trigger-gcp?/private", "windmill-trigger-azure?/private", "windmill-trigger-email?/private", "windmill-git-sync/private", "windmill-autoscaling?/private", "windmill-object-store/private", "windmill-api-npm-proxy/private"]
private = ["windmill-audit/private", "windmill-common/private", "windmill-api-auth/private", "windmill-store/private", "windmill-api-users/private", "windmill-api-workspaces/private", "windmill-api-groups/private", "windmill-api-configs/private", "windmill-api-settings/private", "windmill-api-assets/private", "windmill-api-agent-workers?/private", "windmill-trigger-kafka?/private", "windmill-trigger-postgres?/private", "windmill-trigger-mqtt?/private", "windmill-trigger-amqp?/private", "windmill-trigger-websocket?/private", "windmill-trigger-nats?/private", "windmill-trigger-sqs?/private", "windmill-trigger-gcp?/private", "windmill-trigger-azure?/private", "windmill-trigger-email?/private", "windmill-git-sync/private", "windmill-autoscaling?/private", "windmill-object-store/private", "windmill-api-npm-proxy/private", "windmill-indexer?/private"]
enterprise = ["windmill-queue/enterprise", "windmill-audit/enterprise", "windmill-git-sync/enterprise", "windmill-common/enterprise", "windmill-worker?/enterprise", "windmill-api-auth/enterprise", "windmill-store/enterprise", "windmill-api-jobs/enterprise", "windmill-api-scripts/enterprise", "windmill-api-flows/enterprise", "windmill-api-users/enterprise", "windmill-api-workspaces/enterprise", "windmill-api-groups/enterprise", "windmill-api-configs/enterprise", "windmill-api-settings/enterprise", "windmill-api-schedule/enterprise", "windmill-api-debug/enterprise", "windmill-api-agent-workers?/enterprise", "windmill-trigger/enterprise", "windmill-trigger-kafka?/enterprise", "windmill-trigger-postgres?/enterprise", "windmill-trigger-mqtt?/enterprise", "windmill-trigger-amqp?/enterprise", "windmill-trigger-websocket?/enterprise", "windmill-trigger-email?/enterprise", "windmill-trigger-nats?/enterprise", "windmill-trigger-sqs?/enterprise", "windmill-trigger-gcp?/enterprise", "windmill-trigger-azure?/enterprise", "windmill-trigger-http?/enterprise", "windmill-native-triggers?/enterprise", "dep:windmill-autoscaling", "windmill-autoscaling/enterprise", "windmill-api-npm-proxy/enterprise", "license"]
stripe = []
run_inline = ["dep:windmill-worker", "windmill-api-configs/run_inline"]
@@ -18,10 +18,12 @@ agent_worker_server = ["dep:windmill-worker", "dep:windmill-api-agent-workers"]
enterprise_saml = ["dep:samael", "dep:libxml"]
benchmark = []
embedding = ["windmill-api-embeddings/embedding"]
parquet = ["dep:datafusion", "windmill-common/parquet", "windmill-object-store/parquet", "windmill-worker?/parquet", "windmill-api-users/parquet", "windmill-api-settings/parquet", "windmill-api-workspaces/parquet", "windmill-api-npm-proxy/parquet", "dep:aws-sigv4", "dep:aws-sdk-config", "dep:quick-xml"]
parquet = ["dep:datafusion", "windmill-common/parquet", "windmill-object-store/parquet", "windmill-worker?/parquet", "windmill-indexer?/parquet", "windmill-api-users/parquet", "windmill-api-settings/parquet", "windmill-api-workspaces/parquet", "windmill-api-npm-proxy/parquet", "dep:aws-sigv4", "dep:aws-sdk-config", "dep:quick-xml"]
prometheus = ["windmill-common/prometheus", "windmill-queue/prometheus", "dep:prometheus", "windmill-worker?/prometheus", "windmill-api-scripts/prometheus"]
openidconnect = ["dep:openidconnect", "windmill-common/openidconnect", "windmill-store/openidconnect"]
tantivy = ["dep:windmill-indexer"]
# The service log search handler reads the columnar store, so tantivy alone is
# not enough for this crate to build on its own.
tantivy = ["dep:windmill-indexer", "parquet"]
kafka = ["dep:windmill-trigger-kafka", "windmill-store/kafka"]
kafka-gssapi = ["kafka", "windmill-trigger-kafka/kafka-gssapi"]
nats = ["dep:windmill-trigger-nats", "windmill-store/nats"]
backend/windmill-api/openapi-deref.json
-196
View File
@@ -16532,202 +16532,6 @@
}
}
},
"/w/{workspace}/ai_skills/list": {
"get": {
"summary": "list the workspace AI chat skills (name + description only)",
"operationId": "listAiSkills",
"tags": [
"workspace"
],
"parameters": [
{
"$ref": "#/components/parameters/WorkspaceId"
}
],
"responses": {
"200": {
"description": "skill listing",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"description"
],
"properties": {
"name": {
"type": "string"
},
"description": {
"type": "string"
}
}
}
}
}
}
}
}
}
},
"/w/{workspace}/ai_skills/get/{name}": {
"get": {
"summary": "get a workspace AI chat skill including its instructions",
"operationId": "getAiSkill",
"tags": [
"workspace"
],
"parameters": [
{
"$ref": "#/components/parameters/WorkspaceId"
},
{
"name": "name",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "skill",
"content": {
"application/json": {
"schema": {
"type": "object",
"required": [
"name",
"description",
"instructions"
],
"properties": {
"name": {
"type": "string"
},
"description": {
"type": "string"
},
"instructions": {
"type": "string"
}
}
}
}
}
}
}
}
},
"/w/{workspace}/ai_skills/upload": {
"post": {
"summary": "upsert workspace AI chat skills (admin only)",
"operationId": "uploadAiSkills",
"tags": [
"workspace"
],
"parameters": [
{
"$ref": "#/components/parameters/WorkspaceId"
}
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"type": "object",
"required": [
"skills"
],
"properties": {
"skills": {
"type": "array",
"maxItems": 50,
"items": {
"type": "object",
"required": [
"name",
"description",
"instructions"
],
"properties": {
"name": {
"type": "string",
"minLength": 1,
"maxLength": 64,
"pattern": "^[a-z0-9-]+$"
},
"description": {
"type": "string",
"minLength": 1,
"maxLength": 1024
},
"instructions": {
"type": "string",
"minLength": 1,
"maxLength": 65536
}
}
}
}
}
}
}
}
},
"responses": {
"200": {
"description": "uploaded",
"content": {
"text/plain": {
"schema": {
"type": "string"
}
}
}
}
}
}
},
"/w/{workspace}/ai_skills/delete/{name}": {
"delete": {
"summary": "delete a workspace AI chat skill (admin only)",
"operationId": "deleteAiSkill",
"tags": [
"workspace"
],
"parameters": [
{
"$ref": "#/components/parameters/WorkspaceId"
},
{
"name": "name",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"responses": {
"200": {
"description": "deleted",
"content": {
"text/plain": {
"schema": {
"type": "string"
}
}
}
}
}
}
},
"/w/{workspace}/apps/get_data/v/{secretWithExtension}": {
"get": {
"summary": "get raw app data by",
backend/windmill-api/openapi-deref.yaml
-135
View File
@@ -17063,141 +17063,6 @@ paths:
text/plain:
schema:
type: string
/w/{workspace}/ai_skills/list:
get:
summary: list the workspace AI chat skills (name + description only)
operationId: listAiSkills
tags:
- workspace
parameters:
- name: workspace
in: path
required: true
schema: *ref_4
responses:
'200':
description: skill listing
content:
application/json:
schema:
type: array
items:
type: object
required:
- name
- description
properties:
name:
type: string
description:
type: string
/w/{workspace}/ai_skills/get/{name}:
get:
summary: get a workspace AI chat skill including its instructions
operationId: getAiSkill
tags:
- workspace
parameters:
- name: workspace
in: path
required: true
schema: *ref_4
- name: name
in: path
required: true
schema:
type: string
responses:
'200':
description: skill
content:
application/json:
schema:
type: object
required:
- name
- description
- instructions
properties:
name:
type: string
description:
type: string
instructions:
type: string
/w/{workspace}/ai_skills/upload:
post:
summary: upsert workspace AI chat skills (admin only)
operationId: uploadAiSkills
tags:
- workspace
parameters:
- name: workspace
in: path
required: true
schema: *ref_4
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- skills
properties:
skills:
type: array
maxItems: 50
items:
type: object
required:
- name
- description
- instructions
properties:
name:
type: string
minLength: 1
maxLength: 64
pattern: ^[a-z0-9-]+$
description:
type: string
minLength: 1
maxLength: 1024
instructions:
type: string
minLength: 1
maxLength: 65536
responses:
'200':
description: uploaded
content:
text/plain:
schema:
type: string
/w/{workspace}/ai_skills/delete/{name}:
delete:
summary: delete a workspace AI chat skill (admin only)
operationId: deleteAiSkill
tags:
- workspace
parameters:
- name: workspace
in: path
required: true
schema: *ref_4
- name: name
in: path
required: true
schema:
type: string
responses:
'200':
description: deleted
content:
text/plain:
schema:
type: string
/w/{workspace}/apps/get_data/v/{secretWithExtension}:
get:
summary: get raw app data by
backend/windmill-api/openapi.yaml
+223 -155
View File
@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.796.0
version: 1.800.1
title: Windmill API
contact:
@@ -3788,6 +3788,38 @@ paths:
schema:
type: boolean
/w/{workspace}/workspaces/billable_seats:
get:
summary: get the billable seats of the workspace the plan is billed on
operationId: getBillableSeats
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
responses:
"200":
description: billable seats
content:
application/json:
schema:
type: object
properties:
developers:
type: integer
description: >-
Omitted when the seats counted are another workspace's, as they are for a
fork resolving to its billing root.
operators:
type: integer
description: >-
Omitted when the seats counted are another workspace's, as they are for a
fork resolving to its billing root.
seats:
type: integer
required:
- seats
/w/{workspace}/workspaces/premium_info:
get:
summary: get premium info
@@ -12502,133 +12534,6 @@ paths:
type: boolean
description: more buckets matched than were returned, so summing them under-reports
/w/{workspace}/ai_skills/list:
get:
summary: list the workspace AI chat skills (name + description only)
operationId: listAiSkills
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
responses:
"200":
description: skill listing
content:
application/json:
schema:
type: array
items:
type: object
required:
- name
- description
properties:
name:
type: string
description:
type: string
/w/{workspace}/ai_skills/get/{name}:
get:
summary: get a workspace AI chat skill including its instructions
operationId: getAiSkill
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: name
in: path
required: true
schema:
type: string
responses:
"200":
description: skill
content:
application/json:
schema:
type: object
required:
- name
- description
- instructions
properties:
name:
type: string
description:
type: string
instructions:
type: string
/w/{workspace}/ai_skills/upload:
post:
summary: upsert workspace AI chat skills (admin only)
operationId: uploadAiSkills
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- skills
properties:
skills:
type: array
maxItems: 50
items:
type: object
required:
- name
- description
- instructions
properties:
name:
type: string
minLength: 1
maxLength: 64
pattern: "^[a-z0-9-]+$"
description:
type: string
minLength: 1
maxLength: 1024
instructions:
type: string
minLength: 1
maxLength: 65536
responses:
"200":
description: uploaded
content:
text/plain:
schema:
type: string
/w/{workspace}/ai_skills/delete/{name}:
delete:
summary: delete a workspace AI chat skill (admin only)
operationId: deleteAiSkill
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: name
in: path
required: true
schema:
type: string
responses:
"200":
description: deleted
content:
text/plain:
schema:
type: string
/w/{workspace}/apps/get_data/v/{secretWithExtension}:
get:
summary: get raw app data by
@@ -23858,7 +23763,7 @@ paths:
items:
type: string
hits:
description: log files that matched the query
description: the log lines that matched the query, newest first
type: array
items:
$ref: "#/components/schemas/LogSearchHit"
@@ -25287,6 +25192,68 @@ paths:
schema:
type: string
/w/{workspace}/hub/projects/{slug}/withdraw:
post:
summary: take a hub project submission back out of review
description: |
Requires the caller to be a workspace admin. Forwards the request to the
configured Hub scoped to the `{workspace}:{folder}` source and returns
the Hub's status code and raw response body. Everything pushed for the
submission is kept, so it can be fixed and submitted again.
operationId: withdrawHubProject
tags:
- hubPublish
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: slug
in: path
required: true
description: hub project slug (3-50 chars, lowercase alphanumeric and hyphens, no leading/trailing hyphen)
schema:
type: string
minLength: 3
maxLength: 50
pattern: "^[a-z0-9][a-z0-9-]*[a-z0-9]$"
- $ref: "#/components/parameters/HubPublishFolder"
responses:
"200":
description: raw Hub response body (status code is passed through from the Hub)
content:
text/plain:
schema:
type: string
/w/{workspace}/hub/projects/{slug}/discard_update:
post:
summary: discard the pending update to a published hub project
description: |
Requires the caller to be a workspace admin. Forwards the request to the
configured Hub scoped to the `{workspace}:{folder}` source and returns
the Hub's status code and raw response body. The published project is
left untouched.
operationId: discardHubProjectUpdate
tags:
- hubPublish
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: slug
in: path
required: true
description: hub project slug (3-50 chars, lowercase alphanumeric and hyphens, no leading/trailing hyphen)
schema:
type: string
minLength: 3
maxLength: 50
pattern: "^[a-z0-9][a-z0-9-]*[a-z0-9]$"
- $ref: "#/components/parameters/HubPublishFolder"
responses:
"200":
description: raw Hub response body (status code is passed through from the Hub)
content:
text/plain:
schema:
type: string
/w/{workspace}/hub/project:
get:
summary: get the hub project linked to a workspace folder
@@ -27162,11 +27129,29 @@ components:
type: integer
minimum: 1
maximum: 2000000
free_tier:
$ref: "#/components/schemas/FreeTierInfo"
model_pricing:
type: object
additionalProperties:
$ref: "#/components/schemas/ModelPriceOverride"
FreeTierInfo:
type: object
description: >-
Read-only. Present when the workspace has no AI provider of its own and is running
on Windmill's free tier. Ignored on write.
properties:
exhausted:
type: boolean
description: The one-time grant is spent; no provider is served and the user must add their own API key.
used_ratio:
type: number
description: Fraction of the grant consumed, 0 to 1.
required:
- exhausted
- used_ratio
ModelPriceOverride:
type: object
description: negotiated rates in USD per million tokens, keyed `provider:model`
@@ -27928,7 +27913,10 @@ components:
type: string
args:
$ref: "#/components/schemas/ScriptArgs"
result: {}
result:
description: |
For large results, this may be the placeholder string 'WINDMILL_TOO_BIG'.
Use the completed job result endpoint to retrieve the full result.
logs:
type: string
deleted:
@@ -29497,6 +29485,12 @@ components:
type: string
is_fileset:
type: boolean
format_extension:
type: string
nullable: true
description: >-
File extension for a type whose value is one file rather than a set
of fields. Omit to leave it unchanged; send null to clear it.
TriggerHistoryEntry:
type: object
@@ -30202,7 +30196,9 @@ components:
description: If true, passes the request body as a raw string instead of parsing as JSON
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -30291,7 +30287,9 @@ components:
description: If true, passes the request body as a raw string instead of parsing as JSON
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -30387,7 +30385,9 @@ components:
description: If true, passes the request body as a raw string instead of parsing as JSON
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -30579,7 +30579,9 @@ components:
description: Optional periodic heartbeat message configuration
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
description: Arguments to pass to the error handler
$ref: "#/components/schemas/ScriptArgs"
@@ -30644,7 +30646,9 @@ components:
description: Optional periodic heartbeat message configuration
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
description: Arguments to pass to the error handler
$ref: "#/components/schemas/ScriptArgs"
@@ -30720,7 +30724,9 @@ components:
description: Optional periodic heartbeat message configuration
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
description: Arguments to pass to the error handler
$ref: "#/components/schemas/ScriptArgs"
@@ -30848,7 +30854,9 @@ components:
description: Last error message if the trigger failed
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -30899,7 +30907,9 @@ components:
$ref: "#/components/schemas/TriggerMode"
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -30963,7 +30973,9 @@ components:
$ref: "#/components/schemas/TriggerMode"
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -31046,7 +31058,9 @@ components:
description: Last error message if the trigger failed
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -31087,7 +31101,9 @@ components:
$ref: "#/components/schemas/TriggerMode"
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -31141,7 +31157,9 @@ components:
$ref: "#/components/schemas/TriggerMode"
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -31539,7 +31557,9 @@ components:
description: Last error message if the trigger failed
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -31641,7 +31661,9 @@ components:
$ref: "#/components/schemas/TriggerMode"
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -31697,7 +31719,9 @@ components:
$ref: "#/components/schemas/TriggerMode"
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -31825,7 +31849,9 @@ components:
description: Timestamp of last server heartbeat (internal)
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -31865,7 +31891,9 @@ components:
description: Configuration for creating/managing the publication (tables, operations)
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -31917,7 +31945,9 @@ components:
description: Configuration for creating/managing the publication (tables, operations)
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -31994,7 +32024,9 @@ components:
description: Last error message if the trigger failed
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -32058,7 +32090,9 @@ components:
$ref: "#/components/schemas/TriggerMode"
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -32133,7 +32167,9 @@ components:
description: True if script_path points to a flow, false if it points to a script
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -32196,7 +32232,9 @@ components:
description: Last error message if the trigger failed
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -32244,7 +32282,9 @@ components:
$ref: "#/components/schemas/TriggerMode"
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -32303,7 +32343,9 @@ components:
description: True if script_path points to a flow, false if it points to a script
error_handler_path:
type: string
description: Path to a script or flow to run when the triggered job fails
description: Path to a script to run when the triggered job fails. A bare
path, without the script/ or flow/ prefix a schedule error handler takes;
it cannot be a flow.
error_handler_args:
$ref: "#/components/schemas/ScriptArgs"
description: Arguments to pass to the error handler
@@ -34186,8 +34228,34 @@ components:
LogSearchHit:
type: object
properties:
dancer:
ts:
description: timestamp of the log line itself, not of the file containing it
type: string
format: date-time
host:
type: string
level:
type: string
enum: [TRACE, DEBUG, INFO, WARN, ERROR]
target:
description: the tracing target that emitted the line
type: string
nullable: true
message:
type: string
file_path:
description: the log file the line came from
type: string
line_no:
description: offset of the line within its file
type: integer
required:
- ts
- host
- level
- message
- file_path
- line_no
AutoscalingEvent:
type: object
backend/windmill-api/src/ai.rs
+182 -97
View File
@@ -409,6 +409,19 @@ impl ExpiringProviderCredentials {
}
}
/// Set on the copilot config when the workspace has no AI provider of its own and is
/// running on Windmill's free tier, so the client can label the lent model as free, warn
/// before the grant runs out, and tell the user to add their own key once it has — rather
/// than showing the same "no provider configured" state a never-configured workspace gets.
#[derive(Serialize, Deserialize, Debug, Default, Clone)]
pub struct FreeTierInfo {
/// The grant is spent: no provider is served and the user must bring their own key.
pub exhausted: bool,
/// Fraction of the grant consumed, 0.0..=1.0. A ratio, not a dollar amount — the
/// pricing model stays server-side.
pub used_ratio: f64,
}
#[derive(Serialize, Deserialize, Debug, Default)]
pub struct AIConfig {
#[serde(skip_serializing_if = "Option::is_none")]
@@ -423,6 +436,11 @@ pub struct AIConfig {
pub custom_prompts: Option<HashMap<String, String>>,
#[serde(skip_serializing_if = "Option::is_none")]
pub max_tokens_per_model: Option<HashMap<String, i32>>,
/// Response-only: this same struct is the request body for saving a workspace's AI
/// config, and `skip_deserializing` is what stops a client from storing a forged
/// free-tier marker. Only the server sets it, per-request.
#[serde(skip_serializing_if = "Option::is_none", skip_deserializing)]
pub free_tier: Option<FreeTierInfo>,
/// Per-model price overrides, keyed `provider:model` like `max_tokens_per_model`.
/// Only models whose rates differ from the built-in table are stored.
#[serde(skip_serializing_if = "Option::is_none")]
@@ -1013,83 +1031,119 @@ async fn proxy(
check_scopes(&authed, || format!("resources:read:{}", resource_path))?;
}
let mut credentials = match workspace_cache {
Some(request_cache) if !request_cache.is_expired() && forced_resource_path.is_none() => {
request_cache.credentials
}
_ => {
let (resource_path, save_to_cache, resource_workspace, instance_ai_config_revision) =
if let Some(resource_path) = forced_resource_path {
// forced resource path
(resource_path, false, w_id.clone(), None)
} else {
let workspace_ai_config = sqlx::query_scalar!(
"SELECT ai_config FROM workspace_settings WHERE workspace_id = $1",
&w_id
)
.fetch_one(&db)
.await?;
// Set when serving the request through Windmill's free AI tier (the lent key). Holds
// the per-user concurrency lock and drives response metering.
let mut free_lease: Option<crate::ai_free_tier_oss::FreeTierLease> = None;
let mut credentials = 'cred: {
match workspace_cache {
Some(request_cache)
if !request_cache.is_expired() && forced_resource_path.is_none() =>
{
request_cache.credentials
}
_ => {
let (resource_path, save_to_cache, resource_workspace, instance_ai_config_revision) =
if let Some(resource_path) = forced_resource_path {
// forced resource path
(resource_path, false, w_id.clone(), None)
} else {
let workspace_ai_config = sqlx::query_scalar!(
"SELECT ai_config FROM workspace_settings WHERE workspace_id = $1",
&w_id
)
.fetch_one(&db)
.await?;
let (ai_config_value, resource_workspace, instance_ai_config_revision) = {
let ws_has_config = workspace_ai_config
.as_ref()
.and_then(|v| serde_json::from_value::<AIConfig>(v.clone()).ok())
.is_some_and(|config| config.has_providers());
let (ai_config_value, resource_workspace, instance_ai_config_revision) = {
let ws_has_config = workspace_ai_config
.as_ref()
.and_then(|v| serde_json::from_value::<AIConfig>(v.clone()).ok())
.is_some_and(|config| config.has_providers());
if ws_has_config {
(workspace_ai_config.unwrap(), w_id.clone(), None)
} else {
let instance_config = sqlx::query_scalar!(
"SELECT value FROM global_settings WHERE name = 'ai_config'"
)
.fetch_optional(&db)
.await?;
if ws_has_config {
(workspace_ai_config.unwrap(), w_id.clone(), None)
} else {
let instance_config = sqlx::query_scalar!(
"SELECT value FROM global_settings WHERE name = 'ai_config'"
)
.fetch_optional(&db)
.await?;
match instance_config {
Some(config) => (
config,
"admins".to_string(),
Some(current_instance_ai_config_revision()),
),
None => {
return Err(Error::internal_err(
"AI resource not configured".to_string(),
));
let instance_has_config =
instance_config.as_ref().is_some_and(|v| {
serde_json::from_value::<AIConfig>(v.clone())
.ok()
.is_some_and(|c| c.has_providers())
});
match instance_config {
// An instance `ai_config` row with no usable provider (e.g. `{}`
// or `{"providers":{}}`) is treated as unconfigured, exactly as
// build_copilot_settings_state does — otherwise its mere presence
// would suppress the free-tier fallback below.
Some(config) if instance_has_config => (
config,
"admins".to_string(),
Some(current_instance_ai_config_revision()),
),
_ => {
// Nothing configured: fall back to Windmill's free AI tier
// (EE-only) if a lent key is set and both the user's
// one-time grant and the instance's daily cap have room.
// Errors once the grant is spent, the day is capped, or the
// user already has a request in flight; None otherwise.
// Ineligible identities (e.g. service accounts) are refused
// inside the helper, so every path treats them alike.
let free =
crate::ai_free_tier_oss::resolve_free_tier_credentials(
&provider,
&db,
&ai_path,
&authed.email,
&body,
)
.await?;
if let Some((free_credentials, lease)) = free {
free_lease = Some(lease);
break 'cred free_credentials;
}
return Err(Error::internal_err(
"AI resource not configured".to_string(),
));
}
}
}
};
let mut ai_config = serde_json::from_value::<AIConfig>(ai_config_value)
.map_err(|e| Error::BadRequest(e.to_string()))?;
let provider_config = ai_config
.providers
.as_mut()
.and_then(|providers| providers.remove(&provider))
.ok_or_else(|| {
Error::BadRequest(format!("Provider {:?} not configured", provider))
})?;
if provider_config.resource_path.is_empty() {
return Err(Error::BadRequest("Resource path is empty".to_string()));
}
(
provider_config.resource_path,
true,
resource_workspace,
instance_ai_config_revision,
)
};
let mut ai_config = serde_json::from_value::<AIConfig>(ai_config_value)
.map_err(|e| Error::BadRequest(e.to_string()))?;
let provider_config = ai_config
.providers
.as_mut()
.and_then(|providers| providers.remove(&provider))
.ok_or_else(|| {
Error::BadRequest(format!("Provider {:?} not configured", provider))
})?;
if provider_config.resource_path.is_empty() {
return Err(Error::BadRequest("Resource path is empty".to_string()));
}
(
provider_config.resource_path,
true,
resource_workspace,
instance_ai_config_revision,
)
};
// For user-specified resources, fetch through an RLS-scoped
// connection so PostgreSQL row-level security enforces the same
// folder/group boundaries as the regular resource API. For the
// workspace/instance ai_config path, the resource_path was already
// validated by an admin/devops user when configuring the workspace,
// so the raw pool is used.
let resource = if is_user_specified_resource {
// For user-specified resources, fetch through an RLS-scoped
// connection so PostgreSQL row-level security enforces the same
// folder/group boundaries as the regular resource API. For the
// workspace/instance ai_config path, the resource_path was already
// validated by an admin/devops user when configuring the workspace,
// so the raw pool is used.
let resource = if is_user_specified_resource {
let mut tx = user_db.clone().begin(&authed).await?;
let res = sqlx::query_scalar::<_, Option<sqlx::types::Json<Box<RawValue>>>>(
"SELECT value FROM resource WHERE path = $1 AND workspace_id = $2",
@@ -1112,38 +1166,45 @@ async fn proxy(
.ok_or_else(|| Error::NotFound(format!("Could not find the resource {}, update the resource path in the workspace settings", resource_path)))?
.ok_or_else(|| Error::BadRequest(format!("Empty resource value for {}", resource_path)))?;
let resource = serde_json::from_str::<AIResource>(resource.0.get())
.map_err(|e| Error::BadRequest(e.to_string()))?;
let resource = serde_json::from_str::<AIResource>(resource.0.get())
.map_err(|e| Error::BadRequest(e.to_string()))?;
// Enforce RLS on $var: resolution when the resource path was
// user-specified (X-Resource-Path header) so users can only read
// variables they have permission to access.
let enforce_authed = if is_user_specified_resource {
Some(&authed)
} else {
None
};
let credentials = resolve_provider_credentials(
&provider,
&db,
&resource_workspace,
resource,
enforce_authed,
)
.await?;
if save_to_cache {
AI_REQUEST_CACHE.insert(
(w_id.clone(), provider.clone()),
ExpiringProviderCredentials::new(
credentials.clone(),
instance_ai_config_revision,
),
);
// Enforce RLS on $var: resolution when the resource path was
// user-specified (X-Resource-Path header) so users can only read
// variables they have permission to access.
let enforce_authed = if is_user_specified_resource {
Some(&authed)
} else {
None
};
let credentials = resolve_provider_credentials(
&provider,
&db,
&resource_workspace,
resource,
enforce_authed,
)
.await?;
if save_to_cache {
AI_REQUEST_CACHE.insert(
(w_id.clone(), provider.clone()),
ExpiringProviderCredentials::new(
credentials.clone(),
instance_ai_config_revision,
),
);
}
credentials
}
credentials
}
};
// Free tier: pin the model and clamp max_tokens server-side before forwarding,
// since the request body is otherwise client-controlled.
if free_lease.is_some() {
body = crate::ai_free_tier_oss::enforce_free_tier_body(&body)?;
}
if let Some(fim_transform) =
maybe_transform_fim_request(&provider, &ai_path, &credentials.base_url, &body)?
{
@@ -1291,8 +1352,32 @@ async fn proxy(
let status_code = response.status();
let headers = response.headers().clone();
let is_sse = is_sse_response(&headers);
// Free tier: reconcile the cost reserved up-front against what the response actually
// used, holding the per-user lock (via the lease) until it is recorded. The chat
// streams (SSE), where the usage report only arrives in the final chunk; the
// non-streaming JSON path is handled for completeness.
if let Some(lease) = free_lease {
let body = if is_sse {
axum::body::Body::from_stream(inject_keepalives(
Box::pin(crate::ai_free_tier_oss::meter_usage(
response.bytes_stream(),
db.clone(),
lease,
)),
Duration::from_secs(KEEPALIVE_INTERVAL_SECS),
))
} else {
let bytes = response.bytes().await.map_err(to_anyhow)?;
crate::ai_free_tier_oss::record_json_usage(db.clone(), lease, &bytes);
axum::body::Body::from(bytes)
};
return Ok((status_code, headers, body));
}
let stream = response.bytes_stream();
let body = if is_sse_response(&headers) {
let body = if is_sse {
axum::body::Body::from_stream(inject_keepalives(
stream,
Duration::from_secs(KEEPALIVE_INTERVAL_SECS),
backend/windmill-api/src/ai_evals/datasets.rs
+2
View File
@@ -190,6 +190,8 @@ pub async fn create_dataset(
}
tx.commit().await?;
windmill_common::feature_usage::log_feature_usage("ai_agent_eval", "dataset_created", "");
Ok(format!("Created eval dataset {}", payload.path))
}
backend/windmill-api/src/ai_evals/run.rs
+11
View File
@@ -707,6 +707,17 @@ pub async fn run_experiment(
.await?;
return Err(e);
}
// Which state of the agent was measured is the whole key vocabulary: it is what separates
// running what is deployed from measuring edits or an older version.
windmill_common::feature_usage::log_feature_usage(
"ai_agent_eval",
"run",
match subject.kind {
EvalSubjectKind::Agent => "agent",
EvalSubjectKind::AgentDraft => "agent_draft",
EvalSubjectKind::AgentVersion => "agent_version",
},
);
Ok(experiment_id.to_string())
}
backend/windmill-api/src/ai_free_tier_oss.rs
+65
View File
@@ -0,0 +1,65 @@
#[cfg(feature = "private")]
#[allow(unused)]
pub use crate::ai_free_tier_ee::*;
// Open-source build: Windmill's free AI tier does not exist. These stubs make the
// callers in `ai.rs` / `workspaces.rs` compile while disabling the feature entirely —
// `resolve_free_tier_credentials` never opts in, so the proxy falls through to its
// normal "AI resource not configured" path and the copilot stays hidden.
//
// Caller contract (enforced by the private impl, restated here for parity): the `email`
// passed to `resolve_free_tier_credentials` / `free_tier_copilot_config` MUST be the
// authenticated caller's own identity (an `ApiAuthed` email), never a client-supplied one —
// it selects whose lent-key grant is spent and whose usage is read.
#[cfg(not(feature = "private"))]
use crate::ai::AIConfig;
#[cfg(not(feature = "private"))]
use crate::db::DB;
#[cfg(not(feature = "private"))]
use axum::body::Bytes;
#[cfg(not(feature = "private"))]
use windmill_ai::ai_providers::AIProvider;
#[cfg(not(feature = "private"))]
use windmill_ai::credentials::ProviderCredentials;
#[cfg(not(feature = "private"))]
use windmill_common::error::Result;
#[cfg(not(feature = "private"))]
pub struct FreeTierLease;
#[cfg(not(feature = "private"))]
pub async fn resolve_free_tier_credentials(
_provider: &AIProvider,
_db: &DB,
_ai_path: &str,
_email: &str,
_body: &Bytes,
) -> Result<Option<(ProviderCredentials, FreeTierLease)>> {
Ok(None)
}
#[cfg(not(feature = "private"))]
pub fn enforce_free_tier_body(body: &Bytes) -> Result<Bytes> {
Ok(body.clone())
}
#[cfg(not(feature = "private"))]
pub async fn free_tier_copilot_config(_db: &DB, _email: &str) -> Result<Option<AIConfig>> {
Ok(None)
}
#[cfg(not(feature = "private"))]
pub fn record_json_usage(_db: DB, _lease: FreeTierLease, _bytes: &[u8]) {}
#[cfg(not(feature = "private"))]
pub fn meter_usage<S>(
upstream: S,
_db: DB,
_lease: FreeTierLease,
) -> impl futures::Stream<Item = std::result::Result<Bytes, reqwest::Error>>
where
S: futures::Stream<Item = std::result::Result<Bytes, reqwest::Error>> + Unpin,
{
upstream
}
backend/windmill-api/src/ai_skills.rs
-394
View File
@@ -1,394 +0,0 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2026
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
use crate::db::{ApiAuthed, DB};
use axum::{
extract::{Extension, Json, Path},
routing::{delete, get, post},
Router,
};
use serde::{Deserialize, Serialize};
use std::collections::HashSet;
use windmill_audit::audit_oss::audit_log;
use windmill_audit::ActionKind;
use windmill_common::{
db::UserDB,
error::{Error, JsonResult, Result},
utils::require_admin,
};
pub fn workspaced_service() -> Router {
Router::new()
.route("/list", get(list_skills))
.route("/get/{name}", get(get_skill))
.route("/upload", post(upload_skills))
.route("/delete/{name}", delete(delete_skill))
}
/// Cheap listing surfaced in the AI chat system prompt — no `instructions` body.
#[derive(Serialize)]
pub struct SkillListItem {
pub name: String,
pub description: String,
}
/// Full skill, including the SKILL.md body, fetched on demand by `read_skill`.
#[derive(Serialize)]
pub struct Skill {
pub name: String,
pub description: String,
pub instructions: String,
}
#[derive(Deserialize)]
pub struct UploadSkills {
pub skills: Vec<SkillUpload>,
}
#[derive(Deserialize)]
pub struct SkillUpload {
pub name: String,
pub description: String,
pub instructions: String,
}
const MAX_SKILLS_PER_UPLOAD: usize = 50;
// Every stored skill's name + description is advertised in the global AI chat
// system prompt, so bound the total a workspace can accumulate across uploads.
const MAX_SKILLS_PER_WORKSPACE: usize = 100;
// `name` and `description` follow the Claude SKILL.md spec
// (https://platform.claude.com/docs/en/agents-and-tools/agent-skills): both are
// loaded into the AI chat system prompt and `name` is the model-facing skill id,
// so matching the upstream limits keeps skills portable with Claude Code.
const MAX_SKILL_NAME_CHARS: usize = 64;
const MAX_SKILL_DESCRIPTION_CHARS: usize = 1_024;
// Not a spec field — a payload bound on the SKILL.md body, so measured in bytes.
const MAX_SKILL_INSTRUCTIONS_BYTES: usize = 64 * 1024;
fn validate_skill(skill: &SkillUpload) -> Result<()> {
let name = skill.name.trim();
if name.is_empty() || name.chars().count() > MAX_SKILL_NAME_CHARS {
return Err(Error::BadRequest(format!(
"skill name must be between 1 and {MAX_SKILL_NAME_CHARS} characters, got {:?}",
skill.name
)));
}
if !name
.chars()
.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-')
{
return Err(Error::BadRequest(format!(
"skill name {name:?} must only contain lowercase letters, digits or '-'"
)));
}
if skill.description.trim().is_empty() {
return Err(Error::BadRequest(format!(
"skill {name:?} is missing a description (the SKILL.md frontmatter `description`)"
)));
}
if skill.description.chars().count() > MAX_SKILL_DESCRIPTION_CHARS {
return Err(Error::BadRequest(format!(
"skill {name:?} description must be at most {MAX_SKILL_DESCRIPTION_CHARS} characters"
)));
}
if skill.instructions.trim().is_empty() {
return Err(Error::BadRequest(format!(
"skill {name:?} has an empty SKILL.md body"
)));
}
if skill.instructions.len() > MAX_SKILL_INSTRUCTIONS_BYTES {
return Err(Error::BadRequest(format!(
"skill {name:?} instructions must be at most {MAX_SKILL_INSTRUCTIONS_BYTES} bytes"
)));
}
Ok(())
}
/// Collect the trimmed skill names, rejecting duplicates within a single upload.
/// The insert upserts by name, so a duplicate would silently keep only the last
/// and make the reported/audited count wrong.
fn collect_upload_names(skills: &[SkillUpload]) -> Result<Vec<String>> {
let mut names = Vec::with_capacity(skills.len());
let mut seen = HashSet::with_capacity(skills.len());
for skill in skills {
let name = skill.name.trim().to_string();
if !seen.insert(name.clone()) {
return Err(Error::BadRequest(format!(
"duplicate skill name {name:?} in upload"
)));
}
names.push(name);
}
Ok(names)
}
/// Reject an upload that would push the workspace past `MAX_SKILLS_PER_WORKSPACE`.
/// Uploads upsert, so names already present (`replacing`) don't count as new.
fn check_workspace_skill_capacity(
existing_total: i64,
replacing: i64,
upload_count: usize,
) -> Result<()> {
let new_count = upload_count as i64 - replacing;
if existing_total + new_count > MAX_SKILLS_PER_WORKSPACE as i64 {
return Err(Error::BadRequest(format!(
"workspace cannot store more than {MAX_SKILLS_PER_WORKSPACE} skills"
)));
}
Ok(())
}
async fn list_skills(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Path(w_id): Path<String>,
) -> JsonResult<Vec<SkillListItem>> {
let mut tx = user_db.begin(&authed).await?;
let rows = sqlx::query!(
"SELECT name, description FROM ai_skill WHERE workspace_id = $1 ORDER BY name",
&w_id
)
.fetch_all(&mut *tx)
.await?;
tx.commit().await?;
Ok(Json(
rows.into_iter()
.map(|r| SkillListItem { name: r.name, description: r.description })
.collect(),
))
}
async fn get_skill(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Path((w_id, name)): Path<(String, String)>,
) -> JsonResult<Skill> {
let mut tx = user_db.begin(&authed).await?;
let row = sqlx::query!(
"SELECT name, description, instructions FROM ai_skill WHERE workspace_id = $1 AND name = $2",
&w_id,
&name
)
.fetch_optional(&mut *tx)
.await?;
tx.commit().await?;
row.map(|r| {
Json(Skill { name: r.name, description: r.description, instructions: r.instructions })
})
.ok_or_else(|| Error::NotFound(format!("no skill named {name:?} in workspace {w_id}")))
}
/// Bulk upsert the uploaded skills by name. Existing skills not in the payload
/// are left untouched — removal goes through `delete_skill`.
async fn upload_skills(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Json(payload): Json<UploadSkills>,
) -> Result<String> {
require_admin(authed.is_admin, &authed.username)?;
if payload.skills.is_empty() {
return Err(Error::BadRequest("no skills to upload".to_string()));
}
if payload.skills.len() > MAX_SKILLS_PER_UPLOAD {
return Err(Error::BadRequest(format!(
"cannot upload more than {MAX_SKILLS_PER_UPLOAD} skills at a time"
)));
}
for skill in &payload.skills {
validate_skill(skill)?;
}
let names = collect_upload_names(&payload.skills)?;
let mut tx = db.begin().await?;
let counts = sqlx::query!(
r#"SELECT
COUNT(*)::bigint AS "total!",
COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS "replacing!"
FROM ai_skill
WHERE workspace_id = $1"#,
&w_id,
&names
)
.fetch_one(&mut *tx)
.await?;
check_workspace_skill_capacity(counts.total, counts.replacing, names.len())?;
for (skill, name) in payload.skills.iter().zip(names.iter()) {
sqlx::query!(
r#"INSERT INTO ai_skill (workspace_id, name, description, instructions, edited_at, edited_by)
VALUES ($1, $2, $3, $4, now(), $5)
ON CONFLICT (workspace_id, name) DO UPDATE
SET description = EXCLUDED.description,
instructions = EXCLUDED.instructions,
edited_at = now(),
edited_by = EXCLUDED.edited_by"#,
&w_id,
name,
skill.description,
skill.instructions,
&authed.username,
)
.execute(&mut *tx)
.await?;
}
let audit_resource = names.join(",");
audit_log(
&mut *tx,
&authed,
"ai_skills.upload",
ActionKind::Update,
&w_id,
Some(&audit_resource),
Some([("skill_count", &names.len().to_string()[..])].into()),
)
.await?;
tx.commit().await?;
Ok(format!(
"Uploaded {} skill(s) to workspace {}",
payload.skills.len(),
&w_id
))
}
async fn delete_skill(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path((w_id, name)): Path<(String, String)>,
) -> Result<String> {
require_admin(authed.is_admin, &authed.username)?;
let mut tx = db.begin().await?;
let deleted = sqlx::query_scalar!(
"DELETE FROM ai_skill WHERE workspace_id = $1 AND name = $2 RETURNING name",
&w_id,
&name
)
.fetch_optional(&mut *tx)
.await?;
if deleted.is_none() {
tx.commit().await?;
return Err(Error::NotFound(format!(
"no skill named {name:?} in workspace {w_id}"
)));
}
audit_log(
&mut *tx,
&authed,
"ai_skills.delete",
ActionKind::Delete,
&w_id,
Some(&name),
None,
)
.await?;
tx.commit().await?;
Ok(format!("Deleted skill {name} from workspace {w_id}"))
}
#[cfg(test)]
mod tests {
use super::*;
fn skill() -> SkillUpload {
SkillUpload {
name: "test-skill".to_string(),
description: "Useful for tests".to_string(),
instructions: "# Test\n\nDo the thing.".to_string(),
}
}
#[test]
fn validate_skill_rejects_oversized_description() {
let mut skill = skill();
skill.description = "x".repeat(MAX_SKILL_DESCRIPTION_CHARS + 1);
assert!(matches!(validate_skill(&skill), Err(Error::BadRequest(_))));
}
#[test]
fn validate_skill_rejects_oversized_instructions() {
let mut skill = skill();
skill.instructions = "x".repeat(MAX_SKILL_INSTRUCTIONS_BYTES + 1);
assert!(matches!(validate_skill(&skill), Err(Error::BadRequest(_))));
}
#[test]
fn validate_skill_rejects_oversized_name() {
let mut skill = skill();
skill.name = "a".repeat(MAX_SKILL_NAME_CHARS + 1);
assert!(matches!(validate_skill(&skill), Err(Error::BadRequest(_))));
}
#[test]
fn validate_skill_rejects_non_slug_name() {
// Uppercase, underscore, space and punctuation are all outside the
// Claude SKILL.md `[a-z0-9-]` name charset.
for bad in ["My-Skill", "my_skill", "my skill", "skill!"] {
let mut skill = skill();
skill.name = bad.to_string();
assert!(
matches!(validate_skill(&skill), Err(Error::BadRequest(_))),
"{bad:?} should be rejected"
);
}
}
#[test]
fn validate_skill_counts_description_in_characters() {
// 1024 two-byte chars exceed the byte limit but sit exactly on the
// character limit, so they must be accepted.
let mut skill = skill();
skill.description = "é".repeat(MAX_SKILL_DESCRIPTION_CHARS);
assert!(validate_skill(&skill).is_ok());
}
#[test]
fn workspace_capacity_allows_replacement_at_cap() {
// Already at the cap, but the upload only replaces an existing skill.
let at_cap = MAX_SKILLS_PER_WORKSPACE as i64;
assert!(check_workspace_skill_capacity(at_cap, 1, 1).is_ok());
}
#[test]
fn workspace_capacity_rejects_new_skill_over_cap() {
let at_cap = MAX_SKILLS_PER_WORKSPACE as i64;
assert!(matches!(
check_workspace_skill_capacity(at_cap, 0, 1),
Err(Error::BadRequest(_))
));
}
#[test]
fn collect_upload_names_trims_and_collects() {
let names = collect_upload_names(&[skill()]).unwrap();
assert_eq!(names, vec!["test-skill".to_string()]);
}
#[test]
fn collect_upload_names_rejects_duplicates() {
// Names are compared after trimming, so whitespace can't smuggle a dup in.
let dup = SkillUpload { name: " test-skill ".to_string(), ..skill() };
assert!(matches!(
collect_upload_names(&[skill(), dup]),
Err(Error::BadRequest(_))
));
}
}
backend/windmill-api/src/db.rs
+8
View File
@@ -102,6 +102,12 @@ lazy_static::lazy_static! {
(20260727151319, include_str!(
"../../migrations/20260727151319_draft_only_listing_indexes.up.sql"
).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY")),
(20260826202939, include_str!(
"../../migrations/20260826202939_queue_suspended_resume_at_index.up.sql"
).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY").replace("DROP INDEX", "DROP INDEX CONCURRENTLY")),
(20260826214706, include_str!(
"../../migrations/20260826214706_queue_suspended_drop_legacy_index.up.sql"
).replace("DROP INDEX", "DROP INDEX CONCURRENTLY")),
].into_iter().collect();
}
@@ -228,6 +234,8 @@ impl Migrate for CustomMigrator {
// CONCURRENTLY operations cannot run inside a transaction block
// or a multi-statement query (PostgreSQL requires top-level execution).
// Split into individual statements and execute each separately.
// The split is naive, so a `;` anywhere in an overridden migration —
// inside a comment or a string literal included — splits mid-statement.
for stmt in migration_sql.split(';') {
let stmt = stmt.trim();
if !stmt.is_empty()
backend/windmill-api/src/hub_publish.rs
+34 -3
View File
@@ -42,6 +42,11 @@ pub fn workspaced_service() -> Router {
.route("/migrations", post(publish_migrations))
.route("/projects/{slug}/export", get(get_project_export))
.route("/projects/{slug}/submit", post(submit_project))
.route("/projects/{slug}/withdraw", post(withdraw_project))
.route(
"/projects/{slug}/discard_update",
post(discard_project_update),
)
.route("/project", get(get_project_by_source))
}
@@ -554,9 +559,35 @@ async fn submit_project(
.await
}
// The Hub has no auth of its own: it validates bearer tokens by calling this
// instance's /api/users/whoami. Forwarding the caller's own token logs them in
// on the Hub as themselves (account auto-created on first use).
// Take a submission back out of review, keeping what was pushed for it.
async fn withdraw_project(
ctx: HubPublishCtx,
Path((_workspace, slug)): Path<(String, ProjectSlug)>,
) -> Result<impl IntoResponse, Error> {
ctx.post(
&format!("/projects/{}/withdraw", slug),
&serde_json::json!({}),
)
.await
}
// Throw away the pending update to an already-published project. The published
// version is untouched — it never saw the update.
async fn discard_project_update(
ctx: HubPublishCtx,
Path((_workspace, slug)): Path<(String, ProjectSlug)>,
) -> Result<impl IntoResponse, Error> {
ctx.post(
&format!("/projects/{}/discard_update", slug),
&serde_json::json!({}),
)
.await
}
// The Hub has no auth of its own: it validates bearer tokens by calling
// /api/users/whoami — on app.windmill.dev for the public Hub, on the paired
// instance for a private one. Forwarding the caller's own token logs them in on
// the Hub as themselves (account auto-created on first use).
async fn get_from_hub(
path: &str,
source_id: &str,
backend/windmill-api/src/lib.rs
+3 -2
View File
@@ -66,7 +66,9 @@ use crate::scim_oss::has_scim_token;
use windmill_common::error::AppError;
mod ai;
mod ai_skills;
#[cfg(feature = "private")]
mod ai_free_tier_ee;
mod ai_free_tier_oss;
mod apps;
mod apps_raw_bundle;
pub use apps::invalidate_app_policy_cache;
@@ -710,7 +712,6 @@ pub async fn run_server(
Router::new()
})
.nest("/ai", ai::workspaced_service())
.nest("/ai_skills", ai_skills::workspaced_service())
.nest("/npm_proxy", windmill_api_npm_proxy::workspaced_service())
.nest(
"/path_autocomplete",
backend/windmill-api/src/service_logs.rs
+86 -18
View File
@@ -88,6 +88,66 @@ async fn list_files(
Ok(Json(rows))
}
/// Rebuild one source log file from the columnar store.
///
/// Not the original bytes: the store holds a line's fields rather than its text,
/// so the JSON is re-serialized here and key order and whitespace are this
/// writer's. Everything a reader can see survives — the drawer this feeds
/// renders a prettified view of each line either way, and a line that was never
/// JSON comes back exactly as it was written.
#[cfg(all(feature = "tantivy", feature = "private"))]
async fn get_log_file_from_store(
db: &DB,
store: &windmill_indexer::service_logs_store_ee::Store,
path: &str,
) -> windmill_common::error::Result<Response> {
let (hostname, file_name) = path
.split_once('/')
.ok_or_else(|| Error::BadRequest("Invalid path".to_string()))?;
// The store is partitioned by day and mode, neither of which the path
// carries. `log_file` names both, and its primary key starts with hostname.
let file = sqlx::query!(
// `mode!` because the column is NOT NULL and only the cast makes sqlx
// think otherwise; a silent default would look up a `mode=` partition
// that matches nothing and read as a missing file.
"SELECT mode::text AS \"mode!\", log_ts FROM log_file WHERE hostname = $1 AND file_path = $2 ORDER BY log_ts DESC LIMIT 1",
hostname,
file_name
)
.fetch_optional(db)
.await?
.ok_or_else(|| Error::NotFound(format!("File {path} not found")))?;
// A row registered by this version carries the minute in the file's own name,
// so the two agree and the second is redundant. One written before the
// uploader derived `log_ts` from the name carries a wall clock instead, and
// those outlive an upgrade by the retention period — which is also what makes
// the `ORDER BY` above worth having. The name is authoritative, so both go.
let mut known_ts = vec![chrono::DateTime::from_naive_utc_and_offset(
file.log_ts,
chrono::Utc,
)];
if let Some(named) = file_name.rsplit('.').next().and_then(|s| {
chrono::NaiveDateTime::parse_from_str(s, windmill_common::tracing_init::LOG_TIMESTAMP_FMT)
.ok()
}) {
known_ts.push(chrono::DateTime::from_naive_utc_and_offset(
named,
chrono::Utc,
));
}
let text = windmill_indexer::service_logs_store_ee::read_log_file(
store, &file.mode, hostname, file_name, &known_ts,
)
.await
.map_err(|e| Error::internal_err(format!("Error reading the service log store: {e}")))?
.ok_or_else(|| Error::NotFound(format!("File {path} not found")))?;
Ok(content_plain(Body::from(text)))
}
async fn get_log_file(
authed: ApiAuthed,
Extension(db): Extension<DB>,
@@ -104,27 +164,30 @@ async fn get_log_file(
let s3_client = windmill_object_store::get_object_store().await;
#[cfg(feature = "parquet")]
if let Some(s3_client) = s3_client {
let path = format!("{}{}", windmill_common::tracing_init::LOGS_SERVICE, path);
let file = s3_client
use windmill_object_store::object_store_reexports::ObjectStoreError;
// The raw file, for as long as it is there. It outlives its ingestion by
// one indexer pass at most, so this covers the most recent minutes of a
// host's logs byte for byte; everything older is rebuilt from the store.
let object_path = format!("{}{}", windmill_common::tracing_init::LOGS_SERVICE, path);
match s3_client
.get(&windmill_object_store::object_store_reexports::Path::from(
path,
object_path,
))
.await;
match file {
Ok(file) => {
let bytes = file.bytes().await;
match bytes {
Ok(bytes) => {
return Ok(content_plain(Body::from(bytes::Bytes::from(bytes))));
}
Err(e) => {
return Err(Error::internal_err(format!(
"Error pulling the bytes: {}",
e
)));
}
.await
{
Ok(file) => match file.bytes().await {
Ok(bytes) => {
return Ok(content_plain(Body::from(bytes::Bytes::from(bytes))));
}
}
Err(e) => {
return Err(Error::internal_err(format!(
"Error pulling the bytes: {}",
e
)));
}
},
Err(ObjectStoreError::NotFound { .. }) => {}
Err(e) => {
return Err(Error::internal_err(format!(
"Error fetching the file: {}",
@@ -132,6 +195,11 @@ async fn get_log_file(
)));
}
}
#[cfg(all(feature = "tantivy", feature = "private"))]
return get_log_file_from_store(&db, &s3_client, &path).await;
#[cfg(not(all(feature = "tantivy", feature = "private")))]
return Err(Error::NotFound(format!("File {path} not found")));
}
let full_path = format!("{}{}", *TMP_WINDMILL_LOGS_SERVICE, path);
// SECURITY (defense in depth): refuse to read through a symlink so a planted
backend/windmill-api/src/token.rs
-1
View File
@@ -99,7 +99,6 @@ fn build_standard_scope_domains() -> Vec<ScopeDomain> {
("configs", "Configs", "Configuration management", false),
("oauth", "OAuth", "OAuth management", false),
("ai", "AI", "AI feature management", false),
("ai_skills", "AI Skills", "AI skill management", false),
(
"ai_evals",
"AI Evals",
backend/windmill-api/src/workspaces.rs
+47 -11
View File
@@ -18,7 +18,6 @@ use crate::teams_oss::{
connect_teams, edit_teams_command, run_teams_message_test_job,
workspaces_list_available_teams_channels, workspaces_list_available_teams_ids,
};
use axum::{
extract::{Extension, Path},
routing::{get, post},
@@ -153,10 +152,23 @@ async fn edit_copilot_config(
.await?;
let settings_state =
build_copilot_settings_state(workspace_has_config, instance_ai_config.as_ref());
// A provider-less instance config (e.g. `{}`) is unconfigured, same as build_copilot_settings_state
// treats it — so it must not shadow the free-tier fallback here either.
let instance_config_with_providers = instance_ai_config
.as_ref()
.and_then(|v| serde_json::from_value::<AIConfig>(v.clone()).ok())
.filter(|c| c.has_providers());
let effective_ai_config = if workspace_has_config {
ai_config
} else if let Some(instance_ai_config) = instance_ai_config {
serde_json::from_value::<AIConfig>(instance_ai_config).unwrap_or_default()
} else if let Some(instance_config) = instance_config_with_providers {
instance_config
} else if let Some(free_config) =
crate::ai_free_tier_oss::free_tier_copilot_config(&db, &authed.email).await?
{
// Same fallback as get_copilot_info: with nothing configured, surface Windmill's free
// tier (EE-only) so clearing a workspace provider activates it immediately, instead of
// returning an empty config that disables AI until the next page reload re-fetches it.
free_config
} else {
AIConfig::default()
};
@@ -179,6 +191,7 @@ struct EditCopilotConfigResponse {
}
async fn get_copilot_info(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
) -> JsonResult<AIConfig> {
@@ -194,16 +207,25 @@ async fn get_copilot_info(
))
})?;
if let Some(workspace_ai_config) = workspace_ai_config.filter(|c| c.0.has_providers()) {
Ok(Json(workspace_ai_config.0))
} else if let Some(instance_config) =
let instance_config =
sqlx::query_scalar!("SELECT value FROM global_settings WHERE name = 'ai_config'")
.fetch_optional(&db)
.await?
.and_then(|v| serde_json::from_value::<AIConfig>(v).ok())
// A provider-less instance config (e.g. `{}`) is unconfigured; don't let it shadow the
// free-tier fallback, matching the proxy and edit_copilot_config paths.
.filter(|c| c.has_providers());
if let Some(workspace_ai_config) = workspace_ai_config.filter(|c| c.0.has_providers()) {
Ok(Json(workspace_ai_config.0))
} else if let Some(instance_config) = instance_config {
Ok(Json(instance_config))
} else if let Some(free_config) =
crate::ai_free_tier_oss::free_tier_copilot_config(&db, &authed.email).await?
{
Ok(Json(
serde_json::from_value::<AIConfig>(instance_config).unwrap_or_default(),
))
// Nothing configured: fall back to Windmill's free tier (EE-only). The config
// carries a `free_tier` marker even once the user's grant is spent — with no
// providers, but telling the client *why* AI is off.
Ok(Json(free_config))
} else {
Ok(Json(AIConfig::default()))
}
@@ -216,7 +238,14 @@ pub async fn get_critical_alerts(
authed: ApiAuthed,
Query(params): Query<crate::utils::AlertQueryParams>,
) -> JsonResult<serde_json::Value> {
require_admin_or_devops(authed.is_admin, &authed.username, &authed.email, authed.job_id.is_some(), &db).await?;
require_admin_or_devops(
authed.is_admin,
&authed.username,
&authed.email,
authed.job_id.is_some(),
&db,
)
.await?;
crate::utils::get_critical_alerts(db, params, Some(w_id)).await
}
@@ -232,7 +261,14 @@ pub async fn acknowledge_critical_alert(
Path((w_id, id)): Path<(String, i32)>,
authed: ApiAuthed,
) -> Result<String> {
require_admin_or_devops(authed.is_admin, &authed.username, &authed.email, authed.job_id.is_some(), &db).await?;
require_admin_or_devops(
authed.is_admin,
&authed.username,
&authed.email,
authed.job_id.is_some(),
&db,
)
.await?;
crate::utils::acknowledge_critical_alert(db, Some(w_id), id).await
}
backend/windmill-common/src/db_params.rs
-45
View File
@@ -1,45 +0,0 @@
use anyhow::Result;
/// Parsed database connection parameters, shared across DB auth providers (IAM RDS, Entra ID, etc.)
#[derive(Debug, Clone)]
pub struct DatabaseParams {
pub hostname: String,
pub port: u64,
pub username: String,
pub database: String,
}
/// Extract database connection parameters from a PostgreSQL URL
pub fn extract_database_params(database_url: &str) -> Result<DatabaseParams> {
let url = url::Url::parse(database_url)
.map_err(|e| anyhow::anyhow!("Failed to parse database URL: {}", e))?;
let hostname = url
.host_str()
.ok_or_else(|| anyhow::anyhow!("Database URL missing hostname"))?
.to_string();
let port = url.port().unwrap_or(5432) as u64;
let username = if url.username().is_empty() {
return Err(anyhow::anyhow!("Database URL missing username"));
} else {
urlencoding::decode(url.username())?.to_string()
};
let database = url
.path()
.trim_start_matches('/')
.split('/')
.next()
.filter(|s| !s.is_empty())
.ok_or_else(|| anyhow::anyhow!("Database URL missing database name"))?
.to_string();
Ok(DatabaseParams {
hostname,
port,
username,
database: urlencoding::decode(&database)?.to_string(),
})
}
backend/windmill-common/src/global_settings.rs
+1
View File
@@ -15,6 +15,7 @@ pub const OAUTH_SETTING: &str = "oauths";
pub const AI_CONFIG_SETTING: &str = "ai_config";
pub const RETENTION_PERIOD_SECS_SETTING: &str = "retention_period_secs";
pub const RETENTION_PERIOD_SECS_OVERRIDES_SETTING: &str = "retention_period_secs_overrides";
pub const SERVICE_LOG_RETENTION_SECS_SETTING: &str = "service_log_retention_secs";
/// Upper bound on how many per-workspace retention overrides may be configured. The periodic monitor
/// sweeps each override workspace in its own transaction every pass, so this keeps a pass bounded
/// (and the feature is a targeted escape hatch for a handful of special workspaces, not a bulk knob).
backend/windmill-common/src/indexer.rs
+72
View File
@@ -94,6 +94,21 @@ pub async fn load_indexer_config(db: &DB) -> error::Result<TantivyIndexerSetting
})
}
/// How far back the service log index reaches, in seconds.
///
/// [`crate::service_log_retention_secs`] is the ceiling: past it a line's `log_file` row is
/// deleted and can no longer be indexed. `max_index_time_window_secs` of `0` means "do not
/// shrink below that ceiling", not "unbounded" — both sites that trim and populate the index
/// derive the window here so the two cannot disagree about it.
pub fn service_log_index_window_secs(max_index_time_window_secs: i64) -> i64 {
let retention = crate::service_log_retention_secs();
if max_index_time_window_secs > 0 {
std::cmp::min(max_index_time_window_secs, retention)
} else {
retention
}
}
pub fn get_env_var(env_var: &str) -> Option<u64> {
match std::env::var(env_var).map(|x| x.parse()) {
Ok(Ok(i)) => Some(i),
@@ -136,3 +151,60 @@ pub fn get_indexer_rates_from_env() -> TantivyIndexerSettings {
settings
}
#[cfg(test)]
mod tests {
use super::*;
// One test rather than several: both halves share the process-wide retention, and the
// setter half writes it, which parallel tests would race.
#[test]
fn retention_rejects_unusable_values_and_the_index_window_clamps_to_it() {
use crate::{
service_log_retention_secs, set_service_log_retention_secs,
DEFAULT_SERVICE_LOG_RETENTION_SECS,
};
// See `set_service_log_retention_secs` for why the two unusable directions land apart:
// too large keeps the intent by capping, non-positive cannot and falls back.
let rejected: Vec<i64> = [0, -1, i64::MIN]
.iter()
.map(|v| {
set_service_log_retention_secs(*v);
service_log_retention_secs()
})
.collect();
let capped: Vec<i64> = [i64::MAX, 60 * 60 * 24 * 365 * 101]
.iter()
.map(|v| {
set_service_log_retention_secs(*v);
service_log_retention_secs()
})
.collect();
set_service_log_retention_secs(60 * 60 * 24 * 3);
let retention = service_log_retention_secs();
let windows = [
// `0` disables the extra shrinking rather than lifting the ceiling — the trap that
// makes an unset setting look unbounded.
service_log_index_window_secs(0),
// Retention is the ceiling: the index cannot reach lines whose `log_file` row is gone.
service_log_index_window_secs(retention * 2),
service_log_index_window_secs(60),
];
set_service_log_retention_secs(DEFAULT_SERVICE_LOG_RETENTION_SECS);
assert_eq!(
rejected,
vec![DEFAULT_SERVICE_LOG_RETENTION_SECS; 3],
"a value that would expire everything must fall back to the default"
);
assert_eq!(
capped,
vec![60 * 60 * 24 * 365 * 100; 2],
"an oversized value must cap, not shorten retention to the default"
);
assert_eq!(retention, 60 * 60 * 24 * 3);
assert_eq!(windows, [retention, retention, 60]);
}
}
backend/windmill-common/src/lib.rs
+63 -6
View File
@@ -42,7 +42,6 @@ pub mod db;
mod db_entra_ee;
#[cfg(all(feature = "enterprise", feature = "private"))]
mod db_iam_ee;
pub mod db_params;
pub mod dbt_manifest;
pub mod deploy_origin;
#[cfg(feature = "private")]
@@ -148,9 +147,53 @@ pub const DEFAULT_MAX_CONNECTIONS_INDEXER: u32 = 5;
pub const DEFAULT_HUB_BASE_URL: &str = "https://hub.windmill.dev";
pub const PRIVATE_HUB_MIN_VERSION: i32 = 10_000_000;
pub const SERVICE_LOG_RETENTION_SECS: i64 = 60 * 60 * 24 * 14; // 2 weeks retention period for logs
pub const DEFAULT_SERVICE_LOG_RETENTION_SECS: i64 = 60 * 60 * 24 * 14; // 2 weeks retention period for logs
pub const WM_DEPLOYERS_GROUP: &str = "wm_deployers";
/// A century. Every consumer has to survive `now - retention`, and the ceilings are much lower
/// than an `i64`: `DateTime` subtraction panics past year 262143, and the `(<n> s)::interval`
/// the cleanup queries build overflows Postgres' microsecond field.
const MAX_SERVICE_LOG_RETENTION_SECS: i64 = 60 * 60 * 24 * 365 * 100;
/// Apply a configured service log retention, in seconds.
///
/// The only way into [`SERVICE_LOG_RETENTION_SECS`], so an unusable value can never reach a
/// cutoff. The two unusable directions are not the same mistake and must not share a landing
/// point: too large still says "keep these for a very long time", so it is capped and the
/// intent survives, whereas falling back would delete logs the operator meant to keep. A
/// non-positive value has no such reading — every cutoff is `now - retention`, so it lands at
/// or after `now` and the next sweep expires the entire history, rows and object-storage files
/// alike. Unlike job retention there is no "keep forever" spelling here, so `0` — what an
/// operator types by analogy with it, and what the settings UI writes into a field that was
/// merely focused — falls back to the default.
pub fn set_service_log_retention_secs(configured: i64) {
let effective = if configured > MAX_SERVICE_LOG_RETENTION_SECS {
tracing::warn!(
"service log retention of {configured}s exceeds the maximum of \
{MAX_SERVICE_LOG_RETENTION_SECS}s, capping it there"
);
MAX_SERVICE_LOG_RETENTION_SECS
} else if configured >= 1 {
configured
} else {
tracing::warn!(
"service log retention of {configured}s would expire every service log, \
falling back to the default of {DEFAULT_SERVICE_LOG_RETENTION_SECS}s"
);
DEFAULT_SERVICE_LOG_RETENTION_SECS
};
SERVICE_LOG_RETENTION_SECS.store(effective, std::sync::atomic::Ordering::Relaxed);
}
/// How long a service log line stays retrievable, in seconds.
///
/// The outer bound on everything service-log: the `log_file` rows, the raw files in object
/// storage, the columnar store queried by retrieval, and — through
/// [`indexer::service_log_index_window_secs`] — the search index.
pub fn service_log_retention_secs() -> i64 {
SERVICE_LOG_RETENTION_SECS.load(std::sync::atomic::Ordering::Relaxed)
}
/// Canonical form of a base URL, used as one of the inputs to the offline-license
/// instance hash (`compute_instance_hash`).
///
@@ -376,6 +419,10 @@ lazy_static::lazy_static! {
/// workspace configured before its override could be read.
pub static ref JOB_RETENTION_SECS_OVERRIDES_LOADED: AtomicBool = AtomicBool::new(false);
pub static ref AUDIT_LOG_RETENTION_DAYS: AtomicI64 = AtomicI64::new(0);
/// Private on purpose: [`set_service_log_retention_secs`] is the only writer, so a value that
/// would expire every service log cannot reach a cutoff. Read it with
/// [`service_log_retention_secs`].
static ref SERVICE_LOG_RETENTION_SECS: AtomicI64 = AtomicI64::new(DEFAULT_SERVICE_LOG_RETENTION_SECS);
pub static ref MONITOR_LOGS_ON_OBJECT_STORE: AtomicBool = AtomicBool::new(false);
@@ -1479,6 +1526,17 @@ pub async fn create_custom_instance_database(
Ok(())
}
/// Connection options parsed from a database URL.
///
/// The only place a database URL becomes `PgConnectOptions`. Providers that mint the password
/// themselves override it on these and keep the rest: options assembled field by field instead
/// would drop every query parameter, `sslmode` and `sslrootcert` above all, leaving the
/// connection on sqlx's default TLS policy rather than the operator's.
pub fn base_connect_options(database_url: &str) -> Result<sqlx::postgres::PgConnectOptions, Error> {
sqlx::postgres::PgConnectOptions::from_str(database_url)
.map_err(|e| Error::InternalErr(format!("Failed to parse database URL: {}", e)))
}
#[derive(Clone)]
pub enum DatabaseUrl {
#[cfg(all(feature = "enterprise", feature = "private"))]
@@ -1509,8 +1567,8 @@ impl DatabaseUrl {
}
/// Get PgConnectOptions for this database URL.
/// For token-based auth (IAM RDS, Entra ID), this returns options built directly from the
/// token to avoid double-encoding issues with temporary credentials.
/// For token-based auth (IAM RDS, Entra ID), this returns options carrying the current
/// token, set on the builder to avoid double-encoding temporary credentials.
/// For static URLs, this parses the URL string.
pub async fn connect_options(&self) -> Result<sqlx::postgres::PgConnectOptions, Error> {
match self {
@@ -1524,8 +1582,7 @@ impl DatabaseUrl {
let guard = entra_url.read().await;
Ok(guard.connect_options())
}
DatabaseUrl::Static(url) => sqlx::postgres::PgConnectOptions::from_str(url)
.map_err(|e| Error::InternalErr(format!("Failed to parse database URL: {}", e))),
DatabaseUrl::Static(url) => base_connect_options(url),
}
}
backend/windmill-common/src/tracing_init.rs
+32 -48
View File
@@ -178,56 +178,40 @@ pub fn initialize_tracing(
.with(logs_bridge.with_filter(otel_logs_filter))
.with(opentelemetry_filtered);
match *JSON_FMT {
true => {
// Stdout layer with its own filter
let stdout_layer = json_layer()
.with_writer(std::io::stdout)
.flatten_event(true)
.with_filter(stdout_env_filter)
.with_filter(create_targets_filter(default_env_filter));
// The service log files are written to be indexed, not tailed, so they always carry the
// JSON format: it is what preserves level, target and the current span as fields rather
// than as text the index would have to recover by regex. JSON_FMT governs stdout only.
let file_layer = json_layer()
.with_writer(log_file_writer)
.flatten_event(true)
.with_filter(file_env_filter)
.with_filter(create_targets_filter(default_env_filter));
// File layer with its own filter
let file_layer = json_layer()
.with_writer(log_file_writer)
.flatten_event(true)
.with_filter(file_env_filter)
.with_filter(create_targets_filter(default_env_filter));
// Boxed so both arms have one type: the file layer is a single value and could not
// otherwise be typed against two different subscriber stacks.
let stdout_layer = match *JSON_FMT {
true => json_layer()
.with_writer(std::io::stdout)
.flatten_event(true)
.with_filter(stdout_env_filter)
.with_filter(create_targets_filter(default_env_filter))
.boxed(),
false => compact_layer()
.with_writer(std::io::stdout)
.with_ansi(style.to_lowercase() != "never")
.with_file(true)
.with_line_number(true)
.with_target(false)
.with_filter(stdout_env_filter)
.with_filter(create_targets_filter(default_env_filter))
.boxed(),
};
base_layer
.with(stdout_layer)
.with(file_layer)
.with(CountingLayer::new())
.init()
}
false => {
// Stdout layer with its own filter
let stdout_layer = compact_layer()
.with_writer(std::io::stdout)
.with_ansi(style.to_lowercase() != "never")
.with_file(true)
.with_line_number(true)
.with_target(false)
.with_filter(stdout_env_filter)
.with_filter(create_targets_filter(default_env_filter));
// File layer with its own filter
let file_layer = compact_layer()
.with_writer(log_file_writer)
.with_ansi(false) // No ANSI codes in log files
.with_file(true)
.with_line_number(true)
.with_target(false)
.with_filter(file_env_filter)
.with_filter(create_targets_filter(default_env_filter));
base_layer
.with(stdout_layer)
.with(file_layer)
.with(CountingLayer::new())
.init()
}
}
base_layer
.with(stdout_layer)
.with(file_layer)
.with(CountingLayer::new())
.init();
(_guard, meter_provider)
}
backend/windmill-common/src/worker.rs
+7 -1
View File
@@ -734,11 +734,17 @@ fn format_pull_query(peek: String) -> String {
r
}
// The `CASE` is `suspend <= 0 OR suspend_until <= now()` written as one indexable
// expression, equivalent only under the `suspend_until IS NOT NULL` guard. It must stay in
// sync with `queue_suspended_v2` (migration 20260826202939): if it no longer matches, the
// test silently reverts to a heap filter over every suspended row on every worker poll.
pub fn make_suspended_pull_query(tags: &[String]) -> String {
format_pull_query(format!(
"SELECT id
FROM v2_job_queue
WHERE suspend_until IS NOT NULL AND (suspend <= 0 OR suspend_until <= now()) AND tag IN ({})
WHERE suspend_until IS NOT NULL
AND (CASE WHEN suspend <= 0 THEN '-infinity'::timestamptz ELSE suspend_until END) <= now()
AND tag IN ({})
ORDER BY priority DESC NULLS LAST, created_at
FOR UPDATE SKIP LOCKED
LIMIT 1",
backend/windmill-common/src/workspaces.rs
+53 -8
View File
@@ -759,15 +759,24 @@ pub async fn count_workspace_forks(db: &crate::DB, root: &str) -> Result<i64> {
Ok(count)
}
/// Approximate paid seats of a workspace as `ceil(developers + operators/2)`, excluding disabled and
/// service-account members. Reuses billing's author/operator weighting, but counts provisioned
/// members rather than the active-user population billing meters, so it only ever loosens the fork
/// cap (never blocks a paid seat) — good enough for a soft guardrail.
/// The billable members of a workspace and the seats they add up to.
#[derive(Clone, Debug, Serialize)]
pub struct BillableSeats {
pub developers: i64,
pub operators: i64,
pub seats: i64,
}
/// Billable members of `w_id` and the seats they cost, as `ceil(developers + operators/2)`. Service
/// accounts cannot log in and do not take a seat; a disabled member is not billed either.
///
/// The workspace is invoiced by a job outside this codebase that counts the same rows with its own
/// SQL. The two must be changed together: this rule disagreeing with that one is what bills a
/// workspace for seats the product never credits it for.
///
/// Unauthenticated metering helper: reads member counts for any `w_id`, so callers must already be
/// authorized for that workspace (or run in trusted server-side code).
#[cfg(feature = "cloud")]
pub async fn count_paid_seats(db: &crate::DB, w_id: &str) -> Result<i64> {
pub async fn billable_seats(db: &crate::DB, w_id: &str) -> Result<BillableSeats> {
let row = sqlx::query!(
r#"SELECT
COUNT(*) FILTER (WHERE NOT operator AND NOT disabled AND NOT is_service_account) AS "developers!",
@@ -777,8 +786,18 @@ pub async fn count_paid_seats(db: &crate::DB, w_id: &str) -> Result<i64> {
)
.fetch_one(db)
.await
.map_err(|e| Error::internal_err(format!("counting paid seats of {w_id}: {e:#}")))?;
Ok(((row.developers as f64) + 0.5 * (row.operators as f64)).ceil() as i64)
.map_err(|e| Error::internal_err(format!("counting billable seats of {w_id}: {e:#}")))?;
Ok(BillableSeats {
developers: row.developers,
operators: row.operators,
seats: ((row.developers as f64) + 0.5 * (row.operators as f64)).ceil() as i64,
})
}
/// Seats only, for the fork cap. See [`billable_seats`].
#[cfg(feature = "cloud")]
pub async fn count_paid_seats(db: &crate::DB, w_id: &str) -> Result<i64> {
Ok(billable_seats(db, w_id).await?.seats)
}
#[cfg(feature = "cloud")]
@@ -2193,6 +2212,32 @@ pub fn lfs_entry_storage_ref(entry: &serde_json::Value) -> Option<String> {
Some(format!("{typ}:{path}"))
}
pub const FILESYSTEM_STORAGE_DEV_ONLY_MSG: &str =
"Filesystem storage is only available in development builds of Windmill: it points the \
workspace at a directory on the server's own disk rather than at a resource. Use an S3, \
Azure Blob or Google Cloud Storage backend instead.";
/// A filesystem workspace storage names a directory on the server's own disk, so it hands whoever
/// configures it — a workspace admin, or any member who can write a `filesystem` resource —
/// whatever the server process can reach, and it only resolves when server and workers share that
/// disk. It is there so local development can skip MinIO, hence debug builds only. Instance object
/// storage on local disk is a separate, superadmin-only setting and stays allowed everywhere.
pub fn filesystem_storage_allowed() -> bool {
cfg!(debug_assertions)
}
/// Guards every site that builds an `ObjectStoreResource::Filesystem`, so nothing downstream can
/// reach a local-disk store: a stored config outlives the build that accepted it, and the resource
/// route never passes through the workspace-storage settings at all.
pub fn ensure_filesystem_storage_allowed() -> Result<()> {
if !filesystem_storage_allowed() {
return Err(Error::BadRequest(
FILESYSTEM_STORAGE_DEV_ONLY_MSG.to_string(),
));
}
Ok(())
}
/// Resolve a `$res:`/`$var:` reference tree to its concrete value (recursively, secrets
/// decrypted). No permission checks — trusted server-side callers only; never echo the result
/// to a user.
backend/windmill-common/tests/billing_workspace.rs
+8 -2
View File
@@ -3,7 +3,7 @@
use sqlx::{Pool, Postgres};
use windmill_common::workspaces::{
count_paid_seats, count_workspace_forks, fork_chain_depth, fork_subtree_height,
billable_seats, count_paid_seats, count_workspace_forks, fork_chain_depth, fork_subtree_height,
get_billing_workspace_id, invalidate_billing_workspace_cache, list_fork_descendants,
};
@@ -106,11 +106,17 @@ async fn paid_seats_and_fork_count(db: Pool<Postgres>) {
insert_member(&db, "seat-root", "dev2@w.dev", false, false, false).await;
insert_member(&db, "seat-root", "op1@w.dev", true, false, false).await;
insert_member(&db, "seat-root", "op2@w.dev", true, false, false).await;
// These must NOT count towards seats.
// These must NOT count towards seats. The service account is a non-operator, so counting it
// would inflate the developer tally the invoice line is written from, not the operator one.
insert_member(&db, "seat-root", "disabled@w.dev", false, true, false).await;
insert_member(&db, "seat-root", "svc@w.dev", false, false, true).await;
assert_eq!(count_paid_seats(&db, "seat-root").await.unwrap(), 3);
let breakdown = billable_seats(&db, "seat-root").await.unwrap();
assert_eq!(
(breakdown.developers, breakdown.operators, breakdown.seats),
(2, 2, 3)
);
insert_ws(&db, "seat-fork1", Some("seat-root"), false).await;
insert_ws(&db, "seat-fork2", Some("seat-root"), false).await;
backend/windmill-indexer/Cargo.toml
+10 -1
View File
@@ -10,7 +10,13 @@ path = "src/lib.rs"
[features]
default = []
parquet = ["windmill-common/parquet", "windmill-object-store/parquet"]
parquet = [
"windmill-common/parquet",
"windmill-object-store/parquet",
"dep:datafusion",
"dep:object_store",
"dep:url",
]
private = ["windmill-common/private"]
enterprise = ["windmill-common/enterprise", "windmill-object-store/enterprise"]
@@ -33,3 +39,6 @@ astral-tokio-tar.workspace = true
lazy_static.workspace = true
const_format.workspace = true
flume.workspace = true
datafusion = { workspace = true, optional = true }
object_store = { workspace = true, optional = true }
url = { workspace = true, optional = true }
backend/windmill-indexer/src/lib.rs
+2
View File
@@ -7,3 +7,5 @@ pub mod indexer_oss;
#[cfg(feature = "private")]
pub mod service_logs_ee;
pub mod service_logs_oss;
#[cfg(all(feature = "private", feature = "parquet"))]
pub mod service_logs_store_ee;
backend/windmill-object-store/src/lib.rs
+1
View File
@@ -1171,6 +1171,7 @@ pub fn lfs_to_object_store_resource(
Ok(ObjectStoreResource::Gcs(gcs_resource))
}
LargeFileStorage::FilesystemStorage(fs) => {
windmill_common::workspaces::ensure_filesystem_storage_allowed()?;
Ok(ObjectStoreResource::Filesystem(FilesystemSettings {
root_path: fs.root_path.clone(),
}))
backend/windmill-queue/src/jobs.rs
+6
View File
@@ -4671,6 +4671,12 @@ pub async fn check_debouncing_within_limits(
}
}
/// Whether the tag's queue name is computed from the job's arguments, so that a caller holding
/// arguments it could not build knows the tag cannot be built either.
pub fn tag_reads_args(tag: &str) -> bool {
RE_ARG_TAG.is_match(tag)
}
pub fn interpolate_args(x: String, args: &PushArgs, workspace_id: &str) -> String {
// Save this value to avoid parsing twice
let workspaced = x.as_str().replace("$workspace", workspace_id).to_string();
backend/windmill-store/src/resources.rs
+38
View File
@@ -130,6 +130,12 @@ pub struct EditResourceType {
pub schema: Option<serde_json::Value>,
pub description: Option<String>,
pub is_fileset: Option<bool>,
/// Doubly optional so an edit can distinguish the two things a plain
/// `Option` conflates: an absent field leaves the extension alone, while an
/// explicit `null` clears it. A hub pull relies on both — a type that stops
/// being a file type has to stop being one locally too.
#[serde(default, deserialize_with = "windmill_common::more_serde::double_option")]
pub format_extension: Option<Option<String>>,
}
#[derive(FromRow, Serialize, Deserialize)]
@@ -2803,10 +2809,42 @@ async fn update_resource_type(
if let Some(is_fileset) = ns.is_fileset {
sqlb.set("is_fileset", if is_fileset { "TRUE" } else { "FALSE" });
}
if let Some(format_extension) = ns.format_extension.clone() {
match format_extension {
Some(ext) => sqlb.set_str("format_extension", ext),
None => sqlb.set("format_extension", "NULL"),
};
}
sqlb.set_str("edited_at", "now()");
let sql = sqlb.sql().map_err(|e| Error::internal_err(e.to_string()))?;
let mut tx = user_db.begin(&authed).await?;
// Creation refuses the pair outright, so an edit must too — otherwise the same
// impossible type (a set of files that is also one file) is reachable by setting
// either half on an existing row. Whichever half the request omits is read from
// the row being edited, inside this transaction and with the row locked: read
// outside it, two concurrent edits each supplying one half would both pass.
let current = sqlx::query!(
"SELECT is_fileset, format_extension FROM resource_type
WHERE name = $1 AND workspace_id = $2 FOR UPDATE",
&name,
&w_id
)
.fetch_optional(&mut *tx)
.await?;
let effective_is_fileset = ns
.is_fileset
.unwrap_or_else(|| current.as_ref().map(|c| c.is_fileset).unwrap_or(false));
let effective_format_extension = match &ns.format_extension {
Some(value) => value.clone(),
None => current.and_then(|c| c.format_extension),
};
if effective_is_fileset && effective_format_extension.is_some() {
return Err(Error::BadRequest(
"A fileset resource type cannot have a format_extension".to_string(),
));
}
sqlx::query(&sql).execute(&mut *tx).await?;
audit_log(
&mut *tx,
backend/windmill-trigger-http/src/handler.rs
+13 -4
View File
@@ -268,6 +268,15 @@ pub async fn create_many_http_triggers(
format!("http_triggers:write:{}", &new_http_trigger.base.path)
})?;
// This route inserts directly, bypassing the shared create handler.
// `error_wrapper` would turn the rejection into a 500.
new_http_trigger.error_handling.validate().map_err(|err| {
Error::BadRequest(format!(
"Error occurred for HTTP route at route path: {}, error: {}",
new_http_trigger.config.route_path, err
))
})?;
handler
.validate_new(&db, &w_id, &new_http_trigger.config)
.await
@@ -570,8 +579,8 @@ impl TriggerCrud for HttpTrigger {
route_path,
&route_path_key,
Some(effective_workspaced),
trigger.config.wrap_body,
trigger.config.raw_string,
trigger.config.wrap_body.unwrap_or(false),
trigger.config.raw_string.unwrap_or(false),
trigger.config.authentication_resource_path,
trigger.base.script_path,
trigger.base.path,
@@ -626,8 +635,8 @@ impl TriggerCrud for HttpTrigger {
workspace_id = $20 AND
path = $21
"#,
trigger.config.wrap_body,
trigger.config.raw_string,
trigger.config.wrap_body.unwrap_or(false),
trigger.config.raw_string.unwrap_or(false),
trigger.config.authentication_resource_path,
trigger.base.script_path,
trigger.base.path,
backend/windmill-trigger/src/handler.rs
+4
View File
@@ -554,6 +554,8 @@ async fn create_trigger<T: TriggerCrud>(
)));
}
new_trigger.error_handling.validate()?;
handler
.validate_new(&db, &workspace_id, &new_trigger.config)
.await?;
@@ -815,6 +817,8 @@ async fn update_trigger<T: TriggerCrud>(
)
})?;
edit_trigger.error_handling.validate()?;
handler
.validate_edit(&db, &workspace_id, &edit_trigger.config, path)
.await?;
backend/windmill-trigger/src/types.rs
+62
View File
@@ -84,6 +84,31 @@ pub struct TriggerErrorHandling {
pub retry: Option<sqlx::types::Json<windmill_common::flows::Retry>>,
}
impl TriggerErrorHandling {
/// Schedule and workspace error handlers encode script-vs-flow as a
/// `script/`/`flow/` prefix; a trigger's handler is always a script, so a
/// prefixed path here would be looked up verbatim as a script name and fail
/// only once the trigger errors, which is when the handler is needed.
pub fn validate(&self) -> windmill_common::error::Result<()> {
let Some(path) = self.error_handler_path.as_deref() else {
return Ok(());
};
if let Some(bare) = path.strip_prefix("script/") {
return Err(windmill_common::error::Error::BadRequest(format!(
"error_handler_path is a plain script path, not the prefixed form a schedule \
error handler takes: got '{path}', use '{bare}'"
)));
}
if path.starts_with("flow/") {
return Err(windmill_common::error::Error::BadRequest(format!(
"error_handler_path must be a script: a trigger error handler cannot be a flow \
(got '{path}')"
)));
}
Ok(())
}
}
#[derive(Serialize, Deserialize, Clone)]
pub struct Trigger<T>
where
@@ -253,6 +278,43 @@ mod tests {
use super::*;
use serde_json::json;
// --- TriggerErrorHandling::validate ---
fn error_handling(path: Option<&str>) -> TriggerErrorHandling {
TriggerErrorHandling {
error_handler_path: path.map(str::to_string),
error_handler_args: None,
retry: None,
}
}
#[test]
fn test_error_handler_path_accepts_bare_and_hub_paths() {
for path in [None, Some("f/team/handler"), Some("u/admin/handler")] {
assert!(error_handling(path).validate().is_ok(), "{path:?}");
}
assert!(error_handling(Some("hub/13953/windmill/handler"))
.validate()
.is_ok());
}
#[test]
fn test_error_handler_path_rejects_prefixed_paths() {
// The rejection names the bare path to use, so the caller can fix it
// without knowing which of the two conventions a trigger follows.
let err = error_handling(Some("script/f/team/handler"))
.validate()
.unwrap_err()
.to_string();
assert!(err.contains("f/team/handler"), "{err}");
let err = error_handling(Some("flow/f/team/handler"))
.validate()
.unwrap_err()
.to_string();
assert!(err.contains("cannot be a flow"), "{err}");
}
// --- TriggerMode serde ---
#[test]
backend/windmill-types/src/jobs.rs
+22
View File
@@ -632,6 +632,28 @@ pub enum JobPayload {
},
}
impl JobPayload {
/// Whether the payload itself declares a dedicated worker, in which case `push` replaces
/// whatever tag it is handed and the caller's tag never reaches the queue.
///
/// This reads what the payload carries, not what `push` will conclude: a `SingleStepFlow`
/// loads the flag from the script row at push time and reports `false` here. That only
/// matters to a caller reasoning about the tag, and for those the answer is the same either
/// way, since `push` replaces the tag in exactly the case this misses.
pub fn is_dedicated_worker(&self) -> bool {
let dedicated_worker = match self {
JobPayload::ScriptHash { dedicated_worker, .. }
| JobPayload::FlowScript { dedicated_worker, .. }
| JobPayload::Dependencies { dedicated_worker, .. }
| JobPayload::FlowDependencies { dedicated_worker, .. }
| JobPayload::Flow { dedicated_worker, .. } => dedicated_worker,
JobPayload::Code(raw) => &raw.dedicated_worker,
_ => &None,
};
dedicated_worker.is_some_and(|x| x)
}
}
#[derive(Clone, Serialize, Deserialize, Debug)]
pub struct SkipHandler {
pub path: String,
backend/windmill-types/src/more_serde.rs
+19
View File
@@ -66,3 +66,22 @@ where
NumericOrNull::Null => Ok(None),
}
}
/// Deserializer for a doubly-optional field, so a struct can tell an absent key
/// (`None`) from an explicit `null` (`Some(None)`).
///
/// Plain serde collapses both into the outer `None`, which makes the distinction
/// unusable exactly where it matters: a payload that omits a field means "leave it
/// alone", while one that sends `null` means "clear it".
///
/// ```ignore
/// #[serde(default, deserialize_with = "double_option", skip_serializing_if = "Option::is_none")]
/// pub field: Option<Option<String>>,
/// ```
pub fn double_option<'de, T, D>(deserializer: D) -> Result<Option<Option<T>>, D::Error>
where
T: serde::Deserialize<'de>,
D: serde::Deserializer<'de>,
{
serde::Deserialize::deserialize(deserializer).map(Some)
}
backend/windmill-worker/src/bun_executor.rs
+55 -4
View File
@@ -1256,6 +1256,7 @@ pub async fn prebundle_bun_script(
token: &str,
occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
temp_script_refs: &Option<HashMap<String, String>>,
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> Result<()> {
let (local_path, remote_path) = compute_bundle_local_and_remote_path(
inner_content,
@@ -1264,6 +1265,7 @@ pub async fn prebundle_bun_script(
db,
w_id,
temp_script_refs,
modules,
)
.await;
if exists_in_cache(&local_path, &remote_path).await {
@@ -1442,6 +1444,7 @@ pub async fn compute_bundle_local_and_remote_path(
db: Option<&DB>,
w_id: &str,
temp_script_refs: &Option<HashMap<String, String>>,
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> (String, String) {
let mut input_src = format!("{inner_content}{lock}",);
@@ -1470,7 +1473,10 @@ pub async fn compute_bundle_local_and_remote_path(
let ws_suffix = crate::workspace_registry_cache_suffix(w_id).await;
input_src.push_str(&ws_suffix);
let hash = windmill_common::utils::calculate_hash(&input_src);
// The loader resolves relative imports against the module files in the job dir, so
// their content is inlined into the bundle this name covers.
let hash = crate::worker::artifact_cache_name(input_src, modules);
let local_path = format!("{}/{hash}", *BUN_BUNDLE_CACHE_DIR);
#[cfg(windows)]
@@ -1569,6 +1575,7 @@ pub async fn handle_bun_job(
Some(db),
&job.workspace_id,
&temp_script_refs,
modules.as_ref(),
)
.await
}
@@ -2962,9 +2969,8 @@ pub async fn handle_wac_v2_output(
version: flow_info.version,
labels: flow_info.labels.clone(),
};
let on_behalf_of = flow_info
.on_behalf_of(&job.workspace_id, db)
.await?;
let on_behalf_of =
flow_info.on_behalf_of(&job.workspace_id, db).await?;
let step_args: HashMap<String, Box<RawValue>> = step
.args
.iter()
@@ -4382,4 +4388,49 @@ export function main(x: number) { return x; }"#;
assert!(wrapper.contains(r#"line.startsWith("exec_preprocess:")"#));
assert!(wrapper.contains(r#"line.startsWith("exec:")"#));
}
/// The bundle cache is global and content-keyed, so a key that ignores the inline
/// modules hands one workspace's bundle — attacker helper code and all — to the next
/// job whose main content and lockfile happen to match.
#[tokio::test]
async fn bundle_cache_key_separates_inline_module_content() {
use windmill_common::scripts::ScriptModule;
async fn key_for(modules: Option<&HashMap<String, ScriptModule>>) -> String {
compute_bundle_local_and_remote_path(
"import { h } from './helper.ts';\nexport async function main() { return h(); }",
"{}\n//bun.lock\n<empty>",
"u/alice/script",
None,
"w1",
&None,
modules,
)
.await
.1
}
fn modules(content: &str) -> HashMap<String, ScriptModule> {
HashMap::from([(
"helper.ts".to_string(),
ScriptModule {
content: content.to_string(),
language: ScriptLang::Bun,
lock: None,
},
)])
}
let attacker = key_for(Some(&modules("export const h = () => 'attacker'"))).await;
let victim = key_for(Some(&modules("export const h = () => 'victim'"))).await;
assert_ne!(attacker, victim);
assert_eq!(
attacker,
key_for(Some(&modules("export const h = () => 'attacker'"))).await,
"same modules must still share a cache slot"
);
// An absent map and an empty one are the same script, so they share a slot.
assert_eq!(key_for(None).await, key_for(Some(&HashMap::new())).await);
assert_ne!(key_for(None).await, attacker);
}
}
backend/windmill-worker/src/common.rs
+1
View File
@@ -1640,6 +1640,7 @@ pub(crate) async fn get_workspace_s3_resource_path(
)
}
Some(LargeFileStorage::FilesystemStorage(fs)) => {
windmill_common::workspaces::ensure_filesystem_storage_allowed()?;
return Ok(Some(
windmill_object_store::ObjectStoreResource::Filesystem(
windmill_object_store::FilesystemSettings { root_path: fs.root_path.clone() },
backend/windmill-worker/src/csharp_executor.rs
+17 -5
View File
@@ -13,7 +13,7 @@ use itertools::Itertools;
#[cfg(feature = "csharp")]
use tokio::{fs::File, io::AsyncReadExt, process::Command};
#[cfg(feature = "csharp")]
use windmill_common::{utils::calculate_hash, worker::write_file};
use windmill_common::worker::write_file;
#[cfg(feature = "csharp")]
use crate::global_cache::save_cache;
@@ -72,13 +72,21 @@ const CSHARP_OBJECT_STORE_PREFIX: &str =
/// Cache key of a C# build. The run path and the deploy-time prebuild must derive it the
/// same way or the prebuilt binary is never found and gets rebuilt on first run.
#[cfg(feature = "csharp")]
async fn csharp_cache_key(code: &str, requirements_o: Option<&str>, w_id: &str) -> String {
let mut hash = calculate_hash(&format!(
async fn csharp_cache_key(
code: &str,
requirements_o: Option<&str>,
w_id: &str,
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> String {
// The SDK project globs every `.cs` under the job dir, so companion modules are
// compiled into the binary this key names and have to be part of it.
let base = format!(
"{}{}{}",
code,
requirements_o.unwrap_or(""),
DOTNET_TARGET_FRAMEWORK.as_str()
));
);
let mut hash = crate::worker::artifact_cache_name(base, modules);
hash.push_str(&crate::workspace_registry_cache_suffix(w_id).await);
hash
}
@@ -487,10 +495,11 @@ pub async fn prebuild_csharp_binary(
worker_name: &str,
base_internal_url: &str,
occupancy_metrics: &mut OccupancyMetrics,
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> error::Result<Option<String>> {
check_executor_binary_exists("dotnet", DOTNET_PATH.as_str(), "C#")?;
let hash = csharp_cache_key(code, Some(lock), &job.workspace_id).await;
let hash = csharp_cache_key(code, Some(lock), &job.workspace_id, modules).await;
let remote_path = format!("{CSHARP_OBJECT_STORE_PREFIX}{hash}");
if crate::global_cache::exists_in_object_store(&remote_path).await {
return Ok(None);
@@ -549,6 +558,7 @@ pub async fn handle_csharp_job(
_worker_name: &str,
_envs: HashMap<String, String>,
_occupancy_metrics: &mut OccupancyMetrics,
_modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> Result<Box<RawValue>, Error> {
Err(anyhow!("C# is not available because the feature is not enabled").into())
}
@@ -568,6 +578,7 @@ pub async fn handle_csharp_job(
worker_name: &str,
envs: HashMap<String, String>,
occupancy_metrics: &mut OccupancyMetrics,
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> Result<Box<RawValue>, Error> {
check_executor_binary_exists("dotnet", DOTNET_PATH.as_str(), "C#")?;
@@ -575,6 +586,7 @@ pub async fn handle_csharp_job(
inner_content,
requirements_o.map(|x| x.as_str()),
&job.workspace_id,
modules,
)
.await;
let bin_path = format!("{}/{hash}", *CSHARP_CACHE_DIR);
backend/windmill-worker/src/go_executor.rs
+13 -4
View File
@@ -230,8 +230,15 @@ fn go_runtime_int32(v: &str) -> Option<i32> {
/// Cache key of a Go build. The run path and the deploy-time prebuild must derive it the
/// same way or the prebuilt binary is never found and gets rebuilt on first run.
fn go_cache_key(code: &str, maybe_lock: &MaybeLock) -> String {
calculate_hash(&format!("{}{:?}v2", code, maybe_lock))
fn go_cache_key(
code: &str,
maybe_lock: &MaybeLock,
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> String {
// A module whose path starts with `go/` lands inside the module dir this builds, so
// its content ends up in the binary the key names.
let base = format!("{}{:?}v2", code, maybe_lock);
crate::worker::artifact_cache_name(base, modules)
}
/// Install the deps, generate the entrypoint wrapper, `go build`, and push the binary to
@@ -459,9 +466,10 @@ pub async fn prebuild_go_binary(
worker_name: &str,
base_internal_url: &str,
occupancy_metrics: &mut OccupancyMetrics,
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> Result<Option<String>, Error> {
let maybe_lock = MaybeLock::Resolved { lock: lock.to_string() };
let hash = go_cache_key(code, &maybe_lock);
let hash = go_cache_key(code, &maybe_lock, modules);
let remote_path = format!("{GO_OBJECT_STORE_PREFIX}{hash}");
if crate::global_cache::exists_in_object_store(&remote_path).await {
return Ok(None);
@@ -512,6 +520,7 @@ pub async fn handle_go_job(
envs: HashMap<String, String>,
occupation_metrics: &mut OccupancyMetrics,
maybe_lock: MaybeLock,
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> Result<Box<RawValue>, Error> {
//go does not like executing modules at temp root
let job_dir = &format!("{job_dir}/go");
@@ -520,7 +529,7 @@ pub async fn handle_go_job(
.create(&job_dir)
.expect("could not create go job dir");
let hash = go_cache_key(inner_content, &maybe_lock);
let hash = go_cache_key(inner_content, &maybe_lock, modules);
let bin_path = format!("{}/{hash}", *GO_BIN_CACHE_DIR);
let remote_path = format!("{GO_OBJECT_STORE_PREFIX}{hash}");
let (cache, cache_logs) = crate::global_cache::load_cache(&bin_path, &remote_path, false).await;
backend/windmill-worker/src/java_executor.rs
+13 -4
View File
@@ -66,6 +66,7 @@ pub(crate) struct JobHandlerInput<'a> {
pub requirements_o: Option<&'a String>,
pub shared_mount: &'a str,
pub worker_name: &'a str,
pub modules: Option<&'a HashMap<String, windmill_common::scripts::ScriptModule>>,
}
pub async fn handle_java_job<'a>(mut args: JobHandlerInput<'a>) -> Result<Box<RawValue>, Error> {
@@ -612,25 +613,33 @@ async fn compile<'a>(
inner_content,
requirements_o,
parent_runnable_path,
modules,
..
}: &mut JobHandlerInput<'a>,
classpath: &'a str,
// plugins: Vec<&'a str>,
) -> Result<(), Error> {
fn compute_hash(code: &str, requirements_o: Option<&String>) -> String {
calculate_hash(&format!(
// The cached artifact is the whole `target/` dir, and companion modules are written
// into the job dir before this runs, so their content can land in it.
fn compute_hash(
code: &str,
requirements_o: Option<&String>,
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> String {
let base = format!(
"{}{}",
code,
requirements_o
.as_ref()
.map(|x| x.to_string())
.unwrap_or_default()
))
);
crate::worker::artifact_cache_name(base, modules)
}
let reserved_variables =
get_reserved_variables(job, &client.token, conn, parent_runnable_path.clone()).await?;
let ws_suffix = crate::workspace_registry_cache_suffix(&job.workspace_id).await;
let mut hash = compute_hash(inner_content, *requirements_o);
let mut hash = compute_hash(inner_content, *requirements_o, *modules);
hash.push_str(&ws_suffix);
let bin_path = format!("{}/{hash}", *JAVA_CACHE_DIR);
let remote_path = format!("java_jar/{hash}");

Some files were not shown because too many files have changed in this diff Show More