starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-09-07 16:03:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into glm/improve-multiselect

Browse Source 
# Conflicts:
#	frontend/src/lib/components/common/table/RawAppRow.svelte
#	frontend/src/lib/components/home/ItemsList.svelte
#	frontend/src/lib/components/home/TreeView.svelte
This commit is contained in:
Guilhem Lemouel
2026-09-03 15:07:48 +02:00
parent 14e25c5b7c 38fc0d3a12
commit 96cee83914
429 changed files with 24026 additions and 7157 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/backend-test-windows.yml
-1
View File
@@ -101,7 +101,6 @@ jobs:
- name: Install OpenSSL via vcpkg
run: |
vcpkg.exe install openssl-windows:x64-windows
vcpkg.exe install openssl:x64-windows-static
vcpkg.exe integrate install
.github/workflows/build_windows_worker_.yml
-1
View File
@@ -62,7 +62,6 @@ jobs:
- name: Cargo build binary windows
timeout-minutes: 180
run: |
vcpkg.exe install openssl-windows:x64-windows
vcpkg.exe install openssl:x64-windows-static
vcpkg.exe integrate install
$env:VCPKGRS_DYNAMIC=1
.github/workflows/git-sync-test.yml
+3 -1
View File
@@ -9,6 +9,7 @@ on:
- "backend/windmill-api-integration-tests/tests/git_sync*"
- "backend/ee-repo-ref.txt"
- "backend/windmill-common/src/workspaces.rs"
- "frontend/src/lib/hubPaths.json"
- "backend/windmill-worker/src/result_processor.rs"
- "backend/windmill-api-workspaces/**"
- "cli/src/commands/sync/**"
@@ -22,6 +23,7 @@ on:
- "backend/windmill-api-integration-tests/tests/git_sync*"
- "backend/ee-repo-ref.txt"
- "backend/windmill-common/src/workspaces.rs"
- "frontend/src/lib/hubPaths.json"
- "backend/windmill-worker/src/result_processor.rs"
- "backend/windmill-api-workspaces/**"
- "cli/src/commands/sync/**"
@@ -59,7 +61,7 @@ jobs:
echo "$CHANGED_FILES"
# Direct git sync file changes — always relevant.
if echo "$CHANGED_FILES" | grep -qE '^(backend/windmill-git-sync/|backend/windmill-worker/src/result_processor\.rs|backend/windmill-api-workspaces/|backend/windmill-api-integration-tests/tests/git_sync|backend/windmill-common/src/workspaces\.rs|cli/src/commands/sync/|cli/src/utils/git\.ts|integration_tests/test/git_sync|\.github/workflows/git-sync-test\.yml)'; then
if echo "$CHANGED_FILES" | grep -qE '^(backend/windmill-git-sync/|backend/windmill-worker/src/result_processor\.rs|backend/windmill-api-workspaces/|backend/windmill-api-integration-tests/tests/git_sync|backend/windmill-common/src/workspaces\.rs|frontend/src/lib/hubPaths\.json|cli/src/commands/sync/|cli/src/utils/git\.ts|integration_tests/test/git_sync|\.github/workflows/git-sync-test\.yml)'; then
echo "should_run=true" >> "$GITHUB_OUTPUT"
echo "Relevant: direct git sync file changes"
exit 0
.github/workflows/publish_windows_worker.yml
-1
View File
@@ -51,7 +51,6 @@ jobs:
- name: Cargo build windows
timeout-minutes: 180
run: |
vcpkg.exe install openssl-windows:x64-windows
vcpkg.exe install openssl:x64-windows-static
vcpkg.exe integrate install
$env:VCPKGRS_DYNAMIC=1
.release-please-manifest.json
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "1.796.0"
".": "1.803.0"
}
AGENTS.md
+9
View File
@@ -165,6 +165,15 @@ $NAV --root backend callees "X" # what does X call?
- Search for existing code to reuse before writing new code
- Follow established patterns in the codebase
- Keep changes focused — don't refactor beyond what's asked
- **A simpler design found late is still the design.** Work already spent is not an argument
for a shape, and neither is a clean review round, a passing suite, or a long PR thread. The
signal to stop and re-derive rather than patch again is a change that keeps growing to defend
its own structure: each review finding fixing an assumption the previous fix broke, the same
class of bug reappearing somewhere new, or most of the diff being consequences of one early
choice rather than the thing you set out to do. When that happens, say plainly what the
simpler design is and what switching costs — a migration, a review cycle restarted from zero,
work discarded — and let the user decide. Do not keep paying down the harder one because it
is nearly finished, and do not present the accumulated cost as a reason to continue.
- **Ship only the tests the PR needs.** A committed test must pin behavior a future change could plausibly break, and be the smallest setup that exercises the new logic. While developing, write as many exhaustive tests and do as much manual testing as you need to convince yourself the change works — then remove that scaffolding before marking the PR ready, keeping only the essential regression guard(s). A test that merely re-exercises pre-existing behavior, or needs elaborate fixtures to assert something trivial, is scaffolding: delete it. If nothing meaningful is left to guard, ship no test rather than a ceremonial one.
- **Comments record constraints, not narration.** Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's "simplified" away. State each invariant once, at the place where someone would break it, in ≤4 lines. Don't describe what the next line does, don't repeat the same rationale at multiple sites, and don't address the PR reviewer (justifying a change belongs in the PR description, not the code). Reference nothing ephemeral — no numbered steps from your dev flow, no "the poller / the test does X" scaffolding, no transient state that won't exist for the next reader; keep only the essential, durable rationale. Describe the code as it is, never its drafting history: "we no longer do X", "unchanged behavior", "instead of the previous approach" are meaningless to a reader who never saw the earlier iteration — before finishing, reread your comments as if the current state is the only state that ever existed.
- **Never attribute work to a specific customer, account, or "requested by a customer" in repo-tracked content** (PR descriptions, commit messages, code comments, docs). Describe changes by their technical motivation instead.
CHANGELOG.md
+152
View File
@@ -1,5 +1,157 @@
# Changelog
## [1.803.0](https://github.com/windmill-labs/windmill/compare/v1.802.0...v1.803.0) (2026-09-03)
### Features
* expose request headers to scripts invoked via MCP ([#10903](https://github.com/windmill-labs/windmill/issues/10903)) ([e474e88](https://github.com/windmill-labs/windmill/commit/e474e8803ce2ff5c2df09a58dab51d45f5c922ca))
* reuse an existing workspace resource in the project import wizard ([#10935](https://github.com/windmill-labs/windmill/issues/10935)) ([582761e](https://github.com/windmill-labs/windmill/commit/582761e37c776e92dc1c6ebfee8c4efe7c35d822))
### Bug Fixes
* bump git sync hub scripts to cli 1.802.1, test the fork ui pull ([#10955](https://github.com/windmill-labs/windmill/issues/10955)) ([ca88009](https://github.com/windmill-labs/windmill/commit/ca8800959aa6a0017cc29bad187c9f49e0d13cc4))
* **cli:** make a sync push into a fork converge on schedules and inline names ([#10951](https://github.com/windmill-labs/windmill/issues/10951)) ([0f5a1db](https://github.com/windmill-labs/windmill/commit/0f5a1db2abba8df30a2f975f4498e269f13cf93d))
* fade the home Build with AI placeholder every 10s instead of typing it ([#10953](https://github.com/windmill-labs/windmill/issues/10953)) ([3d089b5](https://github.com/windmill-labs/windmill/commit/3d089b57344f5814086e6176301c5031dc519674))
* let operators use wmill.datatable() from within running jobs ([#10931](https://github.com/windmill-labs/windmill/issues/10931)) ([9b64a89](https://github.com/windmill-labs/windmill/commit/9b64a89cd46ae718d6c58fa12f925fa041fb1032))
## [1.802.0](https://github.com/windmill-labs/windmill/compare/v1.801.0...v1.802.0) (2026-09-02)
### Features
* add retention cleanup for the otel_traces table ([#10949](https://github.com/windmill-labs/windmill/issues/10949)) ([d472193](https://github.com/windmill-labs/windmill/commit/d472193e5bf5f6428e0096a402eb2c9299634fb2))
* open path links from chat messages in the session preview panel ([#10881](https://github.com/windmill-labs/windmill/issues/10881)) ([f10ac6c](https://github.com/windmill-labs/windmill/commit/f10ac6c2b3644fb16697e650efbc4f7cd3c6944c))
* restore owner and label filter chips on the homepage ([#10942](https://github.com/windmill-labs/windmill/issues/10942)) ([ccf8476](https://github.com/windmill-labs/windmill/commit/ccf84761dd664b9228dfe2f65867e8c32cd20c21))
* **sessions:** offer the item you came from when starting a new session ([#10940](https://github.com/windmill-labs/windmill/issues/10940)) ([d3747d6](https://github.com/windmill-labs/windmill/commit/d3747d62555ebcb09c78cfabcaa3b6177758d6ea))
* workspace setting to hide the AI assistant, agent steps unaffected ([#10941](https://github.com/windmill-labs/windmill/issues/10941)) ([fdd3b36](https://github.com/windmill-labs/windmill/commit/fdd3b36423344a2e1a464674179406581074e926))
### Bug Fixes
* apply object-storage test SSRF validation to all non-super-admins ([#10933](https://github.com/windmill-labs/windmill/issues/10933)) ([4fef119](https://github.com/windmill-labs/windmill/commit/4fef1195adaa9fa036a219884bd6c996460ca37f))
* connect to dev server instead of localhost ([#10912](https://github.com/windmill-labs/windmill/issues/10912)) ([337154b](https://github.com/windmill-labs/windmill/commit/337154b8304a5969f35216add627b5c1153c0f6c))
* preselect first row of AI agent and AI sandbox insert panes ([#10937](https://github.com/windmill-labs/windmill/issues/10937)) ([95b6bbd](https://github.com/windmill-labs/windmill/commit/95b6bbd46ada11d96a914ae5b0e92aba4dd02530))
* record supplied script lock hashes so importers can skip relocking ([#10915](https://github.com/windmill-labs/windmill/issues/10915)) ([17ba521](https://github.com/windmill-labs/windmill/commit/17ba521c352aec65a8270893752bbadd7f3d6eaa))
* sandbox script-controlled content types in result_to_response ([#10932](https://github.com/windmill-labs/windmill/issues/10932)) ([419741e](https://github.com/windmill-labs/windmill/commit/419741e5d226c67c51429094fb6ded9474afed99))
## [1.801.0](https://github.com/windmill-labs/windmill/compare/v1.800.1...v1.801.0) (2026-09-01)
### Features
* **ai-chat:** make reusable skills ai_skill resources you select per workspace ([#10914](https://github.com/windmill-labs/windmill/issues/10914)) ([cfcfe29](https://github.com/windmill-labs/windmill/commit/cfcfe298dd9ab50196bd64926ef78c4563f58c2c))
* **ai-sessions:** show a running session across tabs and reload finished turns ([#10916](https://github.com/windmill-labs/windmill/issues/10916)) ([816dc9d](https://github.com/windmill-labs/windmill/commit/816dc9dcd2c310e499d2d210a0abcd403469f29c))
* edit folders and groups in a drawer that saves once ([#10873](https://github.com/windmill-labs/windmill/issues/10873)) ([5d5ad4e](https://github.com/windmill-labs/windmill/commit/5d5ad4e8974e076ef53a26a5584e4209255a2248))
* make the home Build with AI composer dismissible, quiet the rest of the home page ([#10930](https://github.com/windmill-labs/windmill/issues/10930)) ([772fafe](https://github.com/windmill-labs/windmill/commit/772fafec8316a1e0c0e76b9a0737cc41d40a9a8c))
### Bug Fixes
* let a principal without a login account own a draft ([#10925](https://github.com/windmill-labs/windmill/issues/10925)) ([94af8d0](https://github.com/windmill-labs/windmill/commit/94af8d0fb5aceebe83936fd6761c6c1c02c75323))
* resolve chat path links against the session's operating workspace ([#10924](https://github.com/windmill-labs/windmill/issues/10924)) ([9074de2](https://github.com/windmill-labs/windmill/commit/9074de25ea730ca02653c9a2e2b8b99eda6f3137))
* tolerate string app_id in GHES app config deserialization ([#10923](https://github.com/windmill-labs/windmill/issues/10923)) ([af8ff38](https://github.com/windmill-labs/windmill/commit/af8ff3868748412cb658c803ebc8a71edc3cd8fb))
## [1.800.1](https://github.com/windmill-labs/windmill/compare/v1.800.0...v1.800.1) (2026-09-01)
### Bug Fixes
* add top margin to the home Build with AI section ([#10909](https://github.com/windmill-labs/windmill/issues/10909)) ([bedf5ae](https://github.com/windmill-labs/windmill/commit/bedf5ae57445025729e94e16f1b5f13f6ff38ffa))
* **ai-chat:** consume an @ mention with the message that carried it ([#10907](https://github.com/windmill-labs/windmill/issues/10907)) ([c512110](https://github.com/windmill-labs/windmill/commit/c512110a1f8d0d3437c20048f6446ef62b10222c))
* keep a local dbt descriptor under sync pull --keep-deleted ([#10911](https://github.com/windmill-labs/windmill/issues/10911)) ([4b5be38](https://github.com/windmill-labs/windmill/commit/4b5be386ce0f851a087f43c0b0ac6e4b1b055a47))
* keep windmill-indexer out of builds without tantivy ([#10908](https://github.com/windmill-labs/windmill/issues/10908)) ([db0f004](https://github.com/windmill-labs/windmill/commit/db0f004613e3f90428fea4c824cc53f1b2fc03b0))
## [1.800.0](https://github.com/windmill-labs/windmill/compare/v1.799.0...v1.800.0) (2026-08-31)
### Features
* add --keep-deleted flag to wmill sync pull and push ([#10878](https://github.com/windmill-labs/windmill/issues/10878)) ([66123f3](https://github.com/windmill-labs/windmill/commit/66123f3a9b8978c0084b02f50b44cffba125a13a))
* day-partition the service log index and expire whole chunks ([#10893](https://github.com/windmill-labs/windmill/issues/10893)) ([d91ee46](https://github.com/windmill-labs/windmill/commit/d91ee4614a70f20a194f47e190327129f499ec63))
* free AI tokens + home search/filter revamp ([#10020](https://github.com/windmill-labs/windmill/issues/10020)) ([716ce2e](https://github.com/windmill-labs/windmill/commit/716ce2ece00cd5cfb8641afc6432636bc4aa46e9))
* make the service log retention period an instance setting ([#10889](https://github.com/windmill-labs/windmill/issues/10889)) ([815de49](https://github.com/windmill-labs/windmill/commit/815de49e2322f85ca92b1e41a2bcd22591ebe93f))
* rework the evals dataset drawer and run navigation ([#10884](https://github.com/windmill-labs/windmill/issues/10884)) ([1462f17](https://github.com/windmill-labs/windmill/commit/1462f17643302127b4bd76bacfde80cc03f9d606))
* serve service log context from parquet and retire the raw log files ([#10892](https://github.com/windmill-labs/windmill/issues/10892)) ([338d75c](https://github.com/windmill-labs/windmill/commit/338d75cc5227e352cb84828c99bfd3b984cf0fa5))
* serve service log retrieval from a columnar parquet store ([#10886](https://github.com/windmill-labs/windmill/issues/10886)) ([7c1a785](https://github.com/windmill-labs/windmill/commit/7c1a785f756ed27e4425f6534709b19971a73a97))
### Bug Fixes
* correct the service log ingest flush boundary ([#10898](https://github.com/windmill-labs/windmill/issues/10898)) ([ac56586](https://github.com/windmill-labs/windmill/commit/ac56586c0e56d4022761d3c80306a03d57f8bfcb))
* harden the service log indexer's recovery and read paths ([#10904](https://github.com/windmill-labs/windmill/issues/10904)) ([831370c](https://github.com/windmill-labs/windmill/commit/831370cdde8e06f4298b17baa1a0041bacdd98c7))
* keep raw-app editor selection consistent across sidebar and tabs ([#10885](https://github.com/windmill-labs/windmill/issues/10885)) ([b57e231](https://github.com/windmill-labs/windmill/commit/b57e231c2bf5e5fe007f0aa7b958a51e32b47141))
* register every rotated service log file exactly once ([#10891](https://github.com/windmill-labs/windmill/issues/10891)) ([c817248](https://github.com/windmill-labs/windmill/commit/c8172480b0b1be6c57210212afc71d6ec8711235))
* show a loading indicator while the initial data table migration is generated ([#10900](https://github.com/windmill-labs/windmill/issues/10900)) ([b998267](https://github.com/windmill-labs/windmill/commit/b998267c91b9dcf02787768f6205cc5aeda494fb))
* track outstanding service log files on the rows themselves ([#10894](https://github.com/windmill-labs/windmill/issues/10894)) ([aa4a6ff](https://github.com/windmill-labs/windmill/commit/aa4a6ffd66813010a79c07741b01a984ed4e7df6))
### Performance Improvements
* add service log documents to the index one batch at a time ([#10906](https://github.com/windmill-labs/windmill/issues/10906)) ([0c2eb0a](https://github.com/windmill-labs/windmill/commit/0c2eb0ae3d18f49c21370131d15011e8dd103746))
## [1.799.0](https://github.com/windmill-labs/windmill/compare/v1.798.1...v1.799.0) (2026-08-28)
### Features
* enable Anthropic prompt caching on Vertex AI agent steps ([#10876](https://github.com/windmill-labs/windmill/issues/10876)) ([320f400](https://github.com/windmill-labs/windmill/commit/320f4005124202852e6e9c70b394e7f87231d278))
* instrument AI fill/fix, evals, agents and the debugger ([#10853](https://github.com/windmill-labs/windmill/issues/10853)) ([0bbd559](https://github.com/windmill-labs/windmill/commit/0bbd559ac8a35dba04ba5e8d6f2fd8d1d1124891))
### Bug Fixes
* **datatables:** stop a fork's pg_dump restore from failing silently ([#10830](https://github.com/windmill-labs/windmill/issues/10830)) ([3ce9bbc](https://github.com/windmill-labs/windmill/commit/3ce9bbc7168b837cb2111aabd533bb67803502b8))
* key build artifact caches on a runnable's inline modules ([#10819](https://github.com/windmill-labs/windmill/issues/10819)) ([b72ccc3](https://github.com/windmill-labs/windmill/commit/b72ccc35934165b4bad112b947ca5af064aab26f))
* nested template literals in step inputs, and unresolvable $args tags ([#10856](https://github.com/windmill-labs/windmill/issues/10856)) ([8f349c0](https://github.com/windmill-labs/windmill/commit/8f349c032a0d75fc3350292075e5050a030f6166))
* pre-fill the test panel JSON args editor and align its placeholder ([#10871](https://github.com/windmill-labs/windmill/issues/10871)) ([fb82f36](https://github.com/windmill-labs/windmill/commit/fb82f36e6d6492dd0740984d8d78ea4eaa30361e))
* reject a prefixed error_handler_path on triggers ([#10847](https://github.com/windmill-labs/windmill/issues/10847)) ([d334831](https://github.com/windmill-labs/windmill/commit/d33483173526a3b352d2829ac8a2e1e229cc1127))
* unify billable seat counting and prevent fork subscriptions ([#10818](https://github.com/windmill-labs/windmill/issues/10818)) ([7dd88c4](https://github.com/windmill-labs/windmill/commit/7dd88c470caee5f095dc240667aa7550c55696bc))
## [1.798.1](https://github.com/windmill-labs/windmill/compare/v1.798.0...v1.798.1) (2026-08-27)
### Bug Fixes
* allow job tokens to read the automate_username_creation setting ([#10869](https://github.com/windmill-labs/windmill/issues/10869)) ([c2279db](https://github.com/windmill-labs/windmill/commit/c2279db8a96ac76382eafe254627dafd24d173fd))
## [1.798.0](https://github.com/windmill-labs/windmill/compare/v1.797.0...v1.798.0) (2026-08-27)
### Features
* a wizard for importing a hub project, and finishing what the import cannot ([#10729](https://github.com/windmill-labs/windmill/issues/10729)) ([2913339](https://github.com/windmill-labs/windmill/commit/29133398f99cd2dd5b33057ee9df4492d82e067a))
## [1.797.0](https://github.com/windmill-labs/windmill/compare/v1.796.0...v1.797.0) (2026-08-26)
### Features
* configurable expiry for presigned s3 public url signatures ([#10835](https://github.com/windmill-labs/windmill/issues/10835)) ([8a6dc27](https://github.com/windmill-labs/windmill/commit/8a6dc27236aca67f0efe941d9606b787c2305ea8))
* **frontend:** flag the fork-compare datatable schema diff as legacy ([#10829](https://github.com/windmill-labs/windmill/issues/10829)) ([07c77ea](https://github.com/windmill-labs/windmill/commit/07c77ead7425f1877372d358d867445a4c525c96))
* keep a Hub project live while an update is under review ([#10814](https://github.com/windmill-labs/windmill/issues/10814)) ([c04b570](https://github.com/windmill-labs/windmill/commit/c04b5705745c36ecbb3a551ac59459218d2e3807))
### Bug Fixes
* **cli:** keep svelte component styles in the raw-app bundle ([#10838](https://github.com/windmill-labs/windmill/issues/10838)) ([b8bf539](https://github.com/windmill-labs/windmill/commit/b8bf539c3fe2b4db9c74dd73f04b3029287acdc6))
* **debugger:** parse bun 1.4's UUID inspector token ([#10828](https://github.com/windmill-labs/windmill/issues/10828)) ([4658224](https://github.com/windmill-labs/windmill/commit/46582245926a7f8ea961bcd125a58fbfba3530cf))
* force HTTP router rebuild on trigger-change notification ([#10849](https://github.com/windmill-labs/windmill/issues/10849)) ([ffdf17e](https://github.com/windmill-labs/windmill/commit/ffdf17ef8dc5575dd92d62d0d0ba887c1e378576))
* **frontend:** follow the operating workspace in step input forms ([#10834](https://github.com/windmill-labs/windmill/issues/10834)) ([6b73145](https://github.com/windmill-labs/windmill/commit/6b73145e7220232601538b801ebc9dc73fe79bbb))
* **frontend:** key the GitHub App installation selector on installation_id ([#10831](https://github.com/windmill-labs/windmill/issues/10831)) ([78331fd](https://github.com/windmill-labs/windmill/commit/78331fda8b290a2d9a5dd92b8362ff32c8b39432))
* **frontend:** operator menu opens on hover, pins on click ([#10824](https://github.com/windmill-labs/windmill/issues/10824)) ([665f83e](https://github.com/windmill-labs/windmill/commit/665f83e1f438e34d006429889d51a5fb6a6b6176))
* keep connection string query parameters under token auth ([#10859](https://github.com/windmill-labs/windmill/issues/10859)) ([f131c39](https://github.com/windmill-labs/windmill/commit/f131c3920f50f9fa18cd637eac39609495999aef))
* migrate slack resource-connect oauth to v2 ([#10836](https://github.com/windmill-labs/windmill/issues/10836)) ([9fa8159](https://github.com/windmill-labs/windmill/commit/9fa8159ad16204cab52fd18a34a48ebf13f800f6))
* recover from unresolvable AI session links instead of a dead end ([#10854](https://github.com/windmill-labs/windmill/issues/10854)) ([e38c449](https://github.com/windmill-labs/windmill/commit/e38c449007f27b952808cba5aa812441f2ce5946))
* require admin on workspace tarball settings export ([#10817](https://github.com/windmill-labs/windmill/issues/10817)) ([46c363f](https://github.com/windmill-labs/windmill/commit/46c363ffa4bc72bef6b367ece4bdbeef5e0eadc9))
* restrict filesystem workspace storage to debug builds ([#10864](https://github.com/windmill-labs/windmill/issues/10864)) ([8b80b09](https://github.com/windmill-labs/windmill/commit/8b80b09f33d311f0881678577ca6004c12d97c22))
### Performance Improvements
* index the suspended-job resume test instead of filtering it ([#10863](https://github.com/windmill-labs/windmill/issues/10863)) ([69320b2](https://github.com/windmill-labs/windmill/commit/69320b28f615b897a92f580bd5961c41e5c29951))
## [1.796.0](https://github.com/windmill-labs/windmill/compare/v1.795.0...v1.796.0) (2026-08-24)
CONTEXT.md
+19
View File
@@ -36,3 +36,22 @@ _Avoid_: argument field, param
**Expression input**:
Any other place a property can be picked into: the loop iterator, skip and early-stop predicates, the retry condition, a branch predicate, timeout. Its prop picker opens in a popover from the connect button rather than taking a pane.
_Avoid_: JS field, code input
### Permissions
**Member**:
A user or group granted a role on a folder, a group, or an item's extra ACL. The list of them is
"Members (n)" everywhere it is shown, and one is added with "Add member".
_Avoid_: participant, collaborator, owner, ACL entry, permission (that names the concept, not the people)
**Role**:
The access level a member holds: viewer, writer or admin on a folder; member or admin on a group.
Viewers read, writers also edit, admins also manage the members. A group role of **manager** —
manages the group without belonging to it — is a legacy state the UI shows and can leave, but
offers no way to enter.
_Avoid_: permission level, access level, rank
**Owner**:
Reserved for the path prefix that says where an item lives — `u/alice` or `f/team`. A folder's
`owners` column in the database is its admin members; call those admins, never owners, in the UI.
_Avoid_: using "owner" for a folder admin
ai_evals/adapters/frontend/mockBackend.ts
+72 -5
View File
@@ -7,6 +7,7 @@ import type {
ListableApp,
ListableResource,
ListableVariable,
Resource,
Script
} from '../../../frontend/src/lib/gen'
import type {
@@ -81,6 +82,15 @@ export interface BenchmarkWorkspaceAiProvider {
isDefault?: boolean
}
/** A plain (non-AI) resource of the benchmark workspace, for cases about referencing a
* credential — passing one as a run argument, say. `value` is what `get_resource` returns. */
export interface BenchmarkWorkspaceResource {
path: string
resource_type: string
value?: Record<string, unknown>
description?: string
}
export interface BenchmarkWorkspaceJob {
/** Stable id so a case prompt can reference a specific run (e.g. for get_job_logs). */
id?: string
@@ -98,6 +108,7 @@ export interface BenchmarkWorkspaceRunnables {
apps?: BenchmarkWorkspaceApp[]
variables?: BenchmarkWorkspaceVariable[]
aiProviders?: BenchmarkWorkspaceAiProvider[]
resources?: BenchmarkWorkspaceResource[]
datatables?: BenchmarkDatatableSeed[]
jobs?: BenchmarkWorkspaceJob[]
}
@@ -284,15 +295,71 @@ export function listBenchmarkAiProviderResources(workspace: string): ListableRes
}))
}
/** The value of a seeded AI provider resource. Only the endpoint fields are modelled — a key is
* never needed, because no eval run calls the provider through this resource. */
/** Plain seeded resources of a benchmark workspace, shaped like `ResourceService.listResource`
* rows. Null when the workspace is not a benchmark one. */
export function listBenchmarkPlainResources(workspace: string): ListableResource[] | null {
const runnables = benchmarkWorkspaceRunnables.get(workspace)
if (!runnables) {
return null
}
return (runnables.resources ?? []).map((seed) => ({
workspace_id: workspace,
path: seed.path,
resource_type: seed.resource_type,
description: seed.description,
value: null,
is_oauth: false,
is_linked: false,
is_refreshed: false,
extra_perms: {},
edited_at: BENCHMARK_TIMESTAMP
}))
}
/** A seeded resource with its value, as `ResourceService.getResource` returns it. Covers both
* seed kinds, so it agrees with `existsResource` and `listResource` — both of those report AI
* providers too, and a case that lists resources and then reads one by path would otherwise get
* a row it cannot fetch. */
export function getBenchmarkResource(workspace: string, path: string): Resource | null {
const runnables = benchmarkWorkspaceRunnables.get(workspace)
const seed = runnables?.resources?.find((entry) => entry.path === path)
if (seed) {
return {
workspace_id: workspace,
path: seed.path,
resource_type: seed.resource_type,
description: seed.description,
value: seed.value ?? {},
is_oauth: false,
extra_perms: {}
} as Resource
}
const provider = runnables?.aiProviders?.find((entry) => entry.path === path)
if (!provider) {
return null
}
return {
workspace_id: workspace,
path: provider.path,
resource_type: provider.kind,
value: getBenchmarkResourceValue(workspace, path) ?? {},
is_oauth: false,
extra_perms: {}
} as Resource
}
/** The value of a seeded resource. For an AI provider only the endpoint fields are modelled — a
* key is never needed, because no eval run calls the provider through this resource. */
export function getBenchmarkResourceValue(
workspace: string,
path: string
): Record<string, unknown> | null {
const seed = benchmarkWorkspaceRunnables
.get(workspace)
?.aiProviders?.find((entry) => entry.path === path)
const runnables = benchmarkWorkspaceRunnables.get(workspace)
const plain = runnables?.resources?.find((entry) => entry.path === path)
if (plain) {
return plain.value ?? {}
}
const seed = runnables?.aiProviders?.find((entry) => entry.path === path)
if (!seed) {
return null
}
ai_evals/adapters/frontend/vitestAdapter.test.ts
+11 -3
View File
@@ -70,7 +70,9 @@ vi.mock('$lib/gen', async () => {
getBenchmarkResourceValue,
getBenchmarkVariableByPath,
hasBenchmarkWorkspace,
getBenchmarkResource,
listBenchmarkAiProviderResources,
listBenchmarkPlainResources,
listBenchmarkApps,
listBenchmarkDatatables,
listBenchmarkDrafts,
@@ -359,18 +361,24 @@ vi.mock('$lib/gen', async () => {
hasBenchmarkWorkspace(data.workspace)
? Boolean(getBenchmarkResourceValue(data.workspace, data.path))
: actual.ResourceService.existsResource(data),
// Only AI provider resources are modelled: they are what an AI agent step references.
listResource: async (data: { workspace: string; resourceType?: string }) => {
if (!hasBenchmarkWorkspace(data.workspace)) {
return actual.ResourceService.listResource(data)
}
const seeded = listBenchmarkAiProviderResources(data.workspace) ?? []
const seeded = [
...(listBenchmarkAiProviderResources(data.workspace) ?? []),
...(listBenchmarkPlainResources(data.workspace) ?? [])
]
const wanted = data.resourceType?.split(',')
return wanted ? seeded.filter((r) => wanted.includes(r.resource_type)) : seeded
},
getResource: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`Resource "${data.path}" not found in benchmark workspace`)
const resource = getBenchmarkResource(data.workspace, data.path)
if (!resource) {
throw new Error(`Resource "${data.path}" not found in benchmark workspace`)
}
return resource
}
return actual.ResourceService.getResource(data)
},
ai_evals/cases/global.yaml
+34
View File
@@ -2365,3 +2365,37 @@
- the step uses the workspace's anthropic resource f/evals/global/anthropic_main
- the model is the Opus one the user asked for, taken from the models that resource serves
- the diff flow input reaches the agent
# The failure this pins: passing a resource as `{"$res": "<path>"}` (or as a bare path), which
# reaches the script unresolved because the backend only substitutes a string value that itself
# starts with `$res:`. The mock preview echoes args back and reports success, so nothing in the
# loop corrects a wrong shape — the arg form is the whole test.
- id: global-run-arg-resource-reference
prompt: |-
Run `f/evals/global/github_repo_stats` against the `windmill-labs/windmill` repo, passing our
GitHub credentials at `f/evals/global/github_main` as its `gh_auth` input, and tell me whether
it went through.
initial: ai_evals/fixtures/frontend/global/initial/github_repo_stats_script.json
runtime:
maxTurns: 10
validate:
draftCountExactly: 0
toolExpect:
requiredToolsUsed:
- test_run_script
forbiddenToolsUsed:
- write_script
- deploy_workspace_item
toolCallArgs:
# Exact: the mock never resolves the reference, so a near-miss path like
# `$res:f/evals/global/github_main_backup` would otherwise pass.
- tool: test_run_script
field: args.gh_auth
stringEqualsAnyOf:
- "$res:f/evals/global/github_main"
# The judge only sees drafts, and this case makes none — the deliverable is the shape of the
# run argument, checked deterministically above.
skipJudge: true
judgeChecklist:
- runs the existing script rather than rewriting it
- passes the GitHub resource as the bare string $res:f/evals/global/github_main
ai_evals/core/types.ts
+7
View File
@@ -160,6 +160,13 @@ export interface ToolCallArgumentRule {
field: string;
stringStartsWithAnyOf?: string[];
stringMustNotStartWithAnyOf?: string[];
/**
* Universal over calls: every recorded call to `tool` must carry `field` as
* exactly one of these strings. Use when a near-miss would still satisfy a
* prefix — a resource reference like `$res:f/a/b` shares its prefix with the
* wrong `$res:f/a/b_backup`, and the mock never resolves it to catch that.
*/
stringEqualsAnyOf?: string[];
/**
* Case-insensitive "contains", existential over calls: at least one recorded
* call to `tool` must have `field` containing one of these substrings. Other
ai_evals/core/validators.test.ts
+37
View File
@@ -228,6 +228,43 @@ describe("validateToolExpectations", () => {
});
});
// A resource reference shares its prefix with a wrong sibling path, and the mock
// never resolves it, so only exact matching separates the two.
it("rejects a resource reference whose path merely shares the prefix", () => {
const checks = validateToolExpectations({
run: {
success: true,
actual: {},
assistantMessageCount: 1,
toolCallCount: 1,
toolsUsed: ["test_run_script"],
toolCallDetails: [
{
name: "test_run_script",
arguments: { args: { gh_auth: "$res:f/evals/global/github_main_backup" } },
},
],
skillsInvoked: [],
},
toolExpect: {
toolCallArgs: [
{
tool: "test_run_script",
field: "args.gh_auth",
stringEqualsAnyOf: ["$res:f/evals/global/github_main"],
},
],
},
});
expect(checks).toContainEqual({
name: "test_run_script.args.gh_auth matches an accepted value",
passed: false,
details:
'accepted values: $res:f/evals/global/github_main; values: "$res:f/evals/global/github_main_backup"',
});
});
// The whole point of the same-call rule: the per-field rules are existential over
// calls, so two single-filter pages would satisfy them while never opening the
// combined view the case asks for.
ai_evals/core/validators.ts
+14
View File
@@ -278,6 +278,20 @@ export function validateToolExpectations(input: {
);
}
if (rule.stringEqualsAnyOf && rule.stringEqualsAnyOf.length > 0) {
const invalidValues = values.filter(
(value) =>
typeof value !== "string" || !rule.stringEqualsAnyOf!.includes(value)
);
checks.push(
check(
`${rule.tool}.${rule.field} matches an accepted value`,
invalidValues.length === 0,
`accepted values: ${rule.stringEqualsAnyOf.join(", ")}; values: ${summarizeToolValues(values)}`
)
);
}
if (rule.stringMustNotStartWithAnyOf && rule.stringMustNotStartWithAnyOf.length > 0) {
const invalidValues = values.filter(
(value) =>
ai_evals/fixtures/frontend/global/initial/github_repo_stats_script.json
+37
View File
@@ -0,0 +1,37 @@
{
"workspace": {
"resources": [
{
"path": "f/evals/global/github_main",
"resource_type": "github",
"description": "GitHub credentials",
"value": { "token": "$var:f/evals/global/github_token" }
}
],
"scripts": [
{
"path": "f/evals/global/github_repo_stats",
"summary": "Count open issues on a GitHub repository",
"description": "Reads the open issue count for a repository using GitHub credentials.",
"language": "bun",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"gh_auth": {
"type": "object",
"format": "resource-github",
"description": "GitHub credentials"
},
"repo": {
"type": "string",
"description": "Repository in owner/name form"
}
},
"required": ["gh_auth", "repo"]
},
"content": "type Github = { token: string }\n\nexport async function main(gh_auth: Github, repo: string) {\n const res = await fetch(`https://api.github.com/repos/${repo}/issues?state=open`, {\n headers: { Authorization: `Bearer ${gh_auth.token}` }\n })\n const issues = await res.json()\n return { repo, open_issues: issues.length }\n}\n"
}
]
}
}
backend/.sqlx/query-002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7.json
Generated
-29
View File
@@ -1,29 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n COUNT(*)::bigint AS \"total!\",\n COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS \"replacing!\"\n FROM ai_skill\n WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "total!",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "replacing!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": [
null,
null
]
},
"hash": "002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7"
}
backend/.sqlx/query-032b51ce97c2f31dc2aea8ddf64e6971818ea3b19b4d4866d16e1bf9f7f2ec6f.json
Generated
+65
View File
@@ -0,0 +1,65 @@
{
"db_name": "PostgreSQL",
"query": "SELECT COALESCE(u.username, p.username, CASE WHEN p.email IS NOT NULL THEN d.email END) as \"username?\",\n d.created_at as \"draft_saved_at!\"\n FROM draft d\n LEFT JOIN usr u\n ON u.workspace_id = d.workspace_id\n AND u.email = d.email\n LEFT JOIN password p\n ON p.email = d.email\n AND p.super_admin = true\n WHERE d.workspace_id = $1\n AND d.path = $2\n AND d.typ = $3\n AND (d.email IS NULL OR d.email <> $4)\n AND (d.email IS NULL OR u.username IS NOT NULL OR p.email IS NOT NULL)\n ORDER BY d.email NULLS LAST",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "username?",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "draft_saved_at!",
"type_info": "Timestamptz"
}
],
"parameters": {
"Left": [
"Text",
"Text",
{
"Custom": {
"name": "draft_kind",
"kind": {
"Enum": [
"script",
"flow",
"app",
"raw_app",
"resource",
"variable",
"trigger_schedule",
"trigger_webhook",
"trigger_default_email",
"trigger_email",
"trigger_http",
"trigger_websocket",
"trigger_postgres",
"trigger_kafka",
"trigger_nats",
"trigger_mqtt",
"trigger_sqs",
"trigger_gcp",
"trigger_azure",
"trigger_poll",
"trigger_cli",
"trigger_nextcloud",
"trigger_google",
"trigger_github",
"data_pipeline",
"trigger_amqp"
]
}
}
},
"Text"
]
},
"nullable": [
null,
false
]
},
"hash": "032b51ce97c2f31dc2aea8ddf64e6971818ea3b19b4d4866d16e1bf9f7f2ec6f"
}
backend/.sqlx/query-0a02678e3f062c8854226d6d5eb7e493c229d205048eeac78a7cbe328c689b88.json
Generated
+12
View File
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO draft(workspace_id, path, typ, value, email) VALUES\n ('test-workspace', 'u/ext/s', 'script', '{}'::json, 'ext-jwt@windmill.dev'),\n ('test-workspace', 'u/two/s', 'script', '{\"summary\": \"moving\"}'::json, 'test2@windmill.dev'),\n ('test-workspace', 'u/two/s', 'script', '{\"summary\": \"displaced\"}'::json, 'renamed@windmill.dev'),\n ('test-workspace', 'u/three/s', 'script', '{}'::json, 'test3@windmill.dev')",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "0a02678e3f062c8854226d6d5eb7e493c229d205048eeac78a7cbe328c689b88"
}
backend/.sqlx/query-94da1e7feb4f58cc7ebe99752736f956d47810a94cb052fdcffb5cfe440f8033.json → backend/.sqlx/query-0e03dc960c0a22e042e54af719ac90c4b8506acefc85ef5b2f92a7bc451b1c5e.json
Generated
+3 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval RETURNING file_path, hostname",
"query": "DELETE FROM log_file WHERE (hostname, log_ts) IN (\n SELECT hostname, log_ts FROM log_file\n WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval\n LIMIT $2\n ) RETURNING file_path, hostname",
"describe": {
"columns": [
{
@@ -16,6 +16,7 @@
],
"parameters": {
"Left": [
"Int8",
"Int8"
]
},
@@ -24,5 +25,5 @@
false
]
},
"hash": "94da1e7feb4f58cc7ebe99752736f956d47810a94cb052fdcffb5cfe440f8033"
"hash": "0e03dc960c0a22e042e54af719ac90c4b8506acefc85ef5b2f92a7bc451b1c5e"
}
backend/.sqlx/query-1572b7348a05b7e357031f8d44b5bbee155569488352c10b334ce57d83ce1c0a.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT lockfile_hash FROM lock_hash WHERE workspace_id = $1 AND path = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "lockfile_hash",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false
]
},
"hash": "1572b7348a05b7e357031f8d44b5bbee155569488352c10b334ce57d83ce1c0a"
}
backend/.sqlx/query-16b174aaa944fd94458ce3108f0fec23514ae0419f77b632064b75473b5636c3.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT trace_id FROM otel_traces ORDER BY trace_id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "trace_id",
"type_info": "Bytea"
}
],
"parameters": {
"Left": []
},
"nullable": [
false
]
},
"hash": "16b174aaa944fd94458ce3108f0fec23514ae0419f77b632064b75473b5636c3"
}
backend/.sqlx/query-188d024a53b2ef37442412824f73ab5cd81242501d9e7a476698ea7acccd4aef.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM draft dest\n WHERE dest.email = $1\n AND EXISTS (SELECT 1 FROM draft src\n WHERE src.email = $2\n AND src.workspace_id = dest.workspace_id\n AND src.path = dest.path\n AND src.typ = dest.typ)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": []
},
"hash": "188d024a53b2ef37442412824f73ab5cd81242501d9e7a476698ea7acccd4aef"
}
backend/.sqlx/query-1b244f65ee6a2607ebc1c333d4359fbbf8be5a81276a3050a42770e4a5b5aa5e.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM otel_traces WHERE ctid IN (\n SELECT ctid FROM otel_traces\n WHERE start_time_unix_nano < EXTRACT(\n EPOCH FROM now() - ($1::bigint::text || ' s')::interval\n )::bigint * 1000000000\n LIMIT $2\n )",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int8",
"Int8"
]
},
"nullable": []
},
"hash": "1b244f65ee6a2607ebc1c333d4359fbbf8be5a81276a3050a42770e4a5b5aa5e"
}
backend/.sqlx/query-247486558e023ec3adf0c1e8f5664fc5e65995ce81e622c174a89befc1a527e5.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT cost_nanos FROM ai_free_token_usage WHERE email = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "cost_nanos",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false
]
},
"hash": "247486558e023ec3adf0c1e8f5664fc5e65995ce81e622c174a89befc1a527e5"
}
backend/.sqlx/query-43ea7d0cf7f3c50ec1e79b4d2384d49b6c65bc442263f228912a24c1c5740cc8.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE draft SET email = $1 WHERE email = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Text"
]
},
"nullable": []
},
"hash": "43ea7d0cf7f3c50ec1e79b4d2384d49b6c65bc442263f228912a24c1c5740cc8"
}
backend/.sqlx/query-44b9400fed2082c0df9d57f940923f37b8d452ea8e199ccc207da238368f1996.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_free_token_daily_usage (day, cost_nanos, updated_at)\n VALUES ($2::date, $1::bigint, now())\n ON CONFLICT (day) DO UPDATE\n SET cost_nanos = ai_free_token_daily_usage.cost_nanos + $1::bigint,\n updated_at = now()\n RETURNING cost_nanos",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "cost_nanos",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Int8",
"Date"
]
},
"nullable": [
false
]
},
"hash": "44b9400fed2082c0df9d57f940923f37b8d452ea8e199ccc207da238368f1996"
}
backend/.sqlx/query-4b93550c7836fd3643180ade3548faa875e471d3f9ca37fc669f359e7a1818bb.json
Generated
-16
View File
@@ -1,16 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO resource_type (workspace_id, name, schema, description, edited_at)\n VALUES ('admins', $1, $2, $3, now())\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET schema = EXCLUDED.schema, description = EXCLUDED.description, edited_at = now()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Jsonb",
"Text"
]
},
"nullable": []
},
"hash": "4b93550c7836fd3643180ade3548faa875e471d3f9ca37fc669f359e7a1818bb"
}
backend/.sqlx/query-4bc43a5e1c95cb8989962aeb1393a50df05437a2f1909ad5d303e2c2b89a0668.json
Generated
+14
View File
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM draft WHERE email = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "4bc43a5e1c95cb8989962aeb1393a50df05437a2f1909ad5d303e2c2b89a0668"
}
backend/.sqlx/query-624a7dbc6cc951a199b0e70d86c463a0e7b5248c226ee92d95df94c3099cc400.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE log_file SET indexed_at = now()\n FROM unnest($1::text[], $2::text[]) AS c(hostname, file_path)\n WHERE log_file.indexed_at IS NULL\n AND log_file.hostname = c.hostname\n AND log_file.file_path = c.file_path",
"describe": {
"columns": [],
"parameters": {
"Left": [
"TextArray",
"TextArray"
]
},
"nullable": []
},
"hash": "624a7dbc6cc951a199b0e70d86c463a0e7b5248c226ee92d95df94c3099cc400"
}
backend/.sqlx/query-62ed1fe52bc2c22d199101309cbbadb9842318c4c7a1d2526ac567ce41b7fbdc.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO otel_traces (trace_id, span_id, name, kind, start_time_unix_nano, end_time_unix_nano)\n VALUES ($1, $2, 'GET /', 3, $3, $3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Bytea",
"Bytea",
"Int8"
]
},
"nullable": []
},
"hash": "62ed1fe52bc2c22d199101309cbbadb9842318c4c7a1d2526ac567ce41b7fbdc"
}
backend/.sqlx/query-67a83afb708c90b2132cba81a0701dfc8b5e7aedf2f26fc1fe5be8f685cf709c.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT max(log_ts) FROM log_file\n WHERE hostname = $1 AND log_ts < (SELECT max(log_ts) FROM log_file WHERE hostname = $1)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "max",
"type_info": "Timestamp"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "67a83afb708c90b2132cba81a0701dfc8b5e7aedf2f26fc1fe5be8f685cf709c"
}
backend/.sqlx/query-b5c839baab25c4dcdd503d380cf7a886242277cd50555f20b2e22e13942d2a3a.json → backend/.sqlx/query-6bbcb27a3bb70302076c559c8394b14b842f595f68dd885248abaaeabd2d0bf1.json
Generated
+2 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n hostname,\n mode::text,\n worker_group,\n log_ts,\n file_path,\n ok_lines,\n err_lines,\n json_fmt\n FROM log_file\n WHERE log_ts > $1\n ORDER BY log_ts ASC LIMIT $2",
"query": "SELECT\n hostname,\n mode::text,\n worker_group,\n log_ts,\n file_path,\n ok_lines,\n err_lines,\n json_fmt\n FROM log_file\n WHERE indexed_at IS NULL\n ORDER BY log_ts ASC, hostname ASC LIMIT $1",
"describe": {
"columns": [
{
@@ -46,7 +46,6 @@
],
"parameters": {
"Left": [
"Timestamp",
"Int8"
]
},
@@ -61,5 +60,5 @@
true
]
},
"hash": "b5c839baab25c4dcdd503d380cf7a886242277cd50555f20b2e22e13942d2a3a"
"hash": "6bbcb27a3bb70302076c559c8394b14b842f595f68dd885248abaaeabd2d0bf1"
}
backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json
Generated
-18
View File
@@ -1,18 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_skill (workspace_id, name, description, instructions, edited_at, edited_by)\n VALUES ($1, $2, $3, $4, now(), $5)\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET description = EXCLUDED.description,\n instructions = EXCLUDED.instructions,\n edited_at = now(),\n edited_by = EXCLUDED.edited_by",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Text",
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8"
}
backend/.sqlx/query-872be44352d4a27a2005f6bdc38f927ac4886755fd17ec755037763aa92f6c0e.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "WITH moved AS (\n UPDATE log_file SET indexed_at = CASE\n WHEN log_ts > NOW() - make_interval(secs => $1) THEN NULL\n ELSE now() END\n WHERE indexed_at = 'epoch' RETURNING 1)\n SELECT count(*) AS \"n!\" FROM moved",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "n!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Float8"
]
},
"nullable": [
null
]
},
"hash": "872be44352d4a27a2005f6bdc38f927ac4886755fd17ec755037763aa92f6c0e"
}
backend/.sqlx/query-87e8e034b1cf5ea7ce43870d77d33fa0fd05b067454f0d4686d8b155fe562ffe.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE log_file SET indexed_at = now()\n FROM unnest($1::text[], $2::timestamp[]) AS c(hostname, log_ts)\n WHERE log_file.hostname = c.hostname AND log_file.log_ts = c.log_ts",
"describe": {
"columns": [],
"parameters": {
"Left": [
"TextArray",
"TimestampArray"
]
},
"nullable": []
},
"hash": "87e8e034b1cf5ea7ce43870d77d33fa0fd05b067454f0d4686d8b155fe562ffe"
}
backend/.sqlx/query-1ea97f9085ec018f779e77e0fdbda3d4ecd67b3fbee9a58228ef577f846607ae.json → backend/.sqlx/query-8ad79b80033b38ebddf6c8cd4d8cb160d41bac4c45a0fc74d9c9e96d3ef4486a.json
Generated
+5 -3
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT EXISTS(SELECT 1 FROM resource_type WHERE workspace_id = 'admins' AND name = $1 AND schema IS NOT DISTINCT FROM $2 AND description IS NOT DISTINCT FROM $3)",
"query": "SELECT EXISTS(SELECT 1 FROM resource_type WHERE workspace_id = 'admins' AND name = $1 AND schema IS NOT DISTINCT FROM $2 AND description IS NOT DISTINCT FROM $3 AND ($5 IS NOT TRUE OR format_extension IS NOT DISTINCT FROM $4))",
"describe": {
"columns": [
{
@@ -13,12 +13,14 @@
"Left": [
"Text",
"Jsonb",
"Text"
"Text",
"Text",
"Bool"
]
},
"nullable": [
null
]
},
"hash": "1ea97f9085ec018f779e77e0fdbda3d4ecd67b3fbee9a58228ef577f846607ae"
"hash": "8ad79b80033b38ebddf6c8cd4d8cb160d41bac4c45a0fc74d9c9e96d3ef4486a"
}
backend/.sqlx/query-8d207cc9ed101ff116b617d25a94633c1531170ded1fdf09114718b941f5e1db.json
Generated
-65
View File
@@ -1,65 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n hostname,\n mode::text,\n worker_group,\n log_ts,\n file_path,\n ok_lines,\n err_lines,\n json_fmt\n FROM log_file\n WHERE log_ts > NOW() - make_interval(secs => $1)\n ORDER BY log_ts ASC LIMIT $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "hostname",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "mode",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "worker_group",
"type_info": "Varchar"
},
{
"ordinal": 3,
"name": "log_ts",
"type_info": "Timestamp"
},
{
"ordinal": 4,
"name": "file_path",
"type_info": "Varchar"
},
{
"ordinal": 5,
"name": "ok_lines",
"type_info": "Int8"
},
{
"ordinal": 6,
"name": "err_lines",
"type_info": "Int8"
},
{
"ordinal": 7,
"name": "json_fmt",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Float8",
"Int8"
]
},
"nullable": [
false,
null,
true,
false,
false,
true,
true,
true
]
},
"hash": "8d207cc9ed101ff116b617d25a94633c1531170ded1fdf09114718b941f5e1db"
}
backend/.sqlx/query-8da59f1ace46dc9830cb0fc5a640df68f552e38d4b587839e0e41285a2d55455.json
Generated
+28
View File
@@ -0,0 +1,28 @@
{
"db_name": "PostgreSQL",
"query": "SELECT DISTINCT ON (path) path, lock FROM script\n WHERE workspace_id = $1 AND NOT archived AND NOT deleted AND lock IS NOT NULL\n ORDER BY path, created_at DESC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "lock",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
true
]
},
"hash": "8da59f1ace46dc9830cb0fc5a640df68f552e38d4b587839e0e41285a2d55455"
}
backend/.sqlx/query-972df41db505fbbd20a558b200a2e2e8bc43633707d8365f73130c5bca3923b9.json
Generated
+18
View File
@@ -0,0 +1,18 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO resource_type (workspace_id, name, schema, description, format_extension, edited_at)\n VALUES ('admins', $1, $2, $3, $4, now())\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET schema = EXCLUDED.schema, description = EXCLUDED.description,\n -- A fileset is a set of files, so it cannot also be one file.\n -- Create and update reject the pair; this writer bypasses both, so\n -- it declines the extension rather than persisting the forbidden\n -- combination onto a same-named local fileset.\n format_extension = CASE\n WHEN resource_type.is_fileset THEN NULL\n WHEN $5 THEN EXCLUDED.format_extension\n ELSE resource_type.format_extension END,\n edited_at = now()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Jsonb",
"Text",
"Varchar",
"Bool"
]
},
"nullable": []
},
"hash": "972df41db505fbbd20a558b200a2e2e8bc43633707d8365f73130c5bca3923b9"
}
backend/.sqlx/query-97691b88d43442e1f1984562372590428895b2bab67687c12efefe0b0d48881e.json
Generated
+12
View File
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO draft(workspace_id, path, typ, value, email) VALUES\n ('test-workspace', 'u/ext/s', 'script', '{}'::json, 'ext-jwt@windmill.dev'),\n ('test-workspace', 'u/two/s', 'script', '{}'::json, 'test2@windmill.dev'),\n ('test-workspace', 'u/three/s', 'script', '{}'::json, 'test3@windmill.dev')",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "97691b88d43442e1f1984562372590428895b2bab67687c12efefe0b0d48881e"
}
backend/.sqlx/query-9e503b65fe8bb1509f0d486829ce13ec9d93bb94192ddd85c3dacb9ff16cd032.json
Generated
+26
View File
@@ -0,0 +1,26 @@
{
"db_name": "PostgreSQL",
"query": "SELECT email, value->>'summary' AS summary FROM draft WHERE path = 'u/two/s'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "email",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "summary",
"type_info": "Text"
}
],
"parameters": {
"Left": []
},
"nullable": [
true,
null
]
},
"hash": "9e503b65fe8bb1509f0d486829ce13ec9d93bb94192ddd85c3dacb9ff16cd032"
}
backend/.sqlx/query-a7d450e34084d561f69e588bd76fd56e616ee79d895b7dcc37ad9442789e1574.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "WITH retired AS (\n UPDATE log_file SET indexed_at = now()\n WHERE indexed_at IS NULL\n AND log_ts <= NOW() - make_interval(secs => $1) RETURNING 1)\n SELECT count(*) AS \"n!\" FROM retired",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "n!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Float8"
]
},
"nullable": [
null
]
},
"hash": "a7d450e34084d561f69e588bd76fd56e616ee79d895b7dcc37ad9442789e1574"
}
backend/.sqlx/query-abdf62ef0e4eeb8c3213d2e8837e7032f710fe20dd272b7840c9bfbdb92554db.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT COUNT(*) FROM script WHERE workspace_id = $1 AND path = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "abdf62ef0e4eeb8c3213d2e8837e7032f710fe20dd272b7840c9bfbdb92554db"
}
backend/.sqlx/query-acf4a5f4b67ebd06157819677fee6be4cc3adc6aa859c3dbaa57fa50b63741cf.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_free_token_daily_usage (day, cost_nanos, updated_at)\n VALUES ($2::date, GREATEST(0, $1::bigint), now())\n ON CONFLICT (day) DO UPDATE\n SET cost_nanos = GREATEST(0, ai_free_token_daily_usage.cost_nanos + $1::bigint),\n updated_at = now()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int8",
"Date"
]
},
"nullable": []
},
"hash": "acf4a5f4b67ebd06157819677fee6be4cc3adc6aa859c3dbaa57fa50b63741cf"
}
backend/.sqlx/query-b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384.json
Generated
+3 -3
View File
@@ -98,12 +98,12 @@
null,
null,
null,
true,
false,
null,
null,
null,
true,
true
false,
false
]
},
"hash": "b8e732a03969666444f73397ac153d23ec2af084b2f93da24c920532c1916384"
backend/.sqlx/query-c29d815cc38493f17950f12e9e5fe34b6c4ed1b2a49ed2b61dc9fb494026e956.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_free_token_usage (email, cost_nanos, updated_at)\n VALUES ($1, $2::bigint, now())\n ON CONFLICT (email) DO UPDATE\n SET cost_nanos = ai_free_token_usage.cost_nanos + $2::bigint,\n updated_at = now()\n RETURNING cost_nanos",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "cost_nanos",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Varchar",
"Int8"
]
},
"nullable": [
false
]
},
"hash": "c29d815cc38493f17950f12e9e5fe34b6c4ed1b2a49ed2b61dc9fb494026e956"
}
backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json
Generated
-28
View File
@@ -1,28 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT name, description FROM ai_skill WHERE workspace_id = $1 ORDER BY name",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "description",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
false
]
},
"hash": "c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b"
}
backend/.sqlx/query-cc507843e00c83a42cc4a463999656ce9a8b0499d6b9282a3ecfae3164b17c2a.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO lock_hash (workspace_id, path, lockfile_hash)\n SELECT $1, * FROM UNNEST($2::text[], $3::bigint[])\n ON CONFLICT (workspace_id, path) DO UPDATE SET lockfile_hash = EXCLUDED.lockfile_hash\n WHERE lock_hash.lockfile_hash IS DISTINCT FROM EXCLUDED.lockfile_hash",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"TextArray",
"Int8Array"
]
},
"nullable": []
},
"hash": "cc507843e00c83a42cc4a463999656ce9a8b0499d6b9282a3ecfae3164b17c2a"
}
backend/.sqlx/query-f0070b36f7c4fc84dc9c23bb6c73d8ba80993a28b2c2e5df70968acf6d7cebe4.json → backend/.sqlx/query-cfdd5ac1dfc7276fc37d49ddfe1b8880eaafb2d3fe71d75b676f1719e26f660f.json
Generated
+2 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "VACUUM v2_job, v2_job_completed, job_result_stream_v2, job_stats, job_logs, job_perms, concurrency_key, log_file, metrics",
"query": "VACUUM v2_job, v2_job_completed, job_result_stream_v2, job_stats, job_logs, job_perms, concurrency_key, log_file, metrics, otel_traces",
"describe": {
"columns": [],
"parameters": {
@@ -8,5 +8,5 @@
},
"nullable": []
},
"hash": "f0070b36f7c4fc84dc9c23bb6c73d8ba80993a28b2c2e5df70968acf6d7cebe4"
"hash": "cfdd5ac1dfc7276fc37d49ddfe1b8880eaafb2d3fe71d75b676f1719e26f660f"
}
backend/.sqlx/query-daa5b57290cd1f821a53eebe96434f1befe6b16eee363c843faa1f836d53ca8d.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT mode::text AS \"mode!\", log_ts FROM log_file WHERE hostname = $1 AND file_path = $2 ORDER BY log_ts DESC LIMIT 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "mode!",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "log_ts",
"type_info": "Timestamp"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null,
false
]
},
"hash": "daa5b57290cd1f821a53eebe96434f1befe6b16eee363c843faa1f836d53ca8d"
}
backend/.sqlx/query-dd15827cbac128ec53cf03fa305e4cad34e540a2cc09c92e262491145a0de05a.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM lock_hash WHERE workspace_id = $1 AND path = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": []
},
"hash": "dd15827cbac128ec53cf03fa305e4cad34e540a2cc09c92e262491145a0de05a"
}
backend/.sqlx/query-df13e7bb9c14aa19604c40754509f66af26042464ba199586838e073c318c53a.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT is_fileset, format_extension FROM resource_type\n WHERE name = $1 AND workspace_id = $2 FOR UPDATE",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "is_fileset",
"type_info": "Bool"
},
{
"ordinal": 1,
"name": "format_extension",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
true
]
},
"hash": "df13e7bb9c14aa19604c40754509f66af26042464ba199586838e073c318c53a"
}
backend/.sqlx/query-e3c4922e315b75951b5ea07cdfca4cfb32747b52dc62c9e3eccacf9c69e29b3a.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT email FROM draft WHERE path = 'u/two/s'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "email",
"type_info": "Varchar"
}
],
"parameters": {
"Left": []
},
"nullable": [
true
]
},
"hash": "e3c4922e315b75951b5ea07cdfca4cfb32747b52dc62c9e3eccacf9c69e29b3a"
}
backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json
Generated
-35
View File
@@ -1,35 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT name, description, instructions FROM ai_skill WHERE workspace_id = $1 AND name = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "description",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "instructions",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
false,
false
]
},
"hash": "e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d"
}
backend/.sqlx/query-f08ef43b4523c74fcbfc7879c147e1bf2c9a0d11e9841caafca62d85c9fb4c8f.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_free_token_usage (email, cost_nanos, updated_at)\n VALUES ($1, GREATEST(0, $2::bigint), now())\n ON CONFLICT (email) DO UPDATE\n SET cost_nanos = GREATEST(0, ai_free_token_usage.cost_nanos + $2::bigint),\n updated_at = now()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Int8"
]
},
"nullable": []
},
"hash": "f08ef43b4523c74fcbfc7879c147e1bf2c9a0d11e9841caafca62d85c9fb4c8f"
}
backend/.sqlx/query-92faee8714a45a403b623e04d789f7f99067a05e9dfe270223164db8a1df2e4b.json → backend/.sqlx/query-f277db0459ff311d8a396aa4e03876dac75ba6df5f8eaa9f185300483e3ee36f.json
Generated
+2 -2
View File
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)\n VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)\n ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
"query": "INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)\n VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)\n ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
"describe": {
"columns": [],
"parameters": {
@@ -17,5 +17,5 @@
},
"nullable": []
},
"hash": "92faee8714a45a403b623e04d789f7f99067a05e9dfe270223164db8a1df2e4b"
"hash": "f277db0459ff311d8a396aa4e03876dac75ba6df5f8eaa9f185300483e3ee36f"
}
backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json → backend/.sqlx/query-f47cafb6e9a6ec58ccafb06cf5e806e3fe749119214863b9111b58fff0bb9583.json
Generated
+4 -5
View File
@@ -1,23 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM ai_skill WHERE workspace_id = $1 AND name = $2 RETURNING name",
"query": "SELECT plan FROM workspace_settings WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"name": "plan",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false
true
]
},
"hash": "e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61"
"hash": "f47cafb6e9a6ec58ccafb06cf5e806e3fe749119214863b9111b58fff0bb9583"
}
backend/.sqlx/query-f496f06f117f5c91104fc759df35b883070da938dea7c7975332afb876cc4691.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT path FROM draft ORDER BY path",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path",
"type_info": "Varchar"
}
],
"parameters": {
"Left": []
},
"nullable": [
false
]
},
"hash": "f496f06f117f5c91104fc759df35b883070da938dea7c7975332afb876cc4691"
}
backend/AGENTS.md
+5 -3
View File
@@ -131,9 +131,11 @@ minimal explicit set for dev.
## Workspace object storage in dev — use the local filesystem
For a dev workspace you don't need MinIO/S3: use the built-in **`FilesystemStorage`** large-file
storage (a root path on local disk). It is intentionally hidden from the settings-UI storage
dropdown (dev-only), so set it via the API. Requires the backend built with `parquet` (+ `private`
for the real S3 helpers, + `enterprise` if you want advanced permission rules enforced):
storage (a root path on local disk). It is a **debug-build affordance only** — every site that
builds a filesystem object store calls `ensure_filesystem_storage_allowed`, so release builds
refuse it, and the settings UI never offers it — so set it via the API on a `cargo run`/`cargo
test` binary. Requires the backend built with `parquet` (+ `private` for the real S3 helpers,
+ `enterprise` if you want advanced permission rules enforced):
```bash
curl -X POST "$BASE/api/w/<ws>/workspaces/edit_large_file_storage_config" \
backend/Cargo.lock
Generated
+358 -270
View File
File diff suppressed because it is too large Load Diff
backend/Cargo.toml
+12 -5
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.796.0"
version = "1.803.0"
authors.workspace = true
edition.workspace = true
@@ -88,7 +88,7 @@ members = [
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
[workspace.package]
version = "1.796.0"
version = "1.803.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
@@ -115,7 +115,7 @@ strip = "none"
[features]
default = []
private = ["windmill-api/private", "windmill-api-agent-workers?/private", "windmill-autoscaling/private", "windmill-common/private", "windmill-dep-map/private", "windmill-object-store/private", "windmill-git-sync/private", "windmill-indexer/private", "windmill-operator?/private", "windmill-queue/private", "windmill-worker/private", "windmill-test-utils/private"]
private = ["windmill-api/private", "windmill-api-agent-workers?/private", "windmill-autoscaling/private", "windmill-common/private", "windmill-dep-map/private", "windmill-object-store/private", "windmill-git-sync/private", "windmill-indexer?/private", "windmill-operator?/private", "windmill-queue/private", "windmill-worker/private", "windmill-test-utils/private"]
agent_worker_server = ["windmill-api/agent_worker_server", "dep:windmill-api-agent-workers", "windmill-test-utils/agent_worker_server"]
enterprise = ["windmill-worker/enterprise", "windmill-queue/enterprise", "windmill-api/enterprise", "windmill-api-agent-workers?/enterprise", "dep:windmill-autoscaling", "windmill-autoscaling/enterprise", "windmill-git-sync/enterprise", "windmill-common/prometheus", "windmill-common/enterprise", "windmill-object-store/enterprise", "license"]
local_reports = ["windmill-common/local_reports"]
@@ -131,6 +131,10 @@ quickjs = ["windmill-worker/quickjs", "windmill-api/quickjs"]
openidconnect = ["windmill-api/openidconnect", "windmill-common/openidconnect", "windmill-object-store/openidconnect"]
cloud = ["windmill-queue/cloud", "windmill-worker/cloud", "windmill-common/cloud", "windmill-api/cloud"]
jemalloc = ["windmill-common/jemalloc", "dep:tikv-jemallocator", "dep:tikv-jemalloc-sys", "dep:tikv-jemalloc-ctl"]
# `tantivy` must stay the only feature that enables windmill-indexer: it is the only one that
# also gives it `enterprise` + `parquet`, and its EE sources gate nearly everything on that
# pair, so a bare windmill-indexer is a crate of dead code that `-D warnings` rejects. Any
# other feature wanting one of its features has to use the optional `windmill-indexer?/` form.
tantivy = ["dep:windmill-indexer", "windmill-api/tantivy", "windmill-indexer/enterprise", "windmill-indexer/parquet", "windmill-common/tantivy", "enterprise", "parquet"]
sqlx = ["windmill-worker/sqlx"]
deno_core = ["windmill-worker/deno_core", "dep:windmill-runtime-nativets", "windmill-test-utils/deno_core"]
@@ -477,7 +481,10 @@ rust-embed = { version = "^6", features = ["interpolate-folder-path"] }
mime_guess = "^2"
hex = "^0"
sql-builder = "^3"
argon2 = "^0"
# Minor-pinned rather than the `^0` used elsewhere in this file: argon2's 0.x
# minors are API-breaking (0.6 moved `SaltString` into `phc`, put `rand_core`
# behind a feature and changed `hash_password`), so a float breaks the build.
argon2 = "0.6"
quick_cache = "^0"
rand = "=0.9.0"
rand_core = { version = "^0", features = ["std"] }
@@ -697,7 +704,7 @@ tikv-jemalloc-ctl = { version = "^0.5" }
triomphe = "^0"
pin-project-lite = "^0"
tantivy = { git="https://github.com/windmill-labs/tantivy", rev="6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" }
tantivy = { git="https://github.com/windmill-labs/tantivy", rev="ea3b818c7b93db0c2b5db4f5e7ffef38f339060a" }
backon = "1.3.0"
backend/THREAT_MODEL.md
+2 -2
View File
@@ -85,7 +85,7 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical).
| EP9 Worker sandbox | nsjail / unshare / dind / rootless podman isolating user code | user code → host & cross-tenant filesystem/network | Worker host, isolation, downstream |
| EP10 Worker code generation / wrappers | Entrypoint override, env-var names, workspace env interpolated into generated wrapper code | user-controlled identifier → executable code | Worker host, isolation |
| EP11 OAuth / OIDC / SAML / MCP-OAuth / logout | Login callbacks, MCP OAuth client registration, logout `rd` redirect | untrusted IdP / redirect input → session | Session tokens, accounts |
| EP12 Stored-content rendering | App builder HTML component, markdown, S3 download response headers | stored user content → admin browser (same origin) | Admin session, account takeover |
| EP12 Stored-content rendering | App builder HTML component, markdown, S3 download response headers, script-controlled `wm_content_type`/`wm_headers` on `run_wait_result` and sync HTTP-route responses | stored user content → admin browser (same origin) | Admin session, account takeover |
| EP13 Log/file reading & export endpoints | `service_logs`, `jobs_u/getupdate` log file read (symlinks), workspace/tarball export | authed/unauth request → arbitrary file or admin-only config | Arbitrary files, global settings |
| EP14 Secret-value & resource-value caches | In-memory caches in `windmill-store` keyed (historically un-keyed) by path | cache lookup crossing identity/folder boundary | Secret variables, resource creds |
| EP15 Deployment & runtime config | docker-compose defaults: dind, debugger (`REQUIRE_SIGNED_DEBUG_REQUESTS` now defaults to `true`; can still be overridden to `false`), CORS `Any`, default admin/`changeme`, exposed Postgres, `SUPERADMIN_SECRET`, `ENABLE_NSJAIL=false`, privileged containers | operator/infra default → full instance | All assets |
@@ -106,7 +106,7 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical).
| T8 | Unauthenticated RCE via the Debugger WebSocket: `/ws_debug/*` exposed by the gateway/ingress with the debugger service as the auth boundary; signature gate was bypassable via `program`-mode launches (read+exec an arbitrary server-side file path, never signed) even with signing on, and the WS handshake had no Origin check (CSWSH) | remote_unauth | EP15 | Worker host, all assets | critical | possible | partially_mitigated | `program`-mode launches now rejected when `REQUIRE_SIGNED_DEBUG_REQUESTS` is on (signing covers every launch, not just inline `code`); shipped `docker-compose` now defaults `REQUIRE_SIGNED_DEBUG_REQUESTS=true`; opt-in `DEBUG_ALLOWED_ORIGINS` allowlist rejects cross-origin handshakes. Residual: code default is secure but operators can still set `=false`; origin allowlist is opt-in | GHSA-725h-99vx-9xr4 |
| T9 | Supply-chain compromise via cached hub scripts, GitHub workflow command injection, or vulnerable base-image deps | supply_chain | EP16 | Worker host, build integrity | critical | possible | partially_mitigated | hub-script re-pin to patched versions; HUB_BASE_URL override | GHSA-w2m9-q5f7-3gpq, edf340c4d4, GHSA-8rq7-w7g6-8wvr, GHSA-vch9-39v5-4wg7 (CVE-2024-37371) |
| T10 | Unauthenticated disclosure of job results, args, logs, and admin config via missing-authz public endpoints | remote_unauth | EP2, EP13 | Job results/args/logs, global settings, scripts | high | likely | partially_mitigated | anonymous-job checks, log-endpoint authz hardening | GHSA-qfg7-x243-5hg4, GHSA-v448-fmm4-52fp, 108a88a180, bb90f4ce83 |
| T11 | Stored XSS leading to admin/account takeover via app HTML component, markdown, or S3 download content-type | remote_auth | EP12 | Admin session, accounts | high | likely | partially_mitigated | DOMPurify markdown sanitization, `X-Content-Type-Options: nosniff` + CSP sandbox on downloads | GHSA-9c5c-hh3c-r9mc, GHSA-qxj7-hpx3-r892, GHSA-cf2x-rg8c-v63v, bb78b1c06d, 625b67dff0 |
| T11 | Stored XSS leading to admin/account takeover via app HTML component, markdown, S3 download content-type, or a script-chosen `text/html` content type on `run_wait_result` / sync HTTP-route responses (GET-reachable with the `SameSite=Lax` session cookie) | remote_auth | EP12 | Admin session, accounts | high | likely | partially_mitigated | DOMPurify markdown sanitization, `X-Content-Type-Options: nosniff` + CSP sandbox on downloads and on every `result_to_response` composite result (inserted after `wm_headers`; hop-by-hop names such as `Connection` rejected so a proxy cannot strip them) | GHSA-9c5c-hh3c-r9mc, GHSA-qxj7-hpx3-r892, GHSA-cf2x-rg8c-v63v, bb78b1c06d, 625b67dff0, WIN-2471 |
| T12 | Webhook authentication bypass / signature replay forges trigger invocations and approvals | remote_unauth | EP3 | Job execution integrity, approvals | high | likely | partially_mitigated | HMAC verification on some triggers; signing-oracle fix | GHSA-jw8c-h45c-xpjw, GHSA-hh9x-rcf8-xjr2, GHSA-q9g3-q6fj-hc2x, GHSA-8jc4-wj2p-2vmp, ab2a15b2a8 |
| T13 | Path traversal / arbitrary file read via log-reading and MCP path endpoints (incl. symlink following) | remote_auth | EP13 | Arbitrary files on server, global settings | high | likely | partially_mitigated | traversal checks + no-symlink-follow added | GHSA-4hrf-mgvv-xp9x, bb90f4ce83, df451aa64f, ad5ec293b5, 5f2d3e6812 |
| T14 | Privilege escalation via token rescope/refresh, script-issued JWTs, or operator-permission gaps | remote_auth | EP17, EP5 | Tokens, isolation, accounts | high | likely | partially_mitigated | monotonic-privilege enforcement on token lifecycle; SECURITY DEFINER triggers | GHSA-p62p-67xp-v775, GHSA-vv9w-wx3c-q3x2, 2ddf93de96, 865ab70c89, 33fb08cf3d |
backend/ee-repo-ref.txt
+1 -1
View File
@@ -1 +1 @@
d6aef91c0f7ba556befbf4addeb7674d4a9dd819
f5b783d2f7608e1ff3a817caa8b719e06f8b8981
backend/migrations/20260622072905_ai_free_token_usage.down.sql
+2
View File
@@ -0,0 +1,2 @@
DROP TABLE ai_free_token_daily_usage;
DROP TABLE ai_free_token_usage;
backend/migrations/20260622072905_ai_free_token_usage.up.sql
+19
View File
@@ -0,0 +1,19 @@
-- One-time grant of the Windmill-provided free AI tier, measured as cost in nano-dollars
-- (1e-9 USD) rather than raw tokens — a prompt-cache hit costs a fraction of a fresh input
-- token, so a token count wildly overstates the real bill. The grant never resets: once
-- spent, the user must bring their own API key. Keyed by normalized email so the allowance
-- is shared across a user's workspaces (and is resistant to +tag / gmail-dot aliasing).
CREATE TABLE ai_free_token_usage (
email VARCHAR(255) PRIMARY KEY,
cost_nanos BIGINT NOT NULL DEFAULT 0,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Instance-wide daily cost ceiling (nano-dollars) for the free tier — a kill-switch
-- independent of the per-user grant, bounding the blast radius of a bad day. One row per
-- UTC day.
CREATE TABLE ai_free_token_daily_usage (
day DATE PRIMARY KEY,
cost_nanos BIGINT NOT NULL DEFAULT 0,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
backend/migrations/20260826202939_queue_suspended_resume_at_index.down.sql
+1
View File
@@ -0,0 +1 @@
DROP INDEX IF EXISTS queue_suspended_v2;
backend/migrations/20260826202939_queue_suspended_resume_at_index.up.sql
+20
View File
@@ -0,0 +1,20 @@
-- Serves the suspended-job pull in windmill-common/src/worker.rs, whose resume test is the
-- indexed CASE expression. Two things about the shape are load-bearing:
-- * (priority DESC NULLS LAST, created_at) leads, so the scan yields that query's ORDER BY
-- and stops at the first match rather than sorting.
-- * the index is dropped before it is built rather than relying on IF NOT EXISTS. The
-- OVERRIDDEN_MIGRATIONS rewrite in windmill-api/src/db.rs runs these CONCURRENTLY, and an
-- interrupted concurrent build leaves the index present but invalid, which IF NOT EXISTS
-- would then skip rebuilding. Retiring the index this replaces is left to the migration
-- that follows, so this one can only ever be replayed while that index is still there to
-- cover the rebuild.
DROP INDEX IF EXISTS queue_suspended_v2;
CREATE INDEX IF NOT EXISTS queue_suspended_v2
ON v2_job_queue (
priority DESC NULLS LAST,
created_at,
(CASE WHEN suspend <= 0 THEN '-infinity'::timestamptz ELSE suspend_until END),
tag
)
WHERE suspend_until IS NOT NULL;
backend/migrations/20260826214706_queue_suspended_drop_legacy_index.down.sql
+3
View File
@@ -0,0 +1,3 @@
CREATE INDEX IF NOT EXISTS queue_suspended
ON v2_job_queue (priority DESC NULLS LAST, created_at, suspend_until, suspend, tag)
WHERE suspend_until IS NOT NULL;
backend/migrations/20260826214706_queue_suspended_drop_legacy_index.up.sql
+6
View File
@@ -0,0 +1,6 @@
-- Retires the index queue_suspended_v2 replaces. Separate from the migration that builds it
-- so that one is only ever replayed while this index still exists: sqlx records a migration
-- only after all its statements run, so a process that dies before the record is written
-- replays the build, and its leading DROP would otherwise be destroying the sole usable
-- index rather than an interrupted build.
DROP INDEX IF EXISTS queue_suspended;
backend/migrations/20260830085453_log_file_indexed_at.down.sql
+4
View File
@@ -0,0 +1,4 @@
DROP INDEX IF EXISTS index_log_file_premigration;
DROP INDEX IF EXISTS index_log_file_pending_path;
DROP INDEX IF EXISTS index_log_file_pending;
ALTER TABLE log_file DROP COLUMN IF EXISTS indexed_at;
backend/migrations/20260830085453_log_file_indexed_at.up.sql
+35
View File
@@ -0,0 +1,35 @@
-- The service log ingest walked `log_file` with a cursor over `log_ts`, which is when a line
-- was written rather than when its row appeared. Rows do not arrive in that order — an upload
-- retried after a failure, a host that has just started, a batch the row limit cut mid-minute —
-- and a row that becomes visible behind the cursor is never read: it stays in `log_file` and its
-- lines stay out of search until retention drops them.
--
-- No ordering fixes this. A cursor over arrival order fails the same way, because `nextval` is
-- allocated before its INSERT commits: a row can be assigned a lower value and commit after a
-- higher one has already moved the cursor past it. Which rows are outstanding is a property of
-- the rows, so it is recorded on them.
ALTER TABLE log_file ADD COLUMN indexed_at TIMESTAMPTZ;
-- Rows that already existed are marked, not queued: on a 14-day window most were ingested long
-- ago and their raw files are gone. A sentinel rather than a timestamp, because the indexer has to
-- tell them apart from rows registered since — those start NULL — and it puts the window's worth of
-- them back on the queue on its first pass, keeping only what the columnar store can vouch for.
--
-- Not split here on the cursor the old ingest had reached. Below that cursor sits every row it
-- skipped, which is the loss this migration exists to stop; recording those as done would carry the
-- bug into its own fix.
UPDATE log_file SET indexed_at = 'epoch' WHERE indexed_at IS NULL;
-- The work queue, and the only index the ingest query needs: outstanding rows are a small
-- fraction of the table, so this stays proportional to what is left to do rather than to the
-- retention window.
CREATE INDEX index_log_file_pending ON log_file (log_ts) WHERE indexed_at IS NULL;
-- A rebuild takes rows out of the queue by the file it just read out of the store, which is
-- the one lookup that arrives without a `log_ts`: the primary key is `(hostname, log_ts)`, so
-- nothing else covers it and each batch would scan every outstanding row instead.
CREATE INDEX index_log_file_pending_path ON log_file (hostname, file_path) WHERE indexed_at IS NULL;
-- Reached once per pass while pre-migration rows survive, and never again after the first
-- conversion clears them.
CREATE INDEX index_log_file_premigration ON log_file (log_ts) WHERE indexed_at = 'epoch';
backend/migrations/20260831120907_skills_resource_type.down.sql
+13
View File
@@ -0,0 +1,13 @@
-- The up migration only ever added: `ai_skill` still holds every skill it copied,
-- so there is nothing to restore and nothing to delete. Removing the resources
-- would destroy any a user has since edited or created, and removing a folder
-- would take whatever else was put in it.
--
-- The seeded resource type goes. `created_by` only distinguishes this migration's
-- row from one a user created by hand: a hub sync updates the schema in place and
-- leaves `created_by` alone, so a synced-over row is still removed here and the
-- next sync puts it back.
DELETE FROM resource_type
WHERE workspace_id = 'admins'
AND name = 'ai_skill'
AND created_by = 'system';
backend/migrations/20260831120907_skills_resource_type.up.sql
+88
View File
@@ -0,0 +1,88 @@
-- AI chat skills move from the `ai_skill` table onto ordinary resources, so they
-- gain folder ACLs, version history, workspace export and git-sync. An `ai_skill`
-- resource holds the SKILL.md body in `value.content`; its description lives in
-- the resource's own `description` column and its name is the path basename.
--
-- Nothing here is destructive. `ai_skill` is left in place, unread, for a later
-- release to drop once operators have confirmed the copy. That is what lets every
-- step below skip on conflict rather than resolve one: a skipped row is still in
-- the table, so it is not lost, and the migration needs no record of what it did
-- in order to be reversible.
-- `format_extension` makes the resource editor render `value.content` as a plain
-- .md file. Seeded under 'admins' so every workspace sees it.
INSERT INTO resource_type (workspace_id, name, schema, description, created_by, format_extension, edited_at)
VALUES (
'admins',
'ai_skill',
'{"type": "object", "properties": {"content": {"type": "string"}}}',
'A reusable instruction set for the AI chat, in the SKILL.md format. The resource description is what the assistant sees when deciding whether the skill applies; the file body is the instructions it follows.',
'system',
'md',
now()
)
ON CONFLICT (workspace_id, name) DO NOTHING;
-- Shared home matching the admin-only upload these skills had. A workspace that
-- already has a `skills` folder keeps it untouched, ACL and all: adopting one
-- would hand its own grants — possibly write for everyone — over a set of
-- instructions the assistant follows.
INSERT INTO folder (workspace_id, name, display_name, owners, extra_perms)
SELECT DISTINCT workspace_id, 'skills', 'Skills', ARRAY[]::TEXT[], '{"g/all": false}'::jsonb
FROM ai_skill
ON CONFLICT (workspace_id, name) DO NOTHING;
-- Copied only where the destination is free and the folder matches the one above
-- exactly, owners included: a pre-existing folder carrying the same ACL but an
-- owner would hand that owner update and delete over skills the removed API let
-- only workspace admins touch. Anything else stays in `ai_skill` for an operator
-- to place deliberately.
--
-- What was actually inserted is recorded rather than inferred. Inferring it from
-- "is there an ai_skill resource at the destination" reports nothing when the
-- blocker is itself an ai_skill with different instructions — the one case where
-- the skipped skill is least likely to be noticed.
CREATE TEMP TABLE ai_skill_copied AS
WITH inserted AS (
INSERT INTO resource (workspace_id, path, value, description, resource_type, created_by, edited_at)
SELECT
s.workspace_id,
'f/skills/' || s.name,
jsonb_build_object('content', s.instructions),
s.description,
'ai_skill',
s.edited_by,
s.edited_at
FROM ai_skill s
JOIN folder f
ON f.workspace_id = s.workspace_id
AND f.name = 'skills'
AND f.extra_perms = '{"g/all": false}'::jsonb
AND cardinality(f.owners) = 0
ON CONFLICT (workspace_id, path) DO NOTHING
RETURNING workspace_id, path
)
SELECT workspace_id, path FROM inserted;
-- Anything not copied is still in `ai_skill`, but nothing reads that table any
-- more, so from the app's side the skill is missing until an operator places it.
-- Name them rather than leaving that to be discovered.
DO $$
DECLARE
leftover RECORD;
BEGIN
FOR leftover IN
SELECT s.workspace_id, s.name
FROM ai_skill s
WHERE NOT EXISTS (
SELECT 1 FROM ai_skill_copied c
WHERE c.workspace_id = s.workspace_id
AND c.path = 'f/skills/' || s.name
)
LOOP
RAISE WARNING 'ai_skill %/% was not copied to a resource (its destination or the f/skills folder is already taken); it remains in the ai_skill table',
leftover.workspace_id, leftover.name;
END LOOP;
END $$;
DROP TABLE ai_skill_copied;
backend/migrations/20260901130041_drop_draft_password_fkey.down.sql
+12
View File
@@ -0,0 +1,12 @@
-- Drafts owned by a principal with no login account cannot exist under the constraint; drop them
-- before restoring it.
DELETE FROM draft
WHERE email IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM password WHERE password.email = draft.email);
ALTER TABLE draft
ADD CONSTRAINT draft_password_fkey
FOREIGN KEY (email)
REFERENCES password(email)
ON DELETE CASCADE
ON UPDATE CASCADE;
backend/migrations/20260901130041_drop_draft_password_fkey.up.sql
+3
View File
@@ -0,0 +1,3 @@
-- The delete and rename this cascaded are now explicit, at the sites that remove or rename an
-- account; `windmill_common::user_drafts::delete_drafts_of_email` carries the reasoning.
ALTER TABLE draft DROP CONSTRAINT IF EXISTS draft_password_fkey;
backend/parsers/windmill-parser-wasm/Cargo.lock
Generated
+24 -24
View File
@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windmill-common"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"aho-corasick",
"anyhow",
@@ -6274,7 +6274,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"proc-macro2",
"quote",
@@ -6286,7 +6286,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"convert_case",
"serde",
@@ -6295,7 +6295,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6307,7 +6307,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6319,7 +6319,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"gosyn",
@@ -6331,7 +6331,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6343,7 +6343,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6355,7 +6355,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -6366,7 +6366,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6377,7 +6377,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6389,7 +6389,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6400,7 +6400,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -6422,7 +6422,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6434,7 +6434,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6448,7 +6448,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"convert_case",
@@ -6465,7 +6465,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6478,7 +6478,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"serde",
@@ -6490,7 +6490,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6508,7 +6508,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -6524,7 +6524,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6540,7 +6540,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wasm"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"getrandom 0.2.17",
@@ -6572,7 +6572,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6586,7 +6586,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.796.0"
version = "1.803.0"
dependencies = [
"anyhow",
"bitflags",
backend/parsers/windmill-parser-wasm/Cargo.toml
+1 -1
View File
@@ -12,7 +12,7 @@ resolver = "2"
members = ["."]
[workspace.package]
version = "1.796.0"
version = "1.803.0"
edition = "2021"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
backend/src/main.rs
+92 -28
View File
@@ -7,14 +7,14 @@
*/
use anyhow::Context;
use monitor::{
load_base_url, load_otel, reload_critical_alerts_on_db_oversize,
reload_delete_logs_periodically_setting, reload_indexer_config,
reload_instance_python_version_setting, reload_maven_repos_setting,
flush_pending_log_files_to_object_store, load_base_url, load_otel,
reload_critical_alerts_on_db_oversize, reload_delete_logs_periodically_setting,
reload_indexer_config, reload_instance_python_version_setting, reload_maven_repos_setting,
reload_maven_settings_xml_setting, reload_no_default_maven_setting,
reload_nuget_config_setting, reload_powershell_repo_pat_setting,
reload_powershell_repo_url_setting, reload_ruby_repos_setting,
reload_timeout_wait_result_setting, reload_workspace_registries_setting,
send_current_log_file_to_object_store, send_logs_to_object_store, WORKERS_NAMES,
send_logs_to_object_store, WORKERS_NAMES,
};
use rand::Rng;
use sqlx::{Pool, Postgres};
@@ -53,16 +53,17 @@ use windmill_common::{
KEEP_JOB_DIR_SETTING, LICENSE_KEY_SETTING, MAVEN_REPOS_SETTING, MAVEN_SETTINGS_XML_SETTING,
MONITOR_LOGS_ON_OBJECT_STORE_SETTING, NO_DEFAULT_MAVEN_SETTING,
NPM_CONFIG_REGISTRY_SETTING, NSJAIL_TMPFS_SIZE_MB_SETTING, NSJAIL_TMP_BACKING_SETTING,
NUGET_CONFIG_SETTING, OAUTH_SETTING, OTEL_SETTING, OTEL_TRACING_PROXY_SETTING,
PIP_INDEX_URL_SETTING, POWERSHELL_REPO_PAT_SETTING, POWERSHELL_REPO_URL_SETTING,
PREVIEW_TAGS_OVERRIDE_SETTING, REQUEST_SIZE_LIMIT_SETTING,
NUGET_CONFIG_SETTING, OAUTH_SETTING, OTEL_SETTING, OTEL_TRACES_RETENTION_SECS_SETTING,
OTEL_TRACING_PROXY_SETTING, PIP_INDEX_URL_SETTING, POWERSHELL_REPO_PAT_SETTING,
POWERSHELL_REPO_URL_SETTING, PREVIEW_TAGS_OVERRIDE_SETTING, REQUEST_SIZE_LIMIT_SETTING,
REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RESTART_COORDINATION_SETTING,
RETENTION_PERIOD_SECS_OVERRIDES_SETTING, RETENTION_PERIOD_SECS_SETTING, RUBY_REPOS_SETTING,
SAML_METADATA_SETTING, SANDBOX_IMAGE_CACHE_MAX_MB_SETTING,
SANDBOX_IMAGE_DEFAULT_REGISTRY_SETTING, SANDBOX_IMAGE_MAX_SIZE_MB_SETTING,
SANDBOX_IMAGE_PULL_POLICY_SETTING, SANDBOX_REGISTRY_AUTH_SETTING, SCIM_TOKEN_SETTING,
SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING, TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
UV_EXCLUDE_NEWER_SETTING, UV_INDEX_STRATEGY_SETTING, UV_PYTHON_INSTALL_MIRROR_SETTING,
SERVICE_LOG_RETENTION_SECS_SETTING, SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING,
TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING, UV_EXCLUDE_NEWER_SETTING,
UV_INDEX_STRATEGY_SETTING, UV_PYTHON_INSTALL_MIRROR_SETTING,
WORKSPACE_FAIRNESS_DURATION_SECS_SETTING, WORKSPACE_FAIRNESS_ENABLED_SETTING,
WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING,
WORKSPACE_MAX_QUEUED_JOBS_SETTING, WORKSPACE_REGISTRIES_SETTING,
@@ -139,11 +140,12 @@ use crate::monitor::{
reload_instance_events_webhook_setting, reload_job_default_timeout_setting,
reload_job_isolation_setting, reload_jwt_secret_setting, reload_license_key,
reload_npm_config_registry_setting, reload_nsjail_tmp_backing_setting,
reload_nsjail_tmpfs_size_setting, reload_otel_tracing_proxy_setting,
reload_pip_index_url_setting, reload_retention_period_setting,
reload_sandbox_image_cache_max_setting, reload_sandbox_image_default_registry_setting,
reload_sandbox_image_max_size_setting, reload_sandbox_image_pull_policy_setting,
reload_sandbox_registry_auth_setting, reload_scim_token_setting, reload_smtp_config,
reload_nsjail_tmpfs_size_setting, reload_otel_traces_retention_secs_setting,
reload_otel_tracing_proxy_setting, reload_pip_index_url_setting,
reload_retention_period_setting, reload_sandbox_image_cache_max_setting,
reload_sandbox_image_default_registry_setting, reload_sandbox_image_max_size_setting,
reload_sandbox_image_pull_policy_setting, reload_sandbox_registry_auth_setting,
reload_scim_token_setting, reload_service_log_retention_secs_setting, reload_smtp_config,
reload_store_audit_logs_s3_setting, reload_uv_exclude_newer_setting,
reload_uv_index_strategy_setting, reload_uv_python_install_mirror_setting,
reload_worker_config, MonitorIteration,
@@ -380,6 +382,7 @@ async fn cache_hub_scripts(file_path: Option<String>) -> anyhow::Result<()> {
"",
&mut None,
&None,
None,
)
.await
{
@@ -403,8 +406,12 @@ struct HubResourceTypeRaw {
pub schema: Option<String>,
pub app: String,
pub description: Option<String>,
/// Absent from hubs predating the column, and from caches written before it.
#[serde(default)]
pub format_extension: Option<String>,
}
/// Processed resource type with parsed schema
#[derive(serde::Deserialize, serde::Serialize, Clone)]
pub struct HubResourceType {
@@ -413,6 +420,18 @@ pub struct HubResourceType {
pub schema: Option<serde_json::Value>,
pub app: String,
pub description: Option<String>,
/// Doubly optional on purpose. A cache written before this column has no key at
/// all (`None`) and must leave the stored extension alone; one written since
/// always writes the key, so an explicit null (`Some(None)`) is the hub genuinely
/// dropping it and must clear. A single `Option` conflates the two, and picking
/// either meaning breaks the other — as does plain serde, which folds `null`
/// into the outer `None`, hence the wrapping deserializer.
#[serde(
default,
deserialize_with = "windmill_common::more_serde::double_option",
skip_serializing_if = "Option::is_none"
)]
pub format_extension: Option<Option<String>>,
}
const HUB_RT_CACHE_FILE: &str = "resource_types.json";
@@ -459,6 +478,7 @@ async fn cache_hub_resource_types() -> anyhow::Result<()> {
schema,
app: rt.app,
description: rt.description,
format_extension: Some(rt.format_extension),
})
})
.collect();
@@ -500,9 +520,17 @@ pub async fn sync_cached_resource_types(db: &sqlx::Pool<sqlx::Postgres>) -> anyh
tracing::info!("Found {} cached resource types", cached_types.len());
// Get existing resource types in admins workspace
let existing_types: Vec<(String, Option<serde_json::Value>, Option<String>)> = sqlx::query_as(
"SELECT name, schema, description FROM resource_type WHERE workspace_id = 'admins'",
// Get existing resource types in admins workspace. `format_extension` is part of
// the comparison below, so a type whose only change is gaining or losing it is
// not mistaken for unchanged; `is_fileset` decides whether it may take one.
let existing_types: Vec<(
String,
Option<serde_json::Value>,
Option<String>,
Option<String>,
bool,
)> = sqlx::query_as(
"SELECT name, schema, description, format_extension, is_fileset FROM resource_type WHERE workspace_id = 'admins'",
)
.fetch_all(db)
.await
@@ -510,19 +538,42 @@ pub async fn sync_cached_resource_types(db: &sqlx::Pool<sqlx::Postgres>) -> anyh
let existing_map: std::collections::HashMap<
String,
(Option<serde_json::Value>, Option<String>),
(Option<serde_json::Value>, Option<String>, Option<String>, bool),
> = existing_types
.into_iter()
.map(|(name, schema, desc)| (name, (schema, desc)))
.map(|(name, schema, desc, format_extension, is_fileset)| {
(name, (schema, desc, format_extension, is_fileset))
})
.collect();
let mut synced_count = 0;
let mut skipped_count = 0;
for rt in cached_types {
// Check if resource type already exists with same schema and description
if let Some((existing_schema, existing_desc)) = existing_map.get(&rt.name) {
if existing_schema == &rt.schema && existing_desc == &rt.description {
let existing = existing_map.get(&rt.name);
let is_fileset = existing.map(|(_, _, _, f)| *f).unwrap_or(false);
let stored_extension = existing.and_then(|(_, _, e, _)| e.clone());
// A fileset is a set of files, so it cannot also be one file. Create, update
// and the manual sync all reject the pair; this writer would otherwise
// persist it onto a same-named local fileset.
//
// A cache with no key at all leaves the stored value alone, so the target is
// what is already there — which is also what makes the comparison below
// agree with the write instead of re-upserting the row on every boot.
let format_extension = if is_fileset {
None
} else {
match &rt.format_extension {
Some(from_cache) => from_cache.clone(),
None => stored_extension.clone(),
}
};
if let Some((existing_schema, existing_desc, _, _)) = existing {
if existing_schema == &rt.schema
&& existing_desc == &rt.description
&& stored_extension == format_extension
{
skipped_count += 1;
continue;
}
@@ -530,14 +581,19 @@ pub async fn sync_cached_resource_types(db: &sqlx::Pool<sqlx::Postgres>) -> anyh
// Insert or update resource type
sqlx::query(
"INSERT INTO resource_type (workspace_id, name, schema, description, edited_at)
VALUES ('admins', $1, $2, $3, now())
// `format_extension` is resolved above rather than coalesced here: a
// COALESCE could never clear one, so a hub that dropped an extension
// would leave the stale value behind forever.
"INSERT INTO resource_type (workspace_id, name, schema, description, format_extension, edited_at)
VALUES ('admins', $1, $2, $3, $4, now())
ON CONFLICT (workspace_id, name) DO UPDATE
SET schema = EXCLUDED.schema, description = EXCLUDED.description, edited_at = now()",
SET schema = EXCLUDED.schema, description = EXCLUDED.description,
format_extension = EXCLUDED.format_extension, edited_at = now()",
)
.bind(&rt.name)
.bind(&rt.schema)
.bind(&rt.description)
.bind(&format_extension)
.execute(db)
.await
.with_context(|| format!("Failed to upsert resource type {}", rt.name))?;
@@ -1261,10 +1317,12 @@ Windmill Community Edition {GIT_VERSION}
#[cfg(all(feature = "tantivy", feature = "parquet"))]
let log_indexer_f = {
let log_indexer_rx = killpill_rx.resubscribe();
let log_index_writer2 = log_index_writer.clone();
// Moved, not cloned: sealing a chunk takes sole ownership of its
// tantivy writer, which a second live handle would silently prevent.
let moved_log_index_writer = log_index_writer;
async {
if let Some(db) = conn.as_sql() {
if let Some(log_index_writer) = log_index_writer2 {
if let Some(log_index_writer) = moved_log_index_writer {
windmill_indexer::service_logs_oss::run_indexer(
db.clone(),
log_index_writer,
@@ -1661,7 +1719,7 @@ Windmill Community Edition {GIT_VERSION}
} else {
tracing::info!("Nothing to do, exiting.");
}
send_current_log_file_to_object_store(&conn, &hostname, &mode).await;
flush_pending_log_files_to_object_store(&conn, &hostname, &mode).await;
if let Some(db) = conn.as_sql() {
tracing::info!("Exiting connection pool");
@@ -1952,6 +2010,12 @@ async fn process_notify_event(
}
TIMEOUT_WAIT_RESULT_SETTING => reload_timeout_wait_result_setting(conn).await,
RETENTION_PERIOD_SECS_SETTING => reload_retention_period_setting(conn).await,
SERVICE_LOG_RETENTION_SECS_SETTING => {
reload_service_log_retention_secs_setting(conn).await
}
OTEL_TRACES_RETENTION_SECS_SETTING => {
reload_otel_traces_retention_secs_setting(conn).await
}
RETENTION_PERIOD_SECS_OVERRIDES_SETTING => {
if let Err(e) = load_retention_period_overrides(db).await {
tracing::error!("Error loading per-workspace retention overrides: {e:#}");
backend/src/monitor.rs
+464 -149
View File
@@ -64,24 +64,24 @@ use windmill_common::{
JOB_ISOLATION_SETTING, JWT_SECRET_SETTING, KEEP_JOB_DIR_SETTING, LICENSE_KEY_SETTING,
MONITOR_LOGS_ON_OBJECT_STORE_SETTING, NPMRC_SETTING, NPM_CONFIG_REGISTRY_SETTING,
NSJAIL_TMPFS_SIZE_MB_SETTING, NSJAIL_TMP_BACKING_SETTING, NUGET_CONFIG_SETTING,
OTEL_SETTING, OTEL_TRACING_PROXY_SETTING, PIP_INDEX_URL_SETTING,
POWERSHELL_REPO_PAT_SETTING, POWERSHELL_REPO_URL_SETTING, PREVIEW_TAGS_OVERRIDE_SETTING,
REQUEST_SIZE_LIMIT_SETTING, REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING,
RETENTION_PERIOD_SECS_SETTING, SAML_METADATA_SETTING, SANDBOX_IMAGE_CACHE_MAX_MB_SETTING,
OTEL_SETTING, OTEL_TRACES_RETENTION_SECS_SETTING, OTEL_TRACING_PROXY_SETTING,
PIP_INDEX_URL_SETTING, POWERSHELL_REPO_PAT_SETTING, POWERSHELL_REPO_URL_SETTING,
PREVIEW_TAGS_OVERRIDE_SETTING, REQUEST_SIZE_LIMIT_SETTING,
REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RETENTION_PERIOD_SECS_SETTING,
SAML_METADATA_SETTING, SANDBOX_IMAGE_CACHE_MAX_MB_SETTING,
SANDBOX_IMAGE_DEFAULT_REGISTRY_SETTING, SANDBOX_IMAGE_MAX_SIZE_MB_SETTING,
SANDBOX_IMAGE_PULL_POLICY_SETTING, SANDBOX_REGISTRY_AUTH_SETTING, SCIM_TOKEN_SETTING,
SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
UV_EXCLUDE_NEWER_SETTING, UV_INDEX_STRATEGY_SETTING, UV_PYTHON_INSTALL_MIRROR_SETTING,
WORKSPACE_FAIRNESS_DURATION_SECS_SETTING, WORKSPACE_FAIRNESS_ENABLED_SETTING,
WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING,
WORKSPACE_MAX_QUEUED_JOBS_SETTING,
SERVICE_LOG_RETENTION_SECS_SETTING, SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING,
TIMEOUT_WAIT_RESULT_SETTING, UV_EXCLUDE_NEWER_SETTING, UV_INDEX_STRATEGY_SETTING,
UV_PYTHON_INSTALL_MIRROR_SETTING, WORKSPACE_FAIRNESS_DURATION_SECS_SETTING,
WORKSPACE_FAIRNESS_ENABLED_SETTING, WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING,
WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING, WORKSPACE_MAX_QUEUED_JOBS_SETTING,
},
indexer::load_indexer_config,
jobs::delete_jobs,
jwt::JWT_SECRET,
oauth2::REQUIRE_PREEXISTING_USER_FOR_OAUTH,
server::load_smtp_config,
tracing_init::JSON_FMT,
users::truncate_token,
utils::{empty_as_none, now_from_db, report_critical_error, Mode, HUB_API_SECRET},
worker::{
@@ -98,10 +98,10 @@ use windmill_common::{
KillpillSender, AUDIT_LOG_RETENTION_DAYS, BASE_URL, CRITICAL_ALERTS_ON_DB_OVERSIZE,
CRITICAL_ALERTS_ON_TOKEN_EXPIRY, CRITICAL_ALERT_MUTE_UI_ENABLED,
CRITICAL_ALERT_MUTE_ZOMBIE_JOB_RESTART, CRITICAL_ERROR_CHANNELS, DB, DEFAULT_HUB_BASE_URL,
HUB_BASE_URL, JOB_RETENTION_SECS, JOB_RETENTION_SECS_OVERRIDES,
JOB_RETENTION_SECS_OVERRIDES_LOADED, METRICS_DEBUG_ENABLED, METRICS_ENABLED,
MONITOR_LOGS_ON_OBJECT_STORE, OTEL_LOGS_ENABLED, OTEL_METRICS_ENABLED, OTEL_TRACING_ENABLED,
SERVICE_LOG_RETENTION_SECS, STORE_AUDIT_LOGS_S3,
DEFAULT_OTEL_TRACES_RETENTION_SECS, DEFAULT_SERVICE_LOG_RETENTION_SECS, HUB_BASE_URL,
JOB_RETENTION_SECS, JOB_RETENTION_SECS_OVERRIDES, JOB_RETENTION_SECS_OVERRIDES_LOADED,
METRICS_DEBUG_ENABLED, METRICS_ENABLED, MONITOR_LOGS_ON_OBJECT_STORE, OTEL_LOGS_ENABLED,
OTEL_METRICS_ENABLED, OTEL_TRACING_ENABLED, STORE_AUDIT_LOGS_S3,
};
use windmill_common::{
client::AuthedClient,
@@ -476,6 +476,19 @@ pub async fn initial_load(
|v: Option<String>| async move { HUB_API_SECRET.store(std::sync::Arc::new(v)) },
);
// Outside the `server_mode` guard below: every mode reads this. A worker registers its
// rotated log files against the cutoff, and a dedicated indexer trims the search index to a
// window derived from it — neither is a server.
pass.setting(SERVICE_LOG_RETENTION_SECS_SETTING, true, |v| async move {
windmill_common::set_service_log_retention_secs(parse_setting_value::<i64>(
v,
SERVICE_LOG_RETENTION_SECS_SETTING,
"SERVICE_LOG_RETENTION_SECS",
DEFAULT_SERVICE_LOG_RETENTION_SECS,
|x| x,
))
});
if server_mode {
pass.setting(RETENTION_PERIOD_SECS_SETTING, true, |v| async move {
JOB_RETENTION_SECS.store(
@@ -502,6 +515,15 @@ pub async fn initial_load(
Ordering::Relaxed,
)
});
pass.setting(OTEL_TRACES_RETENTION_SECS_SETTING, true, |v| async move {
windmill_common::set_otel_traces_retention_secs(parse_setting_value::<i64>(
v,
OTEL_TRACES_RETENTION_SECS_SETTING,
"OTEL_TRACES_RETENTION_SECS",
DEFAULT_OTEL_TRACES_RETENTION_SECS,
|x| x,
))
});
pass.setting(STORE_AUDIT_LOGS_S3_SETTING, true, |v| async move {
STORE_AUDIT_LOGS_S3.store(
parse_setting_value::<bool>(
@@ -1221,32 +1243,60 @@ async fn sleep_until_next_minute_start_plus_one_s() {
}
use windmill_common::tracing_init::TMP_WINDMILL_LOGS_SERVICE;
async fn find_two_highest_files(hostname: &str) -> (Option<String>, Option<String>) {
/// The minutely rolling appender names each file `<hostname>.log.<%Y-%m-%d-%H-%M>`;
/// anything else in the directory is not a rotated log file.
fn parse_log_file_ts(file_name: &str) -> Option<NaiveDateTime> {
NaiveDateTime::parse_from_str(
file_name.rsplit('.').next()?,
windmill_common::tracing_init::LOG_TIMESTAMP_FMT,
)
.ok()
}
/// Oldest first. Readdir order is filesystem-dependent — tmpfs hands back the
/// newest entry first, ext4 hashes the names — so the listing has to be sorted
/// before anything picks a file out of it.
fn sorted_log_files(file_names: impl Iterator<Item = String>) -> Vec<(NaiveDateTime, String)> {
let mut files = file_names
.filter_map(|name| parse_log_file_ts(&name).map(|ts| (ts, name)))
.collect::<Vec<_>>();
files.sort();
files
}
/// Every log file but the newest one: that one is still being appended to, every
/// older one is final.
fn rotated_log_files(file_names: impl Iterator<Item = String>) -> Vec<(NaiveDateTime, String)> {
let mut files = sorted_log_files(file_names);
files.pop();
files
}
async fn read_log_file_names(hostname: &str) -> Vec<String> {
let log_dir = format!("{}/{}/", *TMP_WINDMILL_LOGS_SERVICE, hostname);
let rd_dir = tokio::fs::read_dir(log_dir).await;
if let Ok(mut log_files) = rd_dir {
let mut highest_file: Option<String> = None;
let mut second_highest_file: Option<String> = None;
while let Ok(Some(file)) = log_files.next_entry().await {
let file_name = file
.file_name()
.to_str()
.map(|x| x.to_string())
.unwrap_or_default();
if file_name > highest_file.clone().unwrap_or_default() {
second_highest_file = highest_file;
highest_file = Some(file_name);
}
let mut rd_dir = match tokio::fs::read_dir(&log_dir).await {
Ok(rd_dir) => rd_dir,
Err(e) => {
tracing::error!("Error reading log files: {}, {:#?}", log_dir, e);
return vec![];
}
};
let mut file_names = vec![];
while let Ok(Some(file)) = rd_dir.next_entry().await {
if let Some(file_name) = file.file_name().to_str() {
file_names.push(file_name.to_string());
}
(highest_file, second_highest_file)
} else {
tracing::error!(
"Error reading log files: {}, {:#?}",
*TMP_WINDMILL_LOGS_SERVICE,
rd_dir.unwrap_err()
);
(None, None)
}
file_names
}
async fn list_log_files(hostname: &str) -> Vec<(NaiveDateTime, String)> {
sorted_log_files(read_log_file_names(hostname).await.into_iter())
}
async fn list_rotated_log_files(hostname: &str) -> Vec<(NaiveDateTime, String)> {
rotated_log_files(read_log_file_names(hostname).await.into_iter())
}
fn get_worker_group(mode: &Mode) -> Option<String> {
@@ -1266,133 +1316,188 @@ pub fn send_logs_to_object_store(conn: &Connection, hostname: &str, mode: &Mode)
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(10));
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
init_last_log_file_sent(&conn, &hostname).await;
sleep_until_next_minute_start_plus_one_s().await;
loop {
interval.tick().await;
let (_, snd_highest_file) = find_two_highest_files(&hostname).await;
send_log_file_to_object_store(
&hostname,
&mode,
&worker_group,
&conn,
snd_highest_file,
false,
)
.await;
let files = list_rotated_log_files(&hostname).await;
send_log_files_to_object_store(&hostname, &mode, &worker_group, &conn, files).await;
}
});
}
pub async fn send_current_log_file_to_object_store(conn: &Connection, hostname: &str, mode: &Mode) {
tracing::info!("Sending current log file to object store");
let (highest_file, _) = find_two_highest_files(hostname).await;
pub async fn flush_pending_log_files_to_object_store(
conn: &Connection,
hostname: &str,
mode: &Mode,
) {
tracing::info!("Sending pending log files to object store");
let worker_group = get_worker_group(&mode);
send_log_file_to_object_store(hostname, mode, &worker_group, conn, highest_file, true).await;
}
fn get_now_and_str() -> (NaiveDateTime, String) {
let ts = Utc::now().naive_utc();
(
ts,
ts.format(windmill_common::tracing_init::LOG_TIMESTAMP_FMT)
.to_string(),
)
// Nothing rotates after this, so the file still being appended to is registered
// here, along with any rotated one the loop had not reached yet. Bounded like the
// pool close that follows: a backlog against a slow object store would otherwise
// hold the process past its termination grace period. Whatever is left over is
// registered by the next run's catch-up.
let flush = async {
let files = list_log_files(hostname).await;
send_log_files_to_object_store(hostname, mode, &worker_group, conn, files).await;
};
if timeout(Duration::from_secs(15), flush).await.is_err() {
tracing::warn!("Could not send all pending log files in time (15s). Exiting anyway.");
}
}
lazy_static::lazy_static! {
static ref LAST_LOG_FILE_SENT: Arc<Mutex<Option<NaiveDateTime>>> = Arc::new(Mutex::new(None));
/// Serializes the periodic uploader against the shutdown flush. The uploader is a
/// detached task that keeps ticking while the flush runs and both walk the same
/// files, so without this both can clear the watermark for one file and count its
/// lines twice through the additive upsert.
static ref SENDING_LOG_FILES: tokio::sync::Mutex<()> = tokio::sync::Mutex::new(());
}
fn last_log_file_sent() -> Option<NaiveDateTime> {
LAST_LOG_FILE_SENT.lock().ok().and_then(|ts| *ts)
}
/// Resume from what this host already registered, so a previous run's leftovers reach
/// the object store rather than being dropped. Their line counts come out zero, this
/// run having counted none of them, which only flattens their bars in the UI.
///
/// The newest registered minute is left out on purpose: the shutdown flush registers
/// the file that was still open and the appender reopens that minute in append mode,
/// so a restart inside it would otherwise strand everything written afterwards.
///
/// A row rewritten this way restores the object and sums the counters, but it keeps the
/// `indexed_at` it already had, so one the indexers have taken is not offered again and
/// the lines added by the rewrite stay out of search.
async fn init_last_log_file_sent(conn: &Connection, hostname: &str) {
let Some(db) = conn.as_sql() else {
return;
};
match sqlx::query_scalar!(
"SELECT max(log_ts) FROM log_file
WHERE hostname = $1 AND log_ts < (SELECT max(log_ts) FROM log_file WHERE hostname = $1)",
hostname
)
.fetch_one(db)
.await
{
Ok(Some(ts)) => {
if let Err(e) = LAST_LOG_FILE_SENT.lock().map(|mut last_log_file_sent| {
last_log_file_sent.replace(ts);
}) {
tracing::error!("Error initializing last log file sent: {:?}", e);
}
}
Ok(None) => {}
Err(e) => tracing::error!("Error loading last log file sent: {:?}", e),
}
}
async fn send_log_files_to_object_store(
hostname: &str,
mode: &Mode,
worker_group: &Option<String>,
conn: &Connection,
files: Vec<(NaiveDateTime, String)>,
) {
let _guard = SENDING_LOG_FILES.lock().await;
let retention_cutoff = Utc::now().naive_utc()
- chrono::Duration::seconds(windmill_common::service_log_retention_secs());
for (ts, file_name) in files {
if last_log_file_sent().is_some_and(|last| last >= ts) {
continue;
}
// A run coming back from a long outage still finds its predecessor's files on
// disk. Registering one past the retention cutoff inserts a row
// `delete_expired_items` drops on its next pass, once the indexers have already
// paid to parse it.
if ts < retention_cutoff {
continue;
}
// Stop at the first failure rather than moving on, so a file is never
// registered before an older one that has not made it to the store yet.
// The indexers do not depend on that ordering — every row is offered until
// it is marked — but a gap here would still be visible while it lasts.
if !send_log_file_to_object_store(hostname, mode, worker_group, conn, &file_name, ts).await
{
break;
}
}
}
/// Returns whether the file ended up registered in `log_file`.
async fn send_log_file_to_object_store(
hostname: &str,
mode: &Mode,
worker_group: &Option<String>,
conn: &Connection,
snd_highest_file: Option<String>,
use_now: bool,
) {
if let Some(highest_file) = snd_highest_file {
//parse datetime frome file xxxx.yyyy-MM-dd-HH-mm
let (ts, ts_str) = if use_now {
get_now_and_str()
} else {
highest_file
.split(".")
.last()
.and_then(|x| {
NaiveDateTime::parse_from_str(
x,
windmill_common::tracing_init::LOG_TIMESTAMP_FMT,
)
.ok()
.map(|y| (y, x.to_string()))
})
.unwrap_or_else(get_now_and_str)
};
file_name: &str,
ts: NaiveDateTime,
) -> bool {
#[cfg(feature = "parquet")]
if let Some(s3_client) = windmill_object_store::get_object_store().await {
let path = std::path::Path::new(&*TMP_WINDMILL_LOGS_SERVICE)
.join(hostname)
.join(file_name);
let exists = LAST_LOG_FILE_SENT.lock().map(|last_log_file_sent| {
last_log_file_sent
.map(|last_log_file_sent| last_log_file_sent >= ts)
.unwrap_or(false)
});
if exists.unwrap_or(false) {
return;
}
#[cfg(feature = "parquet")]
let s3_client = windmill_object_store::get_object_store().await;
#[cfg(feature = "parquet")]
if let Some(s3_client) = s3_client {
let path = std::path::Path::new(&*TMP_WINDMILL_LOGS_SERVICE)
.join(hostname)
.join(&highest_file);
//read file as byte stream
let bytes = tokio::fs::read(&path).await;
if let Err(e) = bytes {
//read file as byte stream
let bytes = match tokio::fs::read(&path).await {
Ok(bytes) => bytes,
Err(e) => {
tracing::error!("Error reading log file: {:?}", e);
return;
return false;
}
let path = windmill_object_store::object_store_reexports::Path::from_url_path(format!(
"{}{hostname}/{highest_file}",
windmill_common::tracing_init::LOGS_SERVICE
));
if let Err(e) = path {
};
let path = windmill_object_store::object_store_reexports::Path::from_url_path(format!(
"{}{hostname}/{file_name}",
windmill_common::tracing_init::LOGS_SERVICE
));
let path = match path {
Ok(path) => path,
Err(e) => {
tracing::error!("Error creating log file path: {:?}", e);
return;
}
if let Err(e) = s3_client.put(&path.unwrap(), bytes.unwrap().into()).await {
tracing::error!("Error sending logs to object store: {:?}", e);
return false;
}
};
if let Err(e) = s3_client.put(&path, bytes.into()).await {
tracing::error!("Error sending logs to object store: {:?}", e);
return false;
}
}
let (ok_lines, err_lines) = read_log_counters(ts_str);
let ts_str = ts
.format(windmill_common::tracing_init::LOG_TIMESTAMP_FMT)
.to_string();
let (ok_lines, err_lines) = read_log_counters(ts_str);
if let Some(db) = conn.as_sql() {
match timeout(Duration::from_secs(10), sqlx::query!("INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)
VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)
ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
hostname, mode.to_string(), worker_group.clone(), ts, highest_file, ok_lines as i64, err_lines as i64, *JSON_FMT)
.execute(db)).await {
Ok(Ok(_)) => {
if let Err(e) = LAST_LOG_FILE_SENT.lock().map(|mut last_log_file_sent| {
last_log_file_sent.replace(ts);
}) {
tracing::error!("Error updating last log file sent: {:?}", e);
}
tracing::info!("Log file sent: {}", highest_file);
}
Ok(Err(e)) => {
tracing::error!("Error inserting log file: {:?}", e);
}
Err(e) => {
tracing::error!("Error inserting log file, timeout elapsed: {:?}", e);
}
let Some(db) = conn.as_sql() else {
// not sending log file to object store in agent mode
return false;
};
match timeout(Duration::from_secs(10), sqlx::query!("INSERT INTO log_file (hostname, mode, worker_group, log_ts, file_path, ok_lines, err_lines, json_fmt)
VALUES ($1, $2::text::LOG_MODE, $3, $4, $5, $6, $7, $8)
ON CONFLICT (hostname, log_ts) DO UPDATE SET ok_lines = log_file.ok_lines + $6, err_lines = log_file.err_lines + $7",
hostname, mode.to_string(), worker_group.clone(), ts, file_name, ok_lines as i64, err_lines as i64, true)
.execute(db)).await {
Ok(Ok(_)) => {
if let Err(e) = LAST_LOG_FILE_SENT.lock().map(|mut last_log_file_sent| {
last_log_file_sent.replace(ts);
}) {
tracing::error!("Error updating last log file sent: {:?}", e);
}
} else {
// tracing::warn!("Not sending log file to object store in agent mode");
()
tracing::info!("Log file sent: {}", file_name);
true
}
Ok(Err(e)) => {
tracing::error!("Error inserting log file: {:?}", e);
false
}
Err(e) => {
tracing::error!("Error inserting log file, timeout elapsed: {:?}", e);
false
}
}
}
@@ -1580,6 +1685,64 @@ pub async fn trim_resource_versions(db: &DB) -> () {
}
}
/// Matches the batch the settings-page cleanup uses for the same table.
const SERVICE_LOG_DELETE_BATCH: i64 = 2_000;
/// Batches per pass. `monitor_db` runs under a 600s timeout that cancels every maintenance
/// future in the same `join!` and reports a critical error, so a large backlog has to drain
/// across ticks rather than inside one, the way the neighbouring sweeps already do.
const SERVICE_LOG_DELETE_MAX_BATCHES: usize = 10;
/// One span per HTTP request made from a job script, so the table grows far faster than the
/// job table it is keyed against; batched for the same reason the service log sweep is.
const OTEL_TRACES_DELETE_BATCH: i64 = 10_000;
const OTEL_TRACES_DELETE_MAX_BATCHES: usize = 10;
/// Delete HTTP request tracing spans older than `retention_secs`, returning how many went.
///
/// `retention_secs` is a parameter rather than a read of the process-wide setting so a test can
/// pin a window without writing state the other tests in this binary run against concurrently.
async fn delete_expired_otel_traces(db: &DB, retention_secs: i64) -> u64 {
// `start_time_unix_nano` is the proto field stored verbatim, so the cutoff is built in that
// unit rather than compared against `now()`. Truncating the epoch to whole seconds first
// keeps the multiplication inside `bigint`.
//
// Batched on `ctid`, not on the `(trace_id, span_id)` primary key: with the key the planner
// hashes the LIMITed subquery and Seq Scans the whole table to probe it, which at the size
// this table reaches is the cost the batching exists to avoid. `ctid` plans as a Tid Scan, so
// each batch touches only the rows it deletes. Safe because the subquery and the delete share
// one snapshot, and spans are never updated after insert.
let mut deleted = 0;
for _ in 0..OTEL_TRACES_DELETE_MAX_BATCHES {
let batch = sqlx::query!(
"DELETE FROM otel_traces WHERE ctid IN (
SELECT ctid FROM otel_traces
WHERE start_time_unix_nano < EXTRACT(
EPOCH FROM now() - ($1::bigint::text || ' s')::interval
)::bigint * 1000000000
LIMIT $2
)",
retention_secs,
OTEL_TRACES_DELETE_BATCH,
)
.execute(db)
.await;
match batch {
Ok(res) => {
deleted += res.rows_affected();
if (res.rows_affected() as i64) < OTEL_TRACES_DELETE_BATCH {
break;
}
}
Err(e) => {
tracing::error!("Error deleting expired otel trace spans: {:?}", e);
break;
}
}
}
deleted
}
pub async fn delete_expired_items(db: &DB) -> () {
let expired_tokens_r = sqlx::query_as!(
TokenRow,
@@ -1662,23 +1825,54 @@ pub async fn delete_expired_items(db: &DB) -> () {
Err(e) => tracing::error!("Error deleting cache resource {}", e.to_string()),
}
match sqlx::query_as!(
LogFile,
"DELETE FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval RETURNING file_path, hostname",
SERVICE_LOG_RETENTION_SECS,
)
.fetch_all(db)
.await
{
Ok(log_files_to_delete) => {
// Batched: every process rotates a log file a minute, so lowering the retention makes one
// ordinary setting change expire millions of rows at once. An unbounded `DELETE ...
// RETURNING` would materialize all of them, and their deletion futures, in this one tick.
for _ in 0..SERVICE_LOG_DELETE_MAX_BATCHES {
let batch = sqlx::query_as!(
LogFile,
"DELETE FROM log_file WHERE (hostname, log_ts) IN (
SELECT hostname, log_ts FROM log_file
WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval
LIMIT $2
) RETURNING file_path, hostname",
windmill_common::service_log_retention_secs(),
SERVICE_LOG_DELETE_BATCH,
)
.fetch_all(db)
.await;
match batch {
Ok(log_files_to_delete) => {
if log_files_to_delete.is_empty() {
break;
}
let n = log_files_to_delete.len();
let paths = log_files_to_delete
.iter()
.map(|f| format!("{}/{}", f.hostname, f.file_path))
.collect();
delete_log_files_from_disk_and_store(paths, &*TMP_WINDMILL_LOGS_SERVICE, windmill_common::tracing_init::LOGS_SERVICE).await;
delete_log_files_from_disk_and_store(
paths,
&*TMP_WINDMILL_LOGS_SERVICE,
windmill_common::tracing_init::LOGS_SERVICE,
)
.await;
if (n as i64) < SERVICE_LOG_DELETE_BATCH {
break;
}
}
Err(e) => {
tracing::error!("Error deleting log file: {:?}", e);
break;
}
}
Err(e) => tracing::error!("Error deleting log file: {:?}", e),
}
let deleted_spans =
delete_expired_otel_traces(db, windmill_common::otel_traces_retention_secs()).await;
if deleted_spans > 0 {
tracing::info!("deleted {} expired otel trace spans", deleted_spans);
}
let audit_retention_days = audit_log_retention_days().await;
@@ -2785,6 +2979,36 @@ pub async fn reload_retention_period_setting(conn: &Connection) {
}
}
pub async fn reload_service_log_retention_secs_setting(conn: &Connection) {
match load_setting_value::<i64>(
conn,
SERVICE_LOG_RETENTION_SECS_SETTING,
"SERVICE_LOG_RETENTION_SECS",
DEFAULT_SERVICE_LOG_RETENTION_SECS,
|x| x,
)
.await
{
Ok(v) => windmill_common::set_service_log_retention_secs(v),
Err(e) => tracing::error!("Error reloading service log retention period: {:?}", e),
}
}
pub async fn reload_otel_traces_retention_secs_setting(conn: &Connection) {
match load_setting_value::<i64>(
conn,
OTEL_TRACES_RETENTION_SECS_SETTING,
"OTEL_TRACES_RETENTION_SECS",
DEFAULT_OTEL_TRACES_RETENTION_SECS,
|x| x,
)
.await
{
Ok(v) => windmill_common::set_otel_traces_retention_secs(v),
Err(e) => tracing::error!("Error reloading otel traces retention period: {:?}", e),
}
}
pub async fn reload_audit_log_retention_days_setting(conn: &Connection) {
match load_setting_value::<i64>(
conn,
@@ -4697,7 +4921,7 @@ async fn poll_git_fork_branches(
}
async fn vacuuming_tables(db: &Pool<Postgres>) -> error::Result<()> {
sqlx::query!("VACUUM v2_job, v2_job_completed, job_result_stream_v2, job_stats, job_logs, job_perms, concurrency_key, log_file, metrics")
sqlx::query!("VACUUM v2_job, v2_job_completed, job_result_stream_v2, job_stats, job_logs, job_perms, concurrency_key, log_file, metrics, otel_traces")
.execute(db)
.await?;
Ok(())
@@ -6832,3 +7056,94 @@ mod zombie_worker_memory_pct_tests {
);
}
}
#[cfg(test)]
mod otel_traces_retention_tests {
use super::{delete_expired_otel_traces, DB};
async fn insert_span(db: &DB, id: u8, age_secs: i64) {
sqlx::query!(
"INSERT INTO otel_traces (trace_id, span_id, name, kind, start_time_unix_nano, end_time_unix_nano)
VALUES ($1, $2, 'GET /', 3, $3, $3)",
&[id; 16][..],
&[id; 8][..],
(chrono::Utc::now() - chrono::Duration::seconds(age_secs))
.timestamp_nanos_opt()
.unwrap(),
)
.execute(db)
.await
.unwrap();
}
/// The cutoff crosses two units: a retention configured in seconds against a column holding
/// nanoseconds. Getting that conversion wrong is silent in both directions — a window a
/// billion times too wide never deletes anything, one a billion times too narrow deletes
/// every span on the next tick — so pin it on either side of the boundary.
#[sqlx::test(migrations = "./migrations")]
async fn deletes_only_spans_past_the_window(db: DB) -> anyhow::Result<()> {
let day = 60 * 60 * 24;
insert_span(&db, 1, 60).await;
insert_span(&db, 2, 6 * day).await;
insert_span(&db, 3, 8 * day).await;
assert_eq!(delete_expired_otel_traces(&db, 7 * day).await, 1);
let kept = sqlx::query_scalar!("SELECT trace_id FROM otel_traces ORDER BY trace_id")
.fetch_all(&db)
.await?;
assert_eq!(kept, vec![vec![1u8; 16], vec![2u8; 16]]);
Ok(())
}
}
#[cfg(test)]
mod log_file_listing_tests {
use super::{rotated_log_files, sorted_log_files};
fn names(files: Vec<(chrono::NaiveDateTime, String)>) -> Vec<String> {
files.into_iter().map(|(_, n)| n).collect()
}
/// A directory read newest-entry-first is what tmpfs actually hands back.
#[test]
fn orders_by_minute_whatever_order_readdir_used() {
let newest_first = [
"h.log.2026-08-29-06-49",
"h.log.2026-08-29-06-46",
"h.log.2026-08-29-06-48",
"h.log.2026-08-29-06-47",
];
assert_eq!(
names(sorted_log_files(newest_first.iter().map(|x| x.to_string()))),
vec![
"h.log.2026-08-29-06-46",
"h.log.2026-08-29-06-47",
"h.log.2026-08-29-06-48",
"h.log.2026-08-29-06-49",
]
);
assert_eq!(
names(rotated_log_files(
newest_first.iter().map(|x| x.to_string())
)),
vec![
"h.log.2026-08-29-06-46",
"h.log.2026-08-29-06-47",
"h.log.2026-08-29-06-48",
]
);
}
#[test]
fn drops_names_that_are_not_rotated_log_files() {
let files = sorted_log_files(
["h.log", "not-a-log-file", "h.log.2026-08-29-06-46"]
.iter()
.map(|x| x.to_string()),
);
assert_eq!(files.len(), 1);
assert_eq!(files[0].1, "h.log.2026-08-29-06-46");
assert_eq!(files[0].0.to_string(), "2026-08-29 06:46:00");
}
}
backend/summarized_schema.txt
+3 -1
View File
@@ -38,6 +38,8 @@ account: workspace_id(char), id(int), expires_at(ts), refresh_token(char), clien
FK: (workspace_id) -> workspace(id)
agent_token_blacklist: token(char), expires_at(ts), blacklisted_at(ts), blacklisted_by(char)
ai_agent_memory: workspace_id(char), conversation_id(uuid), step_id(char), messages(jsonb), created_at(ts), updated_at(ts)
ai_free_token_daily_usage: day(date), cost_nanos(bigint), updated_at(ts)
ai_free_token_usage: email(char), cost_nanos(bigint), updated_at(ts)
ai_token_usage: workspace_id(char), day(date), email(char), provider(char), model(char), session_id(char), input_tokens(bigint), cache_read_tokens(bigint), cache_write_tokens(bigint), output_tokens(bigint), reported_cost_nano_usd(bigint), requests(bigint), updated_at(ts)
FK: (workspace_id) -> workspace(id)
alerts: id(int), alert_type(char), message(text), created_at(ts), acknowledged(bool), workspace_id(text), acknowledged_workspace(bool), resource(text)
@@ -128,7 +130,7 @@ job_stats: workspace_id(char), job_id(uuid), metric_id(char), metric_name(char),
kafka_pending_commits: id(bigint), workspace_id(char), kafka_trigger_path(char), topic(char), partition(int), offset(bigint), created_at(ts)
FK: (workspace_id, kafka_trigger_path) -> kafka_trigger(workspace_id, path)
kafka_trigger: path(char), kafka_resource_path(char), topics(char), group_id(char), script_path(char), is_flow(bool), workspace_id(char), edited_by(char), email(char), edited_at(ts), extra_perms(jsonb), server_id(char), last_server_ping(ts), error(text), error_handler_path(char), error_handler_args(jsonb), retry(jsonb), mode(trigger_mode), filters(jsonb[]), auto_commit(bool), labels(text[])
log_file: hostname(char), log_ts(ts), ok_lines(bigint), err_lines(bigint), mode(log_mode), worker_group(char), file_path(char), json_fmt(bool)
log_file: hostname(char), log_ts(ts), ok_lines(bigint), err_lines(bigint), mode(log_mode), worker_group(char), file_path(char), json_fmt(bool), indexed_at(ts)
macro_definition: workspace_id(char), name(char), provider_path(char), params(text), body(text), is_table_macro(bool), created_at(ts)
FK: (workspace_id) -> workspace(id)
macro_usage: workspace_id(char), consumer_path(char), macro_name(char)
backend/tests/fixtures/inline_preview_auth.sql
Vendored
+11 -1
View File
@@ -2,7 +2,9 @@
-- Layered on top of `base` (which provides test-workspace and the non-operator
-- `test-user-2`/SECRET_TOKEN_2). Adds an Operator member so we can assert that
-- Operators cannot reach the arbitrary-code inline preview path
-- (`POST /jobs/run_inline/preview`).
-- (`POST /jobs/run_inline/preview`) with their own token, plus two deployed script
-- jobs of the operator: one running, so we can assert that its WM_TOKEN can, and
-- one queued but not yet pulled, so we can assert that "queued" is not enough.
INSERT INTO password(email, password_hash, login_type, super_admin, verified, name)
VALUES ('operator@windmill.dev', 'not-a-real-hash', 'password', false, true, 'Operator User');
@@ -12,3 +14,11 @@ INSERT INTO usr(workspace_id, email, username, is_admin, operator, role) VALUES
INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin) VALUES
(encode(sha256('OPERATOR_TOKEN'::bytea), 'hex'), 'OPERATOR_T', 'OPERATOR_TOKEN', 'operator@windmill.dev', 'operator token', false);
INSERT INTO v2_job(id, workspace_id, kind, runnable_path, created_by, permissioned_as, permissioned_as_email) VALUES
('2aa0c0de-0000-4000-8000-000000000001', 'test-workspace', 'script', 'u/test-user/deployed', 'operator-user', 'u/operator-user', 'operator@windmill.dev'),
('2aa0c0de-0000-4000-8000-000000000002', 'test-workspace', 'script', 'u/test-user/deployed', 'operator-user', 'u/operator-user', 'operator@windmill.dev');
INSERT INTO v2_job_queue(id, workspace_id, scheduled_for, running) VALUES
('2aa0c0de-0000-4000-8000-000000000001', 'test-workspace', now(), true),
('2aa0c0de-0000-4000-8000-000000000002', 'test-workspace', now(), false);
backend/tests/inline_preview_auth.rs
+184 -15
View File
@@ -9,16 +9,33 @@
//! was the incomplete-fix residual of CVE-2026-22683, whose v1.615.0 patch only
//! covered the entity-CRUD endpoints and left this direct inline-exec sink open.
//!
//! The guard on both routes has one exemption: `wmill.datatable()` called from
//! inside a job the operator is running. Operators can only run deployed code,
//! so a request the job's WM_TOKEN authenticates comes from code a non-operator
//! authored, and the exemption is limited to the request shape the helper sends
//! (PostgreSQL against a `datatable://` database) so a leaked WM_TOKEN cannot
//! be replayed to run anything else.
//!
//! This test pins down:
//! - an Operator is rejected by the operator guard (the core fix; pre-fix this
//! reached the inline executor instead of returning 401), and
//! - an Operator's own token is rejected by the operator guard (the core fix;
//! pre-fix this reached the inline executor instead of returning 401),
//! - a regular non-operator passes the guard (the fix must not over-block the
//! legitimate inline preview flow): in the test harness the worker inline
//! utils are not registered, so a caller past the guard gets the distinct
//! "worker inline functions" error rather than the operator rejection.
//! "worker inline functions" error rather than the operator rejection,
//! - an Operator's job token passes the guard for a datatable query while its
//! job is running, on the inline route and on the `/jobs/run/preview`
//! fallback the SDKs use when the worker has no internal server,
//! - the same token is rejected for any other payload (in-process DuckDB, or a
//! `-- database` directive redirecting the query, whether written literally or
//! reached through a `WM_INTERNAL_DB` marker) and for a deferred run,
//! - an Operator's job token for a job that is not running, whether finished or
//! merely queued, is rejected.
use serde_json::json;
use sqlx::{Pool, Postgres};
use windmill_common::auth::create_jwt_token;
use windmill_common::db::Authed;
use windmill_test_utils::*;
fn client() -> reqwest::Client {
@@ -38,11 +55,65 @@ fn inline_preview_body() -> serde_json::Value {
})
}
/// The request `wmill.datatable("main")` sends: PostgreSQL against `datatable://main`.
fn datatable_query_body() -> serde_json::Value {
json!({
"language": "postgresql",
"content": "SELECT 1 AS x;",
"args": { "database": "datatable://main" }
})
}
/// Mint the WM_TOKEN a job hands its own code: an internally-signed job JWT
/// (note the `job_id` claim) for the fixture's operator, exactly as the worker
/// issues it when the operator runs a deployed script.
async fn operator_job_token(job_id: uuid::Uuid) -> String {
let authed = Authed {
email: "operator@windmill.dev".to_string(),
username: "operator-user".to_string(),
is_admin: false,
is_operator: true,
groups: vec![],
folders: vec![],
scopes: None,
token_prefix: None,
};
create_jwt_token(
authed,
"test-workspace",
3600,
Some(job_id),
Some("ephemeral-script".to_string()),
None,
None,
)
.await
.expect("mint operator job token")
}
const OPERATOR_GUARD_MSG: &str = "Operators cannot run preview jobs";
/// The fixture's deployed-script jobs of the operator: one running, one queued.
const RUNNING_JOB_ID: &str = "2aa0c0de-0000-4000-8000-000000000001";
const QUEUED_JOB_ID: &str = "2aa0c0de-0000-4000-8000-000000000002";
async fn post(url: &str, token: &str, body: &serde_json::Value) -> (u16, String) {
let resp = authed(client().post(url), token)
.json(body)
.send()
.await
.expect("request");
let status = resp.status().as_u16();
let body = resp.text().await.expect("body");
(status, body)
}
#[sqlx::test(fixtures("base", "inline_preview_auth"))]
async fn test_inline_preview_authorization(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
// The server decodes WM_TOKENs with the same in-process JWT secret, so
// setting it once lets us mint valid ones below.
set_jwt_secret().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
@@ -51,12 +122,7 @@ async fn test_inline_preview_authorization(db: Pool<Postgres>) -> anyhow::Result
// 1. CORE REGRESSION: an Operator must be rejected by the operator guard.
// Pre-fix this fell through to the inline executor (arbitrary code
// execution); post-fix it returns 401 with the operator guard message.
let resp = authed(client().post(&url), "OPERATOR_TOKEN")
.json(&inline_preview_body())
.send()
.await?;
let status = resp.status();
let body = resp.text().await?;
let (status, body) = post(&url, "OPERATOR_TOKEN", &inline_preview_body()).await;
assert_eq!(
status, 401,
"Operator must be rejected from inline preview (got {status}): {body}"
@@ -71,12 +137,7 @@ async fn test_inline_preview_authorization(db: Pool<Postgres>) -> anyhow::Result
// the worker inline utils, so the request proceeds past the guard and
// fails later with the distinct "worker inline functions" error — proving
// the operator guard did not reject it.
let resp = authed(client().post(&url), "SECRET_TOKEN_2")
.json(&inline_preview_body())
.send()
.await?;
let status = resp.status();
let body = resp.text().await?;
let (status, body) = post(&url, "SECRET_TOKEN_2", &inline_preview_body()).await;
assert_ne!(
status, 401,
"non-operator must not be blocked by the operator guard (got {status}): {body}"
@@ -86,5 +147,113 @@ async fn test_inline_preview_authorization(db: Pool<Postgres>) -> anyhow::Result
"non-operator must not hit the operator guard, got: {body}"
);
// 3. The WM_TOKEN of a deployed-script job the Operator is running passes the
// guard for a datatable query: this is `wmill.datatable()` called from
// inside that job. As in 2, the harness then fails with the "worker inline
// functions" error.
let running_job_token =
operator_job_token(uuid::Uuid::parse_str(RUNNING_JOB_ID).unwrap()).await;
let (status, body) = post(&url, &running_job_token, &datatable_query_body()).await;
assert_ne!(
status, 401,
"operator job token of a running job must pass the guard for a datatable query (got {status}): {body}"
);
assert!(
!body.contains(OPERATOR_GUARD_MSG),
"operator job token of a running job must not hit the operator guard, got: {body}"
);
// 4. The same token is rejected for any other payload: the exemption covers
// the datatable request shape only, never in-process DuckDB, and never a
// `-- database` directive, which the executor honors over `args.database`.
let mut redirected = datatable_query_body();
redirected["content"] = json!("-- database u/test-user/other_db\nSELECT 1 AS x;");
let mut to_s3 = datatable_query_body();
to_s3["content"] = json!("-- s3\nSELECT 1 AS x;");
let mut resource_db = datatable_query_body();
resource_db["args"]["database"] = json!("$res:u/test-user/other_db");
// A marker is a single line the directive regexes cannot match; the directive only
// appears once the executor expands it, so the guard must check the expansion.
let mut marker = datatable_query_body();
marker["content"] = json!(concat!(
r#"-- WM_INTERNAL_DB_SELECT {"table":"t","columnDefs":[{"field":"id","datatype":"int4"}],"#,
r#""whereClause":"true\n-- database u/test-user/other_db\n AND true"}"#
));
for (label, payload) in [
("DuckDB", inline_preview_body()),
("database directive", redirected),
("s3 directive", to_s3),
("resource database", resource_db),
("marker-expanded database directive", marker),
] {
let (status, body) = post(&url, &running_job_token, &payload).await;
assert_eq!(
status, 401,
"operator job token must be rejected for a {label} payload (got {status}): {body}"
);
assert!(
body.contains(OPERATOR_GUARD_MSG),
"rejection for a {label} payload must be the operator guard, got: {body}"
);
}
// 5. An Operator's job token whose job is not running is rejected like the
// operator's own token, whether the job is over (no queue row) or merely
// queued: a WM_TOKEN that leaked through logs cannot be replayed once the
// job is over.
for (label, job_id) in [
("finished", uuid::Uuid::new_v4()),
("queued", uuid::Uuid::parse_str(QUEUED_JOB_ID).unwrap()),
] {
let token = operator_job_token(job_id).await;
let (status, body) = post(&url, &token, &datatable_query_body()).await;
assert_eq!(
status, 401,
"operator job token of a {label} job must be rejected (got {status}): {body}"
);
assert!(
body.contains(OPERATOR_GUARD_MSG),
"rejection for a {label} job must be the operator guard, got: {body}"
);
}
// 6. The SDKs fall back to `/jobs/run/preview` when the worker has no internal
// server (agent workers). The same exemption applies there: the running
// job's token queues the datatable query (201 with the job id), the
// operator's own token is still refused.
let fallback_url = format!("http://localhost:{port}/api/w/test-workspace/jobs/run/preview");
let (status, body) = post(&fallback_url, &running_job_token, &datatable_query_body()).await;
assert_eq!(
status, 201,
"operator job token of a running job must queue a datatable preview (got {status}): {body}"
);
let (status, body) = post(&fallback_url, "OPERATOR_TOKEN", &datatable_query_body()).await;
assert_eq!(
status, 401,
"Operator must be rejected from the preview fallback (got {status}): {body}"
);
assert!(
body.contains(OPERATOR_GUARD_MSG),
"rejection must be the operator guard, got: {body}"
);
// 7. A deferred run on the fallback would outlive the running job the
// exemption keys off, so the running job's token cannot schedule one.
for deferral in [
"scheduled_in_secs=86400",
"scheduled_for=2099-01-01T00:00:00Z",
] {
let deferred_url = format!("{fallback_url}?{deferral}");
let (status, body) = post(&deferred_url, &running_job_token, &datatable_query_body()).await;
assert_eq!(
status, 401,
"operator job token must not schedule a deferred preview with {deferral} (got {status}): {body}"
);
assert!(
body.contains(OPERATOR_GUARD_MSG),
"rejection for {deferral} must be the operator guard, got: {body}"
);
}
Ok(())
}
backend/tests/object_storage_test_ssrf.rs
+97
View File
@@ -0,0 +1,97 @@
//! `POST /api/settings/test_object_storage_config` runs the probe on the API server and reflects the
//! upstream response, so every non-super-admin must be rejected for private/loopback endpoints and
//! the Filesystem backend on every deployment (`CLOUD_HOSTED` is unset here), while a super admin's
//! Filesystem probe still round-trips. Requires the `parquet` feature, like the route.
#![cfg(feature = "parquet")]
use serde_json::json;
use sqlx::{Pool, Postgres};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use windmill_test_utils::*;
const SUPER_ADMIN_TOKEN: &str = "SECRET_TOKEN";
const USER_TOKEN: &str = "SECRET_TOKEN_2";
async fn test_object_storage(
url: &str,
token: &str,
body: serde_json::Value,
) -> anyhow::Result<(u16, String)> {
let resp = reqwest::Client::new()
.post(url)
.header("Authorization", format!("Bearer {token}"))
.json(&body)
.send()
.await?;
Ok((resp.status().as_u16(), resp.text().await?))
}
#[sqlx::test(fixtures("base"))]
async fn object_storage_test_is_restricted_for_non_super_admins_off_cloud(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let url = format!(
"http://localhost:{}/api/settings/test_object_storage_config",
server.addr.port()
);
// A loopback "S3 endpoint" standing in for an internal service: the probe must be rejected
// before the server opens a connection to it.
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await?;
let internal_port = listener.local_addr()?.port();
let connected = Arc::new(AtomicBool::new(false));
tokio::spawn({
let connected = connected.clone();
async move {
while listener.accept().await.is_ok() {
connected.store(true, Ordering::SeqCst);
}
}
});
let internal_s3 = json!({
"type": "S3",
"bucket": "bucket",
"region": "us-east-1",
"access_key": "key",
"secret_key": "secret",
"endpoint": format!("http://127.0.0.1:{internal_port}"),
"allow_http": true,
"path_style": true,
});
let (status, body) = test_object_storage(&url, USER_TOKEN, internal_s3).await?;
assert_eq!(
status, 401,
"non-super-admin must be rejected for a loopback endpoint (got {status}): {body}"
);
assert!(
body.contains("requires a super admin"),
"unexpected rejection: {body}"
);
assert!(
!connected.load(Ordering::SeqCst),
"the server must not connect to the rejected endpoint"
);
let tmp = tempfile::tempdir()?;
let filesystem = json!({ "type": "Filesystem", "root_path": tmp.path().to_str().unwrap() });
let (status, body) = test_object_storage(&url, USER_TOKEN, filesystem.clone()).await?;
assert_eq!(
status, 401,
"non-super-admin must be rejected for a Filesystem backend (got {status}): {body}"
);
assert!(
body.contains("requires a super admin"),
"unexpected rejection: {body}"
);
// Super admins keep the unrestricted path.
let (status, body) = test_object_storage(&url, SUPER_ADMIN_TOKEN, filesystem).await?;
assert_eq!(
status, 200,
"super admin must be able to test a Filesystem backend (got {status}): {body}"
);
Ok(())
}
backend/tests/suspended_pull_index.rs
+77
View File
@@ -0,0 +1,77 @@
//! Pins the plan of the suspended-job pull. Its resume test degrades silently: once the
//! query expression and `queue_suspended_v2` stop matching, Postgres still returns the right
//! job, just by falling back to a heap filter and fetching one tuple per suspended row on
//! every worker poll. No functional test can see that, so assert on the plan instead.
use serde_json::Value;
use sqlx::{Pool, Postgres};
use windmill_common::worker::make_suspended_pull_query;
/// Depth-first walk of an `EXPLAIN (FORMAT JSON)` plan tree.
fn nodes(plan: &Value, out: &mut Vec<Value>) {
out.push(plan.clone());
for child in plan["Plans"].as_array().unwrap_or(&vec![]) {
nodes(child, out);
}
}
#[sqlx::test(fixtures("base"))]
async fn suspended_pull_tests_resume_time_inside_the_index(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
sqlx::query(
"INSERT INTO v2_job_queue (id, workspace_id, created_at, scheduled_for, running, suspend, suspend_until, tag)
SELECT gen_random_uuid(), 'test-workspace', now() - make_interval(secs => i),
now(), true, 1 + (i % 3), now() + interval '7 day', 'flow'
FROM generate_series(1, 2000) i",
)
.execute(&db)
.await?;
sqlx::query("ANALYZE v2_job_queue").execute(&db).await?;
// Both plans are cheap on a 2000-row table, and which one wins there says nothing
// about a queue with a large suspended backlog. Force the index path, which is the
// one production takes, and assert on how it evaluates the resume test.
let mut conn = db.acquire().await?;
sqlx::query("SET enable_seqscan = off")
.execute(&mut *conn)
.await?;
let version: String = sqlx::query_scalar("SELECT version()")
.fetch_one(&mut *conn)
.await?;
// FORMAT JSON rather than the default: `Index Cond` and `Filter` are separate keys on the
// node, so this does not ride on EXPLAIN's line layout staying put across a major bump.
let explained: Value = sqlx::query_scalar(&format!(
"EXPLAIN (FORMAT JSON) {}",
make_suspended_pull_query(&["flow".to_string()])
))
.bind("test-worker")
.fetch_one(&mut *conn)
.await?;
let mut all = vec![];
nodes(&explained[0]["Plan"], &mut all);
let pretty = serde_json::to_string_pretty(&explained)?;
let scan = all
.iter()
.find(|n| n["Index Name"] == "queue_suspended_v2")
.unwrap_or_else(|| {
panic!("suspended pull did not scan queue_suspended_v2 on {version}:\n{pretty}")
});
// Only `Index Cond` is checked against the index tuple, so that is where the resume test
// has to land — as a `Filter` it would cost a heap fetch per suspended row. The residual
// `suspend_until IS NOT NULL` filter is not that: it is always true for rows the partial
// index holds, and only ever runs on the row LIMIT 1 already fetched.
let cond = scan["Index Cond"].as_str().unwrap_or_else(|| {
panic!("no Index Cond on the suspended pull scan on {version}:\n{pretty}")
});
assert!(
cond.contains("CASE WHEN"),
"resume test is not an index condition on {version}:\n{pretty}"
);
assert!(
!scan["Filter"].as_str().unwrap_or("").contains("CASE WHEN"),
"resume test fell back to a heap filter on {version}:\n{pretty}"
);
Ok(())
}
backend/tests/wm_token_confinement.rs
+27
View File
@@ -318,6 +318,33 @@ async fn test_wm_token_is_confined_to_its_workspace(db: Pool<Postgres>) -> anyho
resp.text().await?
);
}
// ...and the one `settings/global` key on the allowlist, which the CLI reads before
// creating a user on a git-sync push. `ws_base_url` is the control: the handler leaves
// it as ungated as `automate_username_creation`, so only the allowlist stops it.
let resp = authed(
client().get(format!("{api}/settings/global/automate_username_creation")),
&user_wm,
)
.send()
.await?;
assert_eq!(
resp.status(),
200,
"WM_TOKEN must still read automate_username_creation: {}",
resp.text().await?
);
let resp = authed(
client().get(format!("{api}/settings/global/ws_base_url")),
&user_wm,
)
.send()
.await?;
assert_eq!(
resp.status(),
403,
"WM_TOKEN must not read any other global setting: {}",
resp.text().await?
);
let resp = authed(client().post(format!("{api}/schedules/preview")), &user_wm)
.json(&json!({ "schedule": "0 0 12 * * *", "timezone": "UTC" }))
.send()
backend/windmill-ai/src/ai_providers.rs
+10
View File
@@ -24,6 +24,16 @@ lazy_static::lazy_static! {
.ok()
.map(|v| v == "true" || v == "1")
.unwrap_or(false);
/// Drops the cache breakpoints from agent-step requests on every Anthropic platform,
/// not just the one that motivates it: a Google Cloud project can have explicit prompt
/// caching turned off (by request to Cloud support), and Vertex then rejects any request
/// carrying breakpoints. An instance that sets this to unblock such a project also gives
/// up caching on its direct-Anthropic and Foundry resources.
pub static ref DISABLE_ANTHROPIC_PROMPT_CACHING: bool =
std::env::var("DISABLE_ANTHROPIC_PROMPT_CACHING")
.ok()
.map(|v| v == "true" || v == "1")
.unwrap_or(false);
}
pub const OPENAI_BASE_URL: &str = "https://api.openai.com/v1";
backend/windmill-ai/src/providers/anthropic.rs
+47 -11
View File
@@ -1,7 +1,7 @@
use super::{anthropic_model_rejects_sampling_params, REASONING_OFF_SENTINEL};
use crate::{
ai_google::parse_data_url,
ai_providers::{AIPlatform, AIProvider},
ai_providers::{AIPlatform, AIProvider, DISABLE_ANTHROPIC_PROMPT_CACHING},
image_handler::prepare_messages_for_api,
proxy::{
add_user_to_body, common_outbound_headers, credential_header, ProxyBuildArgs, ProxyRequest,
@@ -632,15 +632,13 @@ impl AnthropicQueryBuilder {
}
}
let caching = !*DISABLE_ANTHROPIC_PROMPT_CACHING;
let system = collect_system_prompt(&prepared_messages, args.system_prompt).map(|text| {
vec![AnthropicSystemContent {
r#type: "text".to_string(),
text,
cache_control: if self.is_vertex() {
None
} else {
Some(CacheControl::ephemeral())
},
cache_control: caching.then(CacheControl::ephemeral),
}]
});
@@ -665,7 +663,7 @@ impl AnthropicQueryBuilder {
let max_tokens = Some(args.max_tokens.unwrap_or(64000));
// Apply cache_control on the last custom tool
if !self.is_vertex() {
if caching {
if let Some(ref mut tools_vec) = tools_option {
if let Some(AnthropicTool::Custom(ref mut custom)) = tools_vec.last_mut() {
custom.cache_control = Some(CacheControl::ephemeral());
@@ -674,7 +672,7 @@ impl AnthropicQueryBuilder {
}
// Apply cache_control on the last content block of the last message
if !self.is_vertex() {
if caching {
if let Some(last_msg) = anthropic_messages.last_mut() {
if let Some(last_block) = last_msg.content.last_mut() {
match last_block {
@@ -882,10 +880,15 @@ mod tests {
}
}
async fn build_text_body(messages: &[OpenAIMessage], system_prompt: Option<&str>) -> String {
async fn build_text_body_on(
platform: AIPlatform,
messages: &[OpenAIMessage],
system_prompt: Option<&str>,
tools: Option<&[ToolDef]>,
) -> String {
let args = BuildRequestArgs {
messages,
tools: None,
tools,
model: "claude-sonnet-4",
temperature: None,
reasoning_effort: None,
@@ -899,12 +902,16 @@ mod tests {
prompt_cache_key: None,
};
AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::Standard)
AnthropicQueryBuilder::new(AIProvider::Anthropic, platform)
.build_request(&args, &authed_client(), "test-workspace")
.await
.unwrap()
}
async fn build_text_body(messages: &[OpenAIMessage], system_prompt: Option<&str>) -> String {
build_text_body_on(AIPlatform::Standard, messages, system_prompt, None).await
}
/// The worker prepends the system prompt as a system message *and* passes it as
/// `system_prompt`; the request must still carry it exactly once.
#[tokio::test]
@@ -946,6 +953,35 @@ mod tests {
assert!(request.get("system").is_none());
}
/// Vertex serves the same Messages API and honours `cache_control` breakpoints, so
/// its requests must carry the same three the standard platform gets.
#[tokio::test]
async fn sets_cache_breakpoints_on_every_platform() {
let messages = vec![message("system", SYSTEM_PROMPT), message("user", "hi")];
let tools = vec![ToolDef {
r#type: "function".to_string(),
function: ToolDefFunction {
name: "get_weather".to_string(),
description: None,
parameters: RawValue::from_string("{}".to_string()).unwrap(),
},
}];
let ephemeral = serde_json::json!({ "type": "ephemeral" });
for platform in [AIPlatform::Standard, AIPlatform::GoogleVertexAi] {
let body =
build_text_body_on(platform, &messages, Some(SYSTEM_PROMPT), Some(&tools)).await;
let request: serde_json::Value = serde_json::from_str(&body).unwrap();
assert_eq!(request["system"][0]["cache_control"], ephemeral);
let sent_tools = request["tools"].as_array().unwrap();
assert_eq!(sent_tools.last().unwrap()["cache_control"], ephemeral);
let sent = request["messages"].as_array().unwrap();
let content = sent.last().unwrap()["content"].as_array().unwrap();
assert_eq!(content.last().unwrap()["cache_control"], ephemeral);
}
}
fn has_header(headers: &[(String, String)], name: &str, value: &str) -> bool {
headers
.iter()
backend/windmill-api-auth/src/scopes.rs
+8 -49
View File
@@ -288,7 +288,6 @@ pub enum ScopeDomain {
Configs,
OAuth,
AI,
AiSkills,
AiEvals, // AI agent eval datasets
Indexer,
@@ -349,7 +348,6 @@ impl ScopeDomain {
Self::Configs => "configs",
Self::OAuth => "oauth",
Self::AI => "ai",
Self::AiSkills => "ai_skills",
Self::AiEvals => "ai_evals",
Self::Capture => "capture",
Self::Drafts => "drafts",
@@ -405,7 +403,6 @@ impl ScopeDomain {
"configs" => Some(Self::Configs),
"oauth" => Some(Self::OAuth),
"ai" => Some(Self::AI),
"ai_skills" => Some(Self::AiSkills),
"ai_evals" => Some(Self::AiEvals),
"indexer" | "srch" => Some(Self::Indexer),
"teams" => Some(Self::Teams),
@@ -996,6 +993,13 @@ fn scope_grants_access(
/// the caller's own row; `email` and `allowed_domain_auto_invite` are derived from the
/// token itself and touch no table.
///
/// `settings/global/automate_username_creation` is the one instance setting on the list.
/// `get_global_setting` exempts a handful of keys from its own super-admin gate, that one
/// among them, so the boolean is already readable by every authenticated user; it is here
/// because the CLI reads it before creating a user during a git-sync push, which runs as a
/// job. The other ungated keys have no such caller, so they stay confined — being ungated
/// earns a key nothing on its own.
///
/// Deliberately absent, as each crosses that line: `users/list_invites` (returns the
/// workspace ids the identity was invited to), `users/tokens/list` (credential metadata
/// of the borrowed identity), `users/exists/{email}` (an oracle over arbitrary
@@ -1011,6 +1015,7 @@ fn is_global_read_open_to_job_token(route_path: &str) -> bool {
| "/api/users/usage"
| "/api/users/tutorial_progress"
| "/api/workspaces/allowed_domain_auto_invite"
| "/api/settings/global/automate_username_creation"
| "/api/docs/search"
| "/api/docs/page"
| "/api/integrations/hub/list"
@@ -1194,12 +1199,6 @@ mod tests {
assert_eq!(domain, ScopeDomain::FlowConversations);
assert_eq!(kind, None);
assert_eq!(route_suffix, Some("flow_conversations/list".to_string()));
let (domain, kind, route_suffix) =
extract_domain_from_route("/api/w/test_workspace/ai_skills/list").unwrap();
assert_eq!(domain, ScopeDomain::AiSkills);
assert_eq!(kind, None);
assert_eq!(route_suffix, Some("ai_skills/list".to_string()));
}
#[test]
@@ -1360,11 +1359,6 @@ mod tests {
ScopeDomain::from_str("flow_conversations"),
Some(ScopeDomain::FlowConversations)
);
assert_eq!(
ScopeDomain::from_str("ai_skills"),
Some(ScopeDomain::AiSkills)
);
// Test canonical string conversion
assert_eq!(ScopeDomain::Acls.as_str(), "acls");
assert_eq!(ScopeDomain::RawApps.as_str(), "raw_apps");
@@ -1373,41 +1367,6 @@ mod tests {
ScopeDomain::FlowConversations.as_str(),
"flow_conversations"
);
assert_eq!(ScopeDomain::AiSkills.as_str(), "ai_skills");
}
#[test]
fn test_ai_skills_scope_access() {
let read_scopes = vec!["ai_skills:read".to_string()];
assert!(
check_route_access(&read_scopes, "/api/w/test_workspace/ai_skills/list", "GET").is_ok()
);
assert!(check_route_access(
&read_scopes,
"/api/w/test_workspace/ai_skills/get/foo",
"GET"
)
.is_ok());
assert!(check_route_access(
&read_scopes,
"/api/w/test_workspace/ai_skills/upload",
"POST"
)
.is_err());
let write_scopes = vec!["ai_skills:write".to_string()];
assert!(check_route_access(
&write_scopes,
"/api/w/test_workspace/ai_skills/upload",
"POST"
)
.is_ok());
assert!(check_route_access(
&write_scopes,
"/api/w/test_workspace/ai_skills/delete/foo",
"DELETE"
)
.is_ok());
}
#[test]
backend/windmill-api-debug/src/lib.rs
+5
View File
@@ -468,6 +468,9 @@ async fn sign_debug_request(
// Parse the language
let script_lang: ScriptLang = request.language.parse().unwrap_or(ScriptLang::Bun);
// Taken from the parsed language, not the request's string: the telemetry key vocabulary has
// to stay the closed set of languages rather than whatever a caller sent.
let lang_key = script_lang.as_str();
// Hash the code (we don't include full code in JWT to keep it small)
let mut hasher = Sha256::new();
@@ -578,6 +581,8 @@ async fn sign_debug_request(
tx.commit().await?;
windmill_common::feature_usage::log_feature_usage("debugger", "session", lang_key);
Ok(Json(SignedDebugPayload {
token,
code: request.code,
backend/windmill-api-flows/src/flows.rs
+2 -1
View File
@@ -157,7 +157,8 @@ async fn list_flows(
FROM draft d \
LEFT JOIN usr u ON u.workspace_id = d.workspace_id AND u.email = d.email \
LEFT JOIN password p ON p.email = d.email AND p.super_admin = true \
WHERE d.workspace_id = o.workspace_id AND d.path = o.path AND d.typ = 'flow') as draft_users",
WHERE d.workspace_id = o.workspace_id AND d.path = o.path AND d.typ = 'flow' \
AND (d.email IS NULL OR u.username IS NOT NULL OR p.email IS NOT NULL)) as draft_users",
"folder_labels(o.workspace_id, o.path) as inherited_labels"
])
.left()
backend/windmill-api-groups/src/granular_acls.rs
+2 -2
View File
@@ -183,9 +183,9 @@ async fn add_granular_acl(
if kind == "folder" {
let change_type = if write.unwrap_or(false) {
"grant_read"
} else {
"grant_write"
} else {
"grant_read"
};
crate::folders::log_folder_permission_change(
&mut *tx,
backend/windmill-api-integration-tests/tests/mcp_preprocessor_headers.rs
+186
View File
@@ -0,0 +1,186 @@
//! Request headers reaching a runnable's preprocessor over MCP.
//!
//! The property this pins is structural rather than a filter: the model writes
//! the tool's arguments, which become `event.body`, while the server writes
//! `event.headers`. A model that guesses a header's name can only ever land in
//! `body`, so an identity read from `headers` is one prompt injection cannot
//! forge. Nothing else in the suite exercises MCP argument shaping end to end.
//!
//! Requires: bun runtime, live database (migrations applied by sqlx::test).
#![cfg(feature = "mcp")]
use serde_json::{json, Value};
use sqlx::{Pool, Postgres};
use windmill_test_utils::*;
const SCRIPT_PATH: &str = "u/test-user/mcp_hdr_probe";
/// Echoes the two halves of the event separately, so the assertions can tell
/// which one a value arrived in.
const PREPROCESSOR_SCRIPT: &str = r#"
export async function preprocessor(event: any) {
return {
kind: event.kind,
from_headers: event.headers?.["x-user-id"] ?? "",
from_body: event.body?.x_user_id ?? "",
header_names: Object.keys(event.headers ?? {}).sort(),
};
}
export async function main(kind: string, from_headers: string, from_body: string, header_names: string[]) {
return { kind, from_headers, from_body, header_names };
}
"#;
async fn insert_mcp_token(db: &Pool<Postgres>) -> anyhow::Result<()> {
sqlx::query(
"INSERT INTO token (token_hash, token_prefix, token, email, label, super_admin, scopes)
VALUES (encode(sha256('MCP_TOKEN'::bytea), 'hex'), 'MCP_TOK', 'MCP_TOKEN', 'test@windmill.dev', 'mcp token', true, ARRAY['mcp:all'])",
)
.execute(db)
.await?;
Ok(())
}
/// POST one JSON-RPC message. The endpoint answers either `application/json` or
/// a single-event SSE stream, so strip the `data: ` framing before parsing.
async fn mcp_post(port: u16, headers: &[(&str, &str)], body: Value) -> anyhow::Result<Value> {
let mut req = reqwest::Client::new()
.post(format!(
"http://localhost:{port}/api/mcp/w/test-workspace/mcp"
))
.header("Authorization", "Bearer MCP_TOKEN")
.header("Accept", "application/json, text/event-stream")
.json(&body);
for (k, v) in headers {
req = req.header(*k, *v);
}
let text = req.send().await?.text().await?;
let payload = text
.lines()
.find_map(|l| l.strip_prefix("data: "))
.unwrap_or(text.trim());
serde_json::from_str(payload).map_err(|e| anyhow::anyhow!("unparseable MCP body {text:?}: {e}"))
}
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_mcp_preprocessor_receives_the_callers_headers(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
initialize_tracing().await;
set_jwt_secret().await;
insert_mcp_token(&db).await?;
let server = ApiServer::start_mcp(db.clone()).await?;
let port = server.addr.port();
let resp = reqwest::Client::new()
.post(format!(
"http://localhost:{port}/api/w/test-workspace/scripts/create"
))
.header("Authorization", "Bearer SECRET_TOKEN")
.json(&json!({
"path": SCRIPT_PATH,
"summary": "mcp header probe",
"description": "",
"content": PREPROCESSOR_SCRIPT,
"language": "bun",
"lock": "",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": { "x_user_id": { "type": "string" } },
"required": []
}
}))
.send()
.await?;
assert_eq!(
resp.status(),
201,
"create script: {}",
resp.text().await.unwrap_or_default()
);
// A script counts as deployed once it has a lock, which normally arrives from
// a dependency job. Planting an empty one keeps the test to the path under
// test instead of a bun resolution whose timing it does not control.
sqlx::query("UPDATE script SET lock = '' WHERE path = $1 AND workspace_id = 'test-workspace'")
.bind(SCRIPT_PATH)
.execute(&db)
.await?;
let tools = mcp_post(
port,
&[],
json!({"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}}),
)
.await?;
let tool_name = tools["result"]["tools"]
.as_array()
.and_then(|list| {
list.iter()
.filter_map(|t| t["name"].as_str())
.find(|n| n.contains("mcp__hdr__probe"))
})
.ok_or_else(|| anyhow::anyhow!("the deployed script was not listed as a tool: {tools}"))?
.to_string();
let result = in_test_worker(
db.clone(),
async {
mcp_post(
port,
// Every name the withheld list covers has to be on the wire, or
// asserting its absence proves nothing. `Authorization` is already
// set by `mcp_post`, and `extract_token` reads it before the
// cookie, so sending one does not disturb auth.
&[
("X-User-Id", "alice@corp.example"),
("Cookie", "session=secret"),
("Proxy-Authorization", "Basic Zm9v"),
],
json!({
"jsonrpc": "2.0", "id": 2, "method": "tools/call",
// The model names the header it wants to spoof. Its value is an
// argument, so it can only ever reach `event.body`.
"params": { "name": tool_name, "arguments": { "x_user_id": "attacker@evil.test" } }
}),
)
.await
},
port,
)
.await?;
let text = result["result"]["content"][0]["text"]
.as_str()
.ok_or_else(|| anyhow::anyhow!("tool call returned no text content: {result}"))?;
let out: Value = serde_json::from_str(text)?;
assert_eq!(out["kind"], "mcp", "preprocessor event kind: {out}");
assert_eq!(
out["from_headers"], "alice@corp.example",
"the caller's header must reach event.headers: {out}"
);
assert_eq!(
out["from_body"], "attacker@evil.test",
"the model's argument must land in event.body, not overwrite the header: {out}"
);
let names: Vec<&str> = out["header_names"]
.as_array()
.map(|a| a.iter().filter_map(|v| v.as_str()).collect())
.unwrap_or_default();
assert!(
names.contains(&"x-user-id"),
"event.headers must carry the request's own headers: {names:?}"
);
for withheld in ["authorization", "cookie", "proxy-authorization"] {
assert!(
!names.contains(&withheld),
"{withheld} is withheld from a preprocessor: {names:?}"
);
}
Ok(())
}
backend/windmill-api-integration-tests/tests/scripts.rs
+89 -5
View File
@@ -38,6 +38,85 @@ fn new_script(path: &str, summary: &str, content: &str) -> serde_json::Value {
})
}
/// A supplied lock queues no dependency job, so if the create does not record its hash nothing
/// ever will, and every importer of this script relocks on each of its deploys forever after.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_create_script_persists_supplied_lock_hash(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let path = "u/test-user/supplied_lock";
let lock = r#"{"version":"4","remote":{}}"#;
let mut script = new_script(
path,
"Supplied lock",
"export async function main() { return 42; }",
);
script["lock"] = json!(lock);
let resp = authed(client().post(format!(
"http://localhost:{port}/api/w/test-workspace/scripts/create"
)))
.json(&script)
.send()
.await?;
assert_eq!(resp.status(), 201, "create: {}", resp.text().await?);
let stored_hash = sqlx::query_scalar!(
"SELECT lockfile_hash FROM lock_hash WHERE workspace_id = $1 AND path = $2",
"test-workspace",
path,
)
.fetch_one(&db)
.await?;
assert_eq!(stored_hash, windmill_common::scripts::hash_script(lock));
// A script deployed before the create recorded hashes has no row, and pushing it unchanged
// creates no version to hang one off. Without the write on that path it would keep its
// importers relocking until someone edited it.
sqlx::query!(
"DELETE FROM lock_hash WHERE workspace_id = $1 AND path = $2",
"test-workspace",
path,
)
.execute(&db)
.await?;
// The no-op comparison covers every field, so the push has to carry what the first deploy
// filled in by itself; `auto_parent` both resolves the parent and keeps the hash distinct.
script["auto_parent"] = json!(true);
script["ws_error_handler_muted"] = json!(false);
script["assets"] = json!([]);
let resp = authed(client().post(format!(
"http://localhost:{port}/api/w/test-workspace/scripts/create?skip_if_noop=true"
)))
.json(&script)
.send()
.await?;
assert_eq!(resp.status(), 201, "no-op push: {}", resp.text().await?);
let versions: i64 = sqlx::query_scalar!(
"SELECT COUNT(*) FROM script WHERE workspace_id = $1 AND path = $2",
"test-workspace",
path,
)
.fetch_one(&db)
.await?
.unwrap_or_default();
assert_eq!(versions, 1, "no-op push must not create a version");
let repaired_hash = sqlx::query_scalar!(
"SELECT lockfile_hash FROM lock_hash WHERE workspace_id = $1 AND path = $2",
"test-workspace",
path,
)
.fetch_one(&db)
.await?;
assert_eq!(repaired_hash, windmill_common::scripts::hash_script(lock));
Ok(())
}
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_script_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
@@ -797,10 +876,12 @@ async fn test_update_script_reports_losing_to_a_concurrent_deploy(
// What a deploy leaves behind: the old head archived, a new one live at the path.
// Copied through a temp table so this does not have to restate every column.
sqlx::query("CREATE TEMP TABLE superseding ON COMMIT DROP AS SELECT * FROM script WHERE hash = $1")
.bind(head)
.execute(&mut *winner)
.await?;
sqlx::query(
"CREATE TEMP TABLE superseding ON COMMIT DROP AS SELECT * FROM script WHERE hash = $1",
)
.bind(head)
.execute(&mut *winner)
.await?;
sqlx::query("UPDATE superseding SET hash = $1, archived = false, parent_hashes = ARRAY[$2]")
.bind(head + 1)
.bind(head)
@@ -818,7 +899,10 @@ async fn test_update_script_reports_losing_to_a_concurrent_deploy(
let resp = tokio::time::timeout(std::time::Duration::from_secs(20), update).await??;
let status = resp.status();
let body = resp.text().await?;
assert_eq!(status, 400, "losing the race should not read as success: {body}");
assert_eq!(
status, 400,
"losing the race should not read as success: {body}"
);
assert!(
body.contains("deployed to concurrently"),
"the loser must say it was superseded, not that the script is missing: {body}"
backend/windmill-api-integration-tests/tests/users.rs
+91 -8
View File
@@ -308,14 +308,17 @@ async fn test_user_endpoints(db: Pool<Postgres>) -> anyhow::Result<()> {
let auth_base = format!("http://localhost:{port}/api/auth");
// --- login (will fail: password hash in fixture is fake) ---
// An unparseable stored hash must read as a failed login, not as a server error
// relaying the hash parser's message to an unauthenticated caller.
let resp = client()
.post(format!("{auth_base}/login"))
.json(&json!({"email": "test@windmill.dev", "password": "wrong-password"}))
.send()
.await
.unwrap();
assert!(
resp.status() == 400 || resp.status() == 401 || resp.status() == 500,
assert_eq!(
resp.status(),
400,
"login: unexpected status {}",
resp.status()
);
@@ -804,12 +807,16 @@ async fn test_change_user_email_leaves_group_identities(db: Pool<Postgres>) -> a
let server = ApiServer::start(db.clone()).await?;
let global_base = format!("http://localhost:{}/api/users", server.addr.port());
sqlx::query!("UPDATE password SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'")
.execute(&db)
.await?;
sqlx::query!("UPDATE usr SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'")
.execute(&db)
.await?;
sqlx::query!(
"UPDATE password SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'"
)
.execute(&db)
.await?;
sqlx::query!(
"UPDATE usr SET email = 'group-ops@windmill.dev' WHERE email = 'test2@windmill.dev'"
)
.execute(&db)
.await?;
sqlx::query!(
"INSERT INTO group_(workspace_id, name, summary, extra_perms) VALUES ('test-workspace', 'ops', '', '{}')"
)
@@ -910,3 +917,79 @@ async fn test_change_user_email_leaves_group_identities(db: Pool<Postgres>) -> a
Ok(())
}
/// An address with no `password` row can own a draft, and the account paths carry the delete and
/// rename that no foreign key does any more.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_drafts_follow_their_owner_without_a_fkey(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let global_base = format!("http://localhost:{port}/api/users");
// The destination of the rename below already holds a draft of the same item — it belongs to
// an accountless principal, so `change_email`'s "address is free" check does not see it.
sqlx::query!(
"INSERT INTO draft(workspace_id, path, typ, value, email) VALUES
('test-workspace', 'u/ext/s', 'script', '{}'::json, 'ext-jwt@windmill.dev'),
('test-workspace', 'u/two/s', 'script', '{\"summary\": \"moving\"}'::json, 'test2@windmill.dev'),
('test-workspace', 'u/two/s', 'script', '{\"summary\": \"displaced\"}'::json, 'renamed@windmill.dev'),
('test-workspace', 'u/three/s', 'script', '{}'::json, 'test3@windmill.dev')"
)
.execute(&db)
.await?;
// A null username is how the legacy workspace-level row is encoded, so an owner nobody can
// name must be absent from the owner circles rather than pose as one.
let resp = authed(client().get(format!(
"http://localhost:{port}/api/w/test-workspace/drafts/list?all_users=true"
)))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let listed = resp.json::<serde_json::Value>().await?;
let ext = listed
.as_array()
.unwrap()
.iter()
.find(|d| d["path"] == "u/ext/s")
.expect("the accountless owner's draft is listed");
assert_eq!(ext.get("draft_users"), None);
let resp = authed(client().post(format!("{global_base}/change_email/test2@windmill.dev")))
.json(&json!({ "new_email": "renamed@windmill.dev" }))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200, "change_email: {}", resp.text().await?);
let moved = sqlx::query!(
"SELECT email, value->>'summary' AS summary FROM draft WHERE path = 'u/two/s'"
)
.fetch_all(&db)
.await?;
assert_eq!(
moved
.iter()
.map(|r| (r.email.as_deref(), r.summary.as_deref()))
.collect::<Vec<_>>(),
vec![(Some("renamed@windmill.dev"), Some("moving"))],
"the moving account's draft wins the unique index it now collides on"
);
let resp = authed(client().delete(format!("{global_base}/delete/test3@windmill.dev")))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200, "delete_user: {}", resp.text().await?);
let remaining = sqlx::query_scalar!("SELECT path FROM draft ORDER BY path")
.fetch_all(&db)
.await?;
assert_eq!(
remaining,
vec!["u/ext/s".to_string(), "u/two/s".to_string()],
"the deleted account's draft goes, the accountless owner's stays"
);
Ok(())
}
backend/windmill-api-integration-tests/tests/workspaces.rs
+58 -4
View File
@@ -889,6 +889,53 @@ async fn test_get_copilot_info_ignores_empty_instance_ai_row(
Ok(())
}
/// A workspace with no provider of its own is served the instance config, but the
/// `copilot_disabled` flag must still come from the workspace's own row.
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_get_copilot_info_keeps_workspace_copilot_disabled_over_instance_fallback(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let base = format!("http://localhost:{port}/api/w/test-workspace/workspaces");
sqlx::query("UPDATE workspace_settings SET ai_config = $1 WHERE workspace_id = $2")
.bind(json!({ "copilot_disabled": true }))
.bind("test-workspace")
.execute(&db)
.await?;
sqlx::query(
"INSERT INTO global_settings (name, value) VALUES ($1, $2) \
ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value",
)
.bind("ai_config")
.bind(json!({
"providers": {
"openai": {
"resource_path": "u/test-user/openai_instance",
"models": ["gpt-4o-mini"]
}
}
}))
.execute(&db)
.await?;
let resp = authed(client().get(format!("{base}/get_copilot_info")))
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200);
let settings = resp.json::<serde_json::Value>().await?;
assert_eq!(
settings["providers"]["openai"]["models"][0], "gpt-4o-mini",
"instance providers are still served"
);
assert_eq!(settings["copilot_disabled"], true);
Ok(())
}
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_error_handler_instance_alerts_fallback(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
@@ -941,7 +988,12 @@ async fn test_error_handler_instance_alerts_fallback(db: Pool<Postgres>) -> anyh
.send()
.await
.unwrap();
assert_eq!(resp.status(), 200, "disable on fork: {}", resp.text().await?);
assert_eq!(
resp.status(),
200,
"disable on fork: {}",
resp.text().await?
);
assert!(!stored().await?);
Ok(())
@@ -1044,9 +1096,11 @@ async fn test_create_service_account_drops_orphaned_group_memberships(
.await?;
// Same username, different workspace, and very much alive — must not be touched.
sqlx::query("INSERT INTO workspace (id, name, owner) VALUES ('other-workspace', 'other', 'svc_acct')")
.execute(&db)
.await?;
sqlx::query(
"INSERT INTO workspace (id, name, owner) VALUES ('other-workspace', 'other', 'svc_acct')",
)
.execute(&db)
.await?;
sqlx::query(
"INSERT INTO group_ (workspace_id, name, summary) VALUES
('other-workspace', 'all', 'All users'),
backend/windmill-api-jobs/src/execution.rs
+86
View File
@@ -416,11 +416,31 @@ pub fn result_to_response(result: Box<RawValue>, success: bool) -> error::Result
let mut headers = HeaderMap::new();
// A reverse proxy consumes hop-by-hop headers instead of forwarding them and
// drops every header named by `Connection`, so a script could use one to strip
// the sandbox headers this function adds before they reach the browser.
const HOP_BY_HOP_HEADERS: [&str; 9] = [
"connection",
"keep-alive",
"proxy-authenticate",
"proxy-authorization",
"proxy-connection",
"te",
"trailer",
"transfer-encoding",
"upgrade",
];
if let Some(windmill_headers) = windmill_headers {
for (k, v) in windmill_headers {
let k = HeaderName::from_str(k.as_str()).map_err(|err| {
Error::internal_err(format!("Invalid header name {k}: {err}"))
})?;
if HOP_BY_HOP_HEADERS.contains(&k.as_str()) {
return Err(Error::ExecutionErr(format!(
"windmill_headers cannot set the hop-by-hop header \"{k}\""
)));
}
let v = HeaderValue::from_str(v.as_str()).map_err(|err| {
Error::internal_err(format!("Invalid header value {v}: {err}"))
})?;
@@ -428,6 +448,22 @@ pub fn result_to_response(result: Box<RawValue>, success: bool) -> error::Result
}
}
// The script controls the content type and body, and run_wait_result and sync
// HTTP routes are reachable by top-level GET navigation with the session cookie:
// sandbox the document into an opaque origin so HTML can never run with the
// viewer's session. Inserted after `wm_headers` so a script cannot override it.
headers.insert(
http::header::X_CONTENT_TYPE_OPTIONS,
HeaderValue::from_static("nosniff"),
);
headers.insert(
http::header::CONTENT_SECURITY_POLICY,
HeaderValue::from_static(
"sandbox allow-scripts allow-forms allow-popups \
allow-popups-to-escape-sandbox allow-downloads allow-modals",
),
);
if let Some(content_type) = windmill_content_type {
let serialized_json_result = result_value
.map(|val| val.get().to_owned())
@@ -1104,6 +1140,56 @@ mod result_to_response_tests {
resp.headers().get(http::header::CONTENT_TYPE).unwrap(),
"text/html"
);
assert_sandboxed(resp.headers());
assert_eq!(body_bytes(resp).await, b"<h1>hi</h1>");
}
fn assert_sandboxed(headers: &HeaderMap) {
assert_eq!(
headers.get(http::header::X_CONTENT_TYPE_OPTIONS).unwrap(),
"nosniff"
);
let csp = headers
.get(http::header::CONTENT_SECURITY_POLICY)
.expect("content-security-policy")
.to_str()
.unwrap();
assert!(csp.starts_with("sandbox "), "csp: {csp}");
assert!(!csp.contains("allow-same-origin"), "csp: {csp}");
}
#[tokio::test]
async fn custom_headers_cannot_override_sandbox() {
// wm_headers is script-controlled: a content-type set there replaces the JSON
// one even without wm_content_type, and the sandbox headers must survive an
// attempt to override them.
let resp = result_to_response(
raw(
r#"{"wm_headers":{"content-type":"text/html","content-security-policy":"default-src *","x-content-type-options":"none"},"result":"<h1>hi</h1>"}"#,
),
true,
)
.expect("response");
assert_eq!(
resp.headers().get(http::header::CONTENT_TYPE).unwrap(),
"text/html"
);
assert_sandboxed(resp.headers());
}
#[tokio::test]
async fn hop_by_hop_custom_headers_are_rejected() {
// A proxy drops every header named by `Connection`, which would strip the
// sandbox headers on the way to the browser.
for name in ["connection", "Connection", "transfer-encoding", "upgrade"] {
let res = result_to_response(
raw(&format!(
r#"{{"wm_content_type":"text/html","wm_headers":{{"{name}":"content-security-policy, x-content-type-options"}},"result":"<h1>hi</h1>"}}"#
)),
true,
);
assert!(res.is_err(), "hop-by-hop header must be rejected: {name}");
}
}
}
backend/windmill-api-scripts/src/scripts.rs
+27 -3
View File
@@ -39,7 +39,7 @@ use sqlx::{FromRow, Postgres, Transaction};
use std::{collections::HashMap, sync::Arc};
use windmill_audit::audit_oss::{audit_log, AuditAuthorable};
use windmill_audit::ActionKind;
use windmill_dep_map::process_relative_imports;
use windmill_dep_map::{lock_hash::record_lock_hashes, process_relative_imports};
use windmill_dep_map::scoped_dependency_map::ScopedDependencyMap;
use windmill_common::{
@@ -216,12 +216,15 @@ async fn list_scripts(
// a member of has no `usr` row, so fall back to their instance-derived username
// (`password.username`), or their email when derivation is disabled — this keeps the
// raw email out of the payload whenever a derived username exists. The genuine
// NULL-email legacy row stays None (no `usr`/`password` match, `d.email` is NULL).
// NULL-email legacy row stays None (no `usr`/`password` match, `d.email` is NULL),
// which is why an owner that resolves to no name at all — an external JWT's subject
// has neither row — is dropped: None is read as "legacy" downstream.
"(SELECT json_agg(json_build_object('username', COALESCE(u.username, p.username, CASE WHEN p.email IS NOT NULL THEN d.email END)) ORDER BY COALESCE(u.username, p.username, CASE WHEN p.email IS NOT NULL THEN d.email END) NULLS LAST) \
FROM draft d \
LEFT JOIN usr u ON u.workspace_id = d.workspace_id AND u.email = d.email \
LEFT JOIN password p ON p.email = d.email AND p.super_admin = true \
WHERE d.workspace_id = o.workspace_id AND d.path = o.path AND d.typ = 'script') as draft_users",
WHERE d.workspace_id = o.workspace_id AND d.path = o.path AND d.typ = 'script' \
AND (d.email IS NULL OR u.username IS NOT NULL OR p.email IS NOT NULL)) as draft_users",
"folder_labels(o.workspace_id, o.path) as inherited_labels"
])
.left()
@@ -1070,6 +1073,14 @@ fn modules_eq(
}
}
/// Recorded for the empty lock a codebase or a language with no lock generation carries as well as
/// for a real one: the worker writes `hash_script("")` in the same situation, and a path going from
/// a real lock to an empty one has to stop matching what its importers recorded, or they wrongly
/// skip rather than merely relock too often.
fn lock_hash_entry(path: &str, lock: &str) -> [(String, i64); 1] {
[(path.to_string(), hash_script(lock))]
}
async fn create_script_internal<'c>(
mut ns: NewScript,
w_id: String,
@@ -1337,6 +1348,12 @@ async fn create_script_internal<'c>(
parent_hash = %p_hash.0,
"Skipping no-op script deploy (identical to parent)"
);
// The version is unchanged, but the row recording its lock's hash may never have
// been written — nothing else writes it for a supplied lock, and a path only ever
// pushed unchanged would otherwise keep its importers relocking forever.
if let Some(lock) = ps.lock.as_deref() {
record_lock_hashes(&mut tx, &w_id, &lock_hash_entry(&ns.path, lock)).await?;
}
return Ok((p_hash.clone(), tx, None, Vec::new()));
}
@@ -1884,6 +1901,13 @@ async fn create_script_internal<'c>(
.execute(&mut *tx)
.await?;
// A lock that is not left to a dependency job queues none, so this is the only place its hash
// can be recorded. `try_skip_relock` treats a missing hash for an imported script as changed,
// so leaving the row out makes every importer of this path relock on every deploy of it.
if let Some(lock) = lock.as_deref() {
record_lock_hashes(&mut tx, &w_id, &lock_hash_entry(&ns.path, lock)).await?;
}
// Update ci_test_reference table for test scripts
// Delete by both new and old path to handle renames
let old_path = parent_hashes_and_perms.as_ref().map(|x| x.p_path.as_str());
backend/windmill-api-settings/src/lib.rs
+91 -33
View File
@@ -284,15 +284,28 @@ pub async fn test_s3_bucket(
use bytes::Bytes;
use futures::StreamExt;
// The probe executes on the API server itself. On multi-tenant Cloud that is a shared control
// plane, so we constrain untrusted callers to remove the SSRF / credential-exfiltration /
// local-filesystem surface (see validate_object_storage_test). On self-hosted instances the
// object store usually lives on the local/private network and all authenticated users are
// trusted, so testing there stays unrestricted. Super admins keep the unrestricted path too.
// The probe executes on the API server itself and reflects the upstream response into the
// error, so any authenticated caller could otherwise use it as an SSRF / port-scan primitive
// against the server's network, exfiltrate its ambient credentials, or write to its local
// disk (see validate_object_storage_test). That holds on self-hosted instances as much as on
// Cloud, so only super admins get the unrestricted path.
let is_super_admin = windmill_api_auth::is_super_admin_authed(&db, &authed).await?;
let restrict = !is_super_admin && *CLOUD_HOSTED;
let restrict = !is_super_admin;
if restrict {
validate_object_storage_test(&test_s3_bucket).await?;
validate_object_storage_test(&test_s3_bucket)
.await
.map_err(|e| match e {
// A job token never counts as a super admin (it is capped at workspace admin), so
// a super admin calling this route from a script is told why rather than that
// they lack a privilege they hold.
error::Error::NotAuthorized(msg) if authed.job_id.is_some() => {
error::Error::NotAuthorized(format!(
"{msg} A job token ($WM_TOKEN) is never treated as a super admin; call \
this route with a user token instead."
))
}
e => e,
})?;
}
let client = build_object_store_from_settings(test_s3_bucket, Some(&db))
@@ -355,8 +368,8 @@ pub async fn test_s3_bucket(
}
}
// Hardening for the object-storage connectivity test by an untrusted (non-super-admin) caller on
// Cloud. The probe runs on the shared API server, so without these constraints an authenticated
// Hardening for the object-storage connectivity test by an untrusted (non-super-admin) caller.
// The probe runs on the API server, so without these constraints an authenticated
// user could coerce the server into connecting to arbitrary internal endpoints (SSRF), signing
// requests with the instance role (credential exfiltration), or reading/writing the server's local
// disk (filesystem object store).
@@ -366,6 +379,11 @@ async fn validate_object_storage_test(settings: &ObjectSettings) -> error::Resul
opt.as_ref().is_some_and(|s| !s.is_empty())
}
// Every refusal names the way out: the resource usually works in jobs (workers reach the
// endpoint directly), so without it the refusal reads as a broken resource.
const ALTERNATIVE: &str =
"Ask a super admin to run it, or test the resource from a script, which runs on a worker.";
// Reject backends that rely on the server's identity or local filesystem, require explicit
// credentials for the rest (so the server never falls back to its own ambient credentials), and
// resolve the host the client will actually connect to. We derive the *effective* endpoint here
@@ -376,20 +394,25 @@ async fn validate_object_storage_test(settings: &ObjectSettings) -> error::Resul
let effective_endpoint: Option<String> = match settings {
ObjectSettings::Filesystem(_) => {
return Err(error::Error::NotAuthorized(
"Testing a local filesystem object store requires a super admin".to_string(),
"Testing a local filesystem object store requires a super admin: it runs on the \
Windmill server and reads and writes the server's local disk. Ask a super admin \
to run it."
.to_string(),
));
}
ObjectSettings::AwsOidc(_) => {
return Err(error::Error::NotAuthorized(
"Testing OIDC-based object storage requires a super admin".to_string(),
));
return Err(error::Error::NotAuthorized(format!(
"Testing OIDC-based object storage requires a super admin: it runs on the \
Windmill server with the server's own identity. {ALTERNATIVE}"
)));
}
ObjectSettings::S3(s3) => {
if !(non_empty(&s3.access_key) && non_empty(&s3.secret_key)) {
return Err(error::Error::NotAuthorized(
"Testing S3 storage without explicit credentials requires a super admin"
.to_string(),
));
return Err(error::Error::NotAuthorized(format!(
"Testing S3 storage without an explicit access key and secret key requires a \
super admin: it runs on the Windmill server, which would use its own ambient \
credentials. {ALTERNATIVE}"
)));
}
let region = s3
.region
@@ -413,10 +436,11 @@ async fn validate_object_storage_test(settings: &ObjectSettings) -> error::Resul
}
ObjectSettings::Azure(azure) => {
if !non_empty(&azure.access_key) {
return Err(error::Error::NotAuthorized(
"Testing Azure storage without an explicit access key requires a super admin"
.to_string(),
));
return Err(error::Error::NotAuthorized(format!(
"Testing Azure storage without an explicit access key requires a super admin: \
it runs on the Windmill server, which would use its own ambient credentials. \
{ALTERNATIVE}"
)));
}
Some(
azure
@@ -432,10 +456,11 @@ async fn validate_object_storage_test(settings: &ObjectSettings) -> error::Resul
// otherwise an untrusted caller could probe with the server's identity (the very
// SSRF/credential-exfil this function guards against).
if windmill_object_store::gcs_service_account_key_is_blank(&gcs.service_account_key) {
return Err(error::Error::NotAuthorized(
"Testing GCS storage without a service account key requires a super admin"
.to_string(),
));
return Err(error::Error::NotAuthorized(format!(
"Testing GCS storage without a service account key requires a super admin: \
it runs on the Windmill server, which would use its own ambient credentials. \
{ALTERNATIVE}"
)));
}
// The service-account-key JSON can override the data-plane URL (`gcs_base_url`) and the
// OAuth token endpoint (`token_uri`); the GCS client connects to whatever they point at.
@@ -492,10 +517,15 @@ async fn validate_public_endpoint(endpoint: &str) -> error::Result<()> {
// attempts (a name resolving to both a public and a private address).
for addr in addrs {
if is_forbidden_ip(addr.ip()) {
return Err(error::Error::NotAuthorized(
"Testing object storage at a private, loopback, or link-local endpoint requires a super admin"
.to_string(),
));
// The resolved address stays out of the message: it is the server's resolver's
// answer, and this message is only ever shown to the caller being constrained.
return Err(error::Error::NotAuthorized(format!(
"Testing object storage at '{host}', which resolves to a private, loopback, or \
link-local address, requires a super admin: this test runs on the Windmill \
server, which is not allowed to probe internal addresses for non-super-admins. \
Ask a super admin to run it, or test the resource from a script, which runs on \
a worker."
)));
}
}
Ok(())
@@ -2004,6 +2034,15 @@ struct CachedResourceType {
#[allow(dead_code)]
app: String,
description: Option<String>,
/// Doubly optional, and read through a wrapping deserializer: this struct also
/// decodes the on-disk cache, where an absent key means "written before the
/// column, leave the stored extension alone" and an explicit null means the hub
/// dropped it. Plain serde folds both into `None`.
#[serde(
default,
deserialize_with = "windmill_common::more_serde::double_option"
)]
format_extension: Option<Option<String>>,
}
#[derive(serde::Deserialize)]
@@ -2013,6 +2052,8 @@ struct HubResourceTypeRaw {
schema: Option<String>,
app: String,
description: Option<String>,
#[serde(default)]
format_extension: Option<String>,
}
async fn fetch_resource_types_from_hub() -> error::Result<Vec<CachedResourceType>> {
@@ -2054,6 +2095,7 @@ async fn fetch_resource_types_from_hub() -> error::Result<Vec<CachedResourceType
schema,
app: rt.app,
description: rt.description,
format_extension: Some(rt.format_extension),
})
})
.collect())
@@ -2107,10 +2149,12 @@ async fn sync_cached_resource_types(
for rt in &resource_types {
let exists: Option<bool> = sqlx::query_scalar!(
"SELECT EXISTS(SELECT 1 FROM resource_type WHERE workspace_id = 'admins' AND name = $1 AND schema IS NOT DISTINCT FROM $2 AND description IS NOT DISTINCT FROM $3)",
"SELECT EXISTS(SELECT 1 FROM resource_type WHERE workspace_id = 'admins' AND name = $1 AND schema IS NOT DISTINCT FROM $2 AND description IS NOT DISTINCT FROM $3 AND ($5 IS NOT TRUE OR format_extension IS NOT DISTINCT FROM $4))",
&rt.name,
rt.schema.as_ref(),
rt.description.as_deref(),
rt.format_extension.clone().flatten(),
rt.format_extension.is_some(),
)
.fetch_one(&db)
.await?;
@@ -2120,13 +2164,27 @@ async fn sync_cached_resource_types(
}
sqlx::query!(
"INSERT INTO resource_type (workspace_id, name, schema, description, edited_at)
VALUES ('admins', $1, $2, $3, now())
// Whether the payload carried the key at all is what decides: present
// (even as null) is authoritative and may clear, absent means a cache
// written before the column and must leave the stored value alone.
"INSERT INTO resource_type (workspace_id, name, schema, description, format_extension, edited_at)
VALUES ('admins', $1, $2, $3, $4, now())
ON CONFLICT (workspace_id, name) DO UPDATE
SET schema = EXCLUDED.schema, description = EXCLUDED.description, edited_at = now()",
SET schema = EXCLUDED.schema, description = EXCLUDED.description,
-- A fileset is a set of files, so it cannot also be one file.
-- Create and update reject the pair; this writer bypasses both, so
-- it declines the extension rather than persisting the forbidden
-- combination onto a same-named local fileset.
format_extension = CASE
WHEN resource_type.is_fileset THEN NULL
WHEN $5 THEN EXCLUDED.format_extension
ELSE resource_type.format_extension END,
edited_at = now()",
&rt.name,
rt.schema.as_ref(),
rt.description.as_deref(),
rt.format_extension.clone().flatten(),
rt.format_extension.is_some(),
)
.execute(&db)
.await?;
backend/windmill-api-settings/src/log_cleanup.rs
+7 -6
View File
@@ -32,7 +32,7 @@ use windmill_common::tracing_init::{LOGS_SERVICE, TMP_WINDMILL_LOGS_SERVICE};
use windmill_common::worker::WINDMILL_DIR;
use windmill_common::{
DB, INSTANCE_NAME, JOB_RETENTION_SECS, JOB_RETENTION_SECS_OVERRIDES,
JOB_RETENTION_SECS_OVERRIDES_LOADED, SERVICE_LOG_RETENTION_SECS,
JOB_RETENTION_SECS_OVERRIDES_LOADED,
};
use windmill_object_store::object_store_reexports::{
@@ -249,7 +249,7 @@ async fn cleanup_service_logs(
// Count candidates upfront for progress reporting.
let total: i64 = sqlx::query_scalar!(
"SELECT COUNT(*) FROM log_file WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval",
SERVICE_LOG_RETENTION_SECS,
windmill_common::service_log_retention_secs(),
)
.fetch_one(db)
.await?
@@ -274,7 +274,7 @@ async fn cleanup_service_logs(
WHERE log_ts <= now() - ($1::bigint::text || ' s')::interval
LIMIT $2
) RETURNING file_path, hostname",
SERVICE_LOG_RETENTION_SECS,
windmill_common::service_log_retention_secs(),
SERVICE_LOG_BATCH,
)
.fetch_all(db)
@@ -680,9 +680,10 @@ async fn cleanup_s3_orphans(
) -> error::Result<()> {
let job_retention_secs = JOB_RETENTION_SECS.load(std::sync::atomic::Ordering::Relaxed);
let now = Utc::now();
// Service logs always have a retention (hardcoded SERVICE_LOG_RETENTION_SECS),
// so we scan for service-log orphans regardless of JOB_RETENTION_SECS.
let service_cutoff = now - chrono::Duration::seconds(SERVICE_LOG_RETENTION_SECS);
// Service logs always have a retention, so we scan for service-log orphans regardless of
// JOB_RETENTION_SECS.
let service_cutoff =
now - chrono::Duration::seconds(windmill_common::service_log_retention_secs());
// Job-log orphans are only considered once past a job's effective retention window. That window
// is the instance one OR, for an override workspace (EE), its own — and jobs orphan their logs as
backend/windmill-api-users/src/users.rs
+29 -5
View File
@@ -19,7 +19,7 @@ use windmill_api_auth::ApiAuthed;
pub use windmill_api_auth::Tokened;
use argon2::{Argon2, PasswordHash, PasswordVerifier};
use argon2::{Argon2, PasswordVerifier};
use axum::{
extract::{Extension, Path, Query},
response::{IntoResponse, Response},
@@ -1239,6 +1239,7 @@ async fn leave_instance(Extension(db): Extension<DB>, authed: ApiAuthed) -> Resu
sqlx::query!("DELETE FROM password WHERE email = $1", &authed.email)
.execute(&mut *tx)
.await?;
windmill_common::user_drafts::delete_drafts_of_email(&mut *tx, &authed.email).await?;
audit_log(
&mut *tx,
@@ -1661,6 +1662,7 @@ async fn delete_user(
sqlx::query!("DELETE FROM password WHERE email = $1", &email_to_delete)
.execute(&mut *tx)
.await?;
windmill_common::user_drafts::delete_drafts_of_email(&mut *tx, &email_to_delete).await?;
let usernames = sqlx::query_scalar!(
"DELETE FROM usr WHERE email = $1 RETURNING username",
@@ -1869,7 +1871,7 @@ async fn change_user_email(
.execute(&mut *tx)
.await?;
// ---- account ---- (draft.email follows through its ON UPDATE CASCADE fkey)
// ---- account ----
sqlx::query!(
"UPDATE password SET email = $1 WHERE email = $2",
&new_email,
@@ -1883,6 +1885,7 @@ async fn change_user_email(
}
_ => e.into(),
})?;
windmill_common::user_drafts::rename_drafts_of_email(&mut *tx, &old_email, &new_email).await?;
sqlx::query!(
"UPDATE usr SET email = $1 WHERE email = $2",
@@ -2680,10 +2683,8 @@ async fn login(
.await?;
if let Some((email, hash, super_admin)) = email_w_h {
let parsed_hash =
PasswordHash::new(&hash).map_err(|e| Error::internal_err(e.to_string()))?;
if argon2
.verify_password(password.as_bytes(), &parsed_hash)
.verify_password(password.as_bytes(), hash.as_str())
.is_err()
{
audit_log(
@@ -3541,6 +3542,9 @@ async fn overwrite_global_users(
require_super_admin(&db, &authed).await?;
forbid_superadmin_job_token(&db, &authed.email, job_id).await?;
let mut tx = db.begin().await?;
// Replaces the account table, so — unlike the paths that remove one account — it deliberately
// does not call `delete_drafts_of_email`: the addresses are about to be reinstated, and
// dropping every draft on the instance to restore accounts would be pure collateral.
sqlx::query!("DELETE FROM password")
.execute(&mut *tx)
.await?;
@@ -3710,3 +3714,23 @@ async fn request_password_reset(
}
// NOTE: reset_password is in windmill-api (depends on users_oss::hash_password EE dispatch)
#[cfg(test)]
mod tests {
use super::*;
/// Stored hashes outlive the hashing crate: every instance still holds hashes minted by
/// older argon2 releases, and an upgrade that stopped reading them locks their users out.
#[test]
fn verifies_a_hash_minted_by_an_older_argon2() {
// The seeded admin hash from migration 20220508150023, m=4096,t=3,p=1.
let seeded = "$argon2id$v=19$m=4096,t=3,p=1$oLJo/lPn/gezXCuFOEyaNw$i0T2tCkw3xUFsrBIKZwr8jVNHlIfoxQe+HfDnLtd12I";
assert!(Argon2::default()
.verify_password(b"changeme", seeded)
.is_ok());
assert!(Argon2::default()
.verify_password(b"not-the-password", seeded)
.is_err());
}
}

Some files were not shown because too many files have changed in this diff Show More