mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-21 00:02:23 +00:00
v1.771.0
1813 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ruben Fiszel
|
e80fee86b3 |
feat: record and replay raw app sessions step by step (#10318)
* feat: record and replay raw app sessions step by step * fix: address review findings on raw app session recorder * fix: stamp replay target before pruning the snapshot clone * fix: redact step metadata, lock down replayed frames, fix control pre-state * feat: add a checkpoint timeline to the app recording player * fix: parser-based replay CSP, fold label clicks, drop stale frame indices * fix: scrub redacted attributes, keep scroll, neutralize replay navigation * fix: bound replay payloads, strip namespaced nav links, keep control pre-frames * fix: strip SMIL navigation, redact metadata sources, capture pre-edit on beforeinput * fix: redact template content, drop shadow templates, make replays inert * test: pin snapshot redaction and replay sanitization with DOM tests * fix: allow-list no-record attributes and cover a marked document root * fix: classify input types positively so pickers get pre-change frames * fix: one step per control interaction and bound step metadata * fix: keep button inputs recordable and coalesce only continuous controls * fix: no frames for coalesced repeats and drop inline styles when redacting * fix: fold only the label's own click and keep marked stylesheets out * fix: keep label-forwarded and radio-group pre-frames, fold submitter clicks * fix: bound key pre-frames to their gesture and clear ancestor pointer frames * fix: age-bound pre-frames and treat a radio group as one target * fix: consume pre-frames per interaction and coalesce on the browser repeat flag * fix: spend only the pre-frame a step actually used * fix: settle a step from its successor's pre-state and drop stale pointer frames * fix: bound remote frame payloads and snapshot stylesheets as rendered * fix: let a control change spend its own frame and dedupe Enter activations * fix: record Escape on controls and drop disabled stylesheets * feat: collapse the replay step list by default behind a toggle * fix: neutralize disabled sheets in place and fold Enter submissions * fix: withhold redacted control state, fold key repeats, validate remote metadata * fix: drop noscript markup and fold implicit form submissions * fix: mask a select whose chosen option is redacted * fix: mask redacted select choices before the clone diverges * fix: run clone-paired passes before removals and fold only Enter submissions * feat: record a raw app demo from the publish flow instead of the viewer * fix: wait for in-flight runnable jobs before settling a step * feat: record from the editor menu and replay publicly at /replay * feat: export the app recording player and its loader for the hub * feat: publish from folders only, drop iframe sharing * fix: observe runnable responses where they land and mount the hub recording route * fix: respect the app's sandbox opt-in when recording a session * fix: let stop wait for the runnable the last step is still running * fix: filter redacted class/id to styled tokens and gate publish on admin * fix: drop marked sheets from the token vocabulary and bound the replay error * test: pin the remote app-recording validator * fix: carry in-flight runnables across a reload and fold held keys into one step * fix: bind runnable responses off the request and honor base in the replay handoff * fix: close the settling step when a new fill starts and always re-read stylesheets * fix: empty the no-record marker so it carries nothing of its own * fix: decode css escapes so utility classes survive redaction * fix: read keyDriven from the frame the change starts from * docs: condense recorder comments to the invariant each protects * fix: rewrite only real url() tokens and accept leading css escapes * feat: play flow, script and pipeline recordings on the public /replay page (#10327) * feat: play flow, script and pipeline recordings on the public /replay page * fix: render a recorded approval result inert while replaying * fix: bound an asset sample's cell product and validate recording headers * fix: make a replayed approval step inert and bound nested recording structures * fix: stop recorded markup from fetching and bound flow/script render trees * fix: gate recorded markdown at its renderer and close remaining render-budget gaps * fix: replace per-key render caps with one structural budget per recorded value * fix: bound component fan-out and text alongside the structural budget * fix: make component fan-out cumulative and cap the parsed data-test checklist * fix: bound the whole recording, graph contents, metadata strings and timer bursts * fix: keep the published loader path, charge object keys, refuse huge serialized fan-out * fix: cap flat maps a renderer turns into rows (args, schema properties) * fix: refuse structure hidden past the depth ceiling and bound errored samples * fix: count array-shaped argument collections against the row cap * feat: paint canvas pixels into the snapshot * fix: budget canvas encoding per snapshot and bound the unknown-kind error * fix: cap flow graph overlay fan-out and condense budget comments * docs: teach the raw-app prompt about data-wm-no-record |
||
|
Ruben Fiszel
|
2bf7746cdd |
fix: operators cannot archive or delete flows and apps (#10322)
`create_flow`/`update_flow` and `create_app`/`update_app` reject operators, but `archive_flow_by_path`, `delete_flow_by_path` and `delete_app` did not — so an operator with folder write could delete a flow or app they were not allowed to edit. Scripts already get this right (archive is guarded, delete is admin-only). Verified on a live instance: all three returned 2xx for an operator before, 401 after, and a non-operator member with the same folder write is unaffected. |
||
|
Ruben Fiszel
|
4d3ff0299f |
feat: mark failed jobs as resolved so handled failures stop showing red (#10319)
* feat: mark failed jobs as resolved so handled failures stop showing red Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: constrain auto-resolve to the proven retry chain and honor resolved filter everywhere Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: apply resolved filter to queue-union, concurrency and delete paths, bound note Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: sweep resolutions on workspace delete, verify helper args, enforce UI limits Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: count resolution note in characters on both sides of the API Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: skip the queue lookup for cancel-all under the resolved-only filter Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: converge retry auto-resolution from either commit order, keep notes on re-resolve Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct the idempotency claim on the retry auto-resolve sweep Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: gate resolution notes and attribution behind enterprise, add note popover Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: hide resolution from operators, exclude flow steps, enforce EE licence at runtime Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: add job_resolution.automatic to the summarized schema Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: preserve stored attribution when re-resolving without a valid licence Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: condense the attribution-preservation comment to four lines Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: validate resolution notes by code point instead of a UTF-16 maxlength Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the resolution popover open when a note is rejected Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: offer to resolve the original failure after a successful re-run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: verify supersession server-side and stop re-runs overwriting notes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: apply tag scope to the superseding run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: exclude obscured cross-workspace runs from resolution actions Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
9cef724ff2 |
feat: bind WAC approval urls to a named wait_for_approval step (#10317)
* feat: bind WAC approval urls to a named wait_for_approval step Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: reject duplicate WAC approval step keys instead of renaming them Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: reject WAC approval links minted for a step that is not awaiting approval Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: bind WAC approval links to the awaiting step and stop step key aliasing Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: reject empty approval keys and scope minted-key writes to the workspace Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: enforce WAC approval binding at consumption and reject colliding keys Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: make WAC approval binding and collision checks atomic, harden TS step keys Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: decrement WAC suspend atomically instead of from a pre-lock snapshot Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore: add sqlx cache entry for the atomic WAC suspend decrement Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: omit empty approver param from python get_approval_urls Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: pin the suspend-snapshot decrement and the colliding-mint race Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: drop the suspend-snapshot interleave test, it cannot both be stable and discriminate Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: reject step keys that cannot be minted as a URL path segment Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
71b7135cf2 |
feat: multiple homepage sort orders via an efficient merged runnables endpoint (#10297)
Adds recently-updated / oldest / name A-Z / name Z-A sort orders to the homepage (WIN-2236), produced server-side by a new merged, index-backed, keyset-paginated GET /w/{workspace}/runnables/list so a chosen order is globally correct across scripts + flows + apps and stays efficient on large workspaces.
- Backend: UNION ALL of script/flow/app ordered by index (Merge Append + LIMIT); keyset (sort_key, path, kind, tiebreak) cursor; per-branch LIMIT bounds correlated projections; starred-first pinning; RLS + scope-token filters in SQL. Archived view returns the latest row per path. Migration adds time + lowered-name indexes (built CONCURRENTLY).
- Frontend: server-side sort/kind/owner filters + hybrid search (instant client + on-demand server pagination); file-explorer tree with every folder and your user namespace as lazy-loaded top-level nodes (per-owner "Load more", nested subfolders, bounded "expand all", in-place re-sort without collapse or flicker); the client sorts by the server fetch ordinal to reproduce the endpoint's exact order; empty state distinguishes an empty workspace from too-narrow filters.
Reviewed clean by Claude and Pi (good to merge) and Codex (mergeable).
|
||
 Diego Imbert
|
28a79ced15 |
feat: add explore button for object storage resources (#10306)
* feat: add explore button for object storage resources in resource list Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015FhmfSxPuTck3yAhDpkfcA * fix: make s3 drawer tooltip reflect explored resource Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015FhmfSxPuTck3yAhDpkfcA * fix: honor workspace prop in global s3 explorer and add resource connection error state Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015FhmfSxPuTck3yAhDpkfcA * fix: use picker's effective workspace in S3FilePreview requests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015FhmfSxPuTck3yAhDpkfcA * fix: pass acting workspace to explore button in ResourcePicker Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015FhmfSxPuTck3yAhDpkfcA * chore: update ee-repo-ref to f78df23339e3136e8b6e9148a509508633448dd2 This commit updates the EE repository reference after PR #686 was merged in windmill-ee-private. Previous ee-repo-ref: efb5e014fec34fc580b9dbb1b260494dd76c5462 New ee-repo-ref: f78df23339e3136e8b6e9148a509508633448dd2 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
 Tristan TR
|
48618cff8c |
feat: Add image when publishing a project (#10310)
* refactor(hub): remove per-item Publish to Hub entry points Publishing to the Hub now happens exclusively through the folder-level deploy-to-hub flow (/folders). Remove the standalone entry points: - script detail page menu item (and the SCRIPT_VIEW_SHOW_PUBLISH_TO_HUB const that gated it) - script list row dropdown item - raw app editor menu item, its zip-download drawer and publishToHub() - long-dead commented block in AppEditorHeader Also drop the now-orphaned URL helpers (scriptToHubUrl, flowToHubUrl, appToHubUrl, rawAppToHubUrl) from lib/hub.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub): upload a custom project logo from the deploy-to-hub drawer Add a Logo field to the bundle metadata form: a drag-and-drop dropzone (png/svg, 512KB client-side cap mirrored server-side by the Hub) that turns into a live replica of the Hub project card once an image is picked, so the logo can be judged in context before publishing. The logo is pushed after the draft's items/migrations via the new POST /projects/{slug}/logo proxy in hub_publish.rs (slug validated by construction, `logo: null` forwarded to clear). Leaving the field empty never touches the Hub's existing logo, so re-publishing a bundle keeps it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub): logo removal, safer mime inference, explicit clear semantics Review follow-ups on the project logo upload: - Removing a published logo is now possible: hubLogo is three-state (undefined = leave the Hub's logo alone, null = clear on publish, object = upload). Rehydration reads has_logo so the drawer shows a "Remove on publish" affordance when the Hub already has one, with an undo banner before publishing. - hub_publish.rs uses a double-Option for the logo field: a missing `logo` key is now a 400 instead of being serialized as `logo: null`, which the Hub interprets as an explicit clear — POSTing `{}` can no longer silently delete a project's logo. - Client mime inference prefers the browser-reported file.type over the filename extension, so a PNG misnamed *.svg no longer produces a broken preview and a guaranteed server-side sniff rejection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update frontend/src/lib/components/workspaceSettings/deployToHubSession.svelte.ts Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> * Update frontend/src/lib/components/workspaceSettings/DeployToHub.svelte Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> * fix(hub): validate logo size/mime/base64 in the proxy, document the endpoint - Enforce the logo constraints in windmill-api itself instead of relying on the browser and remote Hub: a route-level DefaultBodyLimit sized for a max logo in base64 (+JSON envelope) overrides the global request limit, and the handler validates the mime allowlist, base64 alphabet and decoded length (512KB cap) before anything is forwarded. - Add /w/{workspace}/hub/projects/{slug}/logo to openapi.yaml (with the ProjectLogoBody schema) and regenerate the frontend client. - Drop a narrating comment on the hidden file input. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
3cf7a390a3 |
fix: pin validated DNS address to close SSRF DNS-rebinding TOCTOU (#10303)
* [ee] fix: pin validated DNS address to close SSRF DNS-rebinding TOCTOU validate_url_for_ssrf resolved the host, checked every address was public, then discarded them. Callers re-used the hostname and let stock reqwest re-resolve at connect time, so a TTL-0 DNS rebinder that answered a public IP at check-time and an internal one (e.g. 169.254.169.254) at connect-time slipped straight through the guard. Return the resolved addresses as a ValidatedTarget and pin them onto the client that connects, so validate-time and connect-time target the same address. Covers the AI proxy and worker AI-agent base_url (the primary readable-SSRF sink), AI OAuth token_url, MCP server + OAuth registration/discovery/token endpoints, SAML metadata, and the WebSocket trigger connect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 22abd6d4e229f1206a13ebee8a6a9b808cd82a0d This commit updates the EE repository reference after PR #684 was merged in windmill-ee-private. Previous ee-repo-ref: 700feb02ef1b96758ba9425358dbebc83bc02c61 New ee-repo-ref: 22abd6d4e229f1206a13ebee8a6a9b808cd82a0d Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Diego Imbert
|
68daed8501 |
refactor: custom-instance datatable connection handling (#10271)
Attach custom-instance datatables in the DuckDB executor through a DuckDB secret instead of an inline connection string, and route postgres triggers on custom-instance datatables through a dedicated custom_instance_replication_user role (with its own auto-generated password in global_settings). Normalize custom_instance_user attributes on server boot. Claude-Session: https://claude.ai/code/session_01Tp6NNNinCB8dwWqGaFXDRF Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
65e504146d |
feat: data-pipeline recorder, interactive player, and deploy-to-hub recording (WIN-2156) (#10055)
* feat(frontend): add data-pipeline run recorder and interactive player Adds a recorder/player for data pipelines, mirroring the existing flow and script recorders. Arm "Record" on a pipeline, run it, and the resulting cascade is captured into a downloadable JSON that the /replay player can rerun fully offline. Because a pipeline run is a cascade of independent jobs (not a single root SSE job like flows), the recording captures three things: the resolved asset graph, the per-node cascade status timeline (from the orchestrator's onUpdate), and each node's job stream (opened via getupdate_sse on launch). The player renders the graph read-only, animates the recorded node transitions in real time, and lets you click any node to inspect its recorded args, logs and result — reusing the same JobLoader replay path the flow/script players use (setActiveReplay + isReplay gating), so no network calls are made during replay. - recording/types.ts: PipelineRecording, PipelineTimelineFrame, RecordedNodeState - recording/pipelineRecording.svelte.ts: createPipelineRecording() store - recording/PipelineRecordingReplay.svelte: the player component - replay/+page.svelte: dispatch type === 'pipeline' - pipeline/[folder]/+page.svelte: Record toggle + Download recording; capture the whole-pipeline / bounded cascade run Fixes WIN-2156 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): capture DuckLake/datatable data samples in pipeline recordings Follow-up to the pipeline recorder/player: asset nodes are now inspectable offline in the player, showing what each table held after the recorded run. At record finalization, for each ducklake/datatable asset in the pipeline the recorder samples the table (up to 100 rows + columns + row count) reusing the exact live-preview query path (loadAllTablesMetaData + getRows), so a replayed sample matches what the asset-detail pane would have shown. Captures are best-effort and per-asset — a missing/unconfigured table is stored as an error marker, never thrown, so the recording still completes. The player renders the sample as a read-only typed grid when an asset node is clicked (script nodes keep their logs/result/args detail). - recording/types.ts: PipelineAssetSample + assetSamples on PipelineRecording - recording/pipelineAssetSample.ts: capturePipelineAssetSample() helper - recording/pipelineRecording.svelte.ts: recordAssetSample() + assetSamples - recording/PipelineRecordingReplay.svelte: asset-node data-sample panel - pipeline/[folder]/+page.svelte: sample each asset in finalizePipelineRecording Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * recorder * feat(hub): record data pipelines in deploy-to-hub with interactive player Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub): match editor cascade timeout, warn on cycles, reset badge on re-run Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): address review — finalize race, stale replay timers, /replay redirect, bounded sampling, jobs validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): structural recording validation, guard-clear + SSE cleanup on throw paths Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): validate nested graph arrays and timeline frame statuses Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub): scope recording to bundle membership, fail cyclic runs, validate recording elements Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub): prune recorded graph + asset samples to bundle membership Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): validate graph.triggers array and per-job initial_job/events shapes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): guard non-object payloads, event elements, and asset-sample/code maps Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): render error boundary + validate trigger_kind and non-empty sample error Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): validate event.data and recorded-job shapes for all replay types Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): make the replay event timer crash-proof against malformed events Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): await replay completion and boundary-wrap all three players Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): guard flow Play handler, cap ?src= download size, trim comment Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Tristan TR
|
30eedf9ee1 |
feat: Add section to deploy projects to hub (#9332)
* feat: add Deploy to Hub workspace settings tab * Init record logic * Fix wordings * Add publish-app drawer with per-app rate limit mock - Publish drawer on raw_apps/apps exposes public URL, copy-iframe, unpublish - Inline per-app rate limit config (req/min, burst, per-IP toggle) - Rename workspace settings "Default app" tab header to "Apps" to cover both default app and public rate limiting Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Simplify publish drawer to show workspace-wide rate limit only Drop per-app rate limit fields (req/min, burst, per-IP) — none of these are supported by the backend. The drawer now shows the existing workspace-level rate limit read-only with a link to edit it in Workspace settings → Apps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Rename publish-app drawer wording to 'Share as iframe' 'Publish publicly' was ambiguous (publish to Hub vs make public URL). Use 'Share as iframe' for the button and drawer title, and 'Generate iframe' for the confirm action. Intro text now explicitly mentions iframe embedding use cases (Hub, docs page, own site). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire DeployToHub to real workspace data - Fetch apps, raw_apps, flows, scripts, resources via their services - Fetch workspace rate limit via WorkspaceService.getSettings - Share-as-iframe flips app policy.execution_mode to 'anonymous' via AppService.updateApp and resolves the real public URL via getPublicSecretOfApp + computeSecretUrl - Detect already-public apps from listApps execution_mode field - Filter out app_theme resources (noise, present in every workspace) - Hub bundle/version push and recording remain mocked (no backend yet) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire recordings to real jobs with run-preview UX - Recording flow now fetches the real schema, runs the job, and polls getCompletedJobResultMaybe to surface success/failure before saving. - Drawer shows a sticky status box (loader / success / failure) with a result preview, a job link, and an in-context Save CTA. - Only successful runs can be saved as a recording. Failures show the error and offer re-run. - Filter cache/state/app_theme internal resource types (mirrors workspaces_export.rs filter). - Added "What is a recording?" explainer banner above the items list. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add draft/review state machine and submission gating - Phases: predeploy → draft → under_review → live, with workflow step indicator and contextual footer actions per phase - Bundle drawer collects name + readme before pushing the draft - draftItems snapshot frozen at deploy time; workspaceItems keep refreshing without affecting the draft - Folder MultiSelect lets users scope the bundle to one or more folders; empty = whole workspace - Submit-for-review disabled until every script and flow in the draft has a recording (progress bar + counter) - Recordings now run the real job and poll for success/failure; only successful runs can be saved - under_review phase locks editing, sharing, and recording - Dark mode variants on every coloured banner - Steps card shows the full 3-step process always, highlighting the current step Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Make recordings optional, encourage them for discoverability - Submit for review no longer gated on full recordings - Footer hint now frames recordings as boosting approval speed and public Hub featuring, not as a hard requirement - Progress card label switched from 'Recordings needed' to 'Recordings recommended' - Items without a recording display a yellow 'No recording' badge in every phase so the gap stays visible after submission Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Allow per-item selection inside the bundle scope - Items in predeploy now have checkboxes (all selected by default) - Select all / Deselect all act on the current folder filter - manualDeselected resets when the folder filter changes - Bundle button uses the selected count, disabled when zero - Draft snapshot keeps only the selected items - Checkboxes hidden in draft / under_review / live phases Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add diff button once approved by admins * Small fix * Nits * fix(deploy-to-hub): paginate workspace list and cancel stale record polls - loadWorkspace fetches all pages instead of capping at 100 items per kind - pollJobUntilComplete now bails when recordRunSeq advances (new record target, re-run, or drawer close), preventing late completion of a previous run from overwriting current state Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(deploy-to-hub): parallelize public-app URL resolution resolvePublicUrl now runs once per anonymous app via Promise.all instead of serially inside the items loop, removing N round-trips from initial tab load. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(indexer): tell admins when ingress routes search to wrong pod (#9274) * [ee] fix(indexer): tell admins when ingress routes search to wrong pod When the IndexReader is absent on the pod handling a search request but another pod is actively holding the indexer lock, the EE handler now returns a tailored error pointing at the ingress/load-balancer configuration instead of the generic "indexer not running" message. The indexer status endpoint reads the DB lock so it reports "running" from any pod, but search endpoints need the in-memory IndexReader that only exists on the lock holder. In multi-replica deployments this looks like the indexer is healthy but every search 404s. Companion: windmill-labs/windmill-ee-private#TBD Fixes WIN-1968. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to eb18d7b4c0e37fea3f6e1e2cc44e0fddd74ff817 This commit updates the EE repository reference after PR #586 was merged in windmill-ee-private. Previous ee-repo-ref: 7dd43d1850813071cc18ba49ba090583e7321f4b New ee-repo-ref: eb18d7b4c0e37fea3f6e1e2cc44e0fddd74ff817 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> * feat(cli): add `wmill init prompts` and custom override slot (#9266) * feat(cli): add `wmill init prompts` and custom override slot Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): replace init prompts with refresh prompts + AGENTS.md/AGENTS.cli.md split Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): dedupe claude skills via @-includes and add prompts freshness check Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): drop migration-choice flags from `refresh prompts` Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(cli): add 'Running and previewing local changes' section to AGENTS.cli.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): write full skill content to .claude/, drop @-include wrapper Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): reconcile CLAUDE.md the same way as AGENTS.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): address PR review nits — argv parsing, lazy import, comment detection, error propagation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add yolo mode for ai chat tools (#9258) * feat: add yolo mode for ai chat tools * nit * fix: align chat footer controls * feat: add ai chat autonomy modes * feat: add autonomy mode dropdown * fix: highlight yolo autonomy icon * fix: auto accept flow edits * fix: hide unsupported autonomy modes * fix: handle auto-accept flow editor races * fix(debugger): add non-root user support to Dockerfile (#9277) Mirrors the main Windmill Dockerfile pattern: creates a windmill user (UID/GID 1000) and makes cache/work directories world-writable so the image runs cleanly under Kubernetes securityContext.runAsNonRoot or runAsUser: 1000 without permission errors on Bun, pip, or windmill cache writes. Fixes WIN-1969 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ai): enforce RLS and scope check on user-supplied X-Resource-Path (#9276) * fix(ai): enforce RLS and scope check on user-supplied X-Resource-Path The AI proxy handler accepts an X-Resource-Path header to override the configured workspace AI provider. When supplied, the handler loaded the resource value from the resource table using the root DB pool with no resources:read scope check, so any authenticated workspace user could point X-Resource-Path at a restricted AI resource (e.g. one in a folder they cannot read) and the proxy would use that resource's provider credentials for the outbound AI request. For user-supplied resource paths, now require resources:read:{path} scope and fetch the resource through user_db.begin(&authed) so RLS enforces the same folder/group boundary as the resource API. The RLS- scoped $var: resolution stays in place as defense in depth. The admin-configured workspace/instance ai_config path is unchanged. Fixes WIN-1971 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(ai): regression test for X-Resource-Path RLS enforcement Cover all four cases: - non-admin pointing X-Resource-Path at a restricted resource is rejected - non-admin pointing it at a resource they own still works - admin can point it at any resource - workspace-configured proxy flow (no X-Resource-Path) is unchanged Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add userdraft listing primitives (#9268) * feat: add userdraft listing primitives * fix: cancel stale userdraft discard writes * docs: remove global ai userdraft plan * feat(nsjail): optional disk-backed /tmp via instance setting (#9272) * feat(nsjail): optional disk-backed /tmp via instance setting * test(nsjail): unit-test tmp mount resolver and narrow visibility * refactor(nsjail): switch tmp backing to select + conditional UI * ui(nsjail): make tmpfs the visible default in /tmp backing select * fix(nsjail): refuse preexisting jail_tmp to block symlink escape * fix(nsjail): allow jail_tmp reuse on sequential nsjail calls Codex flagged that python/ruby/rust executors invoke nsjail twice per job_dir (install then run). The previous resolver treated any preexisting jail_tmp as hostile and silently fell back to tmpfs on the second call, so disk-backed mode never reached the main script run for those langs. Use symlink_metadata().is_dir() to distinguish a real directory left by an earlier call in the same job_dir (safe to reuse) from a symlink or other entity (still refused, as the codebase-tar escape requires). Also loosen the frontend visibility predicate: only hide nsjail settings when job_isolation is explicitly 'none' or 'unshare', so deployments that enable nsjail via DISABLE_NSJAIL=false with no DB setting can still see the controls. * chore(main): release 1.706.0 (#9270) * chore(main): release 1.706.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix(nsjail): gate unix-symlink test behind cfg(unix) for Windows build (#9280) The disk_backed_refuses_preexisting_symlink_at_jail_tmp test calls std::os::unix::fs::symlink directly, which doesn't exist on Windows targets. Without a cfg gate, `cargo check --tests` fails on Windows with E0433. Other symlink call sites in this crate (php_executor, bun_executor, rust_executor, etc.) already follow this pattern. Fixes WIN-1972 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Reduce slim image vulnerability surface (#9279) * Reduce slim image vulnerability surface * chore(docker): drop apt-get upgrade -y from slim images apt-get upgrade hurts build reproducibility (same Dockerfile + same commit at different times produces divergent images) and trips hadolint DL3005. The freshness it buys is dominated by simply rebuilding against the periodically-refreshed debian:bookworm-slim base image. The --no-install-recommends and apt-list cleanup wins are kept. --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev> * fix(git-sync): bump to hub/28234 with stateless gpg.program wrapper (WIN-1974) (#9282) * fix(git-sync): revert LATEST_GIT_SYNC_SCRIPT_PATH to hub/28230 to restore GPG-signed deploys (WIN-1974) hub/28231 (PR #9230) is the "thin" script that hands the actual `git commit` to the CLI's hidden `sync git-deploy`. The hub script still does the GPG setup (import key into a fresh GNUPGHOME, dummy `gpg -bsau` to warm the agent passphrase cache, then `git config user.signingkey` + `commit.gpgsign` locally), but the commit no longer runs in the same `git_push` flow — it runs minutes later inside the CLI after workspace API resolution, zip pull, file extraction, and lockfile autofill. By the time the spawned `git commit` asks gpg-agent for the cached passphrase, the cache state is no longer reliable (or the spawned `gpg` ends up talking to a fresh agent), so signing fails non-interactively with `gpg failed to sign the data`. hub/28230 is hub/28217's in-script logic rebuilt with windmill-cli@1.703.3: the GPG setup and the in-script `sh_run("git commit ...")` happen back-to-back in `git_push`, so the cache is always fresh. It preserves wm_deploy / fork branch behavior, the EE deployment-callback `main()` signature is unchanged, and the only min-version check in EE (`is_script_meets_min_version(28103)`) is comfortably below 28230 — so this revert is safe. Forward fix (separate PR): publish a new thin script that, alongside the existing GPG setup, writes a `gpg.program` wrapper using `--pinentry-mode loopback --passphrase-file` so signing is independent of the agent's cache state. Re-bump past 28231 then. Fixes WIN-1974 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(git-sync): check in source-of-truth for the next hub script (gpg.program wrapper) This is the script that will be published to hub.windmill.dev once verified on a customer GPG-signed deploy. It replaces hub/28231's agent-cache pre-warm (`gpg -bsau` with --passphrase) with a stateless gpg.program wrapper + chmod-600 passphrase file. Every git-invoked gpg call goes through the wrapper, which always uses --pinentry-mode loopback (and --passphrase-file when a passphrase exists). Signing no longer depends on gpg-agent having a cached passphrase by the time the CLI's `git commit` runs — which closes WIN-1974. Not wired in yet: LATEST_GIT_SYNC_SCRIPT_PATH stays on hub/28230 until this script is uploaded and the new hub id is known. This file is checked in so the diff is reviewable, future bumps have a source of truth, and a CLI regression test can `cat` it for fixture parity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(frontend): skip format/pattern validation for $var/$res/$jsonvar references in ArgInput A resource field with a `pattern` constraint (e.g. the gpg_key.private_key field, whose pattern enforces a `-----BEGIN PGP PRIVATE KEY BLOCK-----` prefix) rejects values like `$var:u/me/gpg-private-key` with an "invalid format" error in the resource editor — even though `$var:`/`$res:`/`$jsonvar:` are placeholders the backend resolves at runtime, not the actual string that needs to match the regex. Bail out of all format/pattern checks (email, ipv4, ipv6, uuid, custom pattern) when the value is one of these references. Required/numeric bounds/array checks still apply since they're shape-level, not regex. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(git-sync): bump LATEST_GIT_SYNC_SCRIPT_PATH to hub/28234 (gpg.program-wrapper fix) hub/28234 is the forward fix for WIN-1974: replaces hub/28231's agent-cache pre-warm (which became stale by the time the CLI's `git commit` ran) with a stateless `gpg.program` wrapper that uses `--pinentry-mode loopback` (and `--passphrase-file` when a passphrase exists) on every gpg invocation. Bundled CLI is windmill-cli@1.705.0. Verified via reproducer at /tmp/git-sync-diff/test-gpg-fix.sh: deliberately killing gpg-agent between GPG setup and `git commit` reproduces the customer's `gpg failed to sign the data` error verbatim under the old flow, and the wrapper signs through it. Holds for passphrase-protected keys, split-subkey [C]+[S] layouts, and unprotected keys. Drops the local source-of-truth copy (`hub-scripts/`) — hub is canonical now that 28234 is published. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(git-sync): drop verbose comment above LATEST_GIT_SYNC_SCRIPT_PATH The git history (this PR) carries the why; the constant name + value carry the what. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): wmill sync git-deploy stops committing; caller owns commit+push (#9284) Single contract for the deployment-callback path: the CLI does branch checkout + pull, the caller (hub script in production, test in test) does git add + commit + push. This restores the WIN-1974 invariant — GPG setup and `git commit` run back-to-back in the same process, so the agent's pre-warmed passphrase cache is still warm at sign time — without needing a `--skip-commit` flag for the hub case and a default "also-commit" for everything else. Same behavior in every call site. Changes: - sync.ts: drop the gitSyncDeployPush call from pull()'s deploy path (both the onlyCreateBranch fast-return and the post-pull commit). `gitSyncDeployPush` stays exported for any caller that wants the same commit/push semantics — just not invoked by the CLI subcommand. - gitsync_promotion.test.ts: e2e test now does its own git add + commit + push after `wmill sync git-deploy`, mirroring what the hub script does in production. Same regression coverage (wm_deploy branch created in Case A, main untouched; main updated in Case B, no new wm_deploy). CLI typecheck unchanged (two pre-existing TarAsZip errors at lines 2578/3307, present before this PR). All 743 unit tests still pass. The accompanying hub script (option-C — CLI for branch+pull, script for commit+push) lives at /tmp/git-sync-diff/sync-script-to-git-repo-windmill.option-C.ts. Once published, a follow-up bumps LATEST_GIT_SYNC_SCRIPT_PATH to its id. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bump git sync to 28236 * fix: fork compare visibility for non-admins and stale-token superadmins (#9283) * fix: use fork-scoped authed for fork visibility in compare_workspaces * test: add EE end-to-end repro for fork rename visibility * chore: restore concurrency_locks sqlx cache lost in cleanup * test: add regression for stale-superadmin-token fork visibility bug * chore: update sqlx cache for new test queries * chore(main): release 1.706.1 (#9281) * chore(main): release 1.706.1 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * feat: add wmill job rerun subcommand (#9275) * feat: add wmill job rerun subcommand * feat: add wmill job restart subcommand for flow restart-at-step * chore(system_prompts): point plugin skills sync at plugins/windmill/ (#9287) * chore(system_prompts): point plugin skills sync at plugins/windmill/ The plugin checkout's plugin folder is being renamed from `plugins/windmill-code-plugin/` to `plugins/windmill/` to shorten the slash-command namespace and align with the matching Cursor plugin layout. Paired with windmill-labs/windmill-claude-plugin#8. That PR must merge first so the next sync run finds the new folder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(system_prompts): update plugin-dir example to plugins/windmill Co-authored-by: centdix <centdix@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: centdix <centdix@users.noreply.github.com> * fix(cli): wmill sync pull updates wmill-lock.yaml for raw apps (#9289) * fix: flow recording teardown crash + rename package to @windmill-labs/components (#9288) * fix: guard against null recording during FlowRecordingReplay teardown Navigating away from a flow recording inside a workspace file-tree view threw `TypeError: Cannot read properties of null (reading 'flow')` from FlowGraphViewer once during the teardown tick. Svelte 5 compiles child component props as live getters that close over `$$props.recording.flow`. When `recording` flips to null on the parent's navigation, an outer `{#if !recording?.flow}` doesn't stop those getters from firing one more time as derived effects re-evaluate before the unmount lands — so the getter dereferences null and throws. Fix at the two layers where the deref actually happens: - FlowRecordingReplay: use `recording?.flow` at the binding sites (FlowViewer + graph-snippet FlowGraphViewer) so the compiler emits an optional-chained getter, and guard the snippet branch with `{:else if recording?.flow}` so it doesn't mount when there's nothing to show. - FlowGraphViewer: finish the optional chaining the rest of the file already used everywhere else (`flow?.value?.skip_expr`, `flow?.value?.cache_ttl`, `flow?.schema`). When the upstream binding returns undefined during teardown, the graph degrades to an empty frame instead of crashing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rename package to @windmill-labs/components - frontend/package.json: rename `windmill-components` → `@windmill-labs/components` - frontend/publish.sh: drop the in-place sed rename dance; the checked-in name now matches what's published, so `npm run package && npm publish` is enough - frontend/package-lock.json, system_prompts/auto-generated/prompts.d.ts: regenerated by `npm run package` under the new name Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(flows): restore Variables and Resources in flow editor prop picker (#9290) The design system overhaul in |
||
|
Ruben Fiszel
|
c50a2abad0 |
fix(jobs): sanitize NUL in completed job result before jsonb insert (#10274)
## Summary
A job whose result contains a real NUL (U+0000) serializes to a `\u0000` JSON escape that the `jsonb`-typed `v2_job_completed.result` column rejects with Postgres `22P05` ("unsupported Unicode escape sequence"). This aborts the `INSERT` in `commit_completed_job`, which then retries 10 times and leaves the job unable to complete (surfaced as `Could not add completed job <id>: ... unsupported Unicode escape sequence`).
The fix sanitizes the serialized result immediately before the insert, with effectively zero overhead on the common NUL-free path.
## Changes
- **Promote `strip_json_nul` into `windmill-common`** (`utils.rs`): `fn strip_json_nul(&str) -> Cow<str>` — a `contains("\\u0000")` fast guard returns the input borrowed when clean; only a genuine odd-parity NUL escape triggers the O(n) rebuild. `Cow::Owned` is returned **only** when a NUL was actually stripped, so a legitimate `\\u0000` (escaped backslash + literal text) borrows through untouched. Replaces the two duplicated copies previously in `windmill-api/src/drafts.rs` (`strip_json_nul`) and `windmill-api/src/apps.rs` (`strip_null_chars`); both call sites now use the shared helper.
- **Add `serialized_json()` to the `ValidableJson` trait** (`windmill-queue/src/jobs.rs`): `Box<RawValue>` returns `Cow::Borrowed(self.get())` (zero-cost, already serialized); other impls serialize on demand via `to_raw_value`.
- **`commit_completed_job`** binds `strip_json_nul(result.serialized_json())` as `$3::text::jsonb` in both the `INSERT ... SELECT` and the `ON CONFLICT ... result = $3` (was `result as Json<&T>`). Stored data is unchanged (Postgres parses JSON text into `jsonb` identically); `wm_labels`/`result_metadata` still operate on the typed `T`.
- **Regenerated the sqlx offline cache** (one query file swapped; EE caches preserved).
- **Doc:** updated the stale `strip_null_chars` reference in `windmill-api-workspaces/src/workspaces.rs` to point at the shared `strip_json_nul`.
## Test plan
- [x] `cargo check -p windmill-queue -p windmill-api -p windmill-common -p windmill-api-workspaces` — clean, no warnings
- [x] `strip_json_nul` unit tests in `windmill-common` (clean-borrow, real-NUL, legit-escape borrow no-op, collision, nested keys/values, odd-run): 6 passed
- [x] End-to-end regression in `backend/tests/nativets_jobs.rs` (`--features deno_core`): a JS job returning a genuine NUL and a literal `\\u0000` completes, storing `"ab"` (stripped) and `"a\\u0000b"` (preserved). Without the fix the insert aborts and the job never completes.
- [x] `backend/tests/drafts_nul.rs` integration test still passes (helper refactor intact)
|
||
|
Ruben Fiszel
|
2d24b3ac49 |
fix(jobs): enforce self_approval_disabled on the UI resume path (#10262)
* fix(jobs): enforce self_approval_disabled on the UI resume path The "Resume" button in the run detail UI calls the resume_suspended endpoint, whose owner shortcut skipped the approval-condition checks entirely. A flow owner/operator who triggered the run could therefore self-approve despite self_approval_disabled, unlike the owner endpoint which enforces it. Only admins should bypass self-approval. - Extract require_not_self_approval and enforce it before the owner shortcut in resume_suspended and can_approve_step (button visibility), matching resume_suspended_flow_as_owner. - Persist approval_conditions when self_approval_disabled is set even without user_auth_required, so the restriction is not silently dropped at the resume boundary for raw-flow/CLI authors. Fixes WIN-2223 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(jobs): keep self-approval capability-based on the secret path; docs/tests Scope the self_approval_disabled enforcement to identity-based resume boundaries only. Possession of the full HMAC resume URL is the authorization on the secret path (the URL is disclosed only to intended approvers, e.g. when a step returns it), so resume_suspended_job intentionally keeps skipping approval conditions and token-only (anonymous) resumes on resume_suspended are not gated either. The logged-in owner/operator self-approval fix stays. - Add extract_approval_conditions helper (WAC vs classic) reused in resume_suspended. - Update can_approve_step doc to reflect that self_approval_disabled bars the triggerer before the owner shortcut (codex nit). - Reword new test comments to state the invariant, not prior behavior (codex nit). - Add test_self_approval_disabled_without_user_auth_required covering the persistence + authenticated self-approval check for a non-owner triggerer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
8310e46b19 |
(windows) lean worker-only build, stop compiling the amqp trigger (#10251)
windmill-trigger-amqp does not compile on Windows: tokio-reactor-trait only implements reactor_trait::Reactor for its Tokio type under #[cfg(unix)]. This broke two Windows CI jobs since the amqp trigger landed (#10230): the ee_windows worker build (via the amqp_trigger feature) and, because the crate is a default workspace member, the backend-test-windows job (`cargo test --all` compiles every member regardless of features). The amqp trigger is a server-only feature never run on Windows workers, so the fix is to stop compiling it on Windows rather than port its reactor. Worker binary (ee_windows): replace the ce_core+ee_core bundle (every trigger + all server-only features) with a worker-only worker_windows_core. A non-agent worker still runs the full windmill-api on localhost for its own operations (main.rs run_server, under `if !is_agent`) and jobs call back into it via the wmill client, so keep every feature the worker's own runtime path or its jobs touch, and drop the rest. Kept: languages, parquet, quickjs, enterprise/license, prometheus, otel, jemalloc, AI-agent execution (windmill-worker/mcp + windmill-store/mcp client and OAuth-MCP refresh, windmill-worker/bedrock for direct AWS Bedrock), OIDC Vault secrets (openidconnect), instance-SMTP email — critical alerts and the error-handler send endpoint (windmill-api/instance_smtp), OAuth refresh (oauth2 — reload_base_url_setting populates OAUTH_CLIENTS, get_value_internal refreshes tokens in the worker's internal API server), inline/preview runs (run_inline — jobs call /jobs/run_inline/*). Dropped: all *_trigger/kafka/nats/sqs listeners plus static_frontend, stripe, embedding, zip, the MCP gateway (windmill-api/mcp), the server Bedrock proxy route (windmill-api/bedrock), and cloud (runtime-gated on CLOUD_HOSTED, never true self-hosted). Split windmill-api's smtp feature: the send_email_with_instance_smtp endpoint (error-handler failure emails) only needs windmill-common's rustls sender, but the smtp feature also bundled the inbound email trigger's openssl + mail-parser + windmill-trigger-email. Add instance_smtp = ["windmill-common/smtp"] gating just the endpoint; smtp now includes it. The worker uses instance_smtp, avoiding openssl (which broke the ee_windows check step) and the email-trigger crate. backend-test-windows: the Windows binary is worker-only, so test the crates a worker runs (windmill-worker/-common/-queue) via -p instead of `cargo test --all`. --all compiled every workspace member regardless of features — pulling in the amqp crate (which does not build on Windows) and linking the whole windmill-api integration-test suite, whose combined size overran the runner disk (LNK1180). Also unset the setup-rust-toolchain default RUSTFLAGS=-D warnings for this job so cross-platform dead-code (cfg(unix)-only helpers unused on Windows) does not fail the run; hygiene stays enforced on the Linux CI and the build_windows_worker_ release build. Full-workspace coverage runs on the Linux CI. Also drop the redundant `mkdir frontend/build` from the Windows worker workflows and stub openapi-deref.json alongside the .yaml to avoid embedding ~2.5MB of openapi spec the worker never serves. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
d2c5d6f4b4 |
feat: make content search a full CE feature (#10252)
Content search (the `#` mode of the home-page Ctrl+K search, which searches scripts/flows/apps/resources by content) was capped on CE to 10 scripts and 3 each of flows/apps/resources, with an "EE feature" warning in the UI. It is now a full CE feature: the CE result caps are lifted to match the previous EE limits (10000 scripts, 1000 each of the rest) and the EE warning is removed. Fixes WIN-2218 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
4a898247a2 |
fix(apps): let entitled viewers read pre-existing S3 files from deployed apps (#10245)
* fix(apps): let entitled viewers read pre-existing S3 files from deployed apps Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): confine S3 viewer fallback to full unscoped sessions A scope-restricted token (e.g. apps:read:<app>, or an app-embed token) is allowed on apps_u/* but rejected by the route-scope middleware on job_helpers/*, so granting it the viewer fallback would be a new capability it cannot obtain directly. Gate the fallback on scopes.is_none() so only full sessions (which can already read via job_helpers) delegate; scoped and anonymous callers stay gated. Add a scoped-token isolation assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): treat filter-tags-only tokens as unscoped for S3 viewer fallback The scopes.is_none() guard wrongly denied the viewer fallback to tokens that are effectively unscoped (empty scope arrays and if_jobs:filter_tags:-only tokens), which the route-scope middleware treats as unrestricted and which can therefore read the same file via job_helpers directly. Reuse that semantics via a shared is_effectively_unscoped helper so the relaxation covers exactly the tokens that gain no new capability, while genuinely scoped tokens stay gated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
68debab877 |
feat(triggers): add AMQP (RabbitMQ) trigger via lapin (#10230)
* feat(triggers): add AMQP (RabbitMQ) trigger using the lapin library Fixes WIN-2214 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(triggers): defer AMQP cross-workspace deploy pending utils-internal publish Revert the amqp_trigger additions to the shared windmill-utils-internal TriggerDeployKind and the frontend cross-workspace deploy adapter: the frontend installs the published npm package, which lacks the new kind until a release is cut. AMQP create/edit/delete/list/sync/capture are unaffected (they use local types); only cross-workspace deploy/merge of AMQP triggers waits on the package bump. Also document the at-most-once ack in the consumer loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): address AMQP review — at-least-once ack, workspace cascade, contracts - ack AMQP deliveries only after successful dispatch; nack+requeue on failure - add ON DELETE CASCADE workspace FK so amqp_trigger rows are cleaned on workspace deletion (and the listener stops) - fix the /amqp_triggers/test OpenAPI body and add amqp_trigger to WorkspaceDiffRow.kind - register AMQP in the generated workspace trigger tool (create_trigger) - drop banned $bindable defaults on optional props in the config section - add build_uri unit tests (encoding, ports, vhost) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): stop AMQP poison-message loop and reconnect on transient drops Chaos testing against a live RabbitMQ broker showed the previous nack(requeue) + immediate re-poll spun a tight redelivery loop (~1000 critical-error reports/sec) on a poison message, and any connection blip permanently disabled the trigger (lapin has no built-in reconnect). - on dispatch failure: nack+requeue then stop consuming; the listener framework re-lists the trigger after its ping goes stale (~15s), backing redelivery off to that cadence instead of a tight loop (verified: rate dropped from ~1000/s to ~1 per ~26s, message preserved) - on connection/stream error: stop and let the framework reconnect instead of disabling; persistent failures are still disabled via get_consumer (verified: a forced connection close now auto-reconnects and resumes) - finish the AI create-trigger action wiring for AMQP: add amqp to CreatedResourceTriggerKind, the action-card registry, and the drawer registry so the result card renders and its "Open" action works Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP frontend registries and defer merge rows - add amqp to capturableTriggerTypes (so AmqpCapture mounts), the Runs jobTriggerKinds filter, and CLOUD_DISABLED_TRIGGER_TYPES - wire AMQP into global AI chat mode: TRIGGER_KINDS, the request union, writeTriggerSchema, triggerServices, and the draft adapter - stop emitting actionable AMQP fork-comparison rows (revert amqp_trigger from TRIGGER_OR_SCHEDULE_TABLES) since cross-workspace deploy is deferred until windmill-utils-internal is published — avoids a deploy that fails with "Unknown kind: amqp_trigger" - use design-system TextInput instead of raw <input> in the config section Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP session/draft registries and constrain prefetch - add amqp to the session-deploy, draft-compare, preview-router, and copilot workspace-item registries so AMQP drafts/deploys/nav/path resolution work - include amqp_count in the MoveDrawer attached-trigger rename warning - replace the raw prefetch <input> with a design-system TextInput bounded to an integer 1-65535 (backend u16) and block save on invalid values Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): make AMQP disconnect/reconnect consistent with the Kafka trigger lapin, like rdkafka, has no transparent reconnect, so the AMQP listener now mirrors the Kafka trigger's explicit reconnect loop instead of relying on the framework re-list (which disabled the trigger once get_consumer failed on a sustained outage): - get_consumer returns cheaply; consume owns a (re)connect loop that retries with a 30s backoff, reports a critical error every 10 failed attempts, and reports a recovered critical error once it reconnects — never disabling the trigger on a connectivity failure - a consumer/stream error breaks out to reconnect rather than disabling - dispatch failure still nacks+requeues (at-least-once) with a short backoff to avoid a tight poison-message loop, keeping the connection alive Verified against a live RabbitMQ broker: killing the broker keeps the trigger enabled and retrying (attempt N), and restarting it auto-reconnects (logs "reconnected after N attempts") and resumes dispatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP capture registries and constrain prefetch contract - add the 'amqp' case to triggerKindToTriggerType so opening the AMQP editor from a capture button no longer throws "Unknown TriggerKind: amqp" - register AmqpIcon in CaptureTable's icon map and add an AMQP entry to the script/flow CaptureButton menu - bound the OpenAPI prefetch_count to an integer 1-65535 (matches the Rust u16) and regenerate clients/prompts - require a non-empty exchange name when the exchange binding is enabled - build_uri: fall back to "/" on a blank vhost and bracket IPv6 hosts (+ tests) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(triggers): wire AMQP into pipeline graph, git-sync, and preprocessor types - asset_graph: discover attached amqp_trigger rows and emit an AMQP TriggerEdge so AMQP triggers render (and can be opened/deleted) on the data-pipeline canvas - frontend pipeline graph: add amqp to NativeTriggerKind, the add-trigger menu, node presentation, event-trigger set, annotation keywords, and the editor/service registrations - git-sync: add the amqp_trigger include pattern (+ test) so an AMQP git-sync deployment stages only its .amqp_trigger.* file, not an unrelated same-path object - preprocessor starters: add the AMQP event to the generated TS/Python/PHP trigger event types (kind/payload/exchange/routing_key/queue_name/redelivered/ delivery_tag) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): finish AMQP pipeline/parser wiring, prefetch validation, source lists - fix a stray edit that corrupted the pre-existing MqttTriggerEditor import ($lib/... path) in PipelineTriggerEditors.svelte - reject prefetch_count = 0 server-side in validate_config (RabbitMQ treats 0 as unlimited) and defensively skip basic_qos(0) in build_consumer (covers the capture path that bypasses CRUD validation) - recognize `// on amqp` in the canonical parser (TriggerSpec::Amqp) and add amqp to the CLI non-autorun/event-trigger sets so a pipeline cascade never runs an AMQP-only node as a manual root without an event - add amqp to the preprocessor intro lists and both pipeline AI instructions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): reject zero AMQP prefetch in all paths and finish guidance lists - extract a shared validate_amqp_options used by both CRUD validate_config and build_consumer, so capture configs (which bypass CRUD validation) also reject prefetch 0 instead of silently connecting with an unlimited buffer (+ unit tests for 0/1/65535/None) - add AMQP to the main script-writing preprocessor-sources prompt and the CLI triggers-skill guidance list Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(triggers): de-duplicate AMQP prefetch comment and fix GET response text - keep the zero-prefetch rationale only on the shared validate_amqp_options doc; drop the redundant call-site comments - correct the getAmqpTrigger OpenAPI 200 description ("deleted" -> "retrieved") Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to aaa6cb89b05b76139252c64f057e53b94d12ac60 This commit updates the EE repository reference after PR #680 was merged in windmill-ee-private. Previous ee-repo-ref: 5da5fd65aca9594b2611837a52e4677b544b0380 New ee-repo-ref: aaa6cb89b05b76139252c64f057e53b94d12ac60 Automated by sync-ee-ref workflow. * chore(migrations): consolidate the four AMQP migrations into one The table and the three enum ADD VALUE statements (trigger_kind, job_trigger_kind, draft_kind) are one atomic feature. ALTER TYPE ... ADD VALUE runs inside the migration transaction on PG >= 14 (Windmill's minimum) since the amqp_trigger table doesn't reference those enum types, so they can share a single migration instead of four. Verified applying cleanly in a single transaction on a fresh DB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
fd51d40f12 | feat(pipelines): catalog declared measures and dimensions (#10190) | ||
 hugocasa
|
ae3d9ce2c0 |
fix(mcp): apply token scopes consistently across mcp endpoint tools (#10162)
* fix(mcp): apply token scopes consistently across mcp endpoint tools Endpoint-tool authorization is now shared between single- and multi-workspace modes, honors the token's script/flow path patterns for every path-taking tool, and the JWT minted for proxied endpoint calls carries scopes derived from the caller's own. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(mcp): review nits: restore create_http_request doc, reword comments Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
7d2c5ceb0f |
fix(flows): make updateFlow body path optional so AI can update flows (#10176)
* fix(mcp): default a body field to its same-named path param so updateFlow works Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: trim mcp path-param fallback helper comment Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(mcp): keep path params un-mangled so update tools take plain `path` Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(flows): default update_flow body path from URL via EditFlow Harmonizes updateFlow with the EditVariable/EditResource/EditApp convention: the flow to update is identified by the URL, so the body path is optional and only needed to rename. Fixes the 422 at the API layer for every client (MCP, the in-app AI chat, raw HTTP), not just the MCP tool schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(mcp): drop redundant body-path fallback now that the server defaults it Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: fix stale generator comment after removing mcp body-path fallback Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(flows): mark updateFlow body path optional in the openapi contract Adds an `EditFlow` schema (path optional) for the update route so the public contract matches the server; createFlow keeps `OpenFlowWPath` (path required). Also trims two test comments to record constraints rather than history. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
 Alexander Petric
|
91d6606868 |
fix(mcp): push granular scope patterns into SQL so scoped scripts/flows aren't truncated (#10140)
* fix(mcp): push granular scope patterns into SQL so scoped scripts/flows aren't truncated MCP `list_tools` fetched scripts/flows capped at the 100 newest by `created_at` and only *then* filtered by the token's granular folder/custom scope in Rust. In a workspace with more than 100 scripts/flows, in-scope items outside that newest-100 window were truncated before the scope filter ran, so a folder- or custom-scoped token could see zero tools even though matching items existed. Push the scope patterns into the query via a new `PathFilter::Patterns` (mirroring `is_resource_allowed`: `*` disables filtering, exact paths match by equality, `x/*` matches the folder or its subtree, empty grants nothing) so the filter applies before the `ITEMS_FETCH_MAX_LIMIT` cap. The existing hashed-name resolution path keeps its prefix behavior via `PathFilter::Prefix`, and the Rust post-filter stays as defense in depth. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(mcp): warn in scope selector when a scope exceeds the MCP tool cap The server exposes at most ITEMS_FETCH_MAX_LIMIT (100) scripts and 100 flows per token; a scope matching more silently drops the overflow, which bloats the assistant's context with a partial, arbitrary tool set. McpScopeSelector now computes how many scripts/flows the current scope would expose (per type, mirroring the backend's is_resource_allowed) and shows a warning Alert when either exceeds the cap, so the user can narrow the scope before generating the URL/token. An async sequence guard keeps rapid scope changes from applying stale counts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): address review — dedup count fetches, boundary-aware folder counting, fix copy Follow-up to the MCP scope-selector truncation warning: - Reuse a single per-type (scripts/flows) cache for both the preview list and the exposed count, instead of a second concurrent fetch of the same rows. - Count a folder scope against the `f/{folder}/*` subtree (via the same boundary-aware matcher), so a folder like `team` no longer over-counts a sibling like `team2` and falsely warns. - Custom-mode counts are derived synchronously from the already-loaded scripts/flows — no fetch. - Reword the warning to "most recent" (flows are ordered by edited_at, not created_at). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): anchor folder count fetch at the folder boundary Follow-up to review: the folder count fetched the unbounded prefix `f/{folder}` (backend `path LIKE 'f/{folder}%'`), so a prefix-sharing sibling like `f/team2` shared the page. With a page limit, enough newer sibling rows could fill the first page ahead of the target folder's older rows; the client-side boundary filter then dropped them all, wrongly suppressing the warning and emptying the preview. Fetch `f/{folder}/` instead so the backend prefix (`LIKE 'f/{folder}/%'`) is anchored at the folder boundary and never returns siblings. The client-side matcher stays as a backstop for folder names whose LIKE wildcards (`_`, `%`) can still let the backend prefix over-match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
Diego Imbert
|
4fc3f304c6 |
feat(forks): add "Hide unchanged drafts" toggle to fork deploy-draft tab (#10022)
A fork clones the parent workspace's drafts on creation, so the fork's Deploy-draft tab listed all of them as deployable even though the fork never touched them. Add a "Hide unchanged drafts" toggle (fork-only, on by default) that hides drafts identical to the parent's. Backend: the drafts-list endpoint accepts an optional `compare_to_workspace` (honored only when it is the workspace's actual parent) and flags each row with `unchanged_from_parent` via a jsonb-equality subquery against the parent's draft at the same (path, kind, owner). Frontend: the fork compare page passes its parent as the compare workspace; CompareDrafts renders the toggle and filters out unchanged rows, which also drops them from the selection/deploy count. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
51d8db6602 |
feat: automatic git-to-windmill sync (polling, webhooks, in-app PRs + checks) (#9552)
* docs: add design doc for automatic git-to-windmill pull sync
* docs: add migration plan and implementation phases to git-sync pull design
* feat(git-sync): add auto_pull settings schema and pull enqueue primitive
Adds AutoPullSettings/AutoPullMode/AutoPullStatus on GitRepositorySettings
(workspace_settings.git_sync JSONB), the GIT_SYNC_PULL_SCRIPT_PATH constant,
and should_pull/effective_poll_interval_s helpers with unit tests. Exports the
EE enqueue_git_pull_job primitive. Foundation for repo→Windmill auto-pull.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): poll repos and auto-pull new commits into the workspace
Phase 1 of automatic repo → Windmill sync. A monitor task (EE-licensed,
single-replica via advisory lock) git ls-remotes each auto-pull-enabled
repository ~every minute and enqueues a pull when the tracked branch moves,
reusing the {workspace_id}:git_sync concurrency key so pulls serialize with
in-flight push commits.
- windmill-store: background (no-authed) resolver get_git_repo_head_for_autopull
that resolves the repo resource (incl. $var: refs) and ls-remotes; GitHub-App
repos are skipped here and will sync via webhooks (phase 2).
- monitor.rs: poll/reconcile/persist with optimistic sha advance and failure
status; targeted jsonb update so concurrent settings edits aren't clobbered.
- edit_git_sync_repository: preserve server-owned auto_pull state on UI save.
- openapi: AutoPullSettings/AutoPullMode/AutoPullStatus + auto_pull field.
- frontend: per-repo "Automatically deploy changes from Git" toggle with last
sync status; demote the GitHub Actions link to an advanced CI option.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): wire webhook lifecycle + receiver; share reconcile logic
OSS side of phase 2 auto-pull webhooks:
- edit_git_sync_repository creates/removes the repo webhook on save (EE-gated,
best-effort → falls back to polling).
- monitor poller now delegates to the shared windmill_git_sync reconcile/persist
helpers (also used by the webhook receiver), removing duplicated logic.
- export the shared reconcile/persist/failure helpers; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref for phase 3 in-app PR creation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): show webhook vs polling status on the auto-pull toggle
When a repo has an active webhook (auto_pull.webhook_id set), the status line
reads "instant via webhook"; otherwise it reads the ~1-minute polling cadence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): post PR diff check on dry-run completion (phase 4)
Worker completion hook in process_completed_job: when a DeploymentCallback job
carrying the __git_sync_pr_check marker finishes, parse the dry-run SyncResponse
and patch the GitHub check run with the diff summary (success/neutral/failure).
Export enqueue_git_pull_dry_run; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref (drop unused GHES webhook_secret)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* revert(git-sync): defer phase 4 PR diff checks (OSS side)
Remove the worker completion hook that posted the PR check run, drop the
enqueue_git_pull_dry_run re-export and the orphaned sqlx cache, bump EE ref.
Phases 1-3 (polling, webhooks, in-app PR creation) are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "revert(git-sync): defer phase 4 PR diff checks (OSS side)"
This reverts commit
|
||
|
Ruben Fiszel
|
8c725d9e44 |
fix(apps): honor presigned S3 signature on app display/preview routes (#10141)
The app provenance gate short-circuits on a valid presigned signature, but only the raw download_s3_file route parsed it. The parquet/csv/table-count/file-preview/metadata routes discarded sig/exp and always fell through to the provenance gate, so a presigned S3 object rendered as a table showed "File restricted" for any viewer who did not produce it. Thread sig/exp through every apps_u S3 display route and forward the presigned bearer from ParqetCsvTableRenderer/DisplayResult. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
f7eb5c460d |
fix(apps): invalidate cached app policy on change or deletion (GHSA-r5v4-cxh9-7qhq) (#10121)
* fix(apps): invalidate cached app policy on change or deletion (GHSA-r5v4-cxh9-7qhq) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(agents): keep PR tests and comments minimal and non-ephemeral Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
4edffeb84b |
fix(mcp): align script auto_kind filter with scripts list API (#10098)
The MCP `get_items` script filter used `auto_kind IS NULL`, which excluded every script with a non-null `auto_kind` (pipeline, test, WAC, ...). These are valid runnable scripts and should surface as MCP tools. Switch to the deny-list `(auto_kind IS NULL OR auto_kind <> 'lib')`, matching the scripts list API (windmill-api-scripts). Only library scripts (no main function) are excluded; pipeline/test/WAC and any future auto_kind values are included. Fixes WIN-2190 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
710a13a59d |
fix(apps): cover script/flow component outputs in deployed-app S3 provenance gate (#10070)
* fix(apps): cover script/flow component outputs in deployed-app S3 provenance gate Deployed apps read S3 files on-behalf of the app author for logged-in viewers (#10048). A confused-deputy guard confines those reads to files the app "produced", but the recent-production check only matched inline `appscript`/ `preview` jobs nested under the app path. Files produced by the deployed script/flow components an app is wired to run (e.g. a SQL query persisted to S3) were therefore denied "File restricted" for every viewer, admins included. Expand the provenance check to also match completed `script`/`flow`/`flowscript`/ `flownode` jobs whose `runnable_path` is one of the app's declared triggerables, and accept the author identity via `permissioned_as = on_behalf_of` (not only `created_by = caller`) so files produced on-behalf of the author are covered. Reads outside the app's declared triggerables stay denied. Adds a regression test seeding a script-kind produced file that reproduces the "File restricted" denial before the fix and passes after. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): key S3 provenance on on-behalf identity + cover flow steps (review) Addresses the CI review on the S3 provenance gate: - P1 (confused deputy): the recent-production check keyed on `created_by = caller`, so a viewer who can run a declared script/flow directly (outside the app, with un-pinned inputs) could craft a result naming an author-only key and read it back through the app as the author. Key provenance instead on the producing job's `permissioned_as` matching the on-behalf identity the download reads as (the author in author-mode); a viewer's direct run has `permissioned_as = viewer` and no longer clears the gate. Drops `created_by` from both the appscript/preview and script/flow branches, closing the same latent hole in the pre-existing inline-script branch. - P2 (dead flow-step branch): `flowscript`/`flownode` jobs have `runnable_path = <flow_path>/<step_id>`, which exact `= ANY(...)` never matched. Split script vs flow triggerable paths; flow kinds now match the flow's own job (bare path) and its step jobs via a `<flow_path>/%` prefix, bounded to declared flows. - P2 (test realism): the regression test now uses the production component-prefixed triggerable key format (`<id>:script/...`), exercises a flow-step-produced key, and asserts a viewer's own direct run of a declared script stays denied (the P1 case). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): tie deployed-app S3 provenance to an app-origination marker (review) Second CI-review round flagged that `permissioned_as` still does not prove a job was app-launched: a runnable configured with its own `on_behalf_of` makes a direct `/jobs/run` resolve `permissioned_as` to that identity (the app author), so a viewer with run access could execute a declared runnable directly, craft an S3 result, and read it back through the app. The flow-path `LIKE fp || '/%'` match also let `_`/`%` in a declared path admit unrelated flows. Introduce a real app-origination marker instead of inferring provenance: - Add `JobTriggerKind::App`; `execute_component` stamps every app-launched job with `trigger_kind = 'app'` + `trigger = <app path>`. A direct `/jobs/run` cannot set this, so it is the authoritative signal that a file was produced *by the app*. - The provenance gate's recent-production check collapses to `trigger_kind = 'app' AND trigger = <this app path>` (+ the 3h window and result containment). This drops the forgeable `created_by`/`permissioned_as`/ `runnable_path`/kind logic entirely and removes the `LIKE` wildcard issue. - Provenance is scoped to THIS app's path, so another app's jobs (even same author) do not authorize this app's reads. Regression test rewritten to the marker model: an app-produced key clears for viewer and admin; a direct run whose `permissioned_as` resolves to the author stays denied (the forgery); another app's output stays denied. Adds `app` to the OpenAPI JobTriggerKind enum. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(apps): assert execute_component stamps trigger_kind='app' at runtime Adds an end-to-end test that runs a real script component through the app runtime (`apps_u/execute_component`) and asserts the enqueued job carries the app-origination marker `trigger_kind = 'app'` + `trigger = <app path>` (not the runnable path). The provenance-gate tests seed the marker directly; this proves the runtime actually produces the exact marker the gate depends on. execute_component commits the job row and returns its id, so the assertion reads the row directly — no worker needed to run the job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): reject trigger_kind=app for suspended-job reassignment (review) `JobTriggerKind::App` (added for the app-origination S3 marker) became a valid value for the resume/cancel suspended-trigger routes, whose handler derives the table name `<kind>_trigger`. There is no `app_trigger` table, so both endpoints would fail with a missing-relation database error (500). Reject `App` in `get_suspended_trigger` alongside webhook/schedule so it returns a clean 400. Adds a regression test asserting the reassignment route returns 400 (not 500) for trigger_kind=app. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): don't stamp app-origination marker on preview runs (review) The app-origination marker (trigger_kind='app') was stamped unconditionally, including preview mode. A preview lets a `jobs:run` caller supply arbitrary `raw_code` against ANY app path without that app's deployed policy (raw_code with no path/id skips all app authorization), so a preview returning `{"s3":"<author-only-key>"}` would forge the exact marker the S3 provenance gate trusts and read the victim app author's file. Gate the marker on `!is_preview`: only deployed, policy-checked executions are app-provenanced. Preview/editor S3 display does not rely on this marker (the editor routes reads through the force_viewer allowlist), so nothing legitimate regresses. Adds a regression test asserting a preview run's job is not stamped trigger_kind='app'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): editor-authorize preview marker + per-viewer S3 provenance isolation (review) Closes the codex P1 (preview forgery) without breaking editor preview downloads, and adds cross-viewer isolation to the provenance gate. - Preview marker now requires app write: `execute_component` stamps the app-origination marker on a preview only when the caller can EDIT that app (`require_is_writer`), instead of never stamping previews. An app editor already wields the app's author identity (they can deploy a component that reads the same file), so marking their own preview is no escalation and keeps preview-produced S3 results downloadable in the editor; a `jobs:run`-only caller who cannot edit the app still cannot forge the marker. Deployed runs are unchanged (always marked). - Per-viewer isolation: the provenance gate now also requires `j.created_by = <this caller>`. The security boundary stays the un-forgeable `trigger_kind='app'` marker; `created_by` is an additional filter ANDed under it, so it only narrows — a viewer can only download keys their OWN app runs produced, not another viewer's result. Restores the per-caller scoping #10048 had, now safe on top of the marker. Tests: preview marked iff caller can edit the app; cross-viewer isolation (another viewer's app-marked key denied, no admin bypass); direct-run and other-app keys still denied; deployed run still stamped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): require apps:write scope (not just writer ACL) to mark preview provenance (review) require_is_writer checks the user's underlying ACL but ignores token scopes, so a writer's token deliberately scoped to apps:run/apps:read/jobs:run but WITHOUT apps:write could still mark a preview and forge provenance — even though that token cannot deploy the app (update_app requires apps:write), breaking the "any marked caller can deploy equivalent code" rationale. Require BOTH apps:write:<path> scope (check_scopes) AND the writer ACL (require_is_writer) before stamping a preview's app-origination marker. Deployed runs unchanged. Adds a scope-restricted-writer token to the test (apps:run/read + jobs:run, no apps:write) and asserts its preview stays unmarked; retains the full-editor positive case and the non-editor negative case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): never app-provenance preview runs; read editor S3 as the caller (review) Simplifies the preview handling: a preview executes as the *caller* (Viewer mode), never as the author, so its results must be read back as the caller — never author-mode — and must never carry the app-origination marker. This removes the whole `require_is_writer` / `apps:write` / `can_preserve_on_behalf_of` reasoning (which was also unsound: a writer's token or session may not be able to deploy a component running as the app's on-behalf identity, so marking their preview could still escalate). - Backend: mark the app-origination marker for deployed runs only (`!is_preview`). - Frontend: `getS3File` (AppImage/AppPdf/AppDownload) now routes editor/preview reads through the viewer-scoped `job_helpers/download_s3_file` endpoint (reads as the caller), matching what DisplayResult/ParqetCsvTableRenderer already do; only a deployed app view uses the provenance-gated `apps_u` endpoint. This is the path that previously relied on marking previews, so nothing regresses. Test: a preview is never app-provenanced (owner's own preview and a non-editor's both stay unmarked). Cross-viewer isolation, deployed marking, and the reassignment guard are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): app components run on-behalf of the app, not the referenced runnable (review) Root-causes codex's on-behalf-preview finding: `execute_component` was overriding the app's resolved on-behalf identity with the referenced script/flow's OWN `on_behalf_of` (its `on_behalf_of_email`). That is wrong in the app context — the app's execution mode should govern: - A Viewer-mode app could execute a component AS the referenced runnable's on_behalf identity (privilege confusion / escalation), instead of as the viewer. - A preview would run as that identity rather than as the caller, so its S3 output could not be read back as the caller — the download-identity mismatch codex flagged. Always use the app-resolved identity (author in author-mode, caller in viewer/preview); a referenced runnable's own `on_behalf_of` no longer leaks into app execution. Direct `/jobs/run` still honors a runnable's `on_behalf_of` (unchanged). With this, previews always run as the caller, so reading editor/preview S3 as the caller (viewer-scoped `job_helpers`) is unconditionally correct. - Test: the deployed-component e2e now seeds the script with a distinct on_behalf and asserts the component job's `permissioned_as` is the app identity, not the script's. - Also reword the getS3File `configuration` param comment to describe current state only (AGENTS.md comment rule). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(apps): surface 'app' trigger kind in Runs UI; condense provenance comments (review) Addresses codex review nits: - Add `app` to `jobTriggerKinds`, `triggerIconMap` (LayoutDashboard), and `triggerDisplayNamesMap` so app-component jobs (which now carry `trigger_kind = 'app'`) are filterable in Runs and render their trigger info. - Condense the app-origination marker, on-behalf-identity, and provenance-gate comments to state each invariant once in <=4 lines at its relevant site (AGENTS.md comment rule). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
ff774c46bf |
feat: add per-workspace job-retention override (#10050)
* feat: add per-workspace job-retention override (EE) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 2ba6a2a75b6fc97858b306b2c98ada481e363c10 This commit updates the EE repository reference after PR #658 was merged in windmill-ee-private. Previous ee-repo-ref: e7fb36acd813cd717bcf05f5aafbf81de271d618 New ee-repo-ref: 2ba6a2a75b6fc97858b306b2c98ada481e363c10 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
04eb7ddd39 |
fix: clearer errors on auto-draft save failure (WIN-2157) (#10053)
* fix: clearer errors on auto-draft save failure (WIN-2157) When an autosave draft save fails, the cloud indicator now surfaces the backend reason on hover (native title tooltip) in addition to the existing click popover, so the cause is discoverable without a click. Backend now returns a clearer, actionable message: - `require_can_write_path` distinguishes a malformed path (unrecognized namespace prefix -> BadRequest) from a genuine permission denial, and the deny message spells out where the user *can* write. - `require_owner_of_path` no longer panics with an out-of-bounds index on a malformed single-segment path (e.g. a bare `u`/`f`); it returns a clear BadRequest instead. Covered by a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: trim narrative comment to invariant in drafts.rs (WIN-2157) Address CI review (AGENTS.md: comments record constraints, not narration, ≤4 lines): keep the malformed-path invariant, drop the motivation tail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: don't let a malformed stored draft 400 the draft listing (WIN-2157) Address CI review (P1): require_can_write_path can now return BadRequest for a malformed path, and list_drafts propagated it — so a single malformed stored draft row (the draft table has no path constraint; legacy/admin-authored rows may be malformed) would make GET /drafts/list return 400. Treat BadRequest like NotAuthorized there: the row is simply not writable. Verified e2e on EE — listing returns 200 with can_write false for the malformed rows. Also trim "unchanged"/"still" drafting-history narration from the regression test comments (P2, AGENTS.md). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: compress list_drafts comment to 4 lines (WIN-2157) Address CI review P2: keep the constraint (draft table has no path constraint) and the invariant (one malformed row must not 400 the listing) within the AGENTS.md ≤4-line limit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
1e192f2d86 |
feat(apps): authorize deployed-app S3 reads on-behalf of the author for logged-in viewers (#10048)
* feat(apps): authorize deployed-app S3 reads on-behalf of the author for logged-in viewers A logged-in user viewing a deployed app now reads S3 files (rich result, table/image/PDF preview, CSV export, download, metadata) the same way an anonymous viewer already does: on-behalf of the app author per the app policy's execution_mode, gated by an app-provenance check — instead of against the viewer's own S3 permissions. This aligns S3 with every other thing an app does (scripts, flows, resources all already run on-behalf of the author) and lets an operator who lacks folder S3 permission still see data rendered inside the app. The raw job_helpers/* S3 API stays viewer-scoped: a viewer who lacks folder permission is still denied there. Only which endpoint the app frontend uses for logged-in deployed viewers changes. Backend: - Add app-scoped, provenance-gated apps_u/* variants for all S3 display ops (download_s3_file already existed; add download_s3_parquet_file_as_csv, load_file_metadata, load_file_preview, load_parquet_preview, load_csv_preview, load_table_count). Each routes through one shared helper (app_s3_on_behalf_and_provenance) that scope-confines an app embed token, resolves the on-behalf identity, and runs the provenance gate ONCE before dispatching to the EE *_internal S3 helpers. - Close the confused-deputy hole in check_if_allowed_to_access_s3_file_from_app: the unconditional Ok() bypass for a logged-in, non-embed session now only applies in viewer execution mode (where the on-behalf identity IS the viewer, so the viewer's own permissions still bound the read downstream). Author-mode reads (anonymous/publisher) always enforce provenance, for anonymous and logged-in viewers alike, so a viewer cannot launder the author's S3 permissions with an arbitrary file_key. Frontend: - Route the deployed-app view through apps_u/* using the app-viewer isEditor signal instead of login state (the old $userStore proxy wrongly sent logged-in deployed viewers to the viewer-scoped job_helpers API). Editor and preview keep viewer identity via job_helpers. execution_mode: viewer remains the escape hatch for per-viewer S3 enforcement. Fixes provenance-gated S3 display for logged-in operators on deployed apps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(backend): document cargo features, restarting the dev backend, and filesystem object storage The dev backend runs `cargo watch --features quickjs` by default, which omits S3, EE, MCP, and non-JS runtimes — feature-gated routes then 404 or return a "requires <feature>" stub at runtime. Add a backend/CLAUDE.md section that: - explains that you must restart the backend with the appropriate features to exercise gated functionality, with the pid/cwd-scoped restart recipe (never pkill target/debug/windmill) and the PORT=$BACKEND_PORT gotcha; - documents what each commonly-toggled feature gate does (private, enterprise, license, parquet, duckdb, language runtimes, mcp, trigger kinds, no_auth) plus common combinations; - documents using the built-in FilesystemStorage large-file storage for dev workspace object storage (hidden from the UI dropdown; set via edit_large_file_storage_config), including the advanced_permissions shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): don't flatten inner query in app-scoped S3 preview routes axum's `Query` uses `serde_urlencoded`, which cannot deserialize the typed (numeric/bool) fields of a `#[serde(flatten)]`-ed struct and 400s on `limit` / `offset` ("invalid type: string, expected u32"). The app-scoped load_csv_preview / load_parquet_preview / load_table_count routes flattened LoadPreviewQuery / LoadCountQuery, so their previews were broken. Restate the fields directly on the outer query structs (with an into_inner() to rebuild the inner query) and extend the CE OSS stub to match. Also bumps ee-repo-ref.txt for the companion EE csv-separator panic fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address CI review — nested DisplayResult routing, byte-range contract, docs, tests - [P1] Thread `appPath` into the nested `DisplayResult`s (render_all children and the expanded-result drawer) so logged-in deployed viewers route nested/expanded S3 tables, images, PDFs, and downloads through `apps_u/*` too, not job_helpers. - [P2] Mark `read_bytes_from`/`read_bytes_length` required on the `apps_u/load_file_preview` route (they are non-optional in LoadFilePreviewQuery), and mirror the full query shape in the CE OSS stub so the byte-range contract is enforced identically on CE and EE. - [P2] Fix the backend retrigger command in backend/CLAUDE.md: cargo watch runs from `backend/`, so `touch README.md` (not `backend/README.md`). - [P2] Trim app_s3_onbehalf.rs comments per AGENTS.md (state the invariant once, no drafting-history narration). - Extend the integration test to cover the table-count, csv-preview (numeric limit/offset deserialization), and file-preview (byte-range required) routes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(apps): tighten S3 provenance-gate comments per AGENTS.md Consolidate the viewer-mode / author-mode rationale to ≤4 lines at each branch of the gate, and drop the repeated explanation from the shared app_s3_on_behalf_and_provenance doc comment (which now just states what the helper does). No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to f292a1040da6a667ce7c22abf63ec0debfdd480f This commit updates the EE repository reference after PR #657 was merged in windmill-ee-private. Previous ee-repo-ref: a582389084eb363997cb5e8053f29220e0d3eaec New ee-repo-ref: f292a1040da6a667ce7c22abf63ec0debfdd480f Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
5844c32ac5 |
fix: enforce read authorization when signing S3 objects (#10049)
`sign_s3_objects` minted a long-lived HMAC bearer signature for any S3 key handed to it, by any authenticated workspace member, with no check that the caller was allowed to read that key. Since `validate_s3_signature` only verifies the HMAC and expiry at fetch time, any member (operators included) could mint a transferable capability to read arbitrary S3 keys, bypassing the advanced S3 permission rules (`check_lfs_object_path_permissions`). Authorize the read at mint time: add an `ApiAuthed` extractor and, before signing each key, require the caller's own `S3Permission::READ` via `get_workspace_s3_resource_and_check_paths`. A caller can no longer sign a key they cannot themselves read. The fetch-side validators are left unchanged. The only legitimate caller is the wmill SDK invoked from an app-author job, whose token authenticates as the executing (author) identity — which can read the key — so authorized app display is unaffected. Adds an integration test proving an authorized caller can sign a readable key (and the signature validates end-to-end through the presigned fetch route) while an unauthorized caller is refused. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
8343203ec2 |
feat(mcp): add multi-workspace MCP tokens via the gateway endpoint (#10043)
* feat(mcp): add multi-workspace MCP tokens via the gateway endpoint A single MCP token with no bound workspace (workspace_id NULL + mcp scope) now works across every workspace the token owner can access, served through the existing /api/mcp/gateway endpoint. This avoids having to register one MCP server entry per workspace in clients like Claude/Cursor. In multi-workspace mode the runner exposes a synthetic `list_workspaces` tool plus the generic API endpoint tools, each workspace-scoped one gaining a required `workspace_id` argument (mirroring the proxy pattern users built externally). Per-workspace scripts/flows are not enumerated to avoid flooding the tool list — they are run via runScriptByPath/runFlowByPath with an explicit workspace_id. Auth is resolved per tool call: the gateway middleware detects a workspace-less mcp token and marks the request MultiWorkspaceMcp, and the runner resolves a per-workspace ApiAuthed from the raw token via the AuthCache (validating membership; superadmins may act in any workspace). Single-workspace tokens are unchanged. Frontend: the MCP token creation flow gains an "All workspaces" option that produces a workspace-less token and the gateway URL. Fixes WIN-2153 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(mcp): cover multi-workspace endpoint tool transformation Unit tests for endpoint_tool_to_mcp_tool_multi and list_workspaces_tool: workspace-scoped tools gain a required workspace_id arg, global tools are left unchanged, workspace_id is not duplicated, and list_workspaces takes no arguments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): forward script/flow args for runScriptByPath/runFlowByPath These endpoints have an additionalProperties body (no declared properties), so build_request_body previously returned an empty body and dropped every script/flow argument. This was latent for the per-path run endpoints and became load-bearing in multi-workspace mode, where scripts/flows can only be run via runScriptByPath/runFlowByPath — parameterized runs silently lost their arguments. build_request_body now forwards all arguments not consumed by a path/query parameter for pass-through (additionalProperties) bodies, keeping the strict declared-only behavior for endpoints with explicit properties. The runner strips the synthetic workspace_id argument before dispatch so it can't leak into the forwarded body. Reported by Codex review on #10043. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(mcp): note workspace_id requirement in multi-workspace tool descriptions Workspace-scoped tools already gain a required workspace_id parameter (with its own schema description) in multi-workspace mode, but the tool's prose description was unchanged. Append a note so models/clients that read the description text know to pass workspace_id (and to call list_workspaces first). Global tool descriptions are left untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(mcp): trim multi-workspace tool/arg descriptions The workspace_id note repeats across every workspace-scoped tool in each tools/list, so keep it terse: description suffix "Requires `workspace_id`." and arg description "Target workspace id (from list_workspaces)." to avoid spending tokens on repeated boilerplate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): enforce script/flow path scopes for multi-workspace run-by-path In multi-workspace mode runScriptByPath/runFlowByPath are the only way to run scripts/flows, but they were authorized against the endpoint scope only — never the caller's mcp:scripts:/mcp:flows: path scopes. A granular token could run items outside its allowed paths (e.g. mcp:scripts:f/team/* + mcp:endpoints:* running f/other/secret), and a mcp:endpoints:* token could run arbitrary scripts. Now these two endpoints are authorized by the script/flow scope of the requested path (matching single-workspace mode's per-item tools): exposed in list_tools only when the token grants some script/flow (McpScopeConfig::has_any), and at call time the path is checked via is_allowed("script"/"flow", path). Verified e2e: mcp:scripts:f/team/* runs f/team/* but is denied f/other/*; mcp:endpoints:* alone no longer exposes or runs run-by-path. Reported by Codex + Pi review on #10043. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): deny run-by-path for mcp:favorites multi-workspace tokens mcp:favorites sets granular=false, so the previous run-by-path scope check (gated on `granular`) was skipped entirely — a default "Favorites only" all-workspaces token could run any script/flow by naming its path, bypassing the favorites restriction. Favorites are an enumerated set reachable only through per-item tools, not by arbitrary path, so they grant nothing for run-by-path. has_any() now returns true only for mcp:all (not favorites), and the call-time check drops the `granular` gate and relies on is_allowed() directly (already false for favorites, true for mcp:all, pattern-matched for granular). Verified e2e: mcp:favorites no longer exposes or runs run-by-path; mcp:all still runs; granular script scopes still path-enforced. Reported by Codex review on #10043. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
9feda57c15 |
perf: index v2_job(parent_job) to speed up run child-job listing (#10034)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
f28ea9cb99 |
feat(db-health): add connection sizing guidance (#10014)
* feat(db-health): add connection sizing guidance The Database Connections panel showed current/max connections but gave no guidance on how to size max_connections for the deployment. Derive an estimate from the live worker fleet: each worker instance shares a pool sized DEFAULT_MAX_CONNECTIONS_WORKER + (workers - 1), and each server opens up to DEFAULT_MAX_CONNECTIONS_SERVER (both overridable via DATABASE_CONNECTIONS). The endpoint now returns live worker/instance counts, the default per-server and per-worker pool sizes, the estimated peak worker connections, the reserved superuser connections, and a recommended max_connections floor (workers + one server + 25% headroom). Servers do not ping worker_ping, so the recommendation assumes one server and exposes the per-server increment. The panel renders this as a sizing breakdown and warns when max_connections is below the recommended floor. Fixes WIN-2147 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(db-health): single source for pool-size constants + sizing tests Address review: db_connect.rs kept its own copies of DEFAULT_MAX_CONNECTIONS_* that duplicate the windmill_common constants the sizing guidance reads, so tuning the runtime pool size would silently leave the guidance stale. Re-export the windmill_common constants from db_connect.rs so there is one source of truth. Add unit tests for compute_connection_sizing covering the zero-fleet, single worker, multi-instance, and reserved-clamp cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(db-health): 20% headroom and 200-connection minimum floor Lower the sizing headroom from 25% to 20% and never recommend below 200 connections (postgres defaults to 100; cheap headroom for growth/bursts/psql). Update the guidance message and unit tests accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-health): honor DATABASE_CONNECTIONS in sizing recommendation Address Codex P1: the runtime caps every process's pool at DATABASE_CONNECTIONS when set (db_connect.rs), but the sizing guidance always used the default 50/5 pools. For a tuned deployment this under-estimated worker demand and could hide a genuine under-provisioning (e.g. DATABASE_CONNECTIONS=100 with 5 instances is 500 worker connections, not 25). compute_connection_sizing now takes the effective DATABASE_CONNECTIONS override (read the same way db_connect.rs reads it): when set, each worker instance and server pool is that value and the worker estimate is override * instances. The response exposes server_pool_size / worker_pool_size (effective) and database_connections_override; the panel renders both pool rows and labels them (default) vs (DATABASE_CONNECTIONS), and the message states which source is used. Adds a unit test for the override path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-health): exclude agent workers from connection sizing Agent workers reach the API over HTTP (MODE=agent, Connection::Http) and hold no postgres pool, but their pings still land in worker_ping (written server-side by /api/agent_workers/update_ping). Counting them inflated the connection estimate. Filter the fleet query by the worker-name prefixes: DB-connected workers use "wk-" (WORKER_NAME_PREFIX), agent workers use "ag-" (AGENT_WORKER_NAME_PREFIX). Only wk- workers/instances feed the estimate; ag- workers are counted separately and surfaced as context ("N agent workers excluded — they use HTTP, not postgres connections"). Adds a unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Diego Imbert
|
e47aedac0a |
feat: add SQL migrations for data tables (#9693)
* feat: add datatable_migrations table * feat: add route to run datatable migrations * feat: sync datatable migrations as .up.sql/.down.sql files * feat: add datatable migrate up/down commands and post-push run prompt * feat: add datatable migrate new command to scaffold migrations * feat: add datatable migrations management UI * feat: prompt to create migration on DDL in datatable SQL editors * feat: support running a single specific datatable migration * feat: view migration content, run single migration, fix stacked modal * feat: per-row revert button with out-of-order warning * fix: avoid migrations list flicker on refresh after an action * feat: generate initial datatable migration via pg_dump * fix: surface datatable migration API error details in toasts * fix: revert created migration if create-and-run fails to run * fix: include postgres error detail in migration run/rollback failures * feat: sync datatable migrations as files via the workspace export * refactor: move datatable migrations to migrations/datatable/ path * fix: drop redundant datatable_migration label in sync output * fix: exclude datatable migration sql files from script metadata generation * feat: run datatable migrations as user-permissioned labeled jobs * feat: reject invalid datatable migrations on sync push * feat: datatable migrate up/down default to all datatables, --datatable to target one * fix: surface postgres error detail when datatable migrations fail to run * chore: regenerate CLI docs for datatable migrate commands * feat: default new datatable migration to a BEGIN/END transaction template * fix: validate datatable migration name and datatable at the API boundary * fix: ensure detected DDL ends with semicolon when wrapped in transaction * fix: re-prompt instead of stripping DDL when new-migration modal is cancelled * feat: refresh datatable schema after running a migration from the SQL REPL * feat: record db manager DDL on data tables as migrations * feat: make datatable migrations opt-in per data table * fix: make migration view editor read-only so its code can scroll * fix: don't re-prompt DDL guard when creating a migration without running * feat: generate down migrations for db manager DDL (postgres) * fix: correct down migration for db manager alters (no double-wrap, serial) * feat: explain migrations purpose with a tooltip in the migrations modal * compare paeg * feat: add datatable_migration kind to workspace diff pipeline * chore: point ee-repo-ref at datatable_migration git-sync companion * fix: harden datatable migration version allocation and initial-migration bookkeeping, add tests * feat: deploy and run datatable migrations on workspace merge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Refactor + handle datatable setting delete/rename * refactor: move datatable migration rename/delete cascade into module Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(windmill-utils-internal): bump to 1.7.1 for datatable migration deploy provider methods Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(db-manager): add Migrations button to top bar, make Refresh icon-only Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * BEGIN/END placeholder in down migration * feat: autofocus migration name input and flag it red when empty Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(datatable-migrations): allow non-admins to create/run/revert migrations, gate only opt in/out Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * border nits * refresh db manager schema on migrations * BEGIN/END scaffold in CLI * feat(cli): push local datatable migrations before running on migrate up Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: flag invalid migration name with red border, not just empty Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: drop random slug from auto-generated migration names Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: offer revert-and-delete when deleting an installed migration Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: record fork merge as a migration when target datatable opts in * nit * clone migrations on fork * windmill-utils-internal * fix(datatable-migrations): serialize run/rollback with a per-db advisory lock Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-manager): fail closed when migrations-status check errors on DDL apply Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix generate_initial migration ordering comment to match code * chore(datatable-migrations): remove unused update_datatable_migrations endpoint Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: run DDL migration guard on the script editor Test button Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * split * ee-repo-ref * chore(frontend): sync package-lock with package.json (@emnapi deps) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(datatable-migrations): never resolve instance credentials into migration job args datatable_database_arg eagerly resolved instance data-table credentials (including the shared instance-wide Postgres password) and passed them as the migration job's plaintext `database` arg, landing in v2_job.args. Since the run route has no admin gate, a non-admin could run a migration and read args.database to recover the password, granting cross-workspace psql access to all instance data-table DBs. Pass a `datatable://<name>` reference for both resource-backed and instance data tables instead; the pg executor already resolves it to real credentials server-side at run time, so nothing sensitive is ever stored in the job args. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * fix: handle dollar-quoting and comments when splitting SQL statements * feat: deploy datatable migrations on merge with explicit opt-in error * fix(frontend): sync package-lock with npm 11 peer-dep resolution npm ci failed with 'Missing: @emnapi/core@1.11.2 / @emnapi/runtime@1.11.2 from lock file'. @napi-rs/wasm-runtime declares @emnapi/core|runtime ^1.7.1 as peerDependencies while @rolldown/binding-wasm32-wasi pins them to exactly 1.10.0. Newer npm (bundled with node 24 in CI) installs the peer deps at the highest match (1.11.2) alongside rolldown's nested 1.10.0, so the ideal tree needs both versions; the committed lock only had 1.10.0. Regenerate the lock with npm 11.18 so it carries both 1.11.2 (top-level, for the peer deps) and 1.10.0 (nested, for rolldown's pin). Verified npm ci passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit npm publish * fix: fail closed on migrations-status error in fork schema merge * nit CI emnapi/core version * prevent initial_datatable_migration if migrations already exist * fix(datatable-migrations): validate persisted data table names as path segments edit_datatable_config only validated rename segments, not the actual settings.datatables keys, so a data table could be saved directly under a name like '..' or one containing '/'. Since new tables default to migrations_enabled = true, generate_initial_datatable_migration would then insert a migration row and the sync export would build migrations/datatable/<name>/... paths from that name, producing malformed or directory-escaping export paths. Validate every persisted data table name in edit_datatable_config (alongside the existing rename checks) and add validate_datatable_path_segment to generate_initial_datatable_migration for defense in depth. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scope datatable _wm_migrations by data table and cascade renames/deletes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(system_prompts): resolve nested local command groups in CLI docs generator The CLI docs generator anchored on the first `new Command()` in a file and never resolved locally-defined command groups passed as `.command("name", localCmd)`. For datatable this flattened the nested `migrate` group: it emitted `datatable new/up/down` plus a bare `datatable migrate`, and mislabeled the datatable command with the migrate group's description. jobs was broken the same way (its description was pull's, and pull/push rendered empty). Anchor block extraction on the `export default`ed command, recurse into locally-defined `const x = new Command()` groups mounted as subcommands, and render nested sub-subcommands. Regenerated docs now show `datatable migrate new/up/down` and `jobs pull/push` with their real options. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: drop unreleased _wm_migrations legacy-upgrade handling Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: return datatable migration SQL from getItemValue for the diff drawer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(frontend): use windmill-utils-internal 1.8.2 for migration diff drawer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * nit * fix: handle datatable migration renames on push and dedupe timestamps * fix: reject rewriting an already-applied datatable migration on upsert Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): add missing @emnapi/core and @emnapi/runtime lockfile entries Resolves npm ci EUSAGE failure: the optional cpu:wasm32 @rolldown/binding-wasm32-wasi declares deps on @emnapi/core@1.11.2 and @emnapi/runtime@1.11.2 that had no resolved lockfile entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): datatable migrate up/down default to main datatable, not all Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: fail closed when applied status unreadable on datatable migration rewrite Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: surface full error detail in Database Manager DDL/query errors * "See migration" button in the toast * feat: add Enter shortcut to Create-a-migration in the DDL guard * fix(frontend): warn before running a newly-created datatable migration out of order The row-level Run action warns when earlier migrations are still pending, but the create-and-run paths ran a just-created migration with `only` directly, applying it ahead of older pending migrations without that confirmation. Reuse the same "Run migration out of order" confirmation across all create-and-run paths via a shared helper (datatableMigrationUtils): - NewDataTableMigrationModal "Create and run" (and the DDL guard path) - DatatableSchemaDiff fork→parent merge - dbOps schema ops (DB manager create/alter/drop) — the pure factory throws a MigrationRunCancelled sentinel on decline, which DBTableEditor treats as a silent cancel Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep renamed datatable migrations visible in compare view * fix: record per-migration deployment on datatable migrations disable * fix(cli): run deployed datatable migrations after workspace merge The merge command upserted datatable_migration definitions into the target workspace and reported the item as successfully deployed, but never ran the migrations. For forked datatables backed by separate databases, this left the target schema unchanged until someone manually ran `wmill datatable migrate up`, while the CLI reported a successful merge. Collect the datatable migrations deployed (not deleted) into the target and, after the deploy loop, offer to run them via the existing offerToRunNewMigrations helper — the same post-deploy run prompt the push/sync path uses (interactive only; `--yes`/non-TTY skip the mutating run, matching push behavior). Export parseDatatableMigrationDeployPath so the merge path can parse the deployed items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): serialize datatable migration edits/deletes with the run lock A migration run snapshots a migration's code_up from datatable_migrations and only records its version in the data table's _wm_migrations after the job succeeds. upsert_datatable_migration checked _wm_migrations before allowing an edit but took no lock, so a concurrent edit could read "not applied yet", rewrite code_up/code_down, and then the in-flight run would record the version for the old SQL — leaving _wm_migrations pointing at SQL that was never applied (migrate up then skips it; rollback runs a down that doesn't match). Serialize definition rewrites and deletes with the same per-database advisory lock the run/rollback paths use: - Factor the connect+advisory-lock into lock_datatable_migration_runs and the applied-versions read into read_applied_versions_on_client. - run_datatable_migrations now snapshots the definitions AFTER taking the lock, so code_up can't change between snapshot and version-record. - upsert (when changing an existing def) and delete take the lock across the applied-check and the write; delete now rejects deleting an already-applied migration (would orphan its _wm_migrations record), symmetric with upsert. Both fail closed if the data table database is unreachable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): stack the out-of-order migration confirm above the DB editor preview Creating a table on a migrations-enabled data table opened the DB table editor's "Confirm running the following" preview modal, whose confirm triggers applyDdl, which then asks for out-of-order confirmation. Both are ConfirmationModals with a hardcoded z-[9999]; the out-of-order one lives in DBManagerContent (mounted before the editor), so it rendered behind the still-open preview modal. Add an optional zIndexClass prop to ConfirmationModal (default z-[9999], backward-compatible) and give the DB-manager out-of-order confirm z-[10000] so it stacks on top. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 27672e37df5d9dfde94f19963d5ffcdf8dd5448c This commit updates the EE repository reference after PR #623 was merged in windmill-ee-private. Previous ee-repo-ref: 6c287041cd7edd4a77a4bc07ad0e156cec32cce4 New ee-repo-ref: 27672e37df5d9dfde94f19963d5ffcdf8dd5448c Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
97d14d979f |
bun bootstrap housekeeping on the migrator's held connection (#9970)
migrate() and fix_flow_versioning_migration re-acquired a second connection from the pool while already holding one (the migrator's checked-out, advisory-locked connection). That deadlocks any backend limited to one connection at a time — connection-constrained managed Postgres, PgBouncer transaction pooling, or an embedded single-connection dev database. Route those housekeeping queries onto the already-held connection via a new CustomMigrator::connection() accessor. Fewer connections during migration and, for fix_flow_versioning, the existence check and write now run on the same advisory-locked connection. Default multi-connection behavior is unchanged. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
33521505db |
feat(ducklake): scheduled lake maintenance (expiry, compaction, orphan cleanup) (#9916)
* feat(ducklake): scheduled lake maintenance (snapshot expiry, compaction, orphan cleanup) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ducklake): review fixes — starts_with not LIKE, CE license-lapse escape Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(ducklake): auth-contract docs + _unchecked rename per codex review Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(ducklake): move maintenance payload construction into EE module Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ducklake): fall through to script resolution for non-managed reserved-prefix schedules Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(ducklake): document accepted pre-existing-schedule limitation on the reserved prefix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ducklake): CE save-off clears the managed schedule row and queued occurrence Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 2fab310d4f50ed7c34857d69c9b854f4491bf217 This commit updates the EE repository reference after PR #645 was merged in windmill-ee-private. Previous ee-repo-ref: fff1fd830a36beba732486f05941ec243cf6b640 New ee-repo-ref: 2fab310d4f50ed7c34857d69c9b854f4491bf217 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasa
|
d600c7ecfe |
fix(ai): route Azure Foundry Claude models via Anthropic Messages API (#9908)
* fix(ai): route Azure Foundry Claude models via Anthropic Messages API Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai): keep explicit Azure OpenAI deployment base URLs intact build_azure_openai_url only appends /openai/v1 for a bare resource root; any base with an explicit path (e.g. .../openai/deployments/<id>) is preserved. Adds a regression test and a unit test for usesAnthropicMessagesApi. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai): enable Claude extended thinking on Azure Foundry Route azure_foundry+Claude through the Anthropic reasoning branch (adaptive thinking + output_config.effort) instead of the gpt/o gate, and recognize claude-sonnet-5. Live-verified: sonnet-5 and opus-4-8 on Foundry accept the low/medium/high/xhigh/max ladder and render summarized thinking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
3ec1f164be |
fix: strip NUL characters from app values at save time (#9903)
App values are persisted to a json column, which permits the JSON NUL escape (backslash-u-0000), but are later converted to jsonb (e.g. a workspace fork clone_apps, search indexing), which rejects it with "unsupported Unicode escape sequence" -- silently making the app un-forkable. The usual source is a binary file such as .DS_Store accidentally bundled into a raw app file map. A real NUL is unstorable in jsonb either way, and frontend code that needs the character writes it as the source escape (which JSON-encodes to an escaped backslash + literal u0000 and is left untouched), so rather than hard-failing the save we strip genuine NULs and warn. Add strip_null_chars and apply it at both app_version insert sites (create_app_internal and update_app_internal, covering the regular and raw create/update routes). It removes a genuine NUL escape (odd run of backslashes before u0000) while preserving an even run. Returns a borrowed Cow (no allocation) when the value is already clean. Covered by unit tests. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
af01e90b5c |
feat(s3): replace CE 50MB upload cap with 10GiB workspace storage quota (#9874)
* fix(s3_proxy): enforce CE 50MB upload cap on multipart uploads Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(s3): replace CE 50MB upload cap with 10GiB workspace storage quota Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): gate CE quota OSS stubs to not(enterprise) to match callers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): delta-aware CE storage quota + guard usage-load retry loop Account for the overwritten object's size in the quota check so valid same-size overwrites near quota are not rejected (Codex review), and stop the storage-usage $effect from re-firing on persistent API errors (Pi review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): count chunked PUTs; revert overreaching volume quota copy Volumes write to workspace storage via a separate worker-side path with its own 50MB-per-file cap that this PR does not change, so revert the drawer copy that claimed they count toward the 10GiB quota (Codex review). Bump ee-repo-ref for the chunked-PUT accounting fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): add SQLx cache for CE usage-bump/quota queries; exclude volumes Regenerate the missing offline SQLx cache for the not(enterprise) bump and remaining-quota queries so the private CE offline build compiles, and bump ee-repo-ref for the volumes/-prefix exclusion from the counted quota (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): always HEAD for CE upload delta so overwrites don't inflate usage Bump ee-repo-ref for the fast-path overwrite-accounting fix (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): reserve volumes/ prefix on CE write surfaces to close quota bypass Reject direct writes to the reserved volume prefix on the app-upload surface and add the OSS stub; bump ee-repo-ref (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): refuse new multipart work when CE workspace is at quota Bump ee-repo-ref for the multipart-initiate/part quota gate (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(s3): reserve in-flight multipart bytes against CE storage quota Add workspace_multipart_inflight table + grants, SQLx cache for the reservation queries, and bump ee-repo-ref. Bounds abandoned multipart uploads that the list-based recount can't see (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): clear multipart reservation only after a successful complete Add exclude-upload arg to the OSS quota stub/caller and the SQLx cache for the updated remaining-quota query; bump ee-repo-ref (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): per-part multipart reservation; commit only on part success Per-part workspace_multipart_inflight schema (upload_id, part_id) so retries replace rather than double-count; SQLx cache for the reworked queries; bump ee-repo-ref (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(s3): HEAD the multipart overwrite target once per upload, not per part SQLx cache for the stored-credit lookup; bump ee-repo-ref. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to bea5a8b5120d6d69cab1ad4611ebe463559bd200 This commit updates the EE repository reference after PR #640 was merged in windmill-ee-private. Previous ee-repo-ref: 6e6ff86f1939cf74736b7d435bf6851416437523 New ee-repo-ref: bea5a8b5120d6d69cab1ad4611ebe463559bd200 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
1a9debb689 |
fix(jobs): give flow dynselect a path and its worker tag, like scripts (#9867)
Fetching options for a `dynselect`/`dynmultiselect` input was inconsistent between deployed scripts and deployed flows: - scripts ran through `push_script_job_by_path_into_queue` — a `script` job with the script's path, tag, lock and codebase resolution; - flows ran their schema dyn-select code as an anonymous `preview` with no path and no tag (always the language default), and reported access failures as a raw `SqlErr: no rows`. Deployed scripts are left exactly as they were (that path already handles tag/lock/codebase/on-behalf-of correctly). The flow branch now: - carries the flow path on the preview job, - reads the flow's `tag` under RLS and routes the job to it (falling back to the language default when unset), matching the script's worker group, and - runs `check_tag_available_for_workspace` on that tag — the same gate a normal flow run and the script path apply — so a caller who can read the flow but is not allowed to use its (custom/scoped) worker tag is rejected consistently. The flow's tag read runs on every request, so it also serves as the per-request access check, replacing the raw error with a clean `NotAuthorized` / `NotFound`. Entrypoint-name validation now covers all branches (it is interpolated into the generated wrapper). Inline is unchanged: a `preview` with no path on the language default, blocked for operators. Fixes WIN-2118 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
76a9523009 |
feat: use derived username instead of email for non-member superadmins (#9857)
* feat: use derived username instead of email for non-member superadmins Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address review - drop redundant username cache, guard whoami membership by email Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: use explicit non_member boolean instead of role string for superadmin banner Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve email from password table for non-member superadmin permissioned_as Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve non-member superadmin drafts via shared username->email resolver Adds resolve_username_to_email (usr, then super_admin password fallback for both derived-username and email modes) and uses it in get_email_from_permissioned_as and the drafts get/list endpoints, so a non-member superadmin's drafts resolve and no email leaks into the drafts payload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: superadmin-not-in-workspace schedule uses derived username as permissioned_as Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve non-member superadmin identity in draft owner-circles, username_to_email, and home filter Applies the password-fallback username resolution to the script/flow/app/draft owner-circle subqueries and the username_to_email endpoint (was an admins-workspace 'username == email' hack), and switches the home items-list user-folder filter to the non_member flag instead of the now-broken username-contains-@ heuristic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: backfill non-member superadmin favorites from email to derived username Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: propagate DB errors in username resolution instead of leaking email (CI review) Addresses cubic-dev-ai P2: get_instance_username_or_fallback_to_email now returns Result and only falls back to the email for a genuine 'no derived username'; a query error propagates so callers fail closed rather than leaking the raw email as the acting username. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: clarify non-member superadmin popover (username used + admin permissions) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep username_to_email endpoint member-only to not disclose non-member superadmin email (CI review) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: forbid disabling automate_username_creation once usernames assigned (CI review) Makes the setting effectively one-way once instance-wide usernames exist, so the global-uniqueness invariant that keeps stored u/<username> identities (schedules/triggers/drafts/superadmin ownership) unambiguous can never be dropped back to workspace-local uniqueness. Re-saving false on an already-disabled instance stays a no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
1c46f899ca |
fix(mcp): stop double-escaping string query params in build_query_string (#9855)
MCP tool arguments were converted to URL query values via `value.to_string()`
+ `trim_matches('"')`. For string values containing JSON (e.g. the `args`/`result`
filters on job listing, `args` on schedule listing), `to_string()` JSON-encodes the
string and escapes inner quotes with backslashes; stripping the outer quotes leaves
`{\"k\":\"v\"}`, which the backend's `serde_json::from_str` then fails to parse,
falling back to `FALSE` and returning zero results.
Use `value.as_str()` to emit the raw string content for `Value::String`, falling
back to `value.to_string()` for non-string types (numbers, booleans). Adds
regression tests covering JSON-string, non-string, and plain-string params.
Fixes WIN-2114
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Ruben Fiszel
|
75ba81b2d2 |
fix(audit): don't read pg_authid from an elevated context in S3 export migration (#9832)
* fix(audit): don't read pg_authid from an elevated context in S3 export migration Migration 20260626132251 aborted instance startup on managed Postgres (e.g. Cloud SQL) with "Modifying pg_authid or pg_auth_members is not allowed in elevated context": the audit S3 export "oldest in-flight xact_start" floor probe calls pg_has_role(...), which reads pg_authid, and managed providers forbid that read from an elevated context. The migration ran the probe inline in its UPDATE, so the whole migration — and the instance boot — failed. Extract the probe into a shared SQL function audit_logs_s3_oldest_inflight_ts() that returns the oldest in-flight xact_start (when cluster-wide stats are visible) or NULL otherwise. The pg_has_role read is wrapped in a plpgsql BEGIN/EXCEPTION subtransaction, so a pg_authid failure returns NULL (callers fall back to a conservative 7-day window / reject) instead of aborting. is_superuser (a GUC, no catalog read) is checked first to short-circuit. The migration's trigger and UPDATE, the OSS backfill try_start, and the EE exporter/startup anchor (companion windmill-ee-private PR) all route through it. Because 20260626132251 already shipped, it is added to the potentially_stale list in windmill-api/src/db.rs: on startup the stale _sqlx_migrations row (checksum mismatch) is deleted and the fixed, idempotent migration re-applies, so already-migrated instances upgrade without a checksum-mismatch boot failure. Fixes WIN-2108 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 95352c13c4c82247d8cfd80936f9203aeb079802 This commit updates the EE repository reference after PR #635 was merged in windmill-ee-private. Previous ee-repo-ref: 136f49a52af922868acac33abf8198913a9e835c New ee-repo-ref: 95352c13c4c82247d8cfd80936f9203aeb079802 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
0dbd9c1231 |
perf: eliminate dual-connection DB pool contention across worker, queue, and api (#9798)
* perf: eliminate dual-connection DB pool contention across worker, queue, and api Reuse the held transaction (or move pool reads before begin()) instead of checking out a second pool connection while a tx is open, extending the fix from #9789/#7861. Targets the per-worker pool (max 5) hot paths plus several server-pool API handlers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: pass owned pool to get_email_from_permissioned_as in http trigger handler The generified signature takes impl PgExecutor; the http trigger handler passed &db where db is already &DB, yielding &&Pool which does not impl PgExecutor (only surfaced under the full feature set in CI). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep RLS-exposed reads on the non-RLS pool and isolate flow-eval reads in a savepoint Addresses review of the dual-connection sweep: - worker_flow: wrap the stop_after_all_iters_if reads in a SAVEPOINT. The caller swallows the error and keeps using tx, so a DB read failure must not leave the outer transaction aborted (it would fail the later commit). Matches the previous pool-read semantics. - Revert reads that were moved onto an RLS (user_db) transaction back to the non-RLS pool, since RLS row-visibility/role context can change results: push_scheduled_job (email/tag/settings lookups; reachable with a user_db tx from api-schedule/api-flows), push_inner native-retry dedicated_worker routing (RLS isolation variants), resources.rs app-namespace folder auto-create (non-admins must not be blocked), and the script archive/delete UPDATEs. Non-RLS db.begin() reuse and move-before-begin are kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: failpoint proving the stop_after_all_iters_if savepoint isolates an aborted read Adds a worker-crate failpoints feature and a data-driven hook: when the stop_after_all_iters_if expr is the magic sentinel, the in-evaluation read runs SELECT 1/0 to abort its (savepoint) transaction. The test asserts the flow still completes (iteration marked failed) — which only holds if the savepoint keeps the outer status-update transaction committable. Without the savepoint the abort would poison the outer tx and the job would never complete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
ba768fee88 |
feat(api): add structured endpoint for flow logs (#9797)
Add `GET /w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}` as a
JSON alternative to `get_flow_all_logs`. It returns the same flow log
tree as an array of per-job entries (job_id, label, kind, step path,
depth, parent module type, sibling index/count, and resolved logs)
instead of a single delimited text blob, so callers can render or
process logs per-step without parsing the `=== ... ===` markers.
The shared auth, recursive-CTE query, and label-building logic is
extracted into `collect_flow_log_entries`; the existing text endpoint
now formats those entries and produces byte-identical output.
Fixes WIN-2102
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
 centdix
|
9d61e4e59e |
feat: self-host docs search for chat, mcp, cli; drop inkeep (#9772)
* feat: self-host docs search for chat, mcp and cli; remove inkeep
Embed a vendored docs snapshot (llms.txt/llms-full.txt) in the backend and
serve ranking + page rendering from GET /api/docs/{search,page}. The AI chat,
the MCP searchDocs/readDocsPage tools, and 'wmill docs' all consume it, so docs
search works with no runtime egress and is no longer EE-gated. Removes the
inkeep proxy. EE companion deletes inkeep_ee.rs (ee-repo-ref bumped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: name read_docs_page param `url` instead of `path`
search_docs returns each hit's `Source` URL, so the read tool now takes a
`url` argument to match — the AI/MCP loop reads "search gives a Source URL,
read takes that url" rather than copying a `Source:` URL into a `path` slot.
A bare `/docs/...` path is still accepted and canonicalized before lookup.
Regenerated openapi-deref, the MCP endpoint tools, and the frontend client.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: add scheduled workflow to refresh the vendored docs snapshot
The backend embeds docs_snapshot/*.gz at build time, so the in-product docs
corpus is otherwise only as fresh as the last manual fetch.sh run. This adds a
weekly (and manually dispatchable) job that re-runs fetch.sh, sanity-checks the
result against truncation/garbage, and opens a PR via the internal app when the
snapshot changed — so a human reviews the docs diff before it rides into the
next release build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: make docs tool-result strings caller-neutral
The search/page endpoints back three differently-named consumers (the AI chat
`read_docs_page` tool, the MCP `readDocsPage` tool, and the `wmill docs` CLI),
so the shared rendered text shouldn't name one of them. Refer to "the docs
page-reading tool" and its `url` argument instead, and add tests pinning the
caller-neutral follow-up guidance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: point ee-repo-ref at inkeep-removal companion rebased on EE main
The companion branch now carries only the inkeep_ee.rs deletion on top of EE
main (was based on the native-job-retry EE line, which polluted the EE PR diff).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(docs): expose docs:read in token catalog; precompute lowercased corpus
Addresses two review nits on the self-hosted docs PR:
- docs:read was enforced (ScopeDomain::Docs) but missing from the token scope
catalog (token.rs ALL_SCOPES), so it couldn't be selected when creating a
standard scoped token in the UI — leaving scope-restricted CLI/MCP docs use
effectively ungrantable. Add a read-only "Documentation" group (no write
surface) and a test asserting it is exposed.
- search ran page.body.to_lowercase() on the whole corpus per query. Lowercase
body/title/description once at parse time (into the OnceLock corpus) and scan
the precomputed copies instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: update ee-repo-ref to 27a4f41b8e5603d6e444efcfc420bd1c44a07eed
This commit updates the EE repository reference after PR #630 was merged in windmill-ee-private.
Previous ee-repo-ref: c7ec3a0c2fa38d4cb5e50bf0265eef4710de4860
New ee-repo-ref: 27a4f41b8e5603d6e444efcfc420bd1c44a07eed
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
Ruben Fiszel
|
12f92e3ab7 |
[ee] feat(backend): native script retry without one-step-flow wrapping (#9688)
* feat(backend): native script retry without one-step-flow wrapping Schedules and data pipelines that retry a single script previously wrapped it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a v2_job_status row, and UI projection complexity. This adds native retry on a plain JobKind::Script job. - RetrySettings: flatten Retry into a deduped retry_settings table, carried via the existing runnable_settings_handle (lazy, off the hot path). - push() materializes a bare-script-with-retry SingleStepFlow into a native Script job (gated on min-version + no handlers/retry_if). - add_completed_job re-pushes the next attempt on failure with backoff, tracking the attempt counter in v2_job_queue.extras and the chain via parent_job; schedule completion handlers fire only on the terminal attempt. - frontend: ScriptRetryChain shows the attempt chain on the run page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): native retry_if eval + per-occurrence schedule handlers Extends native script retry to the two cases that previously stayed on the one-step-flow path: - retry_if: evaluated natively on the failure path via a feature-gated windmill-jseval dep (quickjs) over the failure result + flow_input; push materializes such policies natively only when quickjs is available. - on_failure_times / on_recovery: apply_schedule_handlers now resolves each past scheduled occurrence's terminal status across its native-retry chain (root OR any parent_job=root child succeeded) and excludes the current occurrence, so the counting is per-occurrence rather than per-attempt. All scheduled-script retries now go native (schedule.rs gate removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): always materialize retry_if natively; unsupported without quickjs retry_if is evaluated by the worker (which always has quickjs), not the pusher, so gating materialization on the pusher's feature was wrong. The flow path was never a real fallback either — the flow runtime needs quickjs to evaluate retry_if too. retry_if now always goes native; on a worker without quickjs it is unsupported and fails closed (no retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): un-park asset-cascade (pipeline) retry Native retry resolves the blocker that parked pipeline retry: a retried subscriber is now a Script job (not a one-step flow / flow step), so it stays eligible for asset dispatch and can trigger its own downstream on recovery. - scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset edges (was dropped with a TODO warning). - asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job, so native-retry attempts dispatch on success while flow steps stay excluded. - tests: retry-bearing subscriber now dispatches as a native Script carrying the policy in runnable_settings_handle; native-retry attempt is eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): cap native retry interval, lazy result serialization, idempotent retry push Hardening from a self-review of the native retry path: - Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path (evaluate_retry); the exponential formula could otherwise schedule up to ~18h vs the flow path's 6h. - Serialize the failure result lazily (only when a retry_if policy needs it), so the common failure no longer pays the serialization on the failure path. - Push each retry with a deterministic id per (root, attempt). If a worker dies between enqueueing the retry and finalizing the current attempt, the reaper re-handles the attempt and lands here again — push rejects the duplicate id, so the retry is enqueued exactly once (no double-retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): defer schedule handlers idempotently on retry-push replay (review P1) Address local-review findings: - P1: retry_pending was derived from the retry push *result*, so on a worker crash + reaper replay the duplicate-id push returned Err → retry_pending flipped to false → apply_schedule_handlers fired for the non-terminal attempt (and the terminal attempt later fired them again). Pre-check whether the deterministic retry id already exists and report it as pending without re-pushing, so the handler-deferral invariant is crash-idempotent too. - P2: refresh the stale 'wrap the script in a one-step flow' comment in the asset-cascade retry push — it now materializes a native Script. - Add RetrySettings <-> Retry round-trip unit tests (clamping edges). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): native retry chain + per-occurrence status sqlx tests Close the two integration-test gaps flagged in local review: - chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts) and asserts crash-replay idempotency (the P1 fix: a replayed completion reports pending without double-enqueueing). - per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but- recovered occurrence counts as success, retries (parent_job set) are excluded from occurrence counting, and the current occurrence is excluded. - canceled_job_does_not_retry: cancellation wins over a pending retry. Runtime sqlx API (no .sqlx cache entry needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): exclude schedule handlers from the retry-attempt chain The retry chain listed all script children of the root by parent_job, but schedule completion handlers (on_failure/on_recovery/on_success) are also script children — when the occurrence has no retries, the handler's parent is the root itself, so a successful, never-retried job rendered a bogus 'Retries (1)' badge pointing at the handler. Filter children to re-runs of the same script (matching script_hash); real retries keep the root's hash, handlers run a different script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): surface schedule handlers on the run page Extend the run-page chain component with schedule completion handlers: - A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/ on_success runs that fired for that occurrence (found as children of the terminal attempt, identified by their synthetic created_by). - A handler's own run page now shows a 'Failure/Recovery/Success handler' label with a link back to the run it handled and its schedule. on_recovery and on_success share created_by, disambiguated by the recovery-only error_started_at arg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare An earlier `cargo sqlx prepare` on this branch ran before #8801's folder_default_permissioned_as test merged in, so it pruned the 3 query caches that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main. * fix(backend): only cascade assets from native retry attempts, not handlers (review P1) is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script child became asset-eligible — including schedule/error/recovery handlers (Script jobs with parent_job set and no flow_step_id). A handler that declares assets would then trigger a cascade the old parent_job IS NULL guard prevented. Gate parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as its chain parent (handlers run a different script). Runtime query, no sqlx cache. * fix(backend): cache the private-gated retry_setting asset-dispatch test query The same prepare-without-private that dropped the folder_default caches also pruned the cache for the retry_setting_dispatches_subscriber_as_native_script test query (asset_trigger_dispatch.rs:721). Regenerated with --features private. * fix(backend): exclude handler children from per-occurrence recovery (review) A scheduled occurrence's on_failure/on_success handler runs as a successful child (parent_job = occurrence), and the per-occurrence success EXISTS counted ANY successful child — so a failed occurrence whose error handler succeeded was marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in the merge) and on_failure_times counting. EE query now scopes the EXISTS to same-runnable children (only native retry attempts); regenerate sqlx cache + bump ee-repo-ref. native_retry_test gains a handler-child regression case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): scheduled-script retry is a native Script, not SingleStepFlow test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a native Script. Update both to assert kind='script' and that the retry policy is carried via runnable_settings_handle. * fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic) Address cubic CI review: - P1: the SingleStepFlow->native Script materialization dropped dedicated_worker, so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve it from the script row in push so the materialized Script keeps the dedicated tag. - P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the u32->i16 // retry count narrowing (scripts.rs) instead of wrapping. * fix(backend): use a retry-specific signal, not runnable equality (codex review) Address Codex CI review: - P1: is_native_retry_attempt treated any same-runnable parented Script child as a retry. WAC v2 inline children have that exact shape, so an inline child of an asset producer would cascade. Use a retry-specific signal instead: the job carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no flow_innermost_root_job. Apply the same flow_innermost guard to the EE per-occurrence EXISTS (WAC inline children must not count as a recovery). - P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate now resolves as 'retry pending' (re-check on the duplicate-id error) instead of flipping retry_pending to false and firing handlers early. - Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): explicit native_retry_attempt marker, drop heuristics Replace the per-site "is this a retry?" inference (parent_job + runnable match + flow_innermost / retry_settings) with one explicit marker: a sparse native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker also carries the attempt counter (previously in v2_job_queue.extras), so it's the single source of truth. - asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker. - EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost. - maybe_enqueue: reads/writes the marker (persistent) instead of queue extras. - Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap. - Eliminates handler / WAC-inline-child misclassification by construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex) The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but the periodic monitor.rs path deleted v2_job rows without it, orphaning markers. Add the same WHERE job_id = ANY(...) sweep there. * fix(backend): widen native_retry_attempt.attempt to integer (cubic) The smallint column was cast to/from u32 and could wrap a retry chain longer than i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32 attempt count, so no narrowing occurs on the maybe_enqueue read/write path. * feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT - Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi), computed from the native_retry_attempt marker. The run-page chain now filters retry attempts by is_retry instead of the script_hash heuristic, so WAC v2 inline children (same script, parent_job) no longer render as retries (codex). - Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep the plain catch-and-continue matching the other side-table deletes; the table is created by a startup migration so it always exists when cleanup runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but does not select is_retry, which errored with "no column found". Only the list endpoint populates the marker; #[sqlx(default)] lets every other query omit the column and default to None. * feat(backend): select is_retry in single-job GET too for consistency The list endpoint already exposes the marker; populate it on the single-job GET (both completed and queued variants) as well so a run loaded directly reflects its retry status. #[sqlx(default)] stays as a safety net for any other query. * feat(backend): reap orphaned native_retry_attempt markers via periodic sweep The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so direct job deletions (workspace/job delete, schedule clearing) would leave marker rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which must then be remembered for every future path), reap orphans in the periodic delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the job). The table is sparse so the anti-join drives off it and probes v2_job by PK — cheap. Retention still sweeps markers inline (keeps the table small so this stays cheap); a transient orphan is harmless (nothing reads is_retry for a gone job). * fix(frontend): include flow handlers in retry chain handler row (codex) Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...), whose handler job is a flow, not a script. The chain fetched children with jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts are still selected by is_retry and handlers by created_by, so both kinds surface. * fix(backend): carry concurrency/debouncing settings into native retries maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings ::default(), dropping the script/pipeline concurrency settings the failed job carried in its runnable_settings_handle. A retry of a concurrency-limited script then inserted no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and pass them in the payload, which push forwards to the materialized retry. Adds a regression test asserting the retry's handle resolves to the concurrency settings. * fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex) P1a (schedule.rs): the scheduled-retry materialization fetched the script's concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings ::default() into the SingleStepFlow payload, so the root attempt's handle held only the retry policy and the whole chain ran unbounded. Pass the fetched settings. Regression test asserts the root handle resolves to retry + concurrency. P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract — it is pub only for the integration test; the sole production caller is the worker completion path passing a DB-derived, already-authorized MiniCompletedJob. * docs(backend): attach native-retry auth contract to the function itself (codex) The doc block was merged with eval_retry_if's doc and bound to that function, leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if keeps its own doc; the native-retry + authorization contract now sits directly above maybe_enqueue_native_script_retry. * docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex) - Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since 1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the served spec couldn't see the field. Now current at 1.739.0. - schedule.rs: a retry_if gate is evaluated at failure time and fails closed without quickjs (no retry); it does not fall back to a flow path. - windmill-types jobs.rs: is_retry is selected by both the list and single-job GET endpoints (not list-only). * docs(backend): fix remaining stale retry_if/quickjs comments (codex) The retry_if block and the push materialization comments claimed push keeps retry_if on a flow path / the worker always has quickjs. The code always materializes native retry and the no-quickjs eval_retry_if path fails closed — correct the comments to that constraint. * docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex) - Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated and the job does not retry (no one-step-flow fallback). - jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence failure/recovery status across the retry chain, so the old 'restricted to schedules whose handlers don't need per-occurrence counting' claim is dropped. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
cd42c6ca18 |
fix: decrypt secret variables via external backend in common resolvers (#9784)
`get_variable_or_self`, `get_variable_or_self_as`, `get_secret_value_as_admin` (and `transform_json_unchecked`'s `$var:` branch) in windmill-common always ran the raw `variable.value` through `decrypt()`. With an external secret backend (HashiCorp Vault / Azure Key Vault / AWS Secrets Manager) configured, that column holds a `$vault:`/`$azure_kv:`/`$aws_sm:` marker rather than base64 ciphertext, so base64 decoding failed with `Invalid byte 36, offset 0` (the `$`). This broke GitHub App git sync (git_sync_ee.rs) and any other consumer of these resolvers when an external backend is active. Move backend resolution (`get_secret_backend`, `get_secret_value`, `is_*_stored_value`, caching) into `windmill-common::secret_backend::resolver` so the low-level variable resolvers can route external markers through the configured backend's `get_secret()` instead of `decrypt()`. The windmill-store and windmill-api `secret_backend_ext` modules now re-export these from windmill-common (single source of truth / single backend cache) and keep only their write-side helpers. No `_ee.rs` files change. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
f5828780fd |
fix(backend): resolve folder_labels search_path on non-public (PG_SCHEMA) schemas (#9758)
* fix(backend): strip search_path=public from folder_labels migrations for non-public schema The folder-labels migrations (20260610151334_folder_labels, 20260614075900_dedup_folder_labels) define `folder_labels(...)` with `SET search_path = public` in their `CREATE FUNCTION` bodies. When Windmill runs in a non-public schema (PG_SCHEMA), PostgreSQL validates the function body against the `public` schema, where the `folder` table lacks the new `labels` column, failing with `column "labels" does not exist`. Add both migrations to OVERRIDDEN_MIGRATIONS, stripping the `SET search_path = public` clause so the function inherits the current search_path (which resolves the correct schema). Same regression and fix pattern as PR #5400. Fixes WIN-2093 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): pin folder_labels search_path FROM CURRENT instead of stripping it Keep the SECURITY DEFINER injection hardening while resolving the correct schema on non-public (PG_SCHEMA) installs: FROM CURRENT snapshots the migration connection's search_path at function creation time (public on normal installs, the custom schema otherwise) instead of dropping the pin and inheriting the caller's search_path at call time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): repair migration to re-pin folder_labels search_path on applied instances Instances that already applied the folder-labels migrations with the hardcoded SET search_path = public have a folder_labels function pinned to public. On a non-public (PG_SCHEMA) schema that reads the wrong folder table at runtime; the OVERRIDDEN_MIGRATIONS fix only helps instances that have not applied them yet. Add a CREATE OR REPLACE ... SET search_path FROM CURRENT migration that re-pins the function to the migration connection's schema. No-op on public installs (re-pins to public) and idempotent on already-correct ones. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |