From 84cc043406d63a1e1472165cf24ce8c09905fc5f Mon Sep 17 00:00:00 2001 From: Guilhem Date: Mon, 22 Jun 2026 09:21:14 +0200 Subject: [PATCH 001/117] feat: link files & folders to the global AI chat (#9520) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: add file attachments to the global AI chat Co-Authored-By: Claude Opus 4.8 * feat: add folder linking and file-type icons to chat attachments Co-Authored-By: Claude Opus 4.8 * feat: persist linked files, add @-menu file tree, and polish chat file UI Persistence (survive reload, scoped to session.id): - IndexedDB store (attachedFilesDB) holding Blob snapshots (every browser) and re-grantable File System Access directory handles (capable browsers) - restore on session activation; re-grant locked handles on the next send; flush in-memory items when the session persists; GC on session delete - capability via feature-detection (fsAccess), never UA sniffing - folders auto-refresh (live re-enumerate + reconcile) on each send @-mention file picker: - Files branch in ChatContextPicker (new DrillPicker architecture); a linked folder's files render as a nested directory tree, picking inserts @filename - attached-file mentions highlight in the input just like context mentions UI polish: - file/folder chips reuse the context-element chip style (icon -> X on hover) - file + context badges sit above the fork/draft bar - disabled dropdown items can surface an explanatory tooltip (DropdownV2Inner) Co-Authored-By: Claude Opus 4.8 * refactor: deepen the attached-files store — folders as first-class objects Two seam fixes from an architecture pass, no behaviour change: - addFolder(dirHandle) now enumerates internally (same junk-filtered walk used on restore/refresh), so callers never pre-enumerate. The dead drop-walkers (collectDroppedEntries, filterFolderPickerFiles) are deleted; isIgnoredPath/MAX_FOLDER_FILES move next to enumerateDir in fsAccess. - The store exposes `folders` (name + aggregate status + children) and `standalone` as derived views, so the bar, the @-menu picker, the folder chip and the system-prompt roster stop re-grouping the flat row list and re-deriving folder status. Placeholder rows (isFolderRoot) become an implementation detail; the roster renders a locked folder as one line. Co-Authored-By: Claude Opus 4.8 * fix: drop the redundant context-badge row in the global chat In GLOBAL mode selected context already appears as a highlighted @mention in the input (deleting the mention deselects), so the hoisted badge row above the chat duplicated it. File chips keep their row — attachments aren't represented in the input. Co-Authored-By: Claude Opus 4.8 * fix: harden attachment edge cases found in review - requestReadPermission/queryReadPermission never reject (the spec rejects with SecurityError when user activation is missing — now mapped to denied/prompt), and sendRequest wraps attachment upkeep in try/catch, so a permission hiccup can never silently swallow a Send. - regrantLocked expands before dropping the locked placeholder: when the re-granted directory is gone from disk, the folder now shows "unavailable" instead of vanishing into a zombie that resurrects locked on the next reload. - addFolder: re-picking a locked/unavailable folder relinks it (natural recovery gesture); a genuine second folder with the same basename gets a visible "already linked" rejection instead of a silent no-op. - fileEngine: readFile clamps its byte slice to maxChars*4 before decoding and streamLines caps its per-line buffer, so newline-sparse files (minified JS, single-line JSONL) can't materialize unbounded strings; corrected the scan-cap comment's claim about catastrophic backtracking. 4 new unit tests (41 total). Co-Authored-By: Claude Opus 4.8 * fix: surface folder-picker failures instead of swallowing them `pickDirectory` caught every `showDirectoryPicker` rejection and returned undefined, so a real failure (an enterprise/browser policy blocking the File System Access API, a lost user-activation, …) was indistinguishable from a no-op — the picker just silently never opened. Now only `AbortError` (user dismissed the dialog, or CDP intercepted it under automation) is treated as a cancel; anything else is rethrown and `linkFolder` surfaces it as a toast. Co-Authored-By: Claude Opus 4.8 (1M context) * feat: support folders in browsers without the File System Access API Folders can now be added in every browser, not just Chromium. Where the File System Access API is absent (Firefox/Safari), a dropped or picked folder's files are snapshotted into the browser (via a webkitGetAsEntry drop-walk or a `webkitdirectory` input) instead of linked as a live handle, and grouped/displayed identically to a File System Access folder. The dropdown item reads "Link folder" when a live link is possible and "Add folder" otherwise, with a tooltip pointing to Chrome/Edge for a live link. Snapshot folder children persist their `folder`/`relPath`, so they regroup into the same folder chip on reload. Removes the arbitrary file-count caps (500 per folder, 100 total) — only the browser's memory / IndexedDB quota now bound a folder. Junk paths (node_modules/.git/dist/dotfiles) are still skipped, folder-contents only, so an explicitly attached standalone dotfile is kept. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: address review feedback — index-race guard + read_file line numbers Both automated reviewers flagged two issues on the attached-files feature: - (P1) Stale async indexing could corrupt a newer file. `#indexFile` applied its unawaited `buildLineIndex` result by display name, so if a row's file was swapped while indexing was in flight (remove + re-add a same-named file, or a folder refresh re-indexing an edited file) the stale result stamped the wrong lineIndex/lineCount — and `read_file` then sliced the new Blob with old offsets. Now patched via `#patchFile`, which applies the result only while the row still holds the exact file object that was indexed. - (P2) `read_file` promised "line-numbered context" but returned raw text. It now prefixes each line with its absolute 1-based number (`→`), matching the tool contract; `numberLines` lives in fileEngine and is unit-tested. Adds regression tests: a deterministic stale-index race test (controlled buildLineIndex ordering) and numberLines coverage. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: address re-review nits — read_file pagination + searchFiles regex state - read_file: when the maxChars cap truncated a window short of its requested end line, the pagination note still reported the full range and gave no/wrong resume point, so the model couldn't reach the unread lines. The note now reports the last line actually returned and resumes at the next unread line (advancing past a single over-long line rather than re-truncating it forever). - searchFiles: reset `regex.lastIndex` before each `.test()` — a caller-supplied `g`/`y` flag makes test() stateful and would silently drop matches. Not reachable from the current caller, but searchFiles is exported. Adds regression tests for both. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: keep an emptied live folder linked and refreshing A live (File System Access) folder carried its directory handle only on its child file rows. When the folder was emptied on disk, refreshFolders/#reconcileFolder removed the last child — dropping the only handle-bearing row — so the folder vanished from the chip bar AND was never re-enumerated again (files added back on disk weren't picked up until a reload). #expandFolder had the same gap on restore. Now #ensureFolderRow leaves one handle-carrying placeholder row when a folder has no readable children (keeps the chip visible and the live source alive), and drops it once children return; refreshFolders collects sources from placeholder rows too, and readyFiles never exposes a placeholder to the read/search tools. Adds a regression test (empty → still visible → file returns → picked up). Co-Authored-By: Claude Opus 4.8 (1M context) * fix: trim read_file char-cap output to match its pagination note When the char cap cut partway into the line after some whole lines, readFile set the note/endLine to the last complete line but still returned the partial next line in `text` — so read_file showed (line-numbered) a line the note said would come on the next read. Trim the returned text back to the last complete newline so the body and the note agree. Test now asserts res.text for that case. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: isolate search_files in a Worker (ReDoS) + path-aware folder dedup - search_files runs a model-supplied regex, and a catastrophic-backtracking pattern (e.g. /^(a+)+$/) can't be interrupted mid-test, freezing the tab. Run the search in a Web Worker (searchFilesInWorker) and terminate it on a timeout, returning "pattern too expensive" instead of hanging. Degrades gracefully to a main-thread search where Workers are unavailable / fail to load. - #isDuplicate keyed its content check on the file basename, so two distinct files sharing a basename under different folder subdirs (proj/a/index.ts vs proj/b/index.ts) were wrongly deduped and silently dropped from snapshotted folders. Key it on the relative path instead. Adds tests: worker result/timeout-and-terminate, and same-basename-different-subdir. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: keep an initially-empty live folder linked (placeholder + persist) addFolder only created rows / persisted the dir-handle when at least one text file was found, so linking a folder that's empty (or all-binary) at pick time was a silent no-op: no chip, nothing persisted, and refreshFolders had no source to re-enumerate when files were added later. Now it always leaves a placeholder (#ensureFolderRow) and persists the handle — matching the became-empty behavior — so the folder stays visible, survives reload, and picks up files added afterward. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: keep empty-folder placeholders out of the real-file name space The placeholder row for an empty live folder uses name = folder, which could collide with a standalone file of the same name: addFiles deduped the file against the placeholder, removeFile(name) dropped both rows, and #uniqueName pushed the file to a "(2)" suffix. Placeholders are managed via removeFolder and never read by the tools, so exclude isFolderRoot rows from #isDuplicate, removeFile, get(), and #uniqueName. Adds a placeholder/standalone collision test. (codex's other nit — @-mentions not highlighting filenames with spaces — left as a known cosmetic limitation per the chosen scope.) Co-Authored-By: Claude Opus 4.8 (1M context) * fix: highlight @-mentions of filenames containing spaces A file mention was inserted verbatim as `@my file.txt`, but the highlighter regex `@[\w/.\-\[\]]+` stops at the space, so only `@my` was parsed/highlighted and the mention didn't behave as advertised. Introduce a small shared `mention` module: names with whitespace are inserted in a bracketed form `@[my file.txt]`, and the shared regex + `mentionTitle` parse both bare and bracketed tokens. Both insertion entry points (the inline `@` picker in ContextTextarea and the toolbar path in AIChatInput) now use `formatMention`, so the full name highlights. Verified in a real browser: `@[my file.txt]` renders as a single highlight span. Unit tests cover format/parse/round-trip. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: search_files reports a requested file's real status, not "not attached" search_files filtered the store down to readyFiles() before validating a requested `file`, so searching an attached-but-not-ready file (indexing / errored / locked / unavailable) while another file was ready returned "No attached file named X" — even though it is attached. Factor read_file's status reporting into a shared notReadyMessage() and have search_files report the same accurate status before searching the ready subset. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: clear attached files on new/loaded chat in the non-session global chat saveAndClear() (the "New chat" button) and loadPastChat() left attachedFiles intact. In an AI session that's intended — files are session-scoped and persist across conversations. But the ephemeral global side-panel chat has no session, so the next, unrelated conversation still got the previous file roster injected and could read_file/search_files against it. Clear attachments on both transitions when `!isSessionChat`; sessions keep them. Adds a lifecycle regression test. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: keep an empty folder linked when regranting access after reload regrantLocked() dropped the locked placeholder unconditionally after #expandFolder. If the regranted folder was empty (or all-binary), #expandFolder's #ensureFolderRow no-op'd (the locked placeholder still existed), so dropping it removed the only handle-bearing row — unlinking the folder and stopping future refreshFolders from ever seeing files added back. Re-ensure a ready placeholder after dropping the locked one. Adds a regression test for the empty-regrant path. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: round-trip @-mentions of filenames containing a closing bracket The bracketed mention form `@[name]` broke when the name contained a `]` (e.g. `notes ] draft.md`): the regex stopped at the first `]` and mentionTitle resolved the wrong name, so it wouldn't highlight. Escape `\` and `]` when bracketing, match escaped chars in MENTION_RE, and unescape in mentionTitle. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: highlight @-mentions of filenames with HTML-sensitive / special chars getHighlightedText() escapes the textarea value to HTML before parsing mentions, then looked the parsed title up against raw attached names — so a file like `R&D notes.md` (escaped to `R&D notes.md`) never matched and wasn't highlighted. Also, names with chars outside the bare set (`<`, `>`, `&`, parens, …) weren't bracketed, so the bare regex truncated them. Now formatMention brackets any non-bare-safe name, and the highlighter HTML-unescapes the parsed title before the store lookup. Verified in a real browser with `R&D notes.md`. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: report the real reason search_files has no readable targets When attachments existed but readyFiles() was empty, search_files always told the model "still being indexed, try again shortly". That's wrong for the placeholder states this PR introduces: an empty or binary-only linked folder leaves only a filtered-out `ready` placeholder, and a locked/unavailable restored folder exposes no readable children. Now the message reflects the actual state — no searchable text, restore access, or re-link — and only says "indexing" when something is. Adds a focused fileTools test for the empty-ready states. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 --- .../src/lib/components/DropdownV2Inner.svelte | 104 +-- .../copilot/chat/AIChatDisplay.svelte | 233 ++++++- .../copilot/chat/AIChatInput.svelte | 8 + .../copilot/chat/AIChatManager.svelte.ts | 37 +- .../copilot/chat/AIChatManager.test.ts | 31 + .../copilot/chat/ChatContextPicker.svelte | 132 +++- .../copilot/chat/ContextTextarea.svelte | 36 +- .../chat/files/AttachedFileChip.svelte | 61 ++ .../chat/files/AttachedFilesBar.svelte | 86 +++ .../chat/files/AttachedFolderChip.svelte | 48 ++ .../chat/files/attachedFiles.svelte.ts | 602 ++++++++++++++++++ .../copilot/chat/files/attachedFiles.test.ts | 476 ++++++++++++++ .../chat/files/attachedFilesDB.test.ts | 28 + .../copilot/chat/files/attachedFilesDB.ts | 118 ++++ .../copilot/chat/files/fileEngine.test.ts | 286 +++++++++ .../copilot/chat/files/fileEngine.ts | 430 +++++++++++++ .../copilot/chat/files/fileTools.test.ts | 66 ++ .../copilot/chat/files/fileTools.ts | 226 +++++++ .../copilot/chat/files/fsAccess.test.ts | 49 ++ .../components/copilot/chat/files/fsAccess.ts | 192 ++++++ .../copilot/chat/files/searchWorker.ts | 38 ++ .../components/copilot/chat/global/core.ts | 7 +- .../components/copilot/chat/mention.test.ts | 54 ++ .../lib/components/copilot/chat/mention.ts | 31 + frontend/src/lib/components/icons/fileIcon.ts | 74 +++ .../components/raw_apps/FileTreeNode.svelte | 83 +-- .../sessions/sessionRuntime.svelte.ts | 5 + .../sessions/sessionState.svelte.ts | 3 + 28 files changed, 3400 insertions(+), 144 deletions(-) create mode 100644 frontend/src/lib/components/copilot/chat/files/AttachedFileChip.svelte create mode 100644 frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte create mode 100644 frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte create mode 100644 frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts create mode 100644 frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts create mode 100644 frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts create mode 100644 frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts create mode 100644 frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts create mode 100644 frontend/src/lib/components/copilot/chat/files/fileEngine.ts create mode 100644 frontend/src/lib/components/copilot/chat/files/fileTools.test.ts create mode 100644 frontend/src/lib/components/copilot/chat/files/fileTools.ts create mode 100644 frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts create mode 100644 frontend/src/lib/components/copilot/chat/files/fsAccess.ts create mode 100644 frontend/src/lib/components/copilot/chat/files/searchWorker.ts create mode 100644 frontend/src/lib/components/copilot/chat/mention.test.ts create mode 100644 frontend/src/lib/components/copilot/chat/mention.ts create mode 100644 frontend/src/lib/components/icons/fileIcon.ts diff --git a/frontend/src/lib/components/DropdownV2Inner.svelte b/frontend/src/lib/components/DropdownV2Inner.svelte index 71cbea82f9..1196589474 100644 --- a/frontend/src/lib/components/DropdownV2Inner.svelte +++ b/frontend/src/lib/components/DropdownV2Inner.svelte @@ -28,6 +28,58 @@ computeItems() +{#snippet menuItem(item: Item)} + item?.action?.(e)} + href={item?.href} + target={item?.hrefTarget} + disabled={item?.disabled} + class={twMerge( + 'px-4 py-2 text-primary font-normal hover:bg-surface-hover cursor-pointer text-xs transition-colors w-full', + 'data-[highlighted]:bg-surface-hover', + 'flex flex-row gap-2 items-center rounded-sm', + // `pointer-events-none` lets a hover fall through the natively-disabled button to the + // wrapper below, so a disabled item's `title` tooltip can still explain *why* it's disabled. + item?.disabled && 'text-disabled cursor-not-allowed pointer-events-none', + item?.type === 'delete' && + !item?.disabled && + 'text-red-600 dark:text-red-400 data-[highlighted]:bg-red-500/10 dark:data-[highlighted]:bg-red-900/80 dark:data-[highlighted]:text-red-300 ' + )} + item={meltItem} + aiId={`${aiId ? `${aiId}-${item.displayName}` : undefined}`} + aiDescription={item.displayName} + > + {#if item.icon} + + {/if} +

+ {item.displayName} +

+ {@render item.extra?.()} + {#if item.shortcut || item.selected} + +
+ {#if item.shortcut} + {item.shortcut} + {/if} + {#if item.selected} + + {/if} +
+ {/if} + {#if item.tooltip && !item.disabled} + + + {#snippet text()} + {item.tooltip} + {/snippet} + + {/if} + +{/snippet} + {#if computedItems}
{#each computedItems ?? [] as item} @@ -36,52 +88,14 @@ {/if} {#if item.submenuItems && builders} + {:else if item.disabled && item.tooltip} + +
+ {@render menuItem(item)} +
{:else} - item?.action?.(e)} - href={item?.href} - target={item?.hrefTarget} - disabled={item?.disabled} - class={twMerge( - 'px-4 py-2 text-primary font-normal hover:bg-surface-hover cursor-pointer text-xs transition-colors w-full', - 'data-[highlighted]:bg-surface-hover', - 'flex flex-row gap-2 items-center rounded-sm', - item?.disabled && 'text-disabled cursor-not-allowed', - item?.type === 'delete' && - !item?.disabled && - 'text-red-600 dark:text-red-400 data-[highlighted]:bg-red-500/10 dark:data-[highlighted]:bg-red-900/80 dark:data-[highlighted]:text-red-300 ' - )} - item={meltItem} - aiId={`${aiId ? `${aiId}-${item.displayName}` : undefined}`} - aiDescription={item.displayName} - > - {#if item.icon} - - {/if} -

- {item.displayName} -

- {@render item.extra?.()} - {#if item.shortcut || item.selected} - -
- {#if item.shortcut} - {item.shortcut} - {/if} - {#if item.selected} - - {/if} -
- {/if} - {#if item.tooltip} - - {#snippet text()} - {item.tooltip} - {/snippet} - - {/if} - + {@render menuItem(item)} {/if} {/each}
diff --git a/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte b/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte index e9ac5eed16..6260a9f6d8 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte +++ b/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte @@ -10,6 +10,8 @@ ChevronDown, ChevronsRight, CheckIcon, + FileText, + Folder, Hand, HistoryIcon, Hourglass, @@ -41,6 +43,15 @@ import QueuedMessageChip from './QueuedMessageChip.svelte' import { getModifierKey } from '$lib/utils' import type { SelectedContext } from './app/core' + import AttachedFilesBar from './files/AttachedFilesBar.svelte' + import { type FileToAttach } from './files/attachedFiles.svelte' + import { + hasFileSystemAccess, + pickDirectory, + handlesFromDataTransfer, + readDroppedEntries + } from './files/fsAccess' + import { sendUserToast } from '$lib/toast' const MAX_YOLO_TOOLTIP_TOOLS = 8 const aiChatManager = getAiChatManager() @@ -252,6 +263,133 @@ aiChatManager.mode === AIMode.GLOBAL || aiChatManager.mode === AIMode.APP ) + + // File attachment is GLOBAL-mode only. + const canAttachFiles = $derived(aiChatManager.mode === AIMode.GLOBAL && !disabled) + // Steers the OS file picker toward text formats (soft hint; content sniff is authoritative). + const TEXT_FILE_ACCEPT = + 'text/*,.txt,.csv,.tsv,.json,.jsonl,.ndjson,.md,.markdown,.log,.yaml,.yml,.toml,.ini,.cfg,.conf,.env,.xml,.html,.htm,.css,.js,.mjs,.cjs,.ts,.tsx,.jsx,.py,.rb,.rs,.go,.java,.kt,.c,.h,.cpp,.cc,.cs,.php,.sh,.bash,.zsh,.sql,.svelte,.vue,.dockerfile' + let fileInputEl = $state(null) + let folderInputEl = $state(null) + let dragDepth = $state(0) + const isDraggingFiles = $derived(dragDepth > 0) + // File System Access API → live re-grantable folder handles (refreshed each turn). + // Otherwise folders are snapshotted into the browser (via webkitdirectory / dropped-entry + // walk), same as files. Either way folders display identically. + const canUseFsAccess = hasFileSystemAccess() + + function reportAddResult(added: string[], rejected: { name: string; reason: string }[]) { + if (rejected.length === 0) return + // Single rejected file (e.g. one dropped image): show the precise reason. + if (added.length === 0 && rejected.length === 1) { + sendUserToast(`Could not attach "${rejected[0].name}": ${rejected[0].reason}`, true) + return + } + // Otherwise (folders / multi-select): summarize to avoid a flood of toasts. The only + // per-file rejection left is non-text content (binary files are skipped). + const lead = added.length + ? `Attached ${added.length}, skipped ${rejected.length}` + : `Skipped ${rejected.length} file${rejected.length === 1 ? '' : 's'}` + sendUserToast(`${lead} (non-text).`, added.length === 0) + } + + async function handleAddFiles(files: FileList | FileToAttach[]) { + const { added, rejected } = await aiChatManager.attachedFiles.addFiles(files) + reportAddResult(added, rejected) + } + + async function addDirHandle(dir: FileSystemDirectoryHandle) { + const { added, rejected } = await aiChatManager.attachedFiles.addFolder(dir) + reportAddResult(added, rejected) + } + + function linkFiles() { + // Files are always snapshotted (every browser), so the universal picker is fine. + fileInputEl?.click() + } + + async function linkFolder() { + if (!canUseFsAccess) { + // No File System Access API → pick a folder via the directory input; its files are + // snapshotted into the browser (no live handle), grouped under the folder name. + folderInputEl?.click() + return + } + let dir: FileSystemDirectoryHandle | undefined + try { + dir = await pickDirectory() + } catch (e) { + // The picker threw instead of opening — surface why (e.g. a browser/enterprise + // policy blocking the File System Access API) rather than appearing to do nothing. + sendUserToast( + `Couldn't open the folder picker: ${e instanceof Error ? e.message : String(e)}`, + true + ) + return + } + if (dir) await addDirHandle(dir) + } + + function dragHasFiles(e: DragEvent): boolean { + return Array.from(e.dataTransfer?.types ?? []).includes('Files') + } + + function onPanelDragEnter(e: DragEvent) { + if (!canAttachFiles || !dragHasFiles(e)) return + e.preventDefault() + dragDepth++ + } + function onPanelDragOver(e: DragEvent) { + if (!canAttachFiles || !dragHasFiles(e)) return + e.preventDefault() + if (e.dataTransfer) e.dataTransfer.dropEffect = 'copy' + } + function onPanelDragLeave(_e: DragEvent) { + if (!canAttachFiles) return + dragDepth = Math.max(0, dragDepth - 1) + } + async function onPanelDrop(e: DragEvent) { + dragDepth = 0 + if (!canAttachFiles || !dragHasFiles(e)) return + e.preventDefault() + const dt = e.dataTransfer + if (!dt) return + if (canUseFsAccess) { + // getAsFileSystemHandle calls are kicked off synchronously inside this call. + const handles = await handlesFromDataTransfer(dt) + for (const h of handles) { + if (h.kind === 'directory') { + // Folders link as a live handle. + await addDirHandle(h as FileSystemDirectoryHandle) + } else { + // Files are always snapshotted (handle discarded). + await handleAddFiles([{ file: await (h as FileSystemFileHandle).getFile() }]) + } + } + } else { + // Fallback (no File System Access API): snapshot dropped files AND folders by walking + // the legacy webkitGetAsEntry tree. readDroppedEntries reads the entries synchronously + // (they're only valid during this event) before its first await; if it yields nothing + // (no entry API), fall back to the flat dt.files. + const entries = await readDroppedEntries(Array.from(dt.items ?? [])) + if (entries.length > 0) await handleAddFiles(entries) + else if (dt.files.length > 0) await handleAddFiles(dt.files) + } + } + + function onFileInputChange(e: Event) { + const input = e.currentTarget as HTMLInputElement + if (input.files && input.files.length > 0) void handleAddFiles(input.files) + input.value = '' // allow re-selecting the same file + } + + function onFolderInputChange(e: Event) { + const input = e.currentTarget as HTMLInputElement + // webkitdirectory files carry webkitRelativePath (`folder/sub/file`); addFiles groups + // them under the folder and skips junk paths. Snapshot, like a dropped folder. + if (input.files && input.files.length > 0) void handleAddFiles(input.files) + input.value = '' + } const availableAutonomyModeOptions = $derived.by(() => autonomyModeOptions.filter((option) => isAutonomyModeAvailable( @@ -339,7 +477,28 @@ -
+
+ {#if isDraggingFiles} +
+
+ + Drop files to attach +
+
+ {/if} {#if !hideHeader}
{/if}
+ {#if aiChatManager.mode === AIMode.GLOBAL} + + + {/if} {#if inputPreface} {@render inputPreface()} {/if} @@ -544,6 +709,7 @@ the panel, or the Escape-to-stop focus check would wrongly reject them. --> bind:this={aiChatInput} bind:selectedContext {availableContext} + showContext={aiChatManager.mode !== AIMode.GLOBAL} disabled={disabled || hasActiveUserQuestion} isFirstMessage={messages.length === 0} /> @@ -595,11 +761,76 @@ the panel, or the Escape-to-stop focus check would wrongly reject them. --> setShowing={(showing) => { if (!showing) close() }} + onSelectFile={(name) => { + aiChatInput?.insertFileMention(name) + close() + }} /> {/if} {/snippet} {/if} + {#if canAttachFiles} + [ + { displayName: 'Attach file', icon: FileText, action: () => linkFiles() }, + { + // A real (live) link needs the File System Access API; without it the + // folder is only snapshotted, so call it "Add folder", not "Link folder". + displayName: canUseFsAccess ? 'Link folder' : 'Add folder', + icon: Folder, + tooltip: canUseFsAccess + ? 'Linked live — the assistant reads the folder’s current files from disk and refreshes each turn.' + : 'Loaded as a snapshot — the folder’s files are copied into your browser (they won’t auto-update). For a live link that refreshes from disk, use a Chromium-based browser (Chrome, Edge).', + action: () => linkFolder() + } + ]} + placement="bottom-start" + fixedHeight={false} + > + {#snippet buttonReplacement()} + + + {file.name} +
diff --git a/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte b/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte new file mode 100644 index 0000000000..50b961c090 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte @@ -0,0 +1,86 @@ + + +{#snippet chip(card: Card)} + {#if card.kind === 'folder'} + removeCard(card)} /> + {:else} + removeCard(card)} /> + {/if} +{/snippet} + +{#if cards.length > 0} +
+ {#each visible as card (card.key)} + {@render chip(card)} + {/each} + + {#if overflow.length > 0} + + {#snippet trigger()} +
+ +{overflow.length} +
+ {/snippet} + {#snippet content()} +
+ {#each overflow as card (card.key)} + {@render chip(card)} + {/each} +
+ {/snippet} + + {/if} + + {#if lockedCount > 0} + + {/if} +
+{/if} diff --git a/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte b/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte new file mode 100644 index 0000000000..068447a878 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte @@ -0,0 +1,48 @@ + + +
(showDelete = true)} + onmouseleave={() => (showDelete = false)} + role="listitem" + title={hoverList} +> + + {folder.name} +
diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts b/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts new file mode 100644 index 0000000000..124b390ec2 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts @@ -0,0 +1,602 @@ +/** + * Session-scoped store of files/folders the user has linked to the GLOBAL AI chat. + * + * Persistence model (survives reload, keyed by session in ./attachedFilesDB): + * - FILES are always stored as a full-byte Blob snapshot — same on every browser, + * no permission re-grant, never "locked". + * - FOLDERS link as a live File System Access directory handle where the API exists + * (one record, re-enumerated live on restore — folder files are read through the + * handle, not copied). Where it doesn't (Firefox/Safari), a dropped/picked folder's + * files are snapshotted individually (each carrying its `folder`/`relPath`) so they + * regroup into the same folder chip on restore. + * + * Storage is bounded by the real browser quota (writes that exceed it are caught and + * the item simply isn't persisted — it stays usable for the session). Persistence is + * gated on the session being persisted (non-transient); links in a transient session + * are buffered and flushed on the first send. + */ +import { createLongHash } from '$lib/editorLangUtils' +import { buildLineIndex, isTextFile, type FileEntry } from './fileEngine' +import { + putItem, + deleteItem, + getItemsForSession, + ensurePersistentStorage, + type PersistedAttachedItem +} from './attachedFilesDB' +import { enumerateDir, isIgnoredPath, queryReadPermission, requestReadPermission } from './fsAccess' + +export type AttachedFileStatus = 'indexing' | 'ready' | 'error' | 'locked' | 'unavailable' + +export interface AttachedFile extends FileEntry { + size: number + status: AttachedFileStatus + error?: string + /** Top-level folder this file came from (first path segment), if part of a folder. */ + folder?: string + /** Persisted source-record id. Folder children share the folder's record id. */ + sourceId: string + /** Parent directory handle (folder children only) — used to re-grant / re-enumerate. */ + handle?: FileSystemDirectoryHandle + /** Relative path within the folder (folder children only) — stable key for refresh diffing. */ + relPath?: string + /** + * Internal: a single placeholder row standing in for a not-yet-expanded folder + * (locked/unavailable). Consumers should read `store.folders` instead of testing this. + */ + isFolderRoot?: boolean +} + +/** A linked folder as a first-class object — consumers read this instead of re-grouping rows. */ +export interface AttachedFolder { + name: string + /** Aggregate status (locked > unavailable > indexing > error > ready). */ + status: AttachedFileStatus + /** Child files; empty while the folder is locked/unavailable after a reload. */ + files: AttachedFile[] +} + +/** Aggregate a folder's rows (children + a possible placeholder) into one status. */ +function folderStatus(rows: AttachedFile[]): AttachedFileStatus { + for (const status of ['locked', 'unavailable', 'indexing', 'error'] as const) { + if (rows.some((f) => f.status === status)) return status + } + return 'ready' +} + +export interface AddFilesResult { + added: string[] + rejected: { name: string; reason: string }[] +} + +/** A file to link: a raw File, or `{ file, path? }` (path = relative display name). */ +export type FileToAttach = File | { file: File; path?: string } + +const EMPTY = new Blob([]) + +export class AttachedFilesStore { + files = $state([]) + + /** Session context, set by the runtime; persistence writes are gated on `#persisted`. */ + sessionId: string | undefined = undefined + #persisted = false + /** Records buffered while the session is transient (flushed on first send). */ + #pending: PersistedAttachedItem[] = [] + + list(): AttachedFile[] { + return this.files + } + get(name: string): AttachedFile | undefined { + // Resolve to a real file — a folder-root placeholder may share the folder's name. + return this.files.find((f) => f.name === name && !f.isFolderRoot) + } + readyFiles(): AttachedFile[] { + // Folder-root placeholders aren't real files — never expose them to the read/search tools. + return this.files.filter((f) => f.status === 'ready' && !f.isFolderRoot) + } + get count(): number { + return this.files.length + } + + /** Linked folders, children grouped and status aggregated (placeholder rows hidden). */ + folders: AttachedFolder[] = $derived.by(() => { + const byName = new Map() + for (const f of this.files) { + if (!f.folder) continue + const rows = byName.get(f.folder) + if (rows) rows.push(f) + else byName.set(f.folder, [f]) + } + return [...byName.entries()].map(([name, rows]) => ({ + name, + status: folderStatus(rows), + files: rows.filter((f) => !f.isFolderRoot) + })) + }) + + /** Files linked on their own (not as part of a folder). */ + standalone: AttachedFile[] = $derived.by(() => this.files.filter((f) => !f.folder)) + + /** Number of locked folders needing a re-grant. */ + get lockedCount(): number { + return this.folders.filter((f) => f.status === 'locked').length + } + + clear(): void { + this.files = [] + this.#pending = [] + } + + removeFile(name: string): void { + // Target the real file only — never a folder-root placeholder that happens to share + // the name (those are managed via removeFolder), else removing a same-named standalone + // file would also drop the folder's placeholder. + const f = this.files.find((x) => x.name === name && !x.isFolderRoot) + if (!f) return + this.files = this.files.filter((x) => !(x.name === name && !x.isFolderRoot)) + void this.#deleteRecord(f.sourceId) + } + + /** Remove every file linked as part of the given folder (and its persisted record). */ + removeFolder(folder: string): void { + const ids = new Set(this.files.filter((f) => f.folder === folder).map((f) => f.sourceId)) + this.files = this.files.filter((f) => f.folder !== folder) + for (const id of ids) void this.#deleteRecord(id) + } + + // ---------------------------------------------------------------- linking + + /** + * Link individual files — always stored as a Blob snapshot. Items carrying a folder + * path (`folder/sub/file`, from a dropped/picked folder) are grouped into a folder and + * have their junk paths (node_modules/.git/dotfiles) skipped; a loose single file is + * kept as-is (so an explicitly attached `.env` isn't filtered out). + */ + async addFiles(input: FileList | FileToAttach[]): Promise { + const result: AddFilesResult = { added: [], rejected: [] } + + for (const item of Array.from(input as ArrayLike)) { + const file = item instanceof File ? item : item.file + const desired = + (item instanceof File ? '' : (item.path ?? '')) || + (file as File & { webkitRelativePath?: string }).webkitRelativePath || + file.name || + 'file' + + const folder = desired.includes('/') ? desired.split('/')[0] : undefined + if (folder && isIgnoredPath(desired)) continue // skip junk inside folders + + if (this.#isDuplicate(desired, file)) continue // silent no-op on re-link + + const reason = await this.#preflight(file) + if (reason) { + result.rejected.push({ name: desired, reason }) + continue + } + + const name = this.#uniqueName(desired) + const relPath = folder ? desired : undefined + const sourceId = createLongHash() + + this.#pushIndexing({ name, file, folder, sourceId, relPath }) + result.added.push(name) + void this.#persist({ + id: sourceId, + sessionId: this.sessionId ?? '', + kind: 'snapshot', + name, + folder, + relPath, + blob: file, + size: file.size, + lastModified: file.lastModified, + addedAt: Date.now() + }) + } + + return result + } + + /** + * Link a folder via a live directory handle (File System Access path only). + * Enumerates the handle internally (junk-filtered, capped) — the same walk used + * on restore and refresh, so callers never pre-enumerate. + */ + async addFolder(dirHandle: FileSystemDirectoryHandle): Promise { + const result: AddFilesResult = { added: [], rejected: [] } + const folder = dirHandle.name + const existing = this.files.filter((f) => f.folder === folder) + if (existing.length > 0) { + const placeholder = existing.length === 1 ? existing.find((f) => f.isFolderRoot) : undefined + if (placeholder) { + // Re-picking a folder that sits locked/unavailable after a reload is a natural + // recovery gesture — replace the stale link with the freshly-granted handle. + this.files = this.files.filter((f) => f.sourceId !== placeholder.sourceId) + void this.#deleteRecord(placeholder.sourceId) + } else { + // Same basename, possibly a different directory — surface it instead of a silent no-op. + result.rejected.push({ name: folder, reason: 'A folder with this name is already linked' }) + return result + } + } + + const files = await enumerateDir(dirHandle) + const sourceId = createLongHash() + for (const { file, path } of files) { + if (!(await this.#sniffText(file))) { + result.rejected.push({ name: path, reason: 'Not a text file' }) + continue + } + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle: dirHandle, relPath: path }) + result.added.push(name) + } + // Keep the folder represented even when it links empty (or all-binary): a placeholder + // carries the handle so the chip stays and refreshFolders picks up files added later. + // Persist unconditionally so an empty-at-link folder also survives a reload. + this.#ensureFolderRow(sourceId, folder, dirHandle) + void this.#persist({ + id: sourceId, + sessionId: this.sessionId ?? '', + kind: 'dir-handle', + name: folder, + folder, + handle: dirHandle, + addedAt: Date.now() + }) + return result + } + + // ------------------------------------------------------------- persistence + + /** Set session context and load any persisted items for it (called on activation). */ + async restore(sessionId: string, persisted: boolean): Promise { + this.sessionId = sessionId + this.#persisted = persisted + this.files = [] + this.#pending = [] + + const items = await getItemsForSession(sessionId) + for (const item of items) { + try { + if (item.kind === 'snapshot') { + if (!item.blob) { + this.#pushPlaceholder(item, 'unavailable') + continue + } + this.#pushIndexing({ + name: item.name, + file: item.blob, + folder: item.folder, + relPath: item.relPath, + sourceId: item.id + }) + } else { + // dir-handle (folder) + const handle = item.handle as FileSystemDirectoryHandle + if ((await queryReadPermission(handle)) === 'granted') { + await this.#expandFolder(handle, item.id) + } else { + this.#pushPlaceholder(item, 'locked', true) + } + } + } catch { + this.#pushPlaceholder(item, 'unavailable', item.kind === 'dir-handle') + } + } + } + + /** Re-grant any locked folder handles. MUST be called within a user gesture (e.g. on send). */ + async regrantLocked(): Promise { + const sources = new Map() + for (const f of this.files) { + if (f.status === 'locked' && f.handle) sources.set(f.sourceId, f) + } + if (sources.size === 0) return + + // Kick off all permission requests within the gesture, then process. A rejected + // request (requestReadPermission never rejects, but stay defensive) counts as denied. + const decided = await Promise.all( + [...sources.values()].map((f) => + requestReadPermission(f.handle!).then( + (perm) => ({ f, perm }), + () => ({ f, perm: 'denied' as PermissionState }) + ) + ) + ) + for (const { f, perm } of decided) { + if (perm !== 'granted') continue + try { + await this.#expandFolder(f.handle as FileSystemDirectoryHandle, f.sourceId) + // Children are in — drop the locked placeholder row, then restore a ready + // placeholder if the folder came back empty/all-binary (else dropping the only + // handle-bearing row would unlink the folder and stop it ever refreshing). + this.files = this.files.filter((x) => !(x.sourceId === f.sourceId && x.isFolderRoot)) + this.#ensureFolderRow(f.sourceId, f.folder ?? f.name, f.handle as FileSystemDirectoryHandle) + } catch { + // Enumeration failed (folder moved/deleted on disk): drop any partially-added + // children and keep the placeholder so the chip shows "unavailable". + this.files = this.files.filter((x) => x.sourceId !== f.sourceId || x.isFolderRoot) + this.#patchSource(f.sourceId, { status: 'unavailable' }) + } + } + } + + /** Flush buffered links once the session becomes persistent (first send). */ + async flushPending(): Promise { + this.#persisted = true + if (!this.sessionId) return + const pending = this.#pending + this.#pending = [] + if (pending.length === 0) return + void ensurePersistentStorage() + for (const item of pending) { + try { + await putItem({ ...item, sessionId: this.sessionId }) + } catch (e) { + console.error('Could not persist linked file', e) + } + } + } + + async #persist(item: PersistedAttachedItem): Promise { + if (this.#persisted && this.sessionId) { + void ensurePersistentStorage() + try { + // A QuotaExceededError just means it won't survive a reload — the item + // stays usable for this session. Swallow + log rather than fail the link. + await putItem({ ...item, sessionId: this.sessionId }) + } catch (e) { + console.error('Could not persist linked file (kept for this session)', e) + } + } else { + this.#pending.push(item) + } + } + + async #deleteRecord(sourceId: string): Promise { + this.#pending = this.#pending.filter((p) => p.id !== sourceId) + if (this.#persisted) { + try { + await deleteItem(sourceId) + } catch { + /* ignore */ + } + } + } + + // ------------------------------------------------------------- internals + + /** Identical re-link (same name, or same File identity) → silent no-op. */ + #isDuplicate(desired: string, file: File): boolean { + return this.files.some( + (f) => + // Folder-root placeholders aren't real files — they must not block attaching a + // standalone file that happens to share the folder's name. + !f.isFolderRoot && + (f.name === desired || + // Identical re-drop at the SAME relative path (its row name may have been + // auto-suffixed). Keyed on the path, NOT the basename — otherwise two distinct + // files sharing a basename under different folder subdirs (proj/a/index.ts vs + // proj/b/index.ts) would be wrongly deduped and silently dropped. + ((f.relPath ?? f.name) === desired && + f.size === file.size && + f.file instanceof File && + f.file.lastModified === file.lastModified)) + ) + } + + /** Returns a rejection reason, or undefined if the file may be linked. */ + async #preflight(file: File): Promise { + if (!(await this.#sniffText(file))) return 'Not a text file' + return undefined + } + + async #sniffText(file: Blob): Promise { + try { + return await isTextFile(file) + } catch { + return false + } + } + + #pushIndexing(p: { + name: string + file: File | Blob + folder?: string + sourceId: string + handle?: FileSystemDirectoryHandle + relPath?: string + }): void { + this.files = [ + ...this.files, + { + name: p.name, + file: p.file, + size: p.file.size, + lineIndex: [], + lineCount: 0, + status: 'indexing', + folder: p.folder, + sourceId: p.sourceId, + handle: p.handle, + relPath: p.relPath + } + ] + void this.#indexFile(p.name, p.file) + } + + #pushPlaceholder( + item: PersistedAttachedItem, + status: AttachedFileStatus, + isFolderRoot = false + ): void { + this.files = [ + ...this.files, + { + name: item.name, + file: EMPTY, + size: item.size ?? 0, + lineIndex: [], + lineCount: 0, + status, + folder: item.folder, + sourceId: item.id, + handle: item.handle as FileSystemDirectoryHandle | undefined, + isFolderRoot + } + ] + } + + async #expandFolder(dirHandle: FileSystemDirectoryHandle, sourceId: string): Promise { + const folder = dirHandle.name + const children = await enumerateDir(dirHandle) + for (const { file, path } of children) { + if (!(await this.#sniffText(file))) continue + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle: dirHandle, relPath: path }) + } + this.#ensureFolderRow(sourceId, folder, dirHandle) + } + + /** + * Re-enumerate granted folder handles to reflect on-disk changes since they were + * linked/last refreshed: added/removed/renamed files and content edits. Called on + * each send so the AI sees the folder's current state. Unchanged files are left as-is + * (diffed by relative path + lastModified); only changed files are re-indexed. + */ + async refreshFolders(): Promise { + const sources = new Map() + for (const f of this.files) { + // Include folder-root placeholders (an emptied folder keeps only its placeholder), + // else the source is lost and the folder never re-enumerates again. + if (f.folder && f.handle) { + sources.set(f.sourceId, { handle: f.handle, folder: f.folder }) + } + } + for (const [sourceId, { handle, folder }] of sources) { + try { + if ((await queryReadPermission(handle)) !== 'granted') continue + const children = await enumerateDir(handle) + await this.#reconcileFolder(sourceId, folder, handle, children) + } catch { + this.#patchSource(sourceId, { status: 'unavailable' }) + } + } + } + + async #reconcileFolder( + sourceId: string, + folder: string, + handle: FileSystemDirectoryHandle, + children: { file: File; path: string }[] + ): Promise { + const existing = new Map() + for (const f of this.files) if (f.sourceId === sourceId && f.relPath) existing.set(f.relPath, f) + const seen = new Set() + + for (const { file, path } of children) { + seen.add(path) + const cur = existing.get(path) + if (!cur) { + // newly added on disk + if (!(await this.#sniffText(file))) continue + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle, relPath: path }) + } else { + const curMod = cur.file instanceof File ? cur.file.lastModified : undefined + if (file.size !== cur.size || file.lastModified !== curMod) { + // content changed → re-read + re-index + this.#patch(cur.name, { file, size: file.size, status: 'indexing' }) + void this.#indexFile(cur.name, file) + } + } + } + // removed/renamed-away on disk → drop from memory + const removed = [...existing.values()].filter((f) => f.relPath && !seen.has(f.relPath)) + if (removed.length > 0) { + const names = new Set(removed.map((f) => f.name)) + this.files = this.files.filter((f) => !names.has(f.name)) + } + this.#ensureFolderRow(sourceId, folder, handle) + } + + /** + * Keep a linked folder represented even with no readable children: leave one + * handle-carrying placeholder row so the chip stays visible AND `refreshFolders` + * keeps the live source (without it, an emptied folder vanishes and never + * re-enumerates). Drop the placeholder as soon as real children exist again. + */ + #ensureFolderRow(sourceId: string, folder: string, handle: FileSystemDirectoryHandle): void { + const hasChild = this.files.some((f) => f.sourceId === sourceId && !f.isFolderRoot) + const hasPlaceholder = this.files.some((f) => f.sourceId === sourceId && f.isFolderRoot) + if (!hasChild && !hasPlaceholder) { + this.files = [ + ...this.files, + { + name: folder, + file: EMPTY, + size: 0, + lineIndex: [], + lineCount: 0, + status: 'ready', + folder, + sourceId, + handle, + isFolderRoot: true + } + ] + } else if (hasChild && hasPlaceholder) { + this.files = this.files.filter((f) => !(f.sourceId === sourceId && f.isFolderRoot)) + } + } + + async #indexFile(name: string, file: File | Blob): Promise { + try { + const { lineIndex, lineCount } = await buildLineIndex(file) + this.#patchFile(name, file, { lineIndex, lineCount, status: 'ready' }) + } catch (e) { + this.#patchFile(name, file, { + status: 'error', + error: e instanceof Error ? e.message : String(e) + }) + } + } + + #patch(name: string, changes: Partial): void { + this.files = this.files.map((f) => (f.name === name ? { ...f, ...changes } : f)) + } + /** + * Patch the row for `name` ONLY while it still holds the exact `file` we indexed. + * `buildLineIndex` is async and unawaited; between its start and finish the row's + * file can be swapped (remove + re-add a same-named file, or a folder refresh + * re-indexing an edited file). Without the identity check a stale completion would + * stamp the wrong lineIndex/lineCount on the new file, and read_file would then slice + * the new Blob with the old offsets. + */ + #patchFile(name: string, file: File | Blob, changes: Partial): void { + this.files = this.files.map((f) => + f.name === name && f.file === file ? { ...f, ...changes } : f + ) + } + #patchSource(sourceId: string, changes: Partial): void { + this.files = this.files.map((f) => (f.sourceId === sourceId ? { ...f, ...changes } : f)) + } + + #uniqueName(original: string): string { + // Uniqueness is only among real files — folder-root placeholders may share a name + // with a standalone file and must not push it to a "(2)" suffix. + const taken = (n: string) => this.files.some((f) => f.name === n && !f.isFolderRoot) + if (!taken(original)) return original + const dot = original.lastIndexOf('.') + const base = dot > 0 ? original.slice(0, dot) : original + const ext = dot > 0 ? original.slice(dot) : '' + let n = 2 + let candidate = `${base} (${n})${ext}` + while (taken(candidate)) { + n++ + candidate = `${base} (${n})${ext}` + } + return candidate + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts b/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts new file mode 100644 index 0000000000..c81bc1f9c4 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts @@ -0,0 +1,476 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' + +// Mock persistence + File System Access so we exercise the in-memory store logic. +vi.mock('./attachedFilesDB', () => ({ + putItem: vi.fn(async () => {}), + deleteItem: vi.fn(async () => {}), + getItemsForSession: vi.fn(async () => []), + ensurePersistentStorage: vi.fn(async () => {}) +})) + +const enumerateDirMock = vi.fn<(h: unknown) => Promise<{ file: File; path: string }[]>>() +vi.mock('./fsAccess', () => ({ + enumerateDir: (h: unknown) => enumerateDirMock(h), + isIgnoredPath: (p: string) => + p.split('/').some((s) => s.startsWith('.') || ['node_modules', 'dist', '.git'].includes(s)), + queryReadPermission: vi.fn(async () => 'granted'), + requestReadPermission: vi.fn(async () => 'granted') +})) + +// buildLineIndex is real by default; a single test flips to 'manual' to control +// completion ordering and exercise the stale-index race guard. +type BuildResult = { lineIndex: number[]; lineCount: number } +const buildDeferreds: Array<{ file: Blob; resolve: (r: BuildResult) => void }> = [] +let buildMode: 'real' | 'manual' = 'real' +vi.mock('./fileEngine', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + buildLineIndex: (file: Blob) => + buildMode === 'real' + ? actual.buildLineIndex(file) + : new Promise((resolve) => buildDeferreds.push({ file, resolve })) + } +}) + +import { AttachedFilesStore } from './attachedFiles.svelte' + +function file(name: string, content: string, lastModified = 1): File { + return new File([content], name, { type: 'text/plain', lastModified }) +} + +const dir = { kind: 'directory', name: 'proj' } as unknown as FileSystemDirectoryHandle + +async function settle(store: AttachedFilesStore) { + for (let i = 0; i < 100 && store.list().some((f) => f.status === 'indexing'); i++) { + await new Promise((r) => setTimeout(r, 2)) + } +} + +const names = (store: AttachedFilesStore) => + store + .list() + .map((f) => f.name) + .sort() + +describe('AttachedFilesStore', () => { + let store: AttachedFilesStore + beforeEach(async () => { + store = new AttachedFilesStore() + await store.restore('s1', false) + }) + + it('links and indexes individual files as snapshots', async () => { + await store.addFiles([file('a.txt', 'one\ntwo\n')]) + await settle(store) + const f = store.get('a.txt') + expect(f?.status).toBe('ready') + expect(f?.lineCount).toBe(2) + expect(f?.handle).toBeUndefined() // files never carry a handle + }) + + it('removes a file', async () => { + await store.addFiles([file('a.txt', 'x')]) + store.removeFile('a.txt') + expect(store.count).toBe(0) + }) + + it('links a folder via a directory handle (enumerating it internally)', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('old.ts', 'y\n'), path: 'proj/old.ts' } + ]) + await store.addFolder(dir) + await settle(store) + expect(enumerateDirMock).toHaveBeenCalledWith(dir) + expect(names(store)).toEqual(['proj/app.ts', 'proj/old.ts']) + expect(store.get('proj/app.ts')?.folder).toBe('proj') + }) + + it('refreshFolders detects rename, add, edit, and delete', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n', 1), path: 'proj/app.ts' }, + { file: file('old.ts', 'y\n', 1), path: 'proj/old.ts' } + ]) + await store.addFolder(dir) + await settle(store) + + // On disk: app.ts edited (mtime bumped), old.ts renamed → new.ts, readme.md added. + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\nedited\n', 2), path: 'proj/app.ts' }, + { file: file('new.ts', 'y\n', 1), path: 'proj/new.ts' }, + { file: file('readme.md', '# hi\n', 1), path: 'proj/readme.md' } + ]) + await store.refreshFolders() + await settle(store) + + // old.ts dropped (renamed away); new.ts + readme.md added; app.ts kept. + expect(names(store)).toEqual(['proj/app.ts', 'proj/new.ts', 'proj/readme.md']) + // edited file re-indexed to its new content (2 lines). + expect(store.get('proj/app.ts')?.status).toBe('ready') + expect(store.get('proj/app.ts')?.lineCount).toBe(2) + }) + + it('exposes folders and standalone as structured views', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await store.addFolder(dir) + await store.addFiles([file('solo.txt', 'one\n')]) + await settle(store) + + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + expect(store.folders[0].status).toBe('ready') + expect(store.folders[0].files.map((f) => f.relPath).sort()).toEqual([ + 'proj/app.ts', + 'proj/sub/b.ts' + ]) + expect(store.standalone.map((f) => f.name)).toEqual(['solo.txt']) + expect(store.lockedCount).toBe(0) + }) + + it('a locked folder surfaces as one folder with no files', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + + expect(s2.folders).toEqual([{ name: 'proj', status: 'locked', files: [] }]) + expect(s2.standalone).toEqual([]) + expect(s2.lockedCount).toBe(1) + }) + + it('re-picking a locked folder relinks it instead of silently no-oping', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.folders[0]?.status).toBe('locked') + + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + const result = await s2.addFolder(dir) + await settle(s2) + expect(result.added).toEqual(['proj/app.ts']) + expect(s2.folders).toHaveLength(1) + expect(s2.folders[0].status).toBe('ready') + }) + + it('rejects linking a second folder with the same name (visible, not silent)', async () => { + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await store.addFolder(dir) + await settle(store) + const result = await store.addFolder(dir) + expect(result.added).toEqual([]) + expect(result.rejected[0]?.reason).toMatch(/already linked/) + expect(store.folders).toHaveLength(1) + }) + + it('regrant keeps the folder visible as unavailable when enumeration fails', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.lockedCount).toBe(1) + + // Permission re-granted, but the directory is gone from disk. + enumerateDirMock.mockRejectedValueOnce(new Error('directory removed')) + await s2.regrantLocked() + expect(s2.folders).toEqual([{ name: 'proj', status: 'unavailable', files: [] }]) + }) + + it('regrant of an empty folder keeps it linked and refreshing (not unlinked)', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.lockedCount).toBe(1) + + // Access re-granted, but the folder is currently empty — it must stay linked (ready + // placeholder), not vanish when the locked placeholder is dropped. + enumerateDirMock.mockResolvedValueOnce([]) + await s2.regrantLocked() + expect(s2.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + expect(s2.lockedCount).toBe(0) + + // A file added afterward is picked up — the handle survived. + enumerateDirMock.mockResolvedValueOnce([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s2.refreshFolders() + await settle(s2) + expect(s2.folders[0].files.map((f) => f.relPath)).toEqual(['proj/app.ts']) + }) + + it('removeFolder drops all of a folder’s files', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/b.ts' } + ]) + await store.addFolder(dir) + store.removeFolder('proj') + expect(store.count).toBe(0) + }) + + it('snapshots a folder via addFiles (paths), grouping it and persisting folder + relPath', async () => { + const { putItem } = await import('./attachedFilesDB') + // A persisted (non-transient) session writes through to IndexedDB immediately. + const s = new AttachedFilesStore() + await s.restore('s1', true) + await s.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await settle(s) + expect(s.folders.map((f) => f.name)).toEqual(['proj']) + expect(s.folders[0].files.map((f) => f.relPath).sort()).toEqual(['proj/a.ts', 'proj/sub/b.ts']) + expect(s.standalone).toEqual([]) + const rec = (putItem as ReturnType).mock.calls + .map((c) => c[0]) + .find((r) => r.name === 'proj/a.ts') + expect(rec).toMatchObject({ kind: 'snapshot', folder: 'proj', relPath: 'proj/a.ts' }) + }) + + it('keeps same-basename files from different folder subdirs (dedup by path, not basename)', async () => { + // Two distinct files with the same basename, size and lastModified, different subdirs. + const res = await store.addFiles([ + { file: file('index.ts', 'a\n', 5), path: 'proj/a/index.ts' }, + { file: file('index.ts', 'a\n', 5), path: 'proj/b/index.ts' } + ]) + await settle(store) + expect(res.added.sort()).toEqual(['proj/a/index.ts', 'proj/b/index.ts']) + expect(store.folders[0].files.map((f) => f.relPath).sort()).toEqual([ + 'proj/a/index.ts', + 'proj/b/index.ts' + ]) + }) + + it('skips junk paths (node_modules/.git/dotfiles) inside a snapshotted folder', async () => { + const res = await store.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('dep.js', 'z\n'), path: 'proj/node_modules/dep.js' }, + { file: file('cfg', 'w\n'), path: 'proj/.git/config' } + ]) + await settle(store) + expect(res.added).toEqual(['proj/a.ts']) + expect(store.folders[0].files).toHaveLength(1) + }) + + it('keeps an explicitly attached standalone dotfile (filter is folder-only)', async () => { + const res = await store.addFiles([file('.env', 'SECRET=1\n')]) + await settle(store) + expect(res.added).toEqual(['.env']) + expect(store.standalone.map((f) => f.name)).toEqual(['.env']) + }) + + it('restores a snapshot folder grouped from its persisted folder/relPath', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 's-a', + sessionId: 's1', + kind: 'snapshot', + name: 'proj/a.ts', + folder: 'proj', + relPath: 'proj/a.ts', + blob: file('a.ts', 'x\n'), + addedAt: 0 + }, + { + id: 's-b', + sessionId: 's1', + kind: 'snapshot', + name: 'proj/b.ts', + folder: 'proj', + relPath: 'proj/b.ts', + blob: file('b.ts', 'y\n'), + addedAt: 0 + } + ]) + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + await settle(s2) + expect(s2.folders.map((f) => f.name)).toEqual(['proj']) + expect(s2.folders[0].files).toHaveLength(2) + expect(s2.standalone).toEqual([]) + }) + + it('imposes no file-count cap on a folder', async () => { + enumerateDirMock.mockResolvedValue( + Array.from({ length: 150 }, (_, i) => ({ + file: file(`f${i}.ts`, 'x\n'), + path: `proj/f${i}.ts` + })) + ) + await store.addFolder(dir) + await settle(store) + expect(store.folders[0].files.length).toBe(150) + }) + + it('removeFolder deletes every snapshot record from storage (persisted session)', async () => { + const { deleteItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + await s.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await settle(s) + const ids = s + .list() + .filter((f) => f.folder === 'proj') + .map((f) => f.sourceId) + expect(ids.length).toBe(2) + ;(deleteItem as ReturnType).mockClear() + s.removeFolder('proj') + expect(s.count).toBe(0) + const deleted = (deleteItem as ReturnType).mock.calls.map((c) => c[0]) + for (const id of ids) expect(deleted).toContain(id) + }) + + it('a stale index completion does not corrupt a re-added same-named file', async () => { + buildMode = 'manual' + try { + const A = file('a.txt', 'AAA\n') + const B = file('a.txt', 'BBB\nBBB\nBBB\n') + await store.addFiles([A]) // row 'a.txt' (file A) → buildLineIndex(A) pending + store.removeFile('a.txt') + await store.addFiles([B]) // new row 'a.txt' (file B) → buildLineIndex(B) pending + + // The old (stale) index for A resolves last — it must NOT touch the row now holding B. + buildDeferreds.find((d) => d.file === A)!.resolve({ lineIndex: [0], lineCount: 99 }) + await Promise.resolve() + expect(store.get('a.txt')?.status).toBe('indexing') + expect(store.get('a.txt')?.lineCount).not.toBe(99) + + // B's own index applies normally. + buildDeferreds.find((d) => d.file === B)!.resolve({ lineIndex: [0, 4, 8], lineCount: 3 }) + await Promise.resolve() + expect(store.get('a.txt')?.status).toBe('ready') + expect(store.get('a.txt')?.lineCount).toBe(3) + } finally { + buildMode = 'real' + buildDeferreds.length = 0 + } + }) + + it('removeFolder deletes the live folder record from storage (persisted session)', async () => { + const { deleteItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s.addFolder(dir) + await settle(s) + const sourceId = s.list().find((f) => f.folder === 'proj')?.sourceId + ;(deleteItem as ReturnType).mockClear() + s.removeFolder('proj') + expect(s.count).toBe(0) + expect((deleteItem as ReturnType).mock.calls.map((c) => c[0])).toContain(sourceId) + }) + + it('an emptied live folder stays visible and refreshes when files return', async () => { + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await store.addFolder(dir) + await settle(store) + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + + // Folder emptied on disk → the last child is removed but the folder persists (placeholder). + enumerateDirMock.mockResolvedValue([]) + await store.refreshFolders() + await settle(store) + expect(store.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + expect(store.get('proj/app.ts')).toBeUndefined() + expect(store.readyFiles()).toEqual([]) // placeholder is never a tool target + + // A file added back on disk is picked up — the live source survived the empty state. + enumerateDirMock.mockResolvedValue([{ file: file('new.ts', 'y\n'), path: 'proj/new.ts' }]) + await store.refreshFolders() + await settle(store) + expect(store.folders[0].files.map((f) => f.relPath)).toEqual(['proj/new.ts']) + }) + + it('an empty-folder placeholder does not collide with a same-named standalone file', async () => { + enumerateDirMock.mockResolvedValue([]) // empty folder "proj" → creates a placeholder named "proj" + await store.addFolder(dir) + await settle(store) + + // A standalone file literally named "proj" must NOT be deduped by the placeholder. + const res = await store.addFiles([file('proj', 'hello\n')]) + await settle(store) + expect(res.added).toEqual(['proj']) + expect(store.standalone.map((f) => f.name)).toEqual(['proj']) + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + + // Removing that standalone leaves the folder's placeholder intact. + store.removeFile('proj') + expect(store.standalone).toEqual([]) + expect(store.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + }) + + it('links an initially empty live folder (kept visible, persisted, refreshes)', async () => { + const { putItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + enumerateDirMock.mockResolvedValue([]) // folder is empty at link time + const res = await s.addFolder(dir) + await settle(s) + expect(res.added).toEqual([]) + expect(s.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + // persisted as a dir-handle so it survives a reload despite being empty + const persisted = (putItem as ReturnType).mock.calls.map((c) => c[0]) + expect(persisted.some((r) => r.kind === 'dir-handle' && r.folder === 'proj')).toBe(true) + + // a file added later is picked up — the source existed from the start. + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s.refreshFolders() + await settle(s) + expect(s.folders[0].files.map((f) => f.relPath)).toEqual(['proj/app.ts']) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts new file mode 100644 index 0000000000..9673e29edc --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts @@ -0,0 +1,28 @@ +import { describe, expect, it } from 'vitest' +import { + getItemsForSession, + putItem, + deleteItem, + deleteItemsForSession, + ensurePersistentStorage +} from './attachedFilesDB' + +// IndexedDB is unavailable in the node test env. The module must degrade gracefully +// (open fails → reads return [], writes/deletes are no-ops) rather than throwing. +describe('attachedFilesDB without IndexedDB', () => { + it('returns [] for reads', async () => { + expect(await getItemsForSession('s1')).toEqual([]) + }) + + it('does not throw on writes/deletes', async () => { + await expect( + putItem({ id: 'a', sessionId: 's1', kind: 'snapshot', name: 'x.txt', addedAt: 0 }) + ).resolves.toBeUndefined() + await expect(deleteItem('a')).resolves.toBeUndefined() + await expect(deleteItemsForSession('s1')).resolves.toBeUndefined() + }) + + it('does not throw when requesting persistent storage', async () => { + await expect(ensurePersistentStorage()).resolves.toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts new file mode 100644 index 0000000000..36f9ac4fbe --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts @@ -0,0 +1,118 @@ +/** + * IndexedDB persistence for AI-chat linked files, keyed by session id. + * + * Two kinds of records survive a reload (see the persistence plan): + * - handle records ('file-handle' / 'dir-handle'): a re-grantable File System + * Access handle is stored (structured-clone), re-read live on restore. + * - 'snapshot' records: a full-byte Blob copy (fallback when the File System + * Access API is unavailable). + * + * Mirrors the `idb` usage in HistoryManager.svelte.ts. + */ +import { openDB, type DBSchema as IDBSchema, type IDBPDatabase } from 'idb' + +export type AttachedItemKind = 'snapshot' | 'dir-handle' + +export interface PersistedAttachedItem { + /** Stable record id. */ + id: string + sessionId: string + /** 'snapshot' = a file copied into IndexedDB; 'dir-handle' = a live folder handle. */ + kind: AttachedItemKind + /** Display name: relative path for files, folder name for dir-handle records. */ + name: string + /** Top-level folder (for grouping); equals `name` for dir-handle records. */ + folder?: string + /** Folder-relative path (snapshot folder children) — restores the folder grouping/tree. */ + relPath?: string + /** Live directory handle (for 'dir-handle'). */ + handle?: FileSystemDirectoryHandle + /** Full-content copy (for 'snapshot'). */ + blob?: Blob + size?: number + lastModified?: number + addedAt: number +} + +interface AttachedFilesSchema extends IDBSchema { + items: { + key: string + value: PersistedAttachedItem + indexes: { 'by-session': string } + } +} + +let dbPromise: Promise | undefined> | undefined + +function getDB(): Promise | undefined> { + if (!dbPromise) { + try { + dbPromise = openDB('copilot-attached-files', 1, { + upgrade(db) { + if (!db.objectStoreNames.contains('items')) { + const store = db.createObjectStore('items', { keyPath: 'id' }) + store.createIndex('by-session', 'sessionId') + } + } + }).catch((err) => { + console.error('Could not open attached-files database', err) + return undefined + }) + } catch (err) { + // IndexedDB unavailable (e.g. private mode / no DOM) — degrade gracefully. + console.error('Could not open attached-files database', err) + dbPromise = Promise.resolve(undefined) + } + } + return dbPromise +} + +export async function putItem(item: PersistedAttachedItem): Promise { + const db = await getDB() + await db?.put('items', item) +} + +export async function getItemsForSession(sessionId: string): Promise { + const db = await getDB() + if (!db) return [] + try { + return await db.getAllFromIndex('items', 'by-session', sessionId) + } catch (err) { + console.error('Could not read attached files', err) + return [] + } +} + +export async function deleteItem(id: string): Promise { + const db = await getDB() + await db?.delete('items', id) +} + +export async function deleteItemsForSession(sessionId: string): Promise { + const db = await getDB() + if (!db) return + try { + const tx = db.transaction('items', 'readwrite') + const index = tx.store.index('by-session') + let cursor = await index.openCursor(sessionId) + while (cursor) { + await cursor.delete() + cursor = await cursor.continue() + } + await tx.done + } catch (err) { + console.error('Could not delete attached files for session', err) + } +} + +/** Ask the browser to keep our storage from being evicted (best-effort, once). */ +let persistRequested = false +export async function ensurePersistentStorage(): Promise { + if (persistRequested) return + persistRequested = true + try { + await navigator.storage?.persist?.() + } catch { + // best-effort; ignore + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts b/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts new file mode 100644 index 0000000000..8ef6d5b827 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts @@ -0,0 +1,286 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + buildLineIndex, + readFile, + searchFiles, + searchFilesInWorker, + isTextFile, + numberLines, + type FileEntry +} from './fileEngine' + +function makeFile(content: string | Uint8Array, name = 'f.txt'): File { + return new File([content as BlobPart], name) +} + +async function makeEntry(content: string, name = 'f.txt'): Promise { + const file = makeFile(content, name) + const { lineIndex, lineCount } = await buildLineIndex(file) + return { name, file, lineIndex, lineCount } +} + +describe('buildLineIndex', () => { + it('counts lines without a trailing newline', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\nb')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 2]) + }) + + it('does not count a single trailing newline as an extra line', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\nb\n')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 2]) + }) + + it('handles an empty file', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('')) + expect(lineCount).toBe(0) + expect(lineIndex).toEqual([]) + }) + + it('handles CRLF line endings (offsets by byte)', async () => { + // bytes: a=0 \r=1 \n=2 b=3 → line starts at 0 and 3 + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\r\nb')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 3]) + }) + + it('counts lines correctly across stream chunk boundaries', async () => { + const lines = Array.from({ length: 5000 }, (_, i) => `line ${i}`) + const { lineCount } = await buildLineIndex(makeFile(lines.join('\n'))) + expect(lineCount).toBe(5000) + }) +}) + +describe('readFile', () => { + it('reads a bounded window and reports pagination', async () => { + const entry = await makeEntry('l1\nl2\nl3') + const res = await readFile(entry, { startLine: 1, endLine: 2 }) + expect(res.text).toBe('l1\nl2\n') + expect(res.startLine).toBe(1) + expect(res.endLine).toBe(2) + expect(res.totalLines).toBe(3) + expect(res.truncated).toBe(true) + expect(res.note).toContain('start_line=3') + }) + + it('reads the final line to end of file', async () => { + const entry = await makeEntry('l1\nl2\nl3') + const res = await readFile(entry, { startLine: 3 }) + expect(res.text).toBe('l3') + expect(res.truncated).toBe(false) + }) + + it('clamps the window to maxLines', async () => { + const entry = await makeEntry(Array.from({ length: 100 }, (_, i) => `l${i}`).join('\n')) + const res = await readFile(entry, { startLine: 1, maxLines: 10 }) + expect(res.endLine).toBe(10) + expect(res.truncated).toBe(true) + }) + + it('char-caps a degenerate single long line', async () => { + const entry = await makeEntry('x'.repeat(10000)) + const res = await readFile(entry, { maxChars: 8000 }) + expect(res.text.length).toBe(8000) + expect(res.truncated).toBe(true) + expect(res.note).toContain('8000 characters') + }) + + it('bounds the byte decode on a newline-sparse window (never reads past maxChars*4 bytes)', async () => { + // One 200k-char line; with maxChars=1000 only ≤4000 bytes are sliced before decode. + const entry = await makeEntry('y'.repeat(200_000)) + const res = await readFile(entry, { maxChars: 1000 }) + expect(res.text).toBe('y'.repeat(1000)) + expect(res.truncated).toBe(true) + }) + + it('clamps a start line beyond the end', async () => { + const entry = await makeEntry('l1\nl2') + const res = await readFile(entry, { startLine: 99 }) + expect(res.startLine).toBe(2) + expect(res.text).toBe('l2') + }) + + it('returns empty for an empty file', async () => { + const entry = await makeEntry('') + const res = await readFile(entry) + expect(res.text).toBe('') + expect(res.totalLines).toBe(0) + }) + + it('char-truncation inside the first line resumes the note at the next line', async () => { + // Line 1 exceeds maxChars; lines 2-3 follow. The window only returns line 1's prefix, + // so the note must report line 1 and point the model at line 2 (not claim lines 1-3). + const entry = await makeEntry('x'.repeat(20000) + '\nl2\nl3') + const res = await readFile(entry, { startLine: 1, endLine: 3, maxChars: 5000 }) + expect(res.endLine).toBe(1) + expect(res.totalLines).toBe(3) + expect(res.truncated).toBe(true) + expect(res.note).toContain('start_line=2') + }) + + it('char-truncation after some whole lines resumes at the truncated line', async () => { + const entry = await makeEntry('a\nb\n' + 'x'.repeat(20000) + '\nd') + const res = await readFile(entry, { startLine: 1, endLine: 4, maxChars: 5000 }) + expect(res.endLine).toBe(2) // a, b whole; line 3 (xxx) cut + expect(res.note).toContain('start_line=3') + // the partial line 3 must NOT leak into the body — it would contradict the note + expect(res.text).toBe('a\nb\n') + expect(numberLines(res.text, res.startLine)).toBe('1→a\n2→b') + }) +}) + +describe('searchFiles', () => { + it('finds matches with 1-based line numbers', async () => { + const entry = await makeEntry('alpha\nbeta\ngamma beta') + const res = await searchFiles([entry], 'beta') + expect(res.error).toBeUndefined() + expect(res.hits).toEqual([ + { file: 'f.txt', line: 2, text: 'beta' }, + { file: 'f.txt', line: 3, text: 'gamma beta' } + ]) + }) + + it('searches across multiple files', async () => { + const a = await makeEntry('needle here', 'a.txt') + const b = await makeEntry('nope\nneedle', 'b.txt') + const res = await searchFiles([a, b], 'needle') + expect(res.hits.map((h) => `${h.file}:${h.line}`)).toEqual(['a.txt:1', 'b.txt:2']) + }) + + it('restricts to a single file with pathFilter', async () => { + const a = await makeEntry('needle', 'a.txt') + const b = await makeEntry('needle', 'b.txt') + const res = await searchFiles([a, b], 'needle', { pathFilter: 'b.txt' }) + expect(res.hits).toEqual([{ file: 'b.txt', line: 1, text: 'needle' }]) + }) + + it('reports an unknown pathFilter as an error', async () => { + const a = await makeEntry('needle', 'a.txt') + const res = await searchFiles([a], 'needle', { pathFilter: 'missing.txt' }) + expect(res.error).toContain('missing.txt') + }) + + it('truncates at maxHits', async () => { + const entry = await makeEntry(Array.from({ length: 10 }, () => 'match').join('\n')) + const res = await searchFiles([entry], 'match', { maxHits: 3 }) + expect(res.hits.length).toBe(3) + expect(res.truncated).toBe(true) + }) + + it('supports case-insensitive flags', async () => { + const entry = await makeEntry('Hello\nworld') + const res = await searchFiles([entry], 'hello', { flags: 'i' }) + expect(res.hits).toEqual([{ file: 'f.txt', line: 1, text: 'Hello' }]) + }) + + it('strips trailing CR from matched CRLF lines', async () => { + const entry = await makeEntry('foo\r\nbar') + const res = await searchFiles([entry], 'foo') + expect(res.hits).toEqual([{ file: 'f.txt', line: 1, text: 'foo' }]) + }) + + it('returns a friendly error for an invalid regex', async () => { + const entry = await makeEntry('anything') + const res = await searchFiles([entry], '(') + expect(res.error).toContain('Invalid regex') + expect(res.hits).toEqual([]) + }) + + it('matches across stream chunk boundaries', async () => { + const lines = Array.from({ length: 5000 }, (_, i) => (i === 4999 ? 'TARGET' : `line ${i}`)) + const entry = await makeEntry(lines.join('\n')) + const res = await searchFiles([entry], 'TARGET') + expect(res.hits).toEqual([{ file: 'f.txt', line: 5000, text: 'TARGET' }]) + }) + + it('a global flag does not drop matches via a stale lastIndex', async () => { + // `.test()` is stateful under the `g` flag; without resetting lastIndex, lines after + // the first match would be tested from a stale offset and silently miss. + const entry = await makeEntry('match\nmatch\nmatch') + const res = await searchFiles([entry], 'match', { flags: 'g' }) + expect(res.hits.map((h) => h.line)).toEqual([1, 2, 3]) + }) +}) + +describe('searchFilesInWorker', () => { + // A controllable stand-in for the search Worker — `reply`, `error`, or `hang` (never responds). + class MockWorker { + onmessage: ((e: MessageEvent) => void) | null = null + onerror: ((e: unknown) => void) | null = null + static mode: 'reply' | 'error' | 'hang' = 'reply' + static reply: unknown = { hits: [], truncated: false } + static terminated = false + constructor(_url: URL | string, _opts?: unknown) {} + postMessage(): void { + if (MockWorker.mode === 'reply') + queueMicrotask(() => this.onmessage?.({ data: MockWorker.reply } as MessageEvent)) + else if (MockWorker.mode === 'error') queueMicrotask(() => this.onerror?.({})) + // 'hang' → never responds, exercising the timeout path. + } + terminate(): void { + MockWorker.terminated = true + } + } + + beforeEach(() => { + MockWorker.terminated = false + vi.stubGlobal('Worker', MockWorker) + }) + afterEach(() => vi.unstubAllGlobals()) + + const entry: FileEntry = { name: 'a.txt', file: makeFile('x'), lineIndex: [], lineCount: 0 } + + it('resolves with the worker result and terminates the worker', async () => { + MockWorker.mode = 'reply' + MockWorker.reply = { hits: [{ file: 'a.txt', line: 1, text: 'x' }], truncated: false } + const res = await searchFilesInWorker([entry], 'x') + expect(res.hits).toEqual([{ file: 'a.txt', line: 1, text: 'x' }]) + expect(MockWorker.terminated).toBe(true) + }) + + it('times out on a non-responding (pathological) pattern and terminates the worker', async () => { + MockWorker.mode = 'hang' + const res = await searchFilesInWorker([entry], '^(a+)+$', {}, 20) + expect(res.error).toContain('timed out') + expect(MockWorker.terminated).toBe(true) + }) +}) + +describe('isTextFile', () => { + it('accepts UTF-8 text', async () => { + expect(await isTextFile(makeFile('hello © world'))).toBe(true) + }) + + it('accepts an empty file', async () => { + expect(await isTextFile(makeFile(''))).toBe(true) + }) + + it('rejects content with NUL bytes', async () => { + expect(await isTextFile(makeFile(new Uint8Array([104, 0, 105]), 'b.bin'))).toBe(false) + }) +}) + +describe('numberLines', () => { + it('prefixes each line with its absolute 1-based number', () => { + expect(numberLines('l3\nl4\n', 3)).toBe('3→l3\n4→l4') + }) + + it('right-aligns numbers to a common width', () => { + expect(numberLines('a\nb', 9)).toBe(' 9→a\n10→b') + }) + + it('numbers a final line that has no trailing newline', () => { + expect(numberLines('only', 5)).toBe('5→only') + }) + + it('matches a readFile window (numbers the returned lines, no phantom line)', async () => { + const file = makeFile('l1\nl2\nl3') + const { lineIndex, lineCount } = await buildLineIndex(file) + const res = await readFile( + { name: 'f.txt', file, lineIndex, lineCount }, + { startLine: 1, endLine: 2 } + ) + expect(numberLines(res.text, res.startLine)).toBe('1→l1\n2→l2') + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fileEngine.ts b/frontend/src/lib/components/copilot/chat/files/fileEngine.ts new file mode 100644 index 0000000000..5dbceb77e2 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileEngine.ts @@ -0,0 +1,430 @@ +/** + * Storage-agnostic streaming engine for reading and searching attached files. + * + * Files are kept as `File` handles (lazy references to bytes on disk). Nothing is + * decoded into the JS heap wholesale: we stream in chunks, so a large file never + * freezes the tab or blows up memory. The only per-file state held in RAM is a + * line-offset index (a flat array of byte offsets, ~8 bytes per line). + * + * Line semantics match `String.split('\n')` except a single trailing newline does + * NOT add an empty final line (so "a\nb\n" is 2 lines, like `wc -l`). Lines are + * 1-based in the public read/search API. + */ + +/** Minimal shape the engine needs. The attached-files store extends this with reactive status. */ +export interface FileEntry { + name: string + /** A File (live link) or a Blob (restored snapshot) — both stream/slice identically. */ + file: File | Blob + lineIndex: number[] + lineCount: number +} + +const CHUNK_NEWLINE = 0x0a // '\n' — in UTF-8 this byte never appears inside a multibyte sequence + +export const DEFAULT_READ_MAX_LINES = 200 +export const DEFAULT_READ_MAX_CHARS = 8000 +export const DEFAULT_SEARCH_MAX_HITS = 50 +/** + * Per-line cap on how much of a degenerate long line the regex is tested against. + * This bounds work for linear-time patterns; it does NOT prevent catastrophic + * backtracking — a nested-quantifier pattern can still go exponential within the + * capped prefix (search runs on the main thread, so that is a self-DoS of the tab). + */ +export const DEFAULT_SEARCH_LINE_SCAN_CAP = 100_000 +/** How much of a matching line we echo back, to keep search results bounded. */ +export const DEFAULT_SEARCH_LINE_ECHO_CAP = 500 + +/** + * Stream the file once and record the byte offset at which each line starts. + * Scans raw bytes for '\n' (no decode needed — 0x0A is unambiguous in UTF-8). + */ +export async function buildLineIndex( + file: Blob +): Promise<{ lineIndex: number[]; lineCount: number }> { + const fileSize = file.size + if (fileSize === 0) { + return { lineIndex: [], lineCount: 0 } + } + + const lineIndex: number[] = [0] + let offset = 0 + const reader = file.stream().getReader() + try { + while (true) { + const { done, value } = await reader.read() + if (done) break + const chunk = value as Uint8Array + for (let i = 0; i < chunk.length; i++) { + if (chunk[i] === CHUNK_NEWLINE) { + lineIndex.push(offset + i + 1) + } + } + offset += chunk.length + } + } finally { + reader.releaseLock() + } + + // A trailing newline points one past the end (a phantom empty line) — drop it. + if (lineIndex.length > 1 && lineIndex[lineIndex.length - 1] === fileSize) { + lineIndex.pop() + } + + return { lineIndex, lineCount: lineIndex.length } +} + +export interface ReadResult { + text: string + startLine: number + endLine: number + totalLines: number + truncated: boolean + note: string +} + +/** + * Read a bounded window of lines, reading only the relevant byte range from disk. + * Clamps the window to `maxLines` and the returned text to `maxChars` (protects + * against degenerate single-line files). Returns a self-describing pagination note. + */ +export async function readFile( + entry: FileEntry, + opts: { + startLine?: number + endLine?: number + maxLines?: number + maxChars?: number + } = {} +): Promise { + const totalLines = entry.lineCount + const maxLines = opts.maxLines ?? DEFAULT_READ_MAX_LINES + const maxChars = opts.maxChars ?? DEFAULT_READ_MAX_CHARS + + if (totalLines === 0) { + return { + text: '', + startLine: 0, + endLine: 0, + totalLines: 0, + truncated: false, + note: 'File is empty.' + } + } + + let start = opts.startLine ?? 1 + if (start < 1) start = 1 + if (start > totalLines) start = totalLines + + const requestedEnd = opts.endLine ?? start + maxLines - 1 + let end = requestedEnd + if (end < start) end = start + const cappedByLines = end - start + 1 > maxLines + if (cappedByLines) end = start + maxLines - 1 + if (end > totalLines) end = totalLines + + const byteStart = entry.lineIndex[start - 1] + const byteEnd = end < totalLines ? entry.lineIndex[end] : entry.file.size + // Bound the decode for newline-sparse files (minified JS, single-line JSONL): the + // window can span the whole file, but we only ever return maxChars characters, and + // a UTF-8 character is at most 4 bytes — so never materialize more than that. + const byteCap = byteStart + maxChars * 4 + const byteCapped = byteCap < byteEnd + + let text: string + try { + text = await entry.file.slice(byteStart, byteCapped ? byteCap : byteEnd).text() + } catch (e) { + throw new FileReadError(entry.name, e instanceof Error ? e.message : String(e)) + } + + let cappedByChars = byteCapped + if (text.length > maxChars) { + text = text.slice(0, maxChars) + cappedByChars = true + } + + // When the char cap truncates the window short of `end`, the text holds fewer lines + // than requested — so the note must report the last line actually returned and resume + // at the next unread one (otherwise it claims lines it didn't return and skips them). + let lastLine = end + let resumeAt: number | undefined = end < totalLines ? end + 1 : undefined + if (cappedByChars) { + const completeLines = (text.match(/\n/g) || []).length + if (completeLines >= 1) { + // lines start..start+completeLines-1 are whole; the next line was cut mid-content. + // Trim that partial line off the returned text so the body matches the note (and + // the model doesn't see a line the note says it'll get on the next read). + lastLine = start + completeLines - 1 + resumeAt = start + completeLines + text = text.slice(0, text.lastIndexOf('\n') + 1) + } else { + // the cap fell inside line `start` itself — it can't be returned in full, so + // advance past it rather than re-truncating the same line forever. + lastLine = start + resumeAt = start + 1 + } + if (resumeAt > totalLines) resumeAt = undefined + } + + const truncated = cappedByChars || resumeAt !== undefined + + let note = `Showing lines ${start}-${lastLine} of ${totalLines}.` + if (cappedByChars) { + note += ` Output truncated to ${maxChars} characters (line(s) very long).` + } + if (resumeAt !== undefined) { + note += ` Call read_file again with start_line=${resumeAt} for more.` + } + + return { text, startLine: start, endLine: lastLine, totalLines, truncated, note } +} + +/** + * Prefix each line of a read window with its absolute 1-based number (`→`), + * so the model can quote/reference exact lines. `startLine` is the window's first line. + */ +export function numberLines(text: string, startLine: number): string { + const lines = text.split('\n') + // readFile's window ends with the trailing newline of its last line when more lines + // follow, so split yields a phantom empty element — drop it before numbering. + if (lines.length > 1 && lines[lines.length - 1] === '') lines.pop() + const width = String(startLine + lines.length - 1).length + return lines.map((l, i) => `${String(startLine + i).padStart(width)}→${l}`).join('\n') +} + +export interface SearchHit { + file: string + line: number + text: string +} + +export interface SearchResult { + hits: SearchHit[] + truncated: boolean + error?: string +} + +/** + * Run a regex across one or more files, streaming each (no full-file load), and + * return matching lines with 1-based line numbers. Stops at `maxHits`. + */ +export async function searchFiles( + entries: FileEntry[], + pattern: string, + opts: { + flags?: string + pathFilter?: string + maxHits?: number + lineScanCap?: number + lineEchoCap?: number + } = {} +): Promise { + const maxHits = opts.maxHits ?? DEFAULT_SEARCH_MAX_HITS + const lineScanCap = opts.lineScanCap ?? DEFAULT_SEARCH_LINE_SCAN_CAP + const lineEchoCap = opts.lineEchoCap ?? DEFAULT_SEARCH_LINE_ECHO_CAP + + let regex: RegExp + try { + regex = new RegExp(pattern, opts.flags ?? '') + } catch (e) { + return { + hits: [], + truncated: false, + error: `Invalid regex: ${e instanceof Error ? e.message : String(e)}` + } + } + + const targets = opts.pathFilter ? entries.filter((e) => e.name === opts.pathFilter) : entries + if (opts.pathFilter && targets.length === 0) { + return { hits: [], truncated: false, error: `No attached file named "${opts.pathFilter}".` } + } + + const hits: SearchHit[] = [] + let truncated = false + + for (const entry of targets) { + if (hits.length >= maxHits) { + truncated = true + break + } + try { + await streamLines(entry.file, (line, lineNo) => { + // Bound backtracking on pathological long lines by only testing a prefix. + const scanned = line.length > lineScanCap ? line.slice(0, lineScanCap) : line + // `regex` may carry a caller-supplied `g`/`y` flag, which makes `.test()` + // stateful (it advances `lastIndex`) — reset so each line matches from 0. + regex.lastIndex = 0 + if (regex.test(scanned)) { + hits.push({ + file: entry.name, + line: lineNo, + text: line.length > lineEchoCap ? line.slice(0, lineEchoCap) + '…' : line + }) + } + return hits.length < maxHits // continue? + }) + } catch (e) { + return { + hits, + truncated, + error: `Error reading "${entry.name}": ${e instanceof Error ? e.message : String(e)}` + } + } + if (hits.length >= maxHits) { + truncated = true + break + } + } + + return { hits, truncated } +} + +/** + * Run `searchFiles` off the main thread. A model-supplied regex can backtrack + * catastrophically (e.g. /^(a+)+$/) and `regex.test()` can't be interrupted — so we run + * it in a Worker and `terminate()` it on a timeout, keeping the tab responsive instead of + * frozen. Falls back to a main-thread search where Workers aren't available (best effort). + */ +export function searchFilesInWorker( + entries: FileEntry[], + pattern: string, + opts: { flags?: string; pathFilter?: string; maxHits?: number } = {}, + timeoutMs = 3000 +): Promise { + let worker: Worker + try { + worker = new Worker(new URL('./searchWorker.ts', import.meta.url), { type: 'module' }) + } catch { + return searchFiles(entries, pattern, opts) + } + return new Promise((resolve) => { + const finish = (r: SearchResult) => { + clearTimeout(timer) + worker.terminate() + resolve(r) + } + const timer = setTimeout( + () => + finish({ + hits: [], + truncated: false, + error: 'Search timed out — the pattern is too expensive. Try a simpler regex.' + }), + timeoutMs + ) + worker.onmessage = (e: MessageEvent) => finish(e.data) + worker.onerror = () => { + // Worker script failed to load/run — fall back to a main-thread search. + clearTimeout(timer) + worker.terminate() + searchFiles(entries, pattern, opts).then(resolve) + } + worker.postMessage({ + files: entries.map((e) => ({ name: e.name, file: e.file })), + pattern, + flags: opts.flags, + pathFilter: opts.pathFilter, + maxHits: opts.maxHits + }) + }) +} + +export class FileReadError extends Error { + constructor( + public fileName: string, + message: string + ) { + super(message) + this.name = 'FileReadError' + } +} + +/** + * Max characters buffered for a single line while streaming. A newline-less file + * (e.g. minified JS) would otherwise accumulate wholesale in `buffer`; past this + * cap excess characters are dropped (the line's intact prefix is preserved) — + * harmless for search, which only tests/echoes a prefix far smaller than this. + */ +const MAX_LINE_BUFFER_CHARS = 1_000_000 + +/** + * Stream a file and invoke `onLine` for each line (1-based). A trailing newline + * does not produce an empty final line. `onLine` returns false to stop early. + * A trailing '\r' (CRLF) is stripped before the callback. Overlong lines are + * passed with at least their first MAX_LINE_BUFFER_CHARS characters intact; + * content beyond the cap may be dropped. + */ +async function streamLines( + file: Blob, + onLine: (line: string, lineNo: number) => boolean +): Promise { + const reader = file.stream().getReader() + const decoder = new TextDecoder('utf-8') + let buffer = '' + let lineNo = 0 + try { + while (true) { + const { done, value } = await reader.read() + if (done) { + buffer += decoder.decode() + break + } + buffer += decoder.decode(value as Uint8Array, { stream: true }) + let start = 0 + let nlIdx: number + while ((nlIdx = buffer.indexOf('\n', start)) !== -1) { + let line = buffer.slice(start, nlIdx) + if (line.endsWith('\r')) line = line.slice(0, -1) + start = nlIdx + 1 + lineNo++ + if (!onLine(line, lineNo)) return + } + buffer = buffer.slice(start) + // The remainder holds no newline — cap how much of an overlong line we keep. + // Dropped characters are line content only, so newline detection and line + // numbering in later chunks are unaffected. + if (buffer.length > MAX_LINE_BUFFER_CHARS) { + buffer = buffer.slice(0, MAX_LINE_BUFFER_CHARS) + } + } + if (buffer.length > 0) { + let line = buffer + if (line.endsWith('\r')) line = line.slice(0, -1) + lineNo++ + onLine(line, lineNo) + } + } finally { + reader.releaseLock() + } +} + +/** + * Sniff the first bytes of a file to decide whether it is text (UTF-8 decodable, + * no NUL bytes). Used to reject binary files at attach time. + */ +export async function isTextFile(file: Blob, sampleBytes = 8192): Promise { + if (file.size === 0) return true + const slice = file.slice(0, Math.min(sampleBytes, file.size)) + const buf = new Uint8Array(await slice.arrayBuffer()) + for (let i = 0; i < buf.length; i++) { + if (buf[i] === 0) return false // NUL byte → binary + } + try { + // `fatal` throws on invalid UTF-8. We may cut a multibyte char at the sample + // boundary, so only treat it as binary if the error is not at the very end. + new TextDecoder('utf-8', { fatal: true }).decode(buf) + return true + } catch { + // Could be a truncated trailing multibyte sequence — retry on a trimmed buffer. + if (buf.length >= 4) { + try { + new TextDecoder('utf-8', { fatal: true }).decode(buf.slice(0, buf.length - 3)) + return true + } catch { + return false + } + } + return false + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts b/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts new file mode 100644 index 0000000000..0bd45153af --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts @@ -0,0 +1,66 @@ +import { describe, expect, it, vi } from 'vitest' + +// '../shared' transitively imports monaco (CSS) which the node test env can't load. +// fileTools only needs createToolDef from it (called at module load), so stub it. +vi.mock('../shared', () => ({ + createToolDef: (_schema: unknown, name: string, description: string) => ({ name, description }) +})) + +import { searchFilesTool } from './fileTools' +import type { AttachedFile, AttachedFilesStore } from './attachedFiles.svelte' + +/** Minimal store stub: searchFilesTool's empty-ready path only reads count/readyFiles/list. */ +function fakeStore(rows: Array>): AttachedFilesStore { + const files = rows as AttachedFile[] + return { + get count() { + return files.length + }, + readyFiles: () => files.filter((f) => f.status === 'ready' && !f.isFolderRoot), + list: () => files + } as unknown as AttachedFilesStore +} + +async function runSearch(store: AttachedFilesStore): Promise { + const res = await searchFilesTool.fn({ + args: { pattern: 'x' }, + helpers: { attachedFiles: store }, + toolId: 't', + toolCallbacks: { setToolStatus: () => {} } + } as any) + return res as string +} + +describe('search_files — attachments present but nothing readable', () => { + it('reports no searchable text for an empty/binary-only linked folder (only ready placeholders)', async () => { + // An empty/all-binary folder leaves a single `ready` placeholder row, filtered out of readyFiles(). + const msg = await runSearch(fakeStore([{ name: 'proj', status: 'ready', isFolderRoot: true }])) + expect(msg).toMatch(/no searchable text files/i) + expect(msg).not.toMatch(/indexed/i) + }) + + it('tells the user to restore access when a restored folder is locked', async () => { + const msg = await runSearch(fakeStore([{ name: 'proj', status: 'locked', isFolderRoot: true }])) + expect(msg).toMatch(/restore access/i) + }) + + it('tells the user to re-link when files are unavailable', async () => { + const msg = await runSearch(fakeStore([{ name: 'gone.txt', status: 'unavailable' }])) + expect(msg).toMatch(/re-link/i) + }) + + it('still reports indexing while a file is genuinely indexing', async () => { + const msg = await runSearch(fakeStore([{ name: 'a.txt', status: 'indexing' }])) + expect(msg).toMatch(/still being indexed/i) + }) + + it('prefers the indexing message when an indexing file coexists with an empty folder', async () => { + const msg = await runSearch( + fakeStore([ + { name: 'proj', status: 'ready', isFolderRoot: true }, + { name: 'a.txt', status: 'indexing' } + ]) + ) + expect(msg).toMatch(/still being indexed/i) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fileTools.ts b/frontend/src/lib/components/copilot/chat/files/fileTools.ts new file mode 100644 index 0000000000..982b858154 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileTools.ts @@ -0,0 +1,226 @@ +/** + * AI tools and system-prompt roster for files attached to the GLOBAL chat. + * + * The model is made aware of attached files via a metadata-only roster appended to + * the system message (see `appendAttachedFilesRoster`). Their contents are NEVER + * inlined — the model pulls only the slices it needs through these two read-only + * tools, which stream from disk via ./fileEngine. + */ +import { z } from 'zod' +import type { ChatCompletionSystemMessageParam } from 'openai/resources/chat/completions.mjs' +import { createToolDef, type Tool } from '../shared' +import { + readFile, + searchFilesInWorker, + numberLines, + FileReadError, + type SearchHit +} from './fileEngine' +import type { AttachedFile, AttachedFilesStore } from './attachedFiles.svelte' + +/** Slice of the GLOBAL tool helpers that exposes the attached-files store. */ +export interface AttachedFilesHelper { + attachedFiles?: AttachedFilesStore +} + +function storeFrom(helpers: unknown): AttachedFilesStore | undefined { + return (helpers as AttachedFilesHelper | undefined)?.attachedFiles +} + +/** + * For a specifically requested attached file, a message describing why it can't be read / + * searched yet (still indexing, locked, unavailable, errored) or that it isn't attached — + * or undefined when it's `ready`. Shared by read_file and search_files so both report the + * same accurate status instead of search_files claiming a non-ready file isn't attached. + */ +function notReadyMessage(store: AttachedFilesStore, file: string): string | undefined { + const entry = store.get(file) + if (entry?.status === 'ready') return undefined + if (entry?.status === 'indexing') + return `File "${file}" is still being indexed. Try again shortly.` + if (entry?.status === 'locked') + return `File "${file}" is locked after a reload. Ask the user to restore access (send a message, or click "Restore access").` + if (entry?.status === 'unavailable') + return `File "${file}" is no longer available (moved, deleted, or its local copy was evicted). Ask the user to re-link it.` + if (entry?.status === 'error') + return `File "${file}" failed to load: ${entry.error ?? 'unknown error'}.` + const names = store + .list() + .map((f) => f.name) + .join(', ') + return `No attached file named "${file}". Attached files: ${names || '(none)'}.` +} + +/** + * When attachments exist but none expose a readable target (`readyFiles()` is empty), + * explain the actual reason instead of always claiming files are still indexing. Empty + * or binary-only linked folders leave only `ready` placeholder rows (filtered out of + * `readyFiles`), while a locked/unavailable restore surfaces those statuses on the rows. + */ +function noReadyFilesMessage(store: AttachedFilesStore): string { + const statuses = new Set(store.list().map((f) => f.status)) + if (statuses.has('indexing')) return 'Attached files are still being indexed. Try again shortly.' + if (statuses.has('locked')) + return 'The attached files are locked after a reload. Ask the user to restore access (send a message, or click "Restore access").' + if (statuses.has('unavailable')) + return 'The attached files are no longer available (moved, deleted, or their local copies were evicted). Ask the user to re-link them.' + if (statuses.has('error')) return 'The attached files failed to load.' + return 'No searchable text files are attached (a linked folder may be empty or contain only non-text files).' +} + +function humanSize(bytes: number): string { + if (bytes < 1024) return `${bytes} B` + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB` + return `${(bytes / (1024 * 1024)).toFixed(1)} MB` +} + +const searchFilesSchema = z.object({ + pattern: z.string().describe('JavaScript regular expression to search for.'), + file: z + .string() + .optional() + .describe( + 'Optional exact filename (as listed under "Attached files") to restrict the search to. Omit to search across all attached files.' + ), + ignore_case: z.boolean().optional().describe('Case-insensitive matching. Defaults to false.') +}) + +const searchFilesToolDef = createToolDef( + searchFilesSchema, + 'search_files', + 'Search the user-attached files with a regular expression and return matching lines with their line numbers. Use this to locate content before reading a specific window with read_file.' +) + +export const searchFilesTool: Tool<{}> = { + def: searchFilesToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const store = storeFrom(helpers) + if (!store || store.count === 0) { + return 'No files are attached to this conversation.' + } + const parsed = searchFilesSchema.parse(args) + // Validate a specifically requested file against the full store first, so a non-ready + // target reports its real status (indexing/locked/…) instead of "not attached". + if (parsed.file) { + const notReady = notReadyMessage(store, parsed.file) + if (notReady) return notReady + } + const ready = store.readyFiles() + if (ready.length === 0) { + return noReadyFilesMessage(store) + } + toolCallbacks.setToolStatus(toolId, { + content: `Searching attached files for /${parsed.pattern}/...` + }) + + // Run in a Worker so a pathological model-supplied regex can't freeze the tab. + const result = await searchFilesInWorker(ready, parsed.pattern, { + flags: parsed.ignore_case ? 'i' : '', + pathFilter: parsed.file + }) + if (result.error) { + return `Error: ${result.error}` + } + const scope = parsed.file ? `"${parsed.file}"` : `${ready.length} file(s)` + if (result.hits.length === 0) { + return `No matches for /${parsed.pattern}/ in ${scope}.` + } + const body = result.hits.map((h: SearchHit) => `${h.file}:${h.line}: ${h.text}`).join('\n') + const header = `Found ${result.hits.length} match(es) in ${scope}:` + const footer = result.truncated + ? '\n\n(Stopped at the result limit — refine your pattern or pass a `file` to narrow the search.)' + : '' + return `${header}\n${body}${footer}` + } +} + +const readFileSchema = z.object({ + file: z.string().describe('Exact filename to read, as listed under "Attached files".'), + start_line: z.number().int().optional().describe('1-based first line to read. Defaults to 1.'), + end_line: z + .number() + .int() + .optional() + .describe('1-based last line to read. The window is capped at 200 lines.') +}) + +const readFileToolDef = createToolDef( + readFileSchema, + 'read_file', + 'Read a bounded window of lines from a user-attached file. Returns each line prefixed with its 1-based number (`→`) plus a pagination note. Files are not in context, so use this to inspect their contents.' +) + +export const readFileTool: Tool<{}> = { + def: readFileToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const store = storeFrom(helpers) + if (!store || store.count === 0) { + return 'No files are attached to this conversation.' + } + const parsed = readFileSchema.parse(args) + const notReady = notReadyMessage(store, parsed.file) + if (notReady) return notReady + const entry = store.get(parsed.file)! + toolCallbacks.setToolStatus(toolId, { content: `Reading "${parsed.file}"...` }) + + try { + const res = await readFile(entry, { + startLine: parsed.start_line, + endLine: parsed.end_line + }) + return res.text ? `${res.note}\n\n${numberLines(res.text, res.startLine)}` : res.note + } catch (e) { + if (e instanceof FileReadError) { + return `Could not read "${parsed.file}": ${e.message}. The file may have been moved or deleted since it was attached.` + } + return `Error reading "${parsed.file}": ${e instanceof Error ? e.message : String(e)}` + } + } +} + +export const fileTools: Tool<{}>[] = [searchFilesTool, readFileTool] + +function rosterLine(f: AttachedFile): string { + if (f.status === 'indexing') return `- ${f.name} (indexing…)` + if (f.status === 'locked') return `- ${f.name} (locked — needs the user to restore access)` + if (f.status === 'unavailable') return `- ${f.name} (unavailable)` + if (f.status === 'error') return `- ${f.name} (failed to load)` + return `- ${f.name} — ${f.lineCount} lines, ${humanSize(f.size)}` +} + +/** Build the `## Attached files` system-prompt section (metadata only, never content). */ +export function buildAttachedFilesRoster(store: AttachedFilesStore): string { + const lines: string[] = [] + for (const folder of store.folders) { + // A locked/unavailable folder has no readable children — one line for the whole folder. + if (folder.status === 'locked') { + lines.push(`- ${folder.name} (locked — needs the user to restore access)`) + } else if (folder.status === 'unavailable') { + lines.push(`- ${folder.name} (unavailable)`) + } else { + lines.push(...folder.files.map(rosterLine)) + } + } + lines.push(...store.standalone.map(rosterLine)) + if (lines.length === 0) return '' + return [ + '## Attached files', + 'The user has attached the following files to this conversation. Their contents are NOT included here.', + 'Use the `search_files` tool to find content with a regex, and `read_file` to read a bounded window of lines.', + '', + lines.join('\n') + ].join('\n') +} + +/** + * Return a copy of the system message with the attached-files roster appended. + * Always derives from the provided base so the roster never accumulates across turns. + */ +export function appendAttachedFilesRoster( + base: ChatCompletionSystemMessageParam, + store: AttachedFilesStore +): ChatCompletionSystemMessageParam { + const roster = buildAttachedFilesRoster(store) + if (!roster || typeof base.content !== 'string') return base + return { ...base, content: `${base.content}\n\n${roster}` } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts b/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts new file mode 100644 index 0000000000..e4536b6eac --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts @@ -0,0 +1,49 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { hasFileSystemAccess, isIgnoredPath, pickDirectory } from './fsAccess' + +describe('hasFileSystemAccess', () => { + it('is false when the File System Access API is absent (node / Firefox / Safari today)', () => { + // The test env exposes none of showOpenFilePicker / showDirectoryPicker / + // DataTransferItem.getAsFileSystemHandle, so the gate must report false. + // The positive path (all three present) is exercised via browser verification. + expect(hasFileSystemAccess()).toBe(false) + }) +}) + +describe('pickDirectory', () => { + afterEach(() => { + delete (window as { showDirectoryPicker?: unknown }).showDirectoryPicker + }) + + it('returns undefined when the user dismisses the picker (AbortError)', async () => { + ;(window as { showDirectoryPicker?: unknown }).showDirectoryPicker = async () => { + throw new DOMException('aborted', 'AbortError') + } + await expect(pickDirectory()).resolves.toBeUndefined() + }) + + it('rethrows any non-abort failure instead of silently no-oping', async () => { + // e.g. a policy that blocks the File System Access API, or a lost user-activation. + ;(window as { showDirectoryPicker?: unknown }).showDirectoryPicker = async () => { + throw new DOMException('blocked by policy', 'SecurityError') + } + await expect(pickDirectory()).rejects.toThrow(/blocked by policy/) + }) +}) + +describe('isIgnoredPath', () => { + it('keeps normal source paths', () => { + expect(isIgnoredPath('myproj/src/app.ts')).toBe(false) + expect(isIgnoredPath('README.md')).toBe(false) + }) + it('skips ignored directories', () => { + expect(isIgnoredPath('myproj/node_modules/lib/index.js')).toBe(true) + expect(isIgnoredPath('myproj/dist/bundle.js')).toBe(true) + expect(isIgnoredPath('a/target/x')).toBe(true) + }) + it('skips dotfiles and dotdirs', () => { + expect(isIgnoredPath('myproj/.env')).toBe(true) + expect(isIgnoredPath('myproj/.git/config')).toBe(true) + expect(isIgnoredPath('.DS_Store')).toBe(true) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fsAccess.ts b/frontend/src/lib/components/copilot/chat/files/fsAccess.ts new file mode 100644 index 0000000000..2c741c0988 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fsAccess.ts @@ -0,0 +1,192 @@ +/** + * Thin wrappers over the File System Access API, used when available so linked + * files/folders can be re-read live after a reload (re-grantable handles). + * Capability is feature-detected (never browser-sniffed): the day Firefox/Safari + * ship the API, the handle path lights up automatically. + */ +const IGNORED_DIRS = new Set([ + 'node_modules', + 'dist', + 'build', + 'out', + 'target', + 'vendor', + 'coverage', + '__pycache__', + '.git', + '.svelte-kit', + '.next', + '.nuxt', + '.venv', + 'venv', + '.idea', + '.vscode', + '.turbo', + '.cache' +]) + +function isIgnoredSegment(name: string): boolean { + return name.startsWith('.') || IGNORED_DIRS.has(name) +} + +/** True if any segment of a relative path is a dotfile/dotdir or an ignored directory. */ +export function isIgnoredPath(path: string): boolean { + return path.split('/').some(isIgnoredSegment) +} + +type FSWindow = Window & { + showDirectoryPicker?: (opts?: { + mode?: 'read' | 'readwrite' + }) => Promise +} + +type FSDataTransferItem = DataTransferItem & { + getAsFileSystemHandle?: () => Promise +} + +/** + * True when the File System Access API needed for FOLDER linking is usable: + * the directory picker plus drag-drop handles. (Files never use the API — they're + * always snapshotted — so showOpenFilePicker is intentionally not required.) + */ +export function hasFileSystemAccess(): boolean { + return ( + typeof window !== 'undefined' && + 'showDirectoryPicker' in window && + typeof DataTransferItem !== 'undefined' && + 'getAsFileSystemHandle' in DataTransferItem.prototype + ) +} + +/** + * Open the directory picker. Returns undefined if the user dismisses it. + * Any other failure (a policy that blocks the File System Access API, a lost + * user-activation, etc.) is rethrown — swallowing it makes the picker silently + * never open, which is indistinguishable from a no-op and impossible to debug. + */ +export async function pickDirectory(): Promise { + const w = window as FSWindow + if (!w.showDirectoryPicker) return undefined + try { + return await w.showDirectoryPicker({ mode: 'read' }) + } catch (e) { + // AbortError means the user dismissed the dialog (and, under browser automation, + // that CDP intercepted the chooser) — a no-op, not a failure. + if (e instanceof DOMException && e.name === 'AbortError') return undefined + throw e + } +} + +/** + * Resolve File System Access handles from a drop's items. The `getAsFileSystemHandle` + * calls are kicked off synchronously (items are only valid during the drop event); + * the returned promise resolves the handles. + */ +export function handlesFromDataTransfer(dt: DataTransfer): Promise { + const pending = Array.from(dt.items) + .filter((it) => it.kind === 'file') + .map((it) => (it as FSDataTransferItem).getAsFileSystemHandle?.() ?? Promise.resolve(null)) + return Promise.all(pending).then((handles) => handles.filter((h): h is FileSystemHandle => !!h)) +} + +export function isFileHandle(h: FileSystemHandle): h is FileSystemFileHandle { + return h.kind === 'file' +} +export function isDirectoryHandle(h: FileSystemHandle): h is FileSystemDirectoryHandle { + return h.kind === 'directory' +} + +/** + * Recursively read a directory handle into a flat list of files with relative paths, + * skipping junk (dotfiles/dotdirs, node_modules, …). No file-count cap — the browser's + * memory/quota are the only limit. Used on link and on live re-enumeration after a reload. + */ +export async function enumerateDir( + dir: FileSystemDirectoryHandle +): Promise<{ file: File; path: string }[]> { + const out: { file: File; path: string }[] = [] + + async function walk(handle: FileSystemDirectoryHandle, prefix: string): Promise { + // @ts-ignore - values() is an async iterator in the File System Access API + for await (const entry of handle.values() as AsyncIterable) { + const path = `${prefix}/${entry.name}` + if (isIgnoredPath(path)) continue + if (isFileHandle(entry)) { + out.push({ file: await entry.getFile(), path }) + } else if (isDirectoryHandle(entry)) { + await walk(entry, path) + } + } + } + + await walk(dir, dir.name) + return out +} + +/** + * Recursively read dropped files AND folders via the legacy `webkitGetAsEntry` API — + * the fallback for browsers without the File System Access API (Firefox/Safari). Folder + * contents are snapshotted into the browser (no live handle). Each result `path` is + * folder-relative (`folder/sub/file` for a dropped folder, bare name for a loose file), + * junk paths skipped, no file-count cap. + * + * `webkitGetAsEntry()` is only valid synchronously during the drop event, so this MUST be + * called from the drop handler — its `.map(...)` runs before the first `await`, capturing + * the entries while the items are still live. + */ +export async function readDroppedEntries( + items: DataTransferItem[] +): Promise<{ file: File; path: string }[]> { + const roots = items + .map((it) => it.webkitGetAsEntry?.() ?? null) + .filter((e): e is FileSystemEntry => !!e) + const out: { file: File; path: string }[] = [] + for (const root of roots) await walkDropEntry(root, out) + return out +} + +async function walkDropEntry( + entry: FileSystemEntry, + out: { file: File; path: string }[] +): Promise { + const path = entry.fullPath.replace(/^\//, '') + if (isIgnoredPath(path)) return + if (entry.isFile) { + const fileEntry = entry as FileSystemFileEntry + const file = await new Promise((res, rej) => fileEntry.file(res, rej)) + out.push({ file, path }) + } else if (entry.isDirectory) { + const reader = (entry as FileSystemDirectoryEntry).createReader() + // readEntries yields in batches and returns [] once exhausted — loop until empty. + while (true) { + const batch = await new Promise((res, rej) => reader.readEntries(res, rej)) + if (batch.length === 0) break + for (const child of batch) await walkDropEntry(child, out) + } + } +} + +/** queryPermission without a user gesture; 'granted' | 'prompt' | 'denied'. Never rejects. */ +export async function queryReadPermission(handle: FileSystemHandle): Promise { + try { + // @ts-ignore - queryPermission is part of the File System Access API + return (await handle.queryPermission?.({ mode: 'read' })) ?? 'prompt' + } catch { + return 'prompt' + } +} + +/** + * requestPermission — MUST be called within a user gesture. Never rejects: the spec + * rejects with SecurityError when user activation is missing (e.g. a second prompt + * after the first consumed the gesture) — that maps to 'denied' here so callers can + * treat it as "still locked" instead of blowing up the send path. + */ +export async function requestReadPermission(handle: FileSystemHandle): Promise { + try { + // @ts-ignore - requestPermission is part of the File System Access API + return (await handle.requestPermission?.({ mode: 'read' })) ?? 'denied' + } catch { + return 'denied' + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/searchWorker.ts b/frontend/src/lib/components/copilot/chat/files/searchWorker.ts new file mode 100644 index 0000000000..68960801da --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/searchWorker.ts @@ -0,0 +1,38 @@ +/** + * Web Worker that runs `search_files` off the main thread. + * + * The regex is model-supplied and `RegExp.prototype.test()` can't be interrupted, so a + * catastrophic-backtracking pattern (e.g. /^(a+)+$/) would otherwise hang the whole tab. + * Running it here lets the caller `terminate()` this worker on a timeout instead. The + * matching itself reuses `searchFiles` from the engine (single source of truth). + */ +import { searchFiles, type FileEntry } from './fileEngine' + +interface SearchRequest { + files: { name: string; file: Blob }[] + pattern: string + flags?: string + pathFilter?: string + maxHits?: number +} + +self.onmessage = async (e: MessageEvent) => { + const { files, pattern, flags, pathFilter, maxHits } = e.data + // searchFiles only reads `name` + `file` (it streams); the index fields are unused here. + const entries: FileEntry[] = files.map((f) => ({ + name: f.name, + file: f.file, + lineIndex: [], + lineCount: 0 + })) + try { + const result = await searchFiles(entries, pattern, { flags, pathFilter, maxHits }) + ;(self as unknown as Worker).postMessage(result) + } catch (err) { + ;(self as unknown as Worker).postMessage({ + hits: [], + truncated: false, + error: err instanceof Error ? err.message : String(err) + }) + } +} diff --git a/frontend/src/lib/components/copilot/chat/global/core.ts b/frontend/src/lib/components/copilot/chat/global/core.ts index 90925255d2..10f2fce49a 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.ts @@ -76,6 +76,8 @@ import { import { searchDocsTool, readDocsPageTool } from '../docs/core' import type { ContextElement } from '../context' import { getDatatableTools } from '../datatableTools' +import { fileTools } from '../files/fileTools' +import type { AttachedFilesStore } from '../files/attachedFiles.svelte' import { UserDraft } from '$lib/userDraft.svelte' import { emptySchema } from '$lib/utils' import { inferArgs } from '$lib/infer' @@ -2112,7 +2114,9 @@ export const globalTools: Tool<{}>[] = [ } }, // Workspace-scoped datatable tools (unrestricted: no whitelist, no creation policy) - ...getDatatableTools() + ...getDatatableTools(), + // Read-only tools over files the user attached to the conversation + ...fileTools ] // Tools that only make sense inside an AI session (they drive the session's @@ -2160,6 +2164,7 @@ export type SessionToolHelpers = { sessionId?: string } export type GlobalToolHelpers = SessionToolHelpers & { testActiveFlow?: (args?: Record) => Promise + attachedFiles?: AttachedFilesStore } function sessionIdFromCtx(ctx: { helpers?: unknown }): string | undefined { diff --git a/frontend/src/lib/components/copilot/chat/mention.test.ts b/frontend/src/lib/components/copilot/chat/mention.test.ts new file mode 100644 index 0000000000..8d379ce3b4 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/mention.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, it } from 'vitest' +import { MENTION_RE, mentionTitle, formatMention } from './mention' + +describe('formatMention', () => { + it('leaves a simple name bare', () => { + expect(formatMention('app.ts')).toBe('@app.ts') + expect(formatMention('proj/sub/a.ts')).toBe('@proj/sub/a.ts') + }) + it('brackets a name containing whitespace', () => { + expect(formatMention('my file.txt')).toBe('@[my file.txt]') + expect(formatMention('my folder/a b.ts')).toBe('@[my folder/a b.ts]') + }) + it('brackets names with HTML-sensitive chars, parens, brackets', () => { + expect(formatMention('R&D notes.md')).toBe('@[R&D notes.md]') + expect(formatMention('a.txt')).toBe('@[a.txt]') + expect(formatMention('report(final).csv')).toBe('@[report(final).csv]') + }) +}) + +describe('mentionTitle', () => { + it('strips the @ from a bare mention', () => { + expect(mentionTitle('@app.ts')).toBe('app.ts') + }) + it('strips the @[ ] from a bracketed mention', () => { + expect(mentionTitle('@[my file.txt]')).toBe('my file.txt') + }) +}) + +describe('MENTION_RE', () => { + it('captures a bracketed (spaced) mention whole alongside bare ones', () => { + const tokens = [...'see @app.ts and @[my file.txt] ok'.matchAll(MENTION_RE)].map((m) => m[0]) + expect(tokens).toEqual(['@app.ts', '@[my file.txt]']) + expect(tokens.map(mentionTitle)).toEqual(['app.ts', 'my file.txt']) + }) + it('round-trips formatMention → MENTION_RE → mentionTitle for a spaced name', () => { + const name = 'my notes (v2).md' + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(mentionTitle(m[0])).toBe(name) + }) + + it('round-trips a name containing both whitespace and a closing bracket', () => { + const name = 'notes ] draft.md' + expect(formatMention(name)).toBe('@[notes \\] draft.md]') + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(m[0]).toBe('@[notes \\] draft.md]') + expect(mentionTitle(m[0])).toBe(name) + }) + + it('round-trips an HTML-sensitive name (highlighter handles HTML-escaping separately)', () => { + const name = 'a & c].txt' + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(mentionTitle(m[0])).toBe(name) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/mention.ts b/frontend/src/lib/components/copilot/chat/mention.ts new file mode 100644 index 0000000000..65a2b3b7c6 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/mention.ts @@ -0,0 +1,31 @@ +/** + * `@mention` formatting shared between the chat input (which inserts mentions) and the + * textarea highlighter (which parses them) so the two never disagree. + * + * A simple name is inserted bare (`@app.ts`); a name containing whitespace is bracketed + * (`@[my file.txt]`) so it's captured whole instead of truncating at the first space. + */ + +/** + * Matches a mention token: a bracketed `@[name with spaces]` (where `\]` and `\\` are + * escaped, so a `]` inside the name doesn't end the token early) first, then a bare `@name`. + */ +export const MENTION_RE = /@\[(?:\\.|[^\]\\\r\n])*\]|@[\w/.\-\[\]]+/g + +/** The title of a mention token (`@name` or `@[name]`), brackets stripped and unescaped. */ +export function mentionTitle(token: string): string { + if (token.startsWith('@[') && token.endsWith(']')) { + return token.slice(2, -1).replace(/\\(.)/g, '$1') + } + return token.slice(1) +} + +/** Chars the bare `@name` regex matches without truncating; anything else needs brackets. */ +const BARE_SAFE = /^[\w/.\-]+$/ + +/** Format a name as a mention token. A bare `@name` only survives for simple names; anything + * with whitespace, HTML-sensitive chars (`< > &`), brackets, parens, etc. is bracketed (with + * `\` and `]` escaped) so the token is captured whole and round-trips through the parser. */ +export function formatMention(name: string): string { + return BARE_SAFE.test(name) ? `@${name}` : `@[${name.replace(/[\\\]]/g, '\\$&')}]` +} diff --git a/frontend/src/lib/components/icons/fileIcon.ts b/frontend/src/lib/components/icons/fileIcon.ts new file mode 100644 index 0000000000..fef893e398 --- /dev/null +++ b/frontend/src/lib/components/icons/fileIcon.ts @@ -0,0 +1,74 @@ +/** + * Resolve a file name (or relative path) to an icon by extension. Shared by the + * raw-app file tree and the AI-chat file attachments so both stay consistent. + */ +import { File, ImageIcon } from 'lucide-svelte' +import TypeScript from '../common/languageIcons/TypeScript.svelte' +import JavaScriptIcon from './JavaScriptIcon.svelte' +import JsonIcon from './JsonIcon.svelte' +import ReactIcon from './ReactIcon.svelte' +import SvelteIcon from './SvelteIcon.svelte' +import VueIcon from './VueIcon.svelte' +import CssIcon from './CssIcon.svelte' +import SassIcon from './SassIcon.svelte' +import LessIcon from './LessIcon.svelte' +import HtmlIcon from './HtmlIcon.svelte' +import MarkdownIcon from './MarkdownIcon.svelte' +import YamlIcon from './YamlIcon.svelte' + +export interface ResolvedFileIcon { + icon: any + className?: string +} + +/** Lowercased extension of a file name or path (basename only); '' if none. */ +export function getFileExtension(filename: string): string { + const base = filename.split('/').pop() ?? filename + const parts = base.split('.') + return parts.length > 1 ? parts[parts.length - 1].toLowerCase() : '' +} + +/** Icon (and optional color class) for a file, by extension. */ +export function getFileIcon(filename: string): ResolvedFileIcon { + switch (getFileExtension(filename)) { + case 'json': + return { icon: JsonIcon } + case 'tsx': + case 'jsx': + return { icon: ReactIcon } + case 'ts': + return { icon: TypeScript } + case 'js': + return { icon: JavaScriptIcon } + case 'svelte': + return { icon: SvelteIcon } + case 'vue': + return { icon: VueIcon } + case 'css': + return { icon: CssIcon } + case 'scss': + case 'sass': + return { icon: SassIcon } + case 'less': + return { icon: LessIcon } + case 'png': + case 'jpg': + case 'jpeg': + case 'gif': + case 'svg': + case 'webp': + case 'ico': + return { icon: ImageIcon, className: 'text-purple-500' } + case 'html': + case 'htm': + return { icon: HtmlIcon } + case 'md': + case 'markdown': + return { icon: MarkdownIcon } + case 'yaml': + case 'yml': + return { icon: YamlIcon } + default: + return { icon: File, className: 'text-tertiary' } + } +} diff --git a/frontend/src/lib/components/raw_apps/FileTreeNode.svelte b/frontend/src/lib/components/raw_apps/FileTreeNode.svelte index 9819ed8796..3d428b9aaf 100644 --- a/frontend/src/lib/components/raw_apps/FileTreeNode.svelte +++ b/frontend/src/lib/components/raw_apps/FileTreeNode.svelte @@ -1,32 +1,11 @@
diff --git a/frontend/src/lib/components/sessions/sessionRuntime.svelte.ts b/frontend/src/lib/components/sessions/sessionRuntime.svelte.ts index a575a19ab8..d7d9f77630 100644 --- a/frontend/src/lib/components/sessions/sessionRuntime.svelte.ts +++ b/frontend/src/lib/components/sessions/sessionRuntime.svelte.ts @@ -274,6 +274,8 @@ function createRuntime(session: Session): SessionRuntime { // session targeting the right workspace. Both calls are idempotent. manager.beforeSend = async () => { materializeTransient(session.id) + // Session is now persisted → flush any linked files buffered while it was transient. + await manager.attachedFiles.flushPending() const committed = await commitSessionWorkspace(session.id, get(workspaceStore) ?? undefined) // commitSessionWorkspace returns undefined only when the session did NOT // commit to a workspace — most importantly when a staged fork failed to @@ -723,6 +725,9 @@ async function initRuntime(runtime: SessionRuntime, session: Session) { const { manager } = runtime await manager.historyManager.init() manager.historyManager.setSessionId(session.id) + // Restore linked files persisted for this session (live handles re-grant on send; + // snapshots restore directly). Non-transient sessions persist immediately. + await manager.attachedFiles.restore(session.id, !session.transient) await ensureChatIdsSeeded(manager.historyManager) if (session.chatId) { diff --git a/frontend/src/lib/components/sessions/sessionState.svelte.ts b/frontend/src/lib/components/sessions/sessionState.svelte.ts index 75a98efff5..64f35fca19 100644 --- a/frontend/src/lib/components/sessions/sessionState.svelte.ts +++ b/frontend/src/lib/components/sessions/sessionState.svelte.ts @@ -13,6 +13,7 @@ import { switchWorkspace } from '$lib/storeUtils' import { getLocalSetting, storeLocalSetting } from '$lib/utils' import { userScopedDb } from '$lib/userScopedDb' import type { DBSchema, IDBPDatabase } from 'idb' +import { deleteItemsForSession } from '../copilot/chat/files/attachedFilesDB' // Switch the global workspace iff the target differs from the active one // and is non-empty. Centralises the "session needs its workspace in focus" @@ -589,6 +590,8 @@ export function deleteSession(id: string) { sessionState.currentSessionId = sessionState.sessions[0]?.id } void deleteSessionRecord(id) + // GC any linked files persisted for this session. + void deleteItemsForSession(id) } export function setSessionChatId(sessionId: string, chatId: string) { From 3a558002244a49c3373a03ea0f8bb52f77391952 Mon Sep 17 00:00:00 2001 From: hugocasa Date: Mon, 22 Jun 2026 09:33:55 +0200 Subject: [PATCH 002/117] add whatsapp business icon (#9541) Co-authored-by: Claude Fable 5 --- .../icons/WhatsappBusinessIcon.svelte | 20 +++++++++++++++++++ frontend/src/lib/components/icons/index.ts | 2 ++ 2 files changed, 22 insertions(+) create mode 100644 frontend/src/lib/components/icons/WhatsappBusinessIcon.svelte diff --git a/frontend/src/lib/components/icons/WhatsappBusinessIcon.svelte b/frontend/src/lib/components/icons/WhatsappBusinessIcon.svelte new file mode 100644 index 0000000000..fc9c2843ec --- /dev/null +++ b/frontend/src/lib/components/icons/WhatsappBusinessIcon.svelte @@ -0,0 +1,20 @@ + + + + + diff --git a/frontend/src/lib/components/icons/index.ts b/frontend/src/lib/components/icons/index.ts index 9c4a70c041..b39df4310e 100644 --- a/frontend/src/lib/components/icons/index.ts +++ b/frontend/src/lib/components/icons/index.ts @@ -185,6 +185,7 @@ import TwitchIcon from './TwitchIcon.svelte' import TwitterIcon from './TwitterIcon.svelte' import VercelIcon from './VercelIcon.svelte' import WebflowIcon from './WebflowIcon.svelte' +import WhatsappBusinessIcon from './WhatsappBusinessIcon.svelte' import WooCommerceIcon from './WooCommerceIcon.svelte' import WordpressIcon from './WordpressIcon.svelte' import XataIcon from './XataIcon.svelte' @@ -413,6 +414,7 @@ export const APP_TO_ICON_COMPONENT = { twitter: TwitterIcon, vercel: VercelIcon, webflow: WebflowIcon, + whatsapp_business: WhatsappBusinessIcon, woocommerce: WooCommerceIcon, wordpress: WordpressIcon, xata: XataIcon, From 4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb Mon Sep 17 00:00:00 2001 From: Diego Imbert <70353967+diegoimbert@users.noreply.github.com> Date: Mon, 22 Jun 2026 10:33:49 +0200 Subject: [PATCH 003/117] feat: scope default instance db name to workspace (dt_/dl_) (#9699) * feat: default instance db name to dt_/dl_ workspace scope Co-Authored-By: Claude Opus 4.8 (1M context) * test: cap instance db name at 63 chars and add unit tests Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .../DataTableSettings.svelte | 22 ++++++--- .../workspaceSettings/DucklakeSettings.svelte | 18 +++++--- .../workspaceSettings/utils.svelte.ts | 28 ++++++++++++ .../workspaceSettings/utils.test.ts | 45 +++++++++++++++++++ 4 files changed, 101 insertions(+), 12 deletions(-) create mode 100644 frontend/src/lib/components/workspaceSettings/utils.test.ts diff --git a/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte b/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte index 60794c6923..218ae19239 100644 --- a/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte +++ b/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte @@ -41,8 +41,6 @@ } return s } - - let DEFAULT_DATATABLE_DB_NAME = 'datatable_db' + + +
+ + + + + {#if skills.length > 0} +
+ {#each skills as skill (skill.name)} +
+
+
{skill.name}
+
{skill.description}
+
+
+ {/each} +
+ {/if} +
+ + + { + const toImport = pendingImport + const skipped = pendingSkipped + pendingImport = undefined + pendingSkipped = [] + if (toImport) await uploadSkills(toImport, skipped) + }} + onCanceled={() => { + pendingImport = undefined + pendingSkipped = [] + }} +> + + Add {pendingImport?.length} skill(s) to the AI chat? + {pendingNamesPreview} + {#if pendingSkipped.length} +
{pendingSkipped.length} file(s) will be skipped. + {/if} + + + + { + const name = toDelete + toDelete = undefined + if (name) await deleteSkill(name) + }} + onCanceled={() => (toDelete = undefined)} +> + + Delete the skill {toDelete}? The AI chat will no longer be able to use it. + + From 3bf5b72afab3241ea41a261a40c2434764bdaf72 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 09:27:02 +0200 Subject: [PATCH 019/117] fix(drafts): stop mis-filing workspace-blind legacy drafts on migration (#9725) --- frontend/src/lib/userDraftDbMigration.ts | 13 +- .../src/lib/userDraftLegacyMigration.test.ts | 231 ++++++------------ frontend/src/lib/userDraftLegacyMigration.ts | 103 +++----- .../src/routes/(root)/(logged)/+layout.svelte | 17 +- 4 files changed, 117 insertions(+), 247 deletions(-) diff --git a/frontend/src/lib/userDraftDbMigration.ts b/frontend/src/lib/userDraftDbMigration.ts index 824cd1c952..4c3079ea94 100644 --- a/frontend/src/lib/userDraftDbMigration.ts +++ b/frontend/src/lib/userDraftDbMigration.ts @@ -1,11 +1,12 @@ /** * One-off migration from the localStorage UserDraft autosave to the - * DB-backed `draft` table. Runs after `migrateLegacyUserDrafts` (which - * produces the `userdraft/w/{workspace}/{kind}/{path}` keys this reads), - * POSTing each to `/drafts/update` and clearing the source key only on - * success — so it's idempotent without a sentinel; failed entries retry next - * mount. Not workspace-gated: keys embed their own workspace and the token - * covers all of them, so gating would orphan other-workspace entries. + * DB-backed `draft` table. Reads the workspace-scoped + * `userdraft/w/{workspace}/{kind}/{path}` keys (written by the editor during + * the interim LS-backed phase, so the embedded workspace is correct), POSTing + * each to `/drafts/update` and clearing the source key only on success — so + * it's idempotent without a sentinel; failed entries retry next mount. Not + * workspace-gated: keys embed their own workspace and the token covers all of + * them, so gating would orphan other-workspace entries. * * Before uploading, each draft is compared against its deployed version * (script / flow / app); a draft that's deep-equal to what's deployed carries diff --git a/frontend/src/lib/userDraftLegacyMigration.test.ts b/frontend/src/lib/userDraftLegacyMigration.test.ts index 7069929897..f7963e55a0 100644 --- a/frontend/src/lib/userDraftLegacyMigration.test.ts +++ b/frontend/src/lib/userDraftLegacyMigration.test.ts @@ -1,6 +1,6 @@ import { describe, it, expect, beforeEach } from 'vitest' import { - migrateLegacyUserDrafts, + purgeLegacyUserDrafts, __resetUserDraftLegacyMigrationForTesting } from './userDraftLegacyMigration' @@ -8,18 +8,12 @@ function encodeLegacy(value: unknown): string { return btoa(encodeURIComponent(JSON.stringify(value))) } -function wrapped(value: V): string { - return JSON.stringify({ value }) -} - -// Read a migrated entry, strip the GC `lastWrittenAt` stamp so assertions -// can match the `{ value }` shape regardless of when the migration ran. -function storedShape(key: string): string | null { - const raw = localStorage.getItem(key) - if (raw == null) return null - const parsed = JSON.parse(raw) - delete parsed.lastWrittenAt - return JSON.stringify(parsed) +const legacyApp = { + grid: [], + fullscreen: false, + theme: undefined, + unusedInlineScripts: [], + hiddenInlineScripts: [] } beforeEach(() => { @@ -27,197 +21,110 @@ beforeEach(() => { __resetUserDraftLegacyMigrationForTesting() }) -describe('migrateLegacyUserDrafts', () => { - it('migrates a legacy app draft to the workspace-scoped key with a { value } wrapper', () => { - // Shape mirrors what the legacy AppEditor wrote: `encodeState($appStore)`, - // i.e. the inner App value, not the wrapping AppWithLastVersion. - const legacyApp = { - grid: [], - fullscreen: false, - theme: undefined, - unusedInlineScripts: [], - hiddenInlineScripts: [] - } +describe('purgeLegacyUserDrafts', () => { + it('drops a recognised legacy app draft without re-creating it under any key', () => { localStorage.setItem('app-u/me/dashboard', encodeLegacy(legacyApp)) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('app-u/me/dashboard')).toBeNull() - expect(storedShape('userdraft/w/main/app/u/me/dashboard')).toBe(wrapped(legacyApp)) + // The workspace-blind key is gone, NOT promoted to a guessed workspace. + expect(localStorage.getItem('userdraft/w/main/app/u/me/dashboard')).toBeNull() }) - it('migrates a legacy empty-path app draft (the `app` literal key)', () => { - const legacyApp = { - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - } + it('drops the empty-path legacy keys (`app` / `flow` / `rawapp` literals)', () => { localStorage.setItem('app', encodeLegacy(legacyApp)) + localStorage.setItem('flow', encodeLegacy({ flow: { summary: '', value: { modules: [] } } })) + localStorage.setItem('rawapp', encodeLegacy({ files: {}, runnables: {}, data: {} })) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('app')).toBeNull() - expect(storedShape('userdraft/w/main/app/')).toBe(wrapped(legacyApp)) + expect(localStorage.getItem('flow')).toBeNull() + expect(localStorage.getItem('rawapp')).toBeNull() }) - it('migrates a legacy flow draft and strips the view-state envelope', () => { - const flow = { summary: 'f', value: { modules: [] }, path: 'u/me/myflow' } - const legacyBundle = { - flow, - path: 'u/me/myflow', - selectedId: 'settings', - draft_triggers: [{ id: 't1' }], - selected_trigger: null, - loadedFromHistory: undefined - } - localStorage.setItem('flow-u/me/myflow', encodeLegacy(legacyBundle)) + it('drops recognised legacy flow and raw-app drafts', () => { + localStorage.setItem( + 'flow-u/me/myflow', + encodeLegacy({ flow: { summary: 'f', value: { modules: [] } }, selectedId: 'settings' }) + ) + localStorage.setItem( + 'rawapp-u/me/site', + encodeLegacy({ files: { 'index.tsx': 'x' }, runnables: {}, data: {} }) + ) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('flow-u/me/myflow')).toBeNull() - // Only the inner Flow survives; the view-state envelope is dropped. - expect(storedShape('userdraft/w/main/flow/u/me/myflow')).toBe(wrapped(flow)) - }) - - it('migrates a legacy raw-app draft, defaulting the new `summary` field', () => { - const legacy = { - files: { 'index.tsx': 'export default () => null' }, - runnables: {}, - data: { tables: [] } - } - localStorage.setItem('rawapp-u/me/site', encodeLegacy(legacy)) - - migrateLegacyUserDrafts('main') - expect(localStorage.getItem('rawapp-u/me/site')).toBeNull() - expect(storedShape('userdraft/w/main/raw_app/u/me/site')).toBe( - wrapped({ ...legacy, summary: '' }) - ) }) - it('preserves an existing new-format entry instead of overwriting it', () => { - // Old and new both exist for the same item — the new one is presumed - // fresher. - localStorage.setItem( - 'app-u/me/dash', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) - const existingNew = wrapped({ value: 'new' }) - localStorage.setItem('userdraft/w/main/app/u/me/dash', existingNew) + it('leaves the workspace-scoped interim keys untouched (migrateUserDraftsToDb owns those)', () => { + const interim = JSON.stringify({ value: { modules: [] } }) + localStorage.setItem('userdraft/w/main/flow/u/me/keep', interim) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() - expect(localStorage.getItem('app-u/me/dash')).toBeNull() - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).toBe(existingNew) - }) - - it('is idempotent — the second invocation is a no-op', () => { - localStorage.setItem( - 'app-u/me/dash', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) - migrateLegacyUserDrafts('main') - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).not.toBeNull() - - // Drop the migrated entry to detect any re-migration attempt. - localStorage.removeItem('userdraft/w/main/app/u/me/dash') - // Drop the source too, so re-running couldn't even find a source. - // (The sentinel alone should be enough; this just clarifies the intent.) - migrateLegacyUserDrafts('main') - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).toBeNull() - }) - - it('skips entirely when no workspace is available', () => { - localStorage.setItem( - 'app-u/me/dash', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) - migrateLegacyUserDrafts('') - - expect(localStorage.getItem('app-u/me/dash')).not.toBeNull() - }) - - it('handles malformed legacy payloads without throwing', () => { - localStorage.setItem('app-u/me/garbled', 'not-base64!!!') - expect(() => migrateLegacyUserDrafts('main')).not.toThrow() - // Migration didn't migrate, didn't crash — leaves the entry alone. - expect(localStorage.getItem('app-u/me/garbled')).toBe('not-base64!!!') + expect(localStorage.getItem('userdraft/w/main/flow/u/me/keep')).toBe(interim) }) it('leaves keys whose path does not match the legacy `u|f/owner/name` shape alone', () => { - // A future feature or neighbouring code might pick a key like - // `app-recent` for its own purposes. The path doesn't look like a - // Windmill item path, so the migration must skip it. + // `app-recent` / `app-some_other_app` look like the legacy prefix but the + // suffix isn't a Windmill item path — a future feature might own them. localStorage.setItem('app-recent', 'whatever') localStorage.setItem('app-some_other_app', 'whatever') - // `flow-u/me/foo` matches the shape and would be migrated, but the - // payload also needs to look like a Windmill draft (asserted below). - localStorage.setItem('flow-u/me/foo', encodeLegacy({ flow: { value: { modules: [] } } })) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('app-recent')).toBe('whatever') expect(localStorage.getItem('app-some_other_app')).toBe('whatever') - expect(localStorage.getItem('userdraft/w/main/flow/u/me/foo')).not.toBeNull() }) - it('skips legacy-shaped keys whose payload does not look like a Windmill draft', () => { - // `app-u/me/dash` matches LEGACY_PATH_SHAPE and decodes to valid JSON, - // but none of the App-shape fields (grid/fullscreen/theme/ - // unusedInlineScripts/hiddenInlineScripts) are present. Treat it as - // unrelated and leave it untouched. - const unrelated = encodeLegacy({ random: 'data', count: 7 }) - localStorage.setItem('app-u/me/dash', unrelated) + it('leaves legacy-shaped keys whose payload does not look like a Windmill draft', () => { + // Matches LEGACY_PATH_SHAPE and decodes to valid JSON, but carries none of + // the App/flow draft fields — treat as unrelated, do not delete. + const unrelatedApp = encodeLegacy({ random: 'data', count: 7 }) + localStorage.setItem('app-u/me/dash', unrelatedApp) const unrelatedFlow = encodeLegacy({ stepsState: {} }) localStorage.setItem('flow-u/me/bar', unrelatedFlow) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() - expect(localStorage.getItem('app-u/me/dash')).toBe(unrelated) - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).toBeNull() + expect(localStorage.getItem('app-u/me/dash')).toBe(unrelatedApp) expect(localStorage.getItem('flow-u/me/bar')).toBe(unrelatedFlow) - expect(localStorage.getItem('userdraft/w/main/flow/u/me/bar')).toBeNull() }) - it('migrates multiple legacy entries in a single invocation', () => { - localStorage.setItem( - 'app-u/me/a', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) + it('leaves a malformed (non-base64) legacy payload alone and does not throw', () => { + localStorage.setItem('app-u/me/garbled', 'not-base64!!!') + + expect(() => purgeLegacyUserDrafts()).not.toThrow() + expect(localStorage.getItem('app-u/me/garbled')).toBe('not-base64!!!') + }) + + it('is idempotent — once the sentinel is set, a later legacy key survives', () => { + localStorage.setItem('app-u/me/a', encodeLegacy(legacyApp)) + purgeLegacyUserDrafts() + expect(localStorage.getItem('app-u/me/a')).toBeNull() + + // A key written after the first run is NOT swept (the sentinel short-circuits). + localStorage.setItem('app-u/me/b', encodeLegacy(legacyApp)) + purgeLegacyUserDrafts() + expect(localStorage.getItem('app-u/me/b')).not.toBeNull() + }) + + it('purges multiple legacy entries in a single invocation', () => { + localStorage.setItem('app-u/me/a', encodeLegacy(legacyApp)) localStorage.setItem( 'flow-u/me/b', - encodeLegacy({ flow: { summary: '', value: { modules: [] }, path: 'u/me/b' } }) - ) - localStorage.setItem( - 'rawapp-u/me/c', - encodeLegacy({ files: {}, runnables: {}, data: { tables: [] } }) + encodeLegacy({ flow: { summary: '', value: { modules: [] } } }) ) + localStorage.setItem('rawapp-u/me/c', encodeLegacy({ files: {}, runnables: {}, data: {} })) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() - expect(localStorage.getItem('userdraft/w/main/app/u/me/a')).not.toBeNull() - expect(localStorage.getItem('userdraft/w/main/flow/u/me/b')).not.toBeNull() - expect(localStorage.getItem('userdraft/w/main/raw_app/u/me/c')).not.toBeNull() + expect(localStorage.getItem('app-u/me/a')).toBeNull() + expect(localStorage.getItem('flow-u/me/b')).toBeNull() + expect(localStorage.getItem('rawapp-u/me/c')).toBeNull() }) }) diff --git a/frontend/src/lib/userDraftLegacyMigration.ts b/frontend/src/lib/userDraftLegacyMigration.ts index d4ee04ee61..4239bc0e8a 100644 --- a/frontend/src/lib/userDraftLegacyMigration.ts +++ b/frontend/src/lib/userDraftLegacyMigration.ts @@ -1,22 +1,29 @@ /** - * One-off migration from the pre-UserDraft localStorage autosave entries to - * the workspace-scoped `userdraft/w/{ws}/{kind}/{path}` format. + * One-shot purge of the pre-UserDraft browser-local autosave keys. * - * Legacy keys (global, not workspace-scoped — assumed to belong to the user's - * current workspace at migration time): + * The original autosave (pre-#9121) wrote workspace-BLIND keys: * - * `flow` / `flow-{path}` base64 of `encodeState({ flow, path, selectedId, draft_triggers, ... })` - * `app` / `app-{path}` base64 of `encodeState(App)` - * `rawapp` / `rawapp-{path}` base64 of `encodeState({ files, runnables, data })` + * `flow` / `flow-{path}` base64 of `encodeState({ flow, path, selectedId, draft_triggers, ... })` + * `app` / `app-{path}` base64 of `encodeState(App)` + * `rawapp` / `rawapp-{path}` base64 of `encodeState({ files, runnables, data })` * - * Target keys: `userdraft/w/{workspace}/{flow|app|raw_app}/{path}` storing - * `JSON.stringify({ value: })`. + * Neither the key nor the decoded value records a workspace (the value carries + * only workspace-agnostic item paths like `u/me/x`), so these drafts cannot be + * attributed to the workspace they were edited in. The current editors are + * DB-backed and never read these keys, so they are dead data with one dangerous + * property: promoting them to the DB would force a GUESS of the workspace, which + * mis-files drafts into whatever workspace happened to be active when the + * migration first ran (a single global sentinel gates it). We therefore drop + * them instead of migrating them. + * + * Only keys that BOTH match the legacy path shape AND decode to a plausible + * legacy draft are removed; unrelated look-alikes (`app-recent`, garbage, + * non-Windmill payloads) are left untouched. The workspace-scoped interim keys + * (`userdraft/w/{ws}/...`, written by the editor with the correct workspace) + * are NOT touched here — `migrateUserDraftsToDb` still pushes those to the DB. * * Idempotent: writes a sentinel under `MIGRATION_FLAG` after the first run so - * subsequent invocations are no-ops. Existing new-format entries are never - * overwritten — when both an old and a new entry exist for the same item, the - * old one is simply dropped on the assumption that the new entry is the more - * recent edit. + * subsequent invocations are no-ops. * * This file is intentionally standalone — it does not import from * `userDraft.svelte.ts` so the new code stays uncluttered by the legacy @@ -71,10 +78,9 @@ function decodeLegacyState(raw: string): unknown { * Per-kind shape gate. The legacy keys (`app-foo`, `flow-foo`, ...) are * unusual enough that nothing else in the codebase has used them, but * matching `LEGACY_PATH_SHAPE` doesn't prove the payload is actually a - * Windmill draft (any base64-of-JSON could pass). Promoting a stray payload - * would silently surface as a phantom "Restored from local storage" toast - * on the next edit, so we reject anything that doesn't carry the fields the - * legacy writers actually produced. + * Windmill draft (any base64-of-JSON could pass). We only delete keys we can + * positively recognise as legacy drafts, so a stray look-alike that happens to + * use this key shape is left untouched rather than silently dropped. */ function isPlausibleLegacyValue(kind: LegacyKind, decoded: unknown): boolean { if (decoded == null || typeof decoded !== 'object') return false @@ -103,32 +109,6 @@ function isPlausibleLegacyValue(kind: LegacyKind, decoded: unknown): boolean { } } -function transformLegacyValue(kind: LegacyKind, decoded: unknown): unknown { - const obj = decoded as Record - switch (kind) { - case 'flow': - // The legacy bundle wrapped the Flow alongside view-state fields - // (selectedId, draft_triggers, ...). The new entry stores only the - // Flow — the view-state lives elsewhere or is re-derived. - return obj.flow - case 'app': - // Legacy stored the App directly. - return obj - case 'raw_app': - // Legacy bundle missed the `summary` field that the new editor adds. - return { - files: obj.files ?? {}, - runnables: obj.runnables ?? {}, - data: obj.data ?? {}, - summary: typeof obj.summary === 'string' ? obj.summary : '' - } - } -} - -function newKey(workspace: string, kind: LegacyKind, path: string): string { - return `userdraft/w/${workspace}/${kind}/${path}` -} - function listLocalStorageKeys(): string[] { const out: string[] = [] for (let i = 0; i < localStorage.length; i++) { @@ -139,16 +119,11 @@ function listLocalStorageKeys(): string[] { } /** - * Run the legacy → new-format migration. Idempotent: returns immediately if a - * previous run completed (signalled by `MIGRATION_FLAG`). - * - * The migration is workspace-scoped because the legacy keys had no notion of - * workspace — we treat the caller's current workspace as the owner of any - * surviving legacy entries. + * Remove the workspace-blind legacy autosave keys (see file header). Idempotent: + * returns immediately if a previous run completed (signalled by `MIGRATION_FLAG`). */ -export function migrateLegacyUserDrafts(workspace: string): void { +export function purgeLegacyUserDrafts(): void { if (typeof localStorage === 'undefined') return - if (!workspace) return if (localStorage.getItem(MIGRATION_FLAG) !== null) return try { @@ -157,32 +132,18 @@ export function migrateLegacyUserDrafts(workspace: string): void { if (!match) continue const raw = localStorage.getItem(key) if (raw == null) continue - - try { - const decoded = decodeLegacyState(raw) - if (!isPlausibleLegacyValue(match.newKind, decoded)) continue - const value = transformLegacyValue(match.newKind, decoded) - const target = newKey(workspace, match.newKind, match.path) - if (value !== undefined && localStorage.getItem(target) == null) { - // `lastWrittenAt` makes the migrated entry visible to - // `gcUserDrafts`. We stamp it as "now" so a freshly-migrated - // autosave gets the full retention window — sweeping it - // immediately on the first GC pass would lose work the - // legacy migration just rescued. - localStorage.setItem(target, JSON.stringify({ value, lastWrittenAt: Date.now() })) - } - localStorage.removeItem(key) - } catch (e) { - console.error('UserDraft legacy migration: failed to migrate', key, e) - } + // Only drop keys we can positively recognise as legacy Windmill + // drafts; leave unrelated or unparseable look-alikes in place. + if (!isPlausibleLegacyValue(match.newKind, decodeLegacyState(raw))) continue + localStorage.removeItem(key) } localStorage.setItem(MIGRATION_FLAG, new Date().toISOString()) } catch (e) { - console.error('UserDraft legacy migration: aborted', e) + console.error('UserDraft legacy purge: aborted', e) } } -/** Test-only: clear the sentinel so the migration can re-run. */ +/** Test-only: clear the sentinel so the purge can re-run. */ export function __resetUserDraftLegacyMigrationForTesting(): void { try { localStorage.removeItem(MIGRATION_FLAG) diff --git a/frontend/src/routes/(root)/(logged)/+layout.svelte b/frontend/src/routes/(root)/(logged)/+layout.svelte index fa9bf2122d..4f6004ce2a 100644 --- a/frontend/src/routes/(root)/(logged)/+layout.svelte +++ b/frontend/src/routes/(root)/(logged)/+layout.svelte @@ -58,7 +58,7 @@ import GlobalSearchModal from '$lib/components/search/GlobalSearchModal.svelte' import MenuButton from '$lib/components/sidebar/MenuButton.svelte' import { loadProtectionRules } from '$lib/workspaceProtectionRules.svelte' - import { migrateLegacyUserDrafts } from '$lib/userDraftLegacyMigration' + import { purgeLegacyUserDrafts } from '$lib/userDraftLegacyMigration' import { migrateUserDraftsToDb } from '$lib/userDraftDbMigration' import DraftMigrationErrorModal from '$lib/components/DraftMigrationErrorModal.svelte' import { setContext, untrack } from 'svelte' @@ -435,16 +435,17 @@ $effect(() => { $workspaceStore && untrack(() => onLoad()) }) - // One-shot UserDraft migration chain. `migrateLegacyUserDrafts` folds - // the legacy `flow` / `app-…` / `rawapp-…` LS keys into the - // `userdraft/w/{ws}/{kind}/{path}` format; `migrateUserDraftsToDb` - // then pushes those onto the server-side draft table and clears LS - // on success. The order matters — the second step only sees what - // the first one normalized. + // One-shot UserDraft migration. `purgeLegacyUserDrafts` drops the oldest + // workspace-blind `flow` / `app-…` / `rawapp-…` LS autosave keys (they + // can't be attributed to a workspace, so promoting them would mis-file + // drafts). `migrateUserDraftsToDb` then pushes the workspace-scoped + // `userdraft/w/{ws}/{kind}/{path}` keys — written by the editor with the + // correct workspace — onto the server-side draft table, clearing LS on + // success. $effect(() => { if ($workspaceStore && $userStore) { untrack(() => { - migrateLegacyUserDrafts($workspaceStore!) + purgeLegacyUserDrafts() void migrateUserDraftsToDb() }) } From e16061df06babeae935a9396de5bdcd46e8119a9 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 09:37:29 +0200 Subject: [PATCH 020/117] fix(health): detect read-only replica via pg_is_in_recovery() (#9722) The /api/health/status database check used `SELECT 1`, which succeeds even on a read-only standby. After a PostgreSQL failover where the primary becomes a secondary, the health check kept reporting healthy while all writes failed with "cannot execute INSERT in a read-only transaction", so Kubernetes liveness probes never restarted the pod. Use `SELECT NOT pg_is_in_recovery()` instead: it returns true on a primary and false on a standby, so a read-only replica is now reported unhealthy. Result handling checks the returned bool (Ok(Some(true))) rather than just query success. Fixes WIN-2085 Co-authored-by: Claude Opus 4.8 (1M context) --- ...ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json} | 6 +++--- backend/windmill-api/src/health.rs | 8 ++++++-- 2 files changed, 9 insertions(+), 5 deletions(-) rename backend/.sqlx/{query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json => query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json} (59%) diff --git a/backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json b/backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json similarity index 59% rename from backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json rename to backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json index 0769d083d6..4e5f9f6ed7 100644 --- a/backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json +++ b/backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json @@ -1,12 +1,12 @@ { "db_name": "PostgreSQL", - "query": "SELECT 1", + "query": "SELECT NOT pg_is_in_recovery()", "describe": { "columns": [ { "ordinal": 0, "name": "?column?", - "type_info": "Int4" + "type_info": "Bool" } ], "parameters": { @@ -16,5 +16,5 @@ null ] }, - "hash": "e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5" + "hash": "282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf" } diff --git a/backend/windmill-api/src/health.rs b/backend/windmill-api/src/health.rs index fab9979c12..4dc8947de6 100644 --- a/backend/windmill-api/src/health.rs +++ b/backend/windmill-api/src/health.rs @@ -219,12 +219,16 @@ struct DatabaseCheckResult { async fn check_database_with_latency(db: &DB) -> DatabaseCheckResult { let start = std::time::Instant::now(); + // `pg_is_in_recovery()` is true on standbys/read-only replicas, so a primary + // returns true here. A read-only replica (e.g. after a failover where the + // primary became a secondary) reports unhealthy, letting liveness probes + // restart the pod instead of silently failing all writes. let healthy = tokio::time::timeout( HEALTH_CHECK_TIMEOUT, - sqlx::query_scalar!("SELECT 1").fetch_one(db), + sqlx::query_scalar!("SELECT NOT pg_is_in_recovery()").fetch_one(db), ) .await - .map(|r| r.is_ok()) + .map(|r| matches!(r, Ok(Some(true)))) .unwrap_or(false); let latency_ms = start.elapsed().as_millis() as i64; From e82a6a68309d862a78090d2c8fe084c354eb7279 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 09:44:35 +0200 Subject: [PATCH 021/117] chore(main): release 1.736.0 (#9720) * chore(main): release 1.736.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> --- CHANGELOG.md | 16 ++ backend/Cargo.lock | 160 +++++++++--------- backend/Cargo.toml | 4 +- .../parsers/windmill-parser-wasm/Cargo.lock | 48 +++--- .../parsers/windmill-parser-wasm/Cargo.toml | 2 +- backend/windmill-api/openapi.yaml | 2 +- benchmarks/lib.ts | 2 +- cli/src/core/constants.ts | 2 +- frontend/package-lock.json | 4 +- frontend/package.json | 2 +- lsp/Pipfile | 2 +- openflow.openapi.yaml | 2 +- .../WindmillClient/WindmillClient.psd1 | 2 +- python-client/wmill/pyproject.toml | 2 +- typescript-client/jsr.json | 2 +- typescript-client/package.json | 2 +- version.txt | 2 +- 17 files changed, 136 insertions(+), 120 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ebd157e41d..47e91fa1a0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,21 @@ # Changelog +## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23) + + +### Features + +* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd)) + + +### Bug Fixes + +* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72)) +* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c)) +* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76)) +* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9)) +* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6)) + ## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22) diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 5a082c9aa3..de575d4aa1 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -7046,9 +7046,9 @@ checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" [[package]] name = "memmap2" -version = "0.9.10" +version = "0.9.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "714098028fe011992e1c3962653c96b2d578c4b4bce9036e15ff220319b1e0e3" +checksum = "d1219ed1b7f229ee7104d281dd01d6802fe28bb6e95d292942c4daacdeb798c0" dependencies = [ "libc", "stable_deref_trait", @@ -13735,7 +13735,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-nats", @@ -13817,7 +13817,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.735.0" +version = "1.736.0" dependencies = [ "async-stream", "async-trait", @@ -13850,7 +13850,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -13863,7 +13863,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "argon2", @@ -14001,7 +14001,7 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14024,7 +14024,7 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14037,7 +14037,7 @@ dependencies = [ [[package]] name = "windmill-api-auth" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14063,7 +14063,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.735.0" +version = "1.736.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14073,7 +14073,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14090,7 +14090,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14112,7 +14112,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14135,7 +14135,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14151,7 +14151,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14172,7 +14172,7 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14193,7 +14193,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14207,7 +14207,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-nats", @@ -14242,7 +14242,7 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14267,7 +14267,7 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "flate2", @@ -14285,7 +14285,7 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14307,7 +14307,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14327,7 +14327,7 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14364,7 +14364,7 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14392,7 +14392,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.735.0" +version = "1.736.0" dependencies = [ "lazy_static", "serde", @@ -14404,7 +14404,7 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.735.0" +version = "1.736.0" dependencies = [ "argon2", "axum 0.8.9", @@ -14429,7 +14429,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14443,7 +14443,7 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.735.0" +version = "1.736.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14476,7 +14476,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.735.0" +version = "1.736.0" dependencies = [ "chrono", "lazy_static", @@ -14490,7 +14490,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14509,7 +14509,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.735.0" +version = "1.736.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -14611,7 +14611,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.735.0" +version = "1.736.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -14630,7 +14630,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.735.0" +version = "1.736.0" dependencies = [ "regex", "serde", @@ -14645,7 +14645,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -14669,7 +14669,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "futures", @@ -14686,7 +14686,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.735.0" +version = "1.736.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -14702,7 +14702,7 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -14723,7 +14723,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -14754,7 +14754,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "arc-swap", @@ -14779,7 +14779,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-stream", @@ -14813,7 +14813,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "futures", @@ -14831,7 +14831,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.735.0" +version = "1.736.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -14840,7 +14840,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "lazy_static", @@ -14852,7 +14852,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde_json", @@ -14864,7 +14864,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "gosyn", @@ -14876,7 +14876,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "lazy_static", @@ -14888,7 +14888,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde_json", @@ -14900,7 +14900,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "nu-parser", @@ -14911,7 +14911,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14922,7 +14922,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14934,7 +14934,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "rustpython-ast", @@ -14945,7 +14945,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-recursion", @@ -14967,7 +14967,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde_json", @@ -14979,7 +14979,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "lazy_static", @@ -14993,7 +14993,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15010,7 +15010,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "lazy_static", @@ -15023,7 +15023,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde", @@ -15035,7 +15035,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "lazy_static", @@ -15053,7 +15053,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15069,7 +15069,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15085,7 +15085,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde", @@ -15096,7 +15096,7 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-recursion", @@ -15134,7 +15134,7 @@ dependencies = [ [[package]] name = "windmill-runtime-nativets" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "const_format", @@ -15173,7 +15173,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.735.0" +version = "1.736.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15184,7 +15184,7 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-recursion", @@ -15216,7 +15216,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15240,7 +15240,7 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15273,7 +15273,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15306,7 +15306,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15326,7 +15326,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15360,7 +15360,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15396,7 +15396,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15419,7 +15419,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15443,7 +15443,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-nats", @@ -15467,7 +15467,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15502,7 +15502,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15530,7 +15530,7 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-trait", @@ -15555,7 +15555,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "bitflags 2.13.0", @@ -15574,7 +15574,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-once-cell", @@ -15684,7 +15684,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.735.0" +version = "1.736.0" dependencies = [ "bytes", "futures", diff --git a/backend/Cargo.toml b/backend/Cargo.toml index 613901a7bc..db53f33726 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.735.0" +version = "1.736.0" authors.workspace = true edition.workspace = true @@ -87,7 +87,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.735.0" +version = "1.736.0" authors = ["Ruben Fiszel "] edition = "2021" diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index 6037b9f8b7..d7f194b2d1 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.735.0" +version = "1.736.0" dependencies = [ "aho-corasick", "anyhow", @@ -6272,7 +6272,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.735.0" +version = "1.736.0" dependencies = [ "proc-macro2", "quote", @@ -6284,7 +6284,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.735.0" +version = "1.736.0" dependencies = [ "convert_case", "serde", @@ -6293,7 +6293,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "lazy_static", @@ -6305,7 +6305,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde_json", @@ -6317,7 +6317,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "gosyn", @@ -6329,7 +6329,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "lazy_static", @@ -6341,7 +6341,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde_json", @@ -6353,7 +6353,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "nu-parser", @@ -6364,7 +6364,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6375,7 +6375,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6387,7 +6387,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6398,7 +6398,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "async-recursion", @@ -6420,7 +6420,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde_json", @@ -6432,7 +6432,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "lazy_static", @@ -6446,7 +6446,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "convert_case", @@ -6463,7 +6463,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "lazy_static", @@ -6476,7 +6476,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde", @@ -6488,7 +6488,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "lazy_static", @@ -6506,7 +6506,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6522,7 +6522,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6538,7 +6538,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6570,7 +6570,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "serde", @@ -6581,7 +6581,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.735.0" +version = "1.736.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index 9e9d9eb7e3..a17cd25e9c 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.735.0" +version = "1.736.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index f1fe8447d4..c98caa43dd 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: - version: 1.735.0 + version: 1.736.0 title: Windmill API contact: diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index 1bd44ae76b..9c39887880 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.735.0"; +export const VERSION = "v1.736.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ diff --git a/cli/src/core/constants.ts b/cli/src/core/constants.ts index 23a8ec4410..d80f1b123d 100644 --- a/cli/src/core/constants.ts +++ b/cli/src/core/constants.ts @@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork"; // (e.g. utils.ts) can read it without importing main.ts and creating a circular // dependency (main → workspace → utils → main) that triggers a TDZ. // Re-exported from main.ts for backwards compatibility. -export const VERSION = "1.735.0"; +export const VERSION = "1.736.0"; diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 88fc25cf99..c1ab7d0fc3 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "@windmill-labs/components", - "version": "1.735.0", + "version": "1.736.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@windmill-labs/components", - "version": "1.735.0", + "version": "1.736.0", "hasInstallScript": true, "license": "AGPL-3.0", "dependencies": { diff --git a/frontend/package.json b/frontend/package.json index beaa1f889e..94a27efcad 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "@windmill-labs/components", - "version": "1.735.0", + "version": "1.736.0", "scripts": { "dev": "vite dev", "dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev", diff --git a/lsp/Pipfile b/lsp/Pipfile index 417ecebb3c..d6dde9126a 100644 --- a/lsp/Pipfile +++ b/lsp/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -wmill = ">=1.735.0" +wmill = ">=1.736.0" sendgrid = "*" mysql-connector-python = "*" pymongo = "*" diff --git a/openflow.openapi.yaml b/openflow.openapi.yaml index 792afd9098..7fb41c38e8 100644 --- a/openflow.openapi.yaml +++ b/openflow.openapi.yaml @@ -1,7 +1,7 @@ openapi: '3.0.3' info: - version: 1.735.0 + version: 1.736.0 title: OpenFlow Spec contact: name: Ruben Fiszel diff --git a/powershell-client/WindmillClient/WindmillClient.psd1 b/powershell-client/WindmillClient/WindmillClient.psd1 index fc0fc4b0f4..9583aa5711 100644 --- a/powershell-client/WindmillClient/WindmillClient.psd1 +++ b/powershell-client/WindmillClient/WindmillClient.psd1 @@ -12,7 +12,7 @@ RootModule = 'WindmillClient.psm1' # Version number of this module. - ModuleVersion = '1.735.0' + ModuleVersion = '1.736.0' # Supported PSEditions # CompatiblePSEditions = @() diff --git a/python-client/wmill/pyproject.toml b/python-client/wmill/pyproject.toml index 23e4643e3c..bf1a825f18 100644 --- a/python-client/wmill/pyproject.toml +++ b/python-client/wmill/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "wmill" -version = "1.735.0" +version = "1.736.0" description = "A client library for accessing Windmill server wrapping the Windmill client API" license = "Apache-2.0" homepage = "https://windmill.dev" diff --git a/typescript-client/jsr.json b/typescript-client/jsr.json index 8d54d7d946..27b20dd666 100644 --- a/typescript-client/jsr.json +++ b/typescript-client/jsr.json @@ -1,6 +1,6 @@ { "name": "@windmill/windmill", - "version": "1.735.0", + "version": "1.736.0", "exports": "./src/index.ts", "publish": { "exclude": ["!src", "./s3Types.ts", "./sqlUtils.ts", "./client.ts"] diff --git a/typescript-client/package.json b/typescript-client/package.json index fdd060be47..197d39772a 100644 --- a/typescript-client/package.json +++ b/typescript-client/package.json @@ -1,7 +1,7 @@ { "name": "windmill-client", "description": "Windmill SDK client for browsers and Node.js", - "version": "1.735.0", + "version": "1.736.0", "author": "Ruben Fiszel", "license": "Apache 2.0", "homepage": "https://github.com/windmill-labs/windmill/tree/main/typescript-client#readme", diff --git a/version.txt b/version.txt index bb6e9e6fd1..20bc4b92da 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.735.0 +1.736.0 From 2879cbb65a4122c86b4a472206d74d9009b07904 Mon Sep 17 00:00:00 2001 From: hugocasa Date: Tue, 23 Jun 2026 10:05:00 +0200 Subject: [PATCH 022/117] feat(apps): opt-in sandbox isolation for published & raw apps (alpha) (#9420) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(apps): sandbox published & raw apps with a scoped embed token Co-Authored-By: Claude Opus 4.8 (1M context) * chore: point ee-repo-ref at embed-token EE commit Co-Authored-By: Claude Opus 4.8 (1M context) * feat(apps): allow top-navigation from the sandboxed app iframe Co-Authored-By: Claude Opus 4.8 (1M context) * feat(apps): share app localStorage across apps via the embedder Co-Authored-By: Claude Opus 4.8 (1M context) * feat(apps): publisher disable-sandbox option with per-version viewer consent Co-Authored-By: Claude Opus 4.8 (1M context) * chore(sqlx): cache for disable-sandbox queries Co-Authored-By: Claude Opus 4.8 (1M context) * chore: bump ee-repo-ref to disable-sandbox EE commit Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): always sandbox the served raw-app wrapper + viewer fixes The raw-app wrapper served by get_raw_app_data now always carries `CSP: sandbox`. The publisher "disable sandbox isolation" opt-out is applied entirely on the viewer side, which (after per-version consent) builds its own same-origin blob wrapper — so the backend-served document stays isolated regardless of how it is reached, never via a relaxed real-origin URL. Also: - CORS on the global /apps_u mount so the opaque viewer can load custom-path public apps cross-origin. - Reject runnable-bridge messages unconditionally until the iframe is bound. - Relay the viewer's in-app hash up to the embedder address bar so deep links stay shareable (hash only; embedder keeps its own pathname). Co-Authored-By: Claude Opus 4.8 * feat(apps): render public raw apps single-iframe (drop embed token) Public raw apps now render directly on the real origin with a single opaque bundle iframe and the page credential, instead of the opaque viewer + scoped-token indirection. The author bundle stays isolated in its own opaque iframe (CSP-sandboxed); low-code apps, whose code runs in the viewer frame, keep the opaque viewer + scoped token. embed_token now reports raw_app and skips minting a token for raw apps; the access check still gates visibility. Also set disable_sandbox: None in the remaining Policy constructors so the full feature build (all_sqlx_features, enterprise, license) compiles. Co-Authored-By: Claude Opus 4.8 * chore: bump ee-repo-ref to single-iframe raw-app EE commit Co-Authored-By: Claude Opus 4.8 * feat(apps): grandfather existing apps as legacy-unsandboxed + authed-only consent Existing apps are stamped by migration as `legacy_unsandboxed` so they keep running same-origin on upgrade — no breakage and no consent prompt. New apps are sandboxed by default; re-deploying an app clears the flag. The publisher `disable_sandbox` consent prompt is now shown only to authenticated viewers — an anonymous viewer has no session to expose, so the prompt was meaningless friction. embed_token reports `legacy_unsandboxed` and `authed`. Co-Authored-By: Claude Opus 4.8 * chore: bump ee-repo-ref to legacy-unsandboxed EE commit Co-Authored-By: Claude Opus 4.8 * feat(apps): deploy-time migration prompt for legacy-unsandboxed apps On the first re-deploy of a grandfathered (legacy-unsandboxed) app, the publisher must explicitly choose: enable sandbox isolation (the flag is cleared → the app becomes sandboxed) or keep running without isolation (→ disable_sandbox, with per-version viewer consent). updatePolicy() no longer carries the legacy flag through a deploy, so the choice is what sticks. Co-Authored-By: Claude Opus 4.8 * fix(apps): disable the sandbox-isolation toggle until the app is deployed The Deploy-drawer "Disable sandbox isolation" toggle called setPublishState() — which updates the app by path — even before the app was first deployed, when the path is empty, throwing an error. Guard it with disabled={!savedApp}, matching the adjacent visibility toggle. Co-Authored-By: Claude Opus 4.8 * feat(apps): sandbox the in-workspace low-code app viewer in an opaque iframe Extend the opaque-origin iframe isolation to the logged-in /apps/get viewer. /apps/get becomes an embedder that keeps the workspace chrome + Edit button and renders the app inside a cookieless, chrome-less /app_embed viewer route, handed a scoped embed token minted from the member's session. The app frame runs in an opaque origin (no allow-same-origin), so it cannot reach the member's session cookie or window.parent. - apps.rs: get_app_embed_token_for_path (authed, by-path, scope + RLS gated); mint_app_embed_token grants a path-scoped apps:read:{path} so the viewer can load its own app definition and no other - lib.rs: CORS on /apps (bearer-token only, no cookies) for the opaque viewer's by-path reads - new /app_embed/[workspace]/[...path] viewer route (private analog of /public) - PublicAppFrame: viewerUrl prop to point the opaque iframe at the viewer route Co-Authored-By: Claude Opus 4.8 (1M context) * feat(apps): unify in-workspace app viewers on the shared sandboxed path Route every in-workspace app display (low-code and raw) through the same PublicAppFrame -> PublicApp machinery as the public viewer, so the sandbox / legacy-unsandboxed / disable-sandbox-consent behavior is identical on every page. - new InWorkspaceAppViewer renders both app types via PublicAppFrame; /apps/get and /apps_raw/get become thin wrappers over it - /apps_raw/get previously rendered RawAppPreview directly (always isolated, with no legacy-grandfathering or consent handling); now consistent with the rest - retire the legacy same-origin raw viewer /apps/get_raw/[version] and re-point the apps-list row to /apps_raw/get; remove the dead /apps_raw/[ws]/[version] route - load the raw bundle secret in the shared viewer (getAppByPath doesn't return it) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): address PR review feedback (scope + policy hardening, nits) - require handler-level apps:read on list_apps / list_search_apps so a scoped embed token cannot read app definitions through the list endpoints. The route layer treats apps:run as satisfying read; the handler check (which does not) closes the gap. - treat legacy_unsandboxed as backend-owned: strip any client-provided value in create/update so it can only be set by the grandfather migration, not the API. - document mint_app_embed_token's caller-verifies-access contract. - use Button's declared onClick prop for the consent action (was onclick, which fell into the rest-spread and bypassed the component's click handling). - test: lock that the embed scopes cannot satisfy domain-level apps:read. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(apps): document embed-token endpoints in openapi + fix doc nit Second-round review nits: - add the three app embed-token endpoints (apps/embed_token/p/{path}, apps_u/embed_token/{secret}, and the EE apps_u/embed_token_by_custom_path) plus the EmbedTokenResponse schema to openapi.yaml; note .html on get_data - mint_app_embed_token doc: "Both" -> "All" (it lists three call sites) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(apps): bound embed-token scopes to the caller's own The embed-token mint now enforces ensure_scopes_within_caller, so the minted scope set is always within the calling credential's own scopes (a no-op for regular unscoped sessions). Adds a unit test locking the boundary and documents the contract on mint_app_embed_token. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): raw-app ctx in external embeds + page credential in direct render - RawAppPreview: engage the storage relay only in opaque frames (probe Web Storage instead of just window.parent), so a public raw app embedded in an external iframe hydrates ctx/storage directly; add a relay-timeout fallback so an unresponsive parent can never stall the ctx handshake. - PublicAppFrame: in direct render, expose the page's own bearer credential through the AuthToken context (JWT public URLs), matching the previous route behavior; opaque-viewer mode keeps the embed token. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(apps): sandbox isolation UI polish + COI embed support for raw apps - Deploy drawer: move the sandbox toggle out of "Public URL" into its own "Sandbox isolation" section (the setting applies to every viewing surface, not just the public URL), with positive phrasing, visible helper text, and state-aware alerts (warning when disabled, info for pre-isolation apps). Toggling it now toasts its own message instead of the login-mode one. - Extract the deploy-time migration prompt into a shared LegacySandboxMigrationModal built on the common Modal component, and wire it into the raw app editor header too (it previously had no prompt, so re-deploying a pre-isolation raw app silently changed behavior). updateRawAppPolicy now also drops the backend-owned legacy flag, matching the low-code updatePolicy. - Viewer consent prompt: use the common ConfirmationModal and show the app path (new appPath prop) instead of the route pathname, falling back to "this app" when the path isn't known yet. - COI embeds: propagate the wm_coep opt-in to the raw-app wrapper document and have the backend assert COEP require-corp on it when the flag is present — required for the bundle iframe to load when the public app page is embedded inside a cross-origin-isolated page. Previously this only worked in dev because the Vite proxy injects the header; the production response lacked it. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(apps): app navigation parity across sandboxed and direct viewers - Navbar component: same-app items relay query + hash to the embedder page (which mirrors them onto the root URL, keeping its own pathname and transport params), app items navigate the top page through a validated wm_embed_navigate relay instead of the cookieless viewer iframe, and external items keep opening a new tab. Selected-item detection now recognizes the /app_embed viewer route and ignores transport params. - Frontend-script `goto` and button `onSuccess: gotoUrl`: same-window navigation goes through a shared appNavigateSameWindow helper that relays to the embedder inside the opaque viewer (same-origin paths SPA-navigate, http(s) URLs do a full load, other schemes rejected) and keeps plain window.location everywhere else. - /apps/get and /apps_raw/get: key the viewer by workspace/path so in-route navigation fully remounts it — previously the URL changed but the app (and in sandbox mode its path-scoped token) did not follow. - wm_embed/wm_embedder_origin added to the reserved query params so they no longer leak into the app's ctx.query. - Raw apps: drop the sandbox attribute entirely for the unsandboxed (grandfathered/consented) blob path, matching the pre-isolation viewer exactly — the attribute added no isolation there and sandboxed popups (e.g. OAuth flows). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): preserve grandfathered policy across updates + in-workspace viewer parity Round of compatibility hardening so pre-existing apps behave exactly as before on every surface: - `legacy_unsandboxed` is now preserved across app updates unless the payload explicitly clears it (`false`, sent by the editor's migration prompt and the sandbox toggle). Unrelated update paths — CLI / git-sync redeploys, publish-mode toggles, cross-workspace promotion — no longer silently drop the grandfathering. Clients still can never SET the flag. - The embed-token endpoints (secret, path, EE custom-path) read only the sandbox-decision policy fields, leniently, and no longer mint a token for raw / legacy / disable_sandbox renders: the token is only consumed by the sandboxed low-code render, and minting for the others wrote a useless token row per view and could fail the render for scope-restricted callers. - In-workspace viewer parity with the pre-sandbox `/apps/get`: new `inWorkspace` mode on PublicApp (no "Powered by Windmill" badge / user overlay, no HTML-result approval gate, column flex wrapper, `hideRefreshBar` honored again), and the page's query/hash are forwarded into the opaque viewer so `ctx.query` / `ctx.hash` reach the app. - Raw apps: `window.ctx` is always `{ctx, workspace}` again (anonymous viewers of pre-existing bundles rely on `ctx.workspace`), and the runnable bridge's job-id scoping now applies only to sandboxed renders (`gateJobIds`) — an unsandboxed bundle holds the same credential as the bridge, so gating there only broke pre-existing apps polling persisted or runnable-returned job ids. - Document `disable_sandbox` / `legacy_unsandboxed` in the openapi Policy schema; add a unit test for the lenient policy read. - bump ee-repo-ref to the matching EE commit. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): keep share-link viewer credentials out of the isolated app context The JWT path segment of authenticated share URLs is an embedder-side credential, consumed only to mint the scoped embed token. Two transport channels still copied it into the isolated frame where app-authored code runs: - the opaque viewer iframe src defaulted to window.location.href — the public and custom-path routes now pass a sanitized viewerUrl (JWT segment stripped, query/hash preserved, captured once so the hash relay does not reload the iframe); - document.referrer on the same-origin iframe navigation carried the full embedder URL — both app iframes now set referrerpolicy="no-referrer" (sandboxed renders only for the raw bundle iframe, keeping exact legacy parity; nothing reads the referrer). Co-Authored-By: Claude Opus 4.8 (1M context) * chore(frontend): drop unused import inherited from main merge `slide` import in AssistantMessage.svelte (from #9539) turns `npm run check` red on this branch. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): redirect the removed raw-app viewer path to the unified viewer The old same-origin raw-app viewer route (/apps/get_raw/{version}/{path}) was removed in favor of the sandboxed unified viewer. Re-add a thin client route at the old path that redirects stale bookmarks to /apps_raw/get/{path}, preserving query + hash (the pinned version is dropped — the unified viewer shows latest). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): narrow embed-token scopes and base consent on browser session - Embed token: resource access is metadata-only (list/type/exists) via a `resources:run` marker — resource values (get/get_value/get_value_interpolated/ list_search) are no longer reachable. Job reads are by-id only: an `app_embed` sentinel blocks the workspace-wide job enumeration/export routes (jobs/list, list_filtered_uuids, queue/list, completed/list, queue/export) while by-id result polling keeps working. - disable_sandbox consent now gates on whether the browser holds any Windmill session (cookie-only whoami) rather than workspace-scoped auth, so a viewer logged into a different workspace is still prompted before a same-origin render. - db-explorer: resolve the MySQL database name server-side (the metadata query already falls back to DATABASE()) instead of reading the resource value client-side; getTablesByResource derives the default db from the schema. Co-Authored-By: Claude Opus 4.8 (1M context) * chore(apps): trim embed-scope and consent comments Reduce duplication — state the resource/job route exclusions and the workspace-session-vs-cookie rationale once at their source and reference them elsewhere; drop contrast/justification phrasing. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(apps): make app sandbox isolation opt-in (alpha) Replace the disable_sandbox + legacy_unsandboxed policy pair and the per-version viewer consent with a single positive `sandbox` opt-in flag. Apps are unsandboxed by default (same-origin, full session — the pre-isolation behavior), so existing apps are unchanged and no migration is needed. Publishers opt an app into isolation from the deploy drawer, flagged alpha. - Policy.sandbox: Option; EmbedTokenResponse -> {token, expiration, raw_app, sandbox}; mint an embed token only for sandboxed low-code apps. - Drop the legacy-unsandboxed migration and the deploy-time migration prompt; remove the consent modal and the browser-session probe. - Deploy drawer: a single "Sandbox isolation" toggle (alpha), off by default, shared by the low-code and raw editors. - Bump ee-repo-ref to the companion EE commit. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): confine embed token to its intended user/folder/job routes The embed token's broad read scopes spanned whole domains while the matching routers are CORS-enabled for the opaque app iframe: - users:read / folders:read were domain-wide, so the token could reach users/list, users/list_usage, users/username_to_email/*, folders/list, etc. Restrict to an app_embed-sentinel allowlist: only users/whoami and folders/listnames; deny the rest of those domains. - jobs:read allowed jobs/completed/export, missed by the job denylist. Add it alongside jobs/queue/export. Extend the embed-scope allow/deny test matrix to cover all of these. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(apps): align sandbox comments with the opt-in model The consent prompt, deploy-time migration, and legacy-unsandboxed grandfathering were removed when sandbox isolation became an opt-in policy flag; update the comments that still described them so they match the two-state (default-unsandboxed / opt-in-sandboxed) reality. Comments only, no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): confine embed-token job reads to runs the app launched App component jobs are stamped `created_by = the viewer`, so an embed token reads its own runs via the launched-by-viewer fast path. The token then also inherited the viewer's broader job access (share links, folder ACLs, admin RLS), letting user-authored app JS reuse it to read unrelated jobs by id. Stop embed tokens at the fast path: only jobs the viewer launched, never those merely visible to them. Return NotFound so the untrusted app can't probe existence. Regression test: an embed token reads its own launched job but is denied the foreign job (result/logs/getupdate) an admin viewer's normal token can read. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): allowlist embed-token apps/jobs routes + scope run to the app The embed token's apps:run/jobs:read reached more than a running app needs. Replace the job denylist with strict per-domain allowlists on the app_embed sentinel: - Apps: only the app's own definition (apps/get/p/) and the public app-serving endpoints (apps_u/*). Denies workspace app inventory (exists, custom_path_exists, list, list_paths*). - Jobs: only the by-id poll routes the frontend JobLoader uses. Denies job counts and the job_signature/resume_urls capability-minting routes (the by-id reads remain confined to the app's own runs). Drop unqualified apps:run from APP_EMBED_SCOPES; mint apps:run: instead and authorize apps:run: first in execute_component, so the token can only run its own app's components, not another app's. Extend the embed-scope route matrix and add a path-scoped run unit test. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(apps): clarify the sandbox toggle vs the on-behalf-of model The deploy-drawer sandbox copy leaned on "session" in a way that collided with the on-behalf-of permissioning right above it. Reword it to say the toggle governs what the app's browser-side code can reach in the viewer's browser — distinct from who its runnables execute as — and rename the label to "Isolate the app from the viewer's browser session". Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): path-scope embed-token S3 download to its own app The apps_u/* allowlist also admitted apps_u/download_s3_file/, whose handler authorized any authenticated caller — so an embed token minted for app A could download app B's S3 files via B's on-behalf policy. Add the same path-scoped guard execute_component uses: download_s3_file_from_app now checks apps:read: first, confining the token to its own app. Other path-taking apps_u routes are already covered (writes lack apps:write; embed_token/p path-checks; public_resource is type-constrained). Extend the path-scoping unit test to cover apps:read (download) alongside apps:run (execute). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): path-scope public-app-by-secret read to the embed token's app The apps_u/* allowlist admitted apps_u/public_app/, whose handler only checked the viewer's read access — so an embed token minted for app A could read app B's definition by secret (confused deputy via the viewer's identity). get_public_app_by_secret now binds a scoped caller to the resolved app with check_scopes(apps:read:), confining it to its own app; unscoped sessions and anonymous access are unchanged. get_raw_app_data needs no binding (pure secret capability, no caller identity). Document the full set of app-resolving handlers the path-scoped read covers. Bump ee-repo-ref for the companion custom-path fix. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): preserve pre-sandbox behavior for db-explorer, edit link, jwt Three behavior-parity fixes for non-sandboxed (existing) apps that the sandbox-isolation refactor changed incidentally: - DB-explorer MySQL table picker: when the connection can see multiple non-system schemas, label the default db's tables unprefixed again. The resource-value read was removed globally, so identify the default db from the introspection script's `DATABASE() AS default_db_name` (carried on SQLSchema.defaultDb) instead of guessing "the single schema key". Equivalent to the prior resource.database match; editor-only (table picker). - In-workspace Edit button: restore `?nodraft=true` on both /apps/get and /apps_raw/get, so opening the editor from the viewer loads the deployed version, not a draft. - Custom-path (/a) viewer: restore the "could not authenticate user with jwt token" toast when a path JWT fails to resolve a user, instead of silently falling through. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): confine embed-token S3 downloads to the app's own keys/outputs download_s3_file_from_app authorized any authenticated caller for any S3 key (opt_authed.is_some() bypass). A sandboxed app's embed token carries the viewer's identity, so app-authored JS could fetch arbitrary S3 keys readable by the on-behalf identity, beyond the app's own declared keys or outputs. Route app embed tokens through the same allowlist as anonymous viewers — the app's declared allowed_s3_keys, or files produced by this app's own component runs — instead of the authed bypass. The produced-files check is parameterized by created_by (the embed viewer for a token, else anonymous) so a sandboxed app's own S3 outputs still render while arbitrary keys are denied. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): let embed tokens cancel their own jobs; gate cancel to launcher A sandboxed low-code app supersedes an in-flight component run on re-run by canceling it, but the embed token only had jobs:read, so cancellation silently failed and prior jobs ran to completion. - Permit the by-id jobs_u/queue/cancel POST for app_embed tokens at the route layer (the only write reachable through the existing by-id allowlist). - Gate cancel_job_api: an app_embed token may cancel ONLY jobs it launched (created_by == viewer). cancel_job_api had no other per-job ownership check, so this also confines the token instead of letting it cancel any job by id. - /app_embed now sets workspaceStore so cancellation targets the right workspace instead of an empty/stale one in the cookieless iframe. Add a shared has_app_embed_sentinel helper; cover cancel in the route matrix and the jobs_read_auth integration test (own job cancelable, foreign denied). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): drop get_root_job_id from the embed-token job allowlist Audit of the embed token's reachable job routes: get_root_job (jobs_u/ get_root_job_id) has no access check in its handler at all — it returns any job's root-job id by id — and the app runtime never calls it. Remove it from the by-id allowlist so the embed token can't probe a foreign job's flow lineage; add a denied-route assertion. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(apps): scope sandboxed-app localStorage per app Sandboxed apps shared one localStorage store (one key on the real origin), so an app could read or clobber another app's keys — and, with job ids stashed there, reuse its embed token to read another app's job. Scope the backing store per app. The embed-token endpoints now return the resolved app_path (EmbedTokenResponse; not a new disclosure — the viewer already receives the path when it loads the app). PublicAppFrame (low-code) and RawAppPreview (raw) key their backing store by it: wm_apps_localstorage:. Same app shares one store across its public and in-workspace surfaces; different apps are isolated. Unsandboxed apps are unaffected (real same-origin localStorage, as before). Bump ee-repo-ref for the companion custom-path change. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): scope embed access checks to embed tokens + key app storage by workspace - Apply the path-scoped read/run checks on the public-by-secret read and the component run path only when the caller is an app embed token, so other caller types keep their prior access. - Key the sandboxed app's backing client storage by workspace + path instead of path alone, and return the resolved workspace from the embed-token endpoints so the custom-path viewer can derive it. - Show a clear message instead of an indefinite loader when the viewer route is opened outside its embedder. Bumps ee-repo-ref to 5b8476b. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): mint embed tokens only from the trusted embedder caller An app embed token must not reach the embed-token mint endpoints; refresh minting stays with the embedder session/JWT. Enforced at the scope route layer and at the mint chokepoint, with a route-matrix regression test. Co-Authored-By: Claude Opus 4.8 * fix(apps): support S3 upload and frontend-script S3 download in sandboxed apps Sandboxed apps run with a scoped embed token (no cookie). Let the app's S3 file-input upload and the frontend-script download({s3}) helper work in that context: upload is reachable with apps:run and re-checked per-app at the handler; the script download routes through the app-scoped apps_u endpoint with the embed token instead of the cookie-authed job_helpers path. Default (unsandboxed) apps are unchanged. Co-Authored-By: Claude Opus 4.8 * chore: update ee-repo-ref to b0cb761bf9852974e571b2978032d310cc998517 This commit updates the EE repository reference after PR #600 was merged in windmill-ee-private. Previous ee-repo-ref: e673c714a4618fdb72353a475f49c748e6016642 New ee-repo-ref: b0cb761bf9852974e571b2978032d310cc998517 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: windmill-internal-app[bot] Co-authored-by: Ruben Fiszel --- ...9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json | 35 + ...1fdf5c006554a70a322ca8d3449cdba7840db.json | 41 + ...63385785f7c2e97364bc04392df11cf364d7.json} | 5 +- backend/ee-repo-ref.txt | 2 +- backend/tests/fixtures/jobs_read_auth.sql | 39 + backend/tests/jobs_read_auth.rs | 86 ++ backend/windmill-api-auth/src/scopes.rs | 120 ++- backend/windmill-api/openapi.yaml | 101 +++ backend/windmill-api/src/apps.rs | 821 +++++++++++++++++- backend/windmill-api/src/jobs.rs | 31 + backend/windmill-api/src/lib.rs | 37 +- frontend/src/app.html | 104 ++- .../components/display/AppNavbarItem.svelte | 20 +- .../display/dbtable/AppDbExplorer.svelte | 37 +- .../components/display/dbtable/metadata.ts | 55 +- .../components/helpers/RunnableWrapper.svelte | 6 +- .../apps/components/helpers/eval.ts | 34 +- .../apps/editor/AppEditorHeader.svelte | 6 +- .../apps/editor/AppEditorHeaderDeploy.svelte | 37 +- .../components/apps/editor/AppPreview.svelte | 3 +- .../apps/editor/InWorkspaceAppViewer.svelte | 146 ++++ .../components/apps/editor/PublicApp.svelte | 77 +- .../apps/editor/PublicAppFrame.svelte | 427 +++++++++ .../lib/components/apps/editor/appPolicy.ts | 3 +- frontend/src/lib/components/apps/types.ts | 6 + frontend/src/lib/components/apps/utils.ts | 25 + .../components/common/table/RawAppRow.svelte | 2 +- .../raw_apps/RawAppBackgroundRunner.svelte | 36 +- .../components/raw_apps/RawAppEditor.svelte | 1 + .../raw_apps/RawAppEditorHeader.svelte | 6 +- .../components/raw_apps/RawAppPreview.svelte | 227 ++++- .../lib/components/raw_apps/rawAppPolicy.ts | 3 +- frontend/src/lib/components/raw_apps/utils.ts | 77 +- frontend/src/lib/stores.ts | 4 + frontend/src/lib/utils.ts | 6 +- .../src/routes/(root)/(logged)/+layout.svelte | 5 +- .../(logged)/apps/get/[...path]/+page.svelte | 103 +-- .../apps/get_raw/[version]/[...path]/+page.js | 5 - .../get_raw/[version]/[...path]/+page.svelte | 48 +- .../apps_raw/get/[...path]/+page.svelte | 81 +- frontend/src/routes/a/[...path]/+page.svelte | 80 +- .../[workspace]/[...path]/+page.svelte | 99 +++ .../[workspace]/[...version]/+page.js | 5 - .../[workspace]/[...version]/+page.svelte | 11 - .../[workspace]/[...secret]/+page.svelte | 105 ++- 45 files changed, 2732 insertions(+), 476 deletions(-) create mode 100644 backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json create mode 100644 backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json rename backend/.sqlx/{query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json => query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json} (64%) create mode 100644 frontend/src/lib/components/apps/editor/InWorkspaceAppViewer.svelte create mode 100644 frontend/src/lib/components/apps/editor/PublicAppFrame.svelte delete mode 100644 frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.js create mode 100644 frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte delete mode 100644 frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js delete mode 100644 frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte diff --git a/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json b/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json new file mode 100644 index 0000000000..20ae255eb1 --- /dev/null +++ b/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app\n FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.path = $1 AND a.workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "policy", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "raw_app", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + false + ] + }, + "hash": "e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd" +} diff --git a/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json b/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json new file mode 100644 index 0000000000..dd7f4da810 --- /dev/null +++ b/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT a.path, a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app\n FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.id = $1 AND a.workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "policy", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 3, + "name": "raw_app", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Int8", + "Text" + ] + }, + "nullable": [ + false, + null, + null, + false + ] + }, + "hash": "ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db" +} diff --git a/backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json b/backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json similarity index 64% rename from backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json rename to backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json index af4ada67f2..84eb4caba9 100644 --- a/backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json +++ b/backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT EXISTS (\n SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE j.workspace_id = $2\n AND (j.kind = 'appscript' OR j.kind = 'preview')\n AND j.created_by = 'anonymous'\n AND c.started_at > now() - interval '3 hours'\n AND j.runnable_path LIKE $3 || '/%'\n AND c.result @> ('{\"s3\":\"' || $1 || '\"}')::jsonb\n )", + "query": "SELECT EXISTS (\n SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE j.workspace_id = $2\n AND (j.kind = 'appscript' OR j.kind = 'preview')\n AND j.created_by = $4\n AND c.started_at > now() - interval '3 hours'\n AND j.runnable_path LIKE $3 || '/%'\n AND c.result @> ('{\"s3\":\"' || $1 || '\"}')::jsonb\n )", "describe": { "columns": [ { @@ -11,6 +11,7 @@ ], "parameters": { "Left": [ + "Text", "Text", "Text", "Text" @@ -20,5 +21,5 @@ null ] }, - "hash": "a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d" + "hash": "f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7" } diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index f746dd3faf..40ad91749d 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -de49fda2320504ad9e7d2d31c7033d71dbf6ca43 +b0cb761bf9852974e571b2978032d310cc998517 diff --git a/backend/tests/fixtures/jobs_read_auth.sql b/backend/tests/fixtures/jobs_read_auth.sql index e6b28fba0c..456ac8c801 100644 --- a/backend/tests/fixtures/jobs_read_auth.sql +++ b/backend/tests/fixtures/jobs_read_auth.sql @@ -19,6 +19,45 @@ INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, sc ARRAY['jobs:read', 'if_jobs:filter_tags:deno'] ); +-- App embed token for the admin viewer (test-user). Mirrors a minted sandboxed +-- low-code app token: carries the `app_embed` sentinel plus the embed scope set. +-- Used to assert the token is confined to jobs the viewer LAUNCHED, not every job +-- the (admin) viewer could otherwise read. +INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, scopes) VALUES ( + encode(sha256('EMBED_APP_TOKEN'::bytea), 'hex'), 'EMBED_APP_', 'EMBED_APP_TOKEN', + 'test@windmill.dev', 'app embed token', false, + ARRAY['apps:run', 'jobs:read', 'app_embed', 'resources:run', 'users:read', 'folders:read'] +); + +-- A completed app-component job LAUNCHED BY the admin viewer (created_by = +-- test-user), running as the app owner. The embed token must keep reading its own +-- launched job (the `created_by == viewer` fast path). +INSERT INTO public.v2_job ( + id, workspace_id, created_by, created_at, permissioned_as, permissioned_as_email, + kind, script_lang, runnable_path, tag, visible_to_owner, args +) VALUES ( + '12121212-1212-1212-1212-121212121212', 'test-workspace', 'test-user', + '2023-01-01 00:00:00', 'u/test-user-2', 'test2@windmill.dev', + 'script', 'deno', 'u/test-user-2/app_component', 'deno', false, + '{"own": "arg"}' +); +INSERT INTO public.v2_job_completed (id, workspace_id, duration_ms, status, result) VALUES + ('12121212-1212-1212-1212-121212121212', 'test-workspace', 1000, 'success'::job_status, + '{"own": "EMBED_OWN_RESULT"}'); + +-- A QUEUED job launched by the admin embed viewer (created_by = test-user). The +-- embed token may cancel its own launched job; it must NOT cancel another user's. +INSERT INTO public.v2_job ( + id, workspace_id, created_by, created_at, permissioned_as, permissioned_as_email, + kind, script_lang, runnable_path, tag, visible_to_owner +) VALUES ( + '13131313-1313-1313-1313-131313131313', 'test-workspace', 'test-user', + '2023-01-01 00:00:00', 'u/test-user-2', 'test2@windmill.dev', + 'script', 'deno', 'u/test-user-2/app_component', 'deno', false +); +INSERT INTO public.v2_job_queue (id, workspace_id, scheduled_for, running, tag) VALUES + ('13131313-1313-1313-1313-131313131313', 'test-workspace', '2023-01-01 00:00:00', false, 'deno'); + -- RUNNING job: queued (no completed row) and owned by test-user-2. Used to check -- that `completed/get_result_maybe?get_started=true` authorizes before disclosing -- running-state to a non-reader. diff --git a/backend/tests/jobs_read_auth.rs b/backend/tests/jobs_read_auth.rs index f19daf0f92..643245d9c5 100644 --- a/backend/tests/jobs_read_auth.rs +++ b/backend/tests/jobs_read_auth.rs @@ -38,6 +38,10 @@ const TOP_SECRET_FLOW: &str = "ffffffff-ffff-ffff-ffff-ffffffffffff"; const DEEP_LEAF_JOB: &str = "88888888-8888-8888-8888-888888888888"; // A queued/running job (no completed row) owned by test-user-2. const RUNNING_JOB: &str = "77777777-7777-7777-7777-777777777777"; +// An app-component job launched BY the admin embed viewer (created_by test-user). +const EMBED_OWN_JOB: &str = "12121212-1212-1212-1212-121212121212"; +// A QUEUED job launched by the embed viewer (created_by test-user) — cancelable by it. +const EMBED_OWN_QUEUED: &str = "13131313-1313-1313-1313-131313131313"; // Secrets that must never leak to an unauthorized viewer. const RESULT_SECRET: &str = "RESULT_SECRET"; @@ -59,6 +63,19 @@ async fn get(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCod (status, body) } +async fn post(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCode, String) { + let mut req = client() + .post(format!("{base}/{path}")) + .json(&serde_json::json!({})); + if let Some(token) = token { + req = req.header("Authorization", format!("Bearer {token}")); + } + let resp = req.send().await.expect("request"); + let status = resp.status(); + let body = resp.text().await.expect("body"); + (status, body) +} + #[sqlx::test(fixtures("base", "jobs_read_auth"))] async fn test_single_job_read_authorization(db: Pool) -> anyhow::Result<()> { initialize_tracing().await; @@ -281,6 +298,75 @@ async fn test_single_job_read_authorization(db: Pool) -> anyhow::Resul "top flow in an unreadable folder must stay denied (got {status}): {body}" ); + // ---- APP EMBED TOKEN: confined to jobs the viewer LAUNCHED, not everything + // the (admin) viewer can otherwise read. The token carries the `app_embed` + // sentinel; an admin's normal token reads VICTIM (asserted above), but the + // embed token must stop at the `created_by == viewer` grant so user-authored + // app JS can't reuse it to read unrelated jobs by UUID. + // Its own launched component job (created_by == viewer) still reads. + let (status, body) = get( + &base, + &format!("completed/get_result/{EMBED_OWN_JOB}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert!( + status.is_success(), + "embed token must read a job it launched (got {status}): {body}" + ); + assert!( + body.contains("EMBED_OWN_RESULT"), + "embed token should get its own launched job result: {body}" + ); + // The VICTIM job — created by another user but readable by this admin viewer's + // normal token (asserted above) — is denied to the embed token across result / + // logs / live update. NotFound (not 403) so the untrusted app can't even probe + // existence, and no secret leaks. + for path in [ + format!("completed/get_result/{VICTIM}"), + format!("get_logs/{VICTIM}"), + format!("getupdate/{VICTIM}?only_result=true"), + ] { + let (status, body) = get(&base, &path, Some("EMBED_APP_TOKEN")).await; + assert_eq!( + status, + reqwest::StatusCode::NOT_FOUND, + "embed token must not read a job it did not launch ({path}, got {status}): {body}" + ); + for secret in [RESULT_SECRET, ARGS_SECRET, LOGS_SECRET] { + assert!( + !body.contains(secret), + "embed token response for {path} leaked `{secret}`: {body}" + ); + } + } + + // ---- APP EMBED TOKEN: cancellation confined to the app's own jobs. The token + // may cancel a job it launched (created_by == viewer), but `cancel_job_api` + // denies (NotFound) a job created by someone else, even though cancel + // otherwise has no per-job ownership check. + let (status, body) = post( + &base, + &format!("queue/cancel/{EMBED_OWN_QUEUED}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert!( + status.is_success(), + "embed token must cancel a job it launched (got {status}): {body}" + ); + let (status, body) = post( + &base, + &format!("queue/cancel/{RUNNING_JOB}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert_eq!( + status, + reqwest::StatusCode::NOT_FOUND, + "embed token must not cancel another user's job (got {status}): {body}" + ); + // ---- UNAUTHENTICATED, unchanged: an anonymous-created job is readable // without a token (public trigger / public app result polling). let (status, body) = get(&base, &format!("completed/get_result/{ANON_JOB}"), None).await; diff --git a/backend/windmill-api-auth/src/scopes.rs b/backend/windmill-api-auth/src/scopes.rs index f9560efc3d..e11607bc91 100644 --- a/backend/windmill-api-auth/src/scopes.rs +++ b/backend/windmill-api-auth/src/scopes.rs @@ -451,6 +451,30 @@ pub fn check_route_access( // Find the domain and kind for this route let (required_domain, required_kind, route_suffix) = extract_domain_from_route(route_path)?; + // App embed tokens (sentinel) carry broad read scopes (`jobs:read`, + // `users:read`, `folders:read`) that exist only for a handful of routes. The + // whole `/users`, `/folders` and `/jobs` routers are CORS-enabled for the + // opaque app iframe, so default-deny everything in those domains except the + // intended routes — otherwise the token could enumerate/export workspace data. + if has_app_embed_sentinel(Some(token_scopes)) { + if let Some(suffix) = route_suffix.as_deref() { + if app_embed_route_denied(required_domain, suffix) { + return Err(Error::PermissionDenied( + "Access denied. App embed token cannot access this route.".to_string(), + )); + } + // The by-id job cancel is a POST (write) that the token's `jobs:read` + // wouldn't satisfy, but cancelling the app's own component runs is + // intended (most components supersede an in-flight run on re-run). Permit + // it here; `cancel_job_api` confines it to jobs the app launched + // (created_by == viewer). A read_only token is still rejected by the + // separate read-only check. + if suffix.starts_with("jobs_u/queue/cancel/") { + return Ok(()); + } + } + } + // MCP scopes (mcp:all, mcp:favorites, mcp:hub:*, etc.) use a custom format // that doesn't fit the standard domain:action model. Verify the token has at // least one mcp: scope; MCP handlers do their own fine-grained checking. @@ -537,7 +561,7 @@ const FLOW_JOBS: [&'static str; 6] = [ lazy_static::lazy_static! { static ref RUN_PATH_ACTIONS: Vec<&'static str> = { - let mut v = vec!["jobs/resume/", "jobs/run/batch_rerun_jobs", "jobs/run/workflow_as_code", "jobs/run/dependencies","jobs/run/flow_dependencies", "apps_u/execute_component"]; + let mut v = vec!["jobs/resume/", "jobs/run/batch_rerun_jobs", "jobs/run/workflow_as_code", "jobs/run/dependencies","jobs/run/flow_dependencies", "apps_u/execute_component", "apps_u/upload_s3_file"]; v.extend(SCRIPT_JOBS); v.extend(FLOW_JOBS); @@ -640,6 +664,92 @@ const RUN_WHITELISTED_GET_PATHS: [&'static str; 20] = [ "jobs/completed/get_result_maybe/", ]; +/// Sentinel scope in app embed tokens. Grants nothing itself; `check_route_access` +/// uses it to deny the workspace-wide job enumeration routes `jobs:read` would +/// otherwise reach, so an embedded app reads only jobs it launched (by id). +pub const APP_EMBED_SENTINEL: &str = "app_embed"; + +/// True if a token's scopes include the app-embed sentinel (a sandboxed app iframe +/// token). Such tokens carry the viewer's identity but represent untrusted app JS, +/// so several handlers confine them to the app's own resources/runs. +pub fn has_app_embed_sentinel(scopes: Option<&[String]>) -> bool { + scopes.is_some_and(|s| s.iter().any(|x| x == APP_EMBED_SENTINEL)) +} + +/// Routes an app embed token (sentinel) is denied. Its broad scopes (`apps:run`, +/// `jobs:read`, `users:read`, `folders:read`) exist only for a fixed set of routes a +/// running app uses, but the whole `/apps`, `/jobs`, `/users`, `/folders` routers are +/// CORS-enabled for the opaque app iframe. Default-deny those domains via an explicit +/// allowlist so the token can't reach workspace inventory, counts, exports, or +/// capability-minting routes (job signatures / resume URLs). +fn app_embed_route_denied(domain: ScopeDomain, suffix: &str) -> bool { + match domain { + ScopeDomain::Apps => !app_embed_apps_route_allowed(suffix), + ScopeDomain::Jobs => !app_embed_job_route_allowed(suffix), + ScopeDomain::Users => suffix != "users/whoami", + ScopeDomain::Folders => suffix != "folders/listnames", + _ => false, + } +} + +/// App routes a running app uses: its own definition (`apps/get/p/`, further +/// path-scoped by `apps:read:`) and the public app-serving endpoints +/// (`apps_u/*`: public_app, public_resource, get_data, and the path-taking +/// `execute_component` / `download_s3_file`, which re-check `apps:run|read:` +/// in their handlers so they stay confined to this app). Everything else in the +/// domain — workspace app inventory (`exists`, `custom_path_exists`, `list`, +/// `list_paths*`, `secret_of`, history, management) — is denied. +fn app_embed_apps_route_allowed(suffix: &str) -> bool { + // The embed-token mint endpoints live under `apps_u/` but they create + // credentials. A running app never calls them — the trusted embedder session/JWT + // mints the token and hands it to the iframe — so deny them here, otherwise an + // app embed token could renew itself indefinitely past the 12h expiry. + if suffix.starts_with("apps_u/embed_token") { + return false; + } + suffix.starts_with("apps/get/p/") || suffix.starts_with("apps_u/") +} + +/// Job routes a running app uses (the by-id poll/cancel surface driven by the +/// frontend JobLoader). Everything else in the jobs domain — enumeration, counts, +/// exports, and the `job_signature`/`resume_urls` capability-minting routes — is +/// denied. By-id reads are further confined to the app's own runs by +/// `require_job_read_access` (the `app_embed` cutoff). +fn app_embed_job_route_allowed(suffix: &str) -> bool { + // `get_root_job_id` is intentionally absent: its handler has no access check at + // all (returns any job's root id by id) and the app never calls it, so denying + // it costs nothing and avoids leaking a foreign job's flow lineage. + const ALLOWED: [&str; 15] = [ + "jobs_u/get/", + "jobs_u/getupdate/", + "jobs_u/getupdate_sse/", + "jobs_u/get_logs/", + "jobs_u/get_completed_logs_tail/", + "jobs_u/get_args/", + "jobs_u/get_flow/", + "jobs_u/get_flow_all_logs/", + "jobs_u/get_flow_debug_info/", + "jobs_u/get_log_file/", + "jobs_u/completed/get/", + "jobs_u/completed/get_result/", + "jobs_u/completed/get_result_maybe/", + "jobs_u/completed/get_timing/", + "jobs_u/queue/cancel/", + ]; + ALLOWED.iter().any(|p| suffix.starts_with(p)) +} + +/// Resource routes a metadata-only `resources:run` scope (app embed tokens) may +/// GET: pickers (`/list`) and type schemas. Excludes every value-returning route +/// (`get`, `get_value`, `get_value_interpolated`, `list_search`) so resource +/// values — which can hold credentials — are never exposed. +fn resource_metadata_route_allowed(suffix: &str) -> bool { + suffix == "resources/list" + || suffix.starts_with("resources/list_names/") + || suffix.starts_with("resources/exists/") + || suffix.starts_with("resources/type/") +} + fn scope_grants_access( scope: &ScopeDefinition, required_domain: ScopeDomain, @@ -659,6 +769,14 @@ fn scope_grants_access( let scope_action = ScopeAction::from_str(&scope.action) .ok_or_else(|| Error::BadRequest(format!("Invalid scope action: {}", scope.action)))?; + // App embed tokens carry `resources:run`: metadata-only resource access via + // default-deny + allowlist (so a new value route is never exposed by accident). + // See `resource_metadata_route_allowed`. + if scope_domain == ScopeDomain::Resources && scope_action == ScopeAction::Run { + return Ok(required_action == ScopeAction::Read + && route_path.is_some_and(resource_metadata_route_allowed)); + } + if !scope_action.includes(&required_action) && !(scope_domain == ScopeDomain::Jobs && required_action == ScopeAction::Read diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index c98caa43dd..102efc765d 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -7501,6 +7501,22 @@ paths: workspace_id: type: string + /apps_u/embed_token_by_custom_path/{custom_path}: + get: + summary: get app embed token by custom path + operationId: getAppEmbedTokenByCustomPath + tags: + - app + parameters: + - $ref: "#/components/parameters/CustomPath" + responses: + "200": + description: embed token + content: + application/json: + schema: + $ref: "#/components/schemas/EmbedTokenResponse" + /scripts/hub/get/{path}: get: summary: get hub script content by path @@ -10569,6 +10585,10 @@ paths: - name: secretWithExtension in: path required: true + description: >- + App version secret suffixed with the requested file type extension. + Supported extensions are `.js` (JavaScript bundle), `.css` + (stylesheet), and `.html` (sandboxed wrapper document). schema: type: string responses: @@ -10578,6 +10598,12 @@ paths: text/javascript: schema: type: string + text/css: + schema: + type: string + text/html: + schema: + type: string /w/{workspace}/apps/list_search: get: @@ -10829,6 +10855,23 @@ paths: - $ref: "#/components/schemas/AppWithLastVersion" - $ref: "#/components/schemas/UserDraftOverlay" + /w/{workspace}/apps/embed_token/p/{path}: + get: + summary: get app embed token by path + operationId: getAppEmbedTokenByPath + tags: + - app + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/ScriptPath" + responses: + "200": + description: embed token + content: + application/json: + schema: + $ref: "#/components/schemas/EmbedTokenResponse" + /w/{workspace}/apps/get/lite/{path}: get: summary: get app lite by path @@ -10946,6 +10989,27 @@ paths: schema: $ref: "#/components/schemas/AppWithLastVersion" + /w/{workspace}/apps_u/embed_token/{secret}: + get: + summary: get app embed token by secret + operationId: getAppEmbedTokenBySecret + tags: + - app + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: secret + in: path + required: true + schema: + type: string + responses: + "200": + description: embed token + content: + application/json: + schema: + $ref: "#/components/schemas/EmbedTokenResponse" + /w/{workspace}/apps_u/public_resource/{path}: get: summary: get public resource @@ -27833,6 +27897,13 @@ components: type: string on_behalf_of_email: type: string + sandbox: + type: boolean + description: > + Publisher opt-in to app sandbox isolation (alpha). When true the app + is isolated from each viewer's Windmill session. When false/absent + the app runs same-origin with the viewer's full session (the + default, pre-isolation behavior). ListableApp: type: object @@ -28052,6 +28123,36 @@ components: required: - version + EmbedTokenResponse: + type: object + properties: + token: + type: string + nullable: true + description: Narrowly-scoped embed token for the iframe. Absent for fully anonymous or raw apps, which load without a scoped token. + expiration: + type: string + format: date-time + nullable: true + description: Expiration of the embed token. + raw_app: + type: boolean + description: Raw apps render single-iframe and skip the opaque-viewer indirection and the embed token entirely. + sandbox: + type: boolean + description: Publisher opted this app into sandbox isolation. When false the viewer runs the app same-origin with its full session. + app_path: + type: string + nullable: true + description: The resolved app path; the embedder uses it to scope the app's backing localStorage per app. + workspace_id: + type: string + nullable: true + description: The resolved workspace; pairs with app_path so apps at the same path in different workspaces don't share a localStorage store. + required: + - raw_app + - sandbox + FlowVersion: type: object properties: diff --git a/backend/windmill-api/src/apps.rs b/backend/windmill-api/src/apps.rs index bfc76eefd8..2912b4397f 100644 --- a/backend/windmill-api/src/apps.rs +++ b/backend/windmill-api/src/apps.rs @@ -75,6 +75,7 @@ use windmill_common::{ use windmill_object_store::object_store_reexports::{Attribute, Attributes}; use windmill_store::resources::get_resource_value_interpolated_internal; +use windmill_api_auth::{create_token_internal, ensure_scopes_within_caller, NewToken}; use windmill_git_sync::{handle_deployment_metadata, DeployedObject}; use windmill_queue::{push, PushArgs, PushArgsOwned, PushIsolationLevel}; @@ -90,6 +91,7 @@ pub fn workspaced_service(raw_app_body_limit: usize) -> Router { .route("/list", get(list_apps)) .route("/list_search", get(list_search_apps)) .route("/get/p/{*path}", get(get_app)) + .route("/embed_token/p/{*path}", get(get_app_embed_token_for_path)) .route("/get/lite/{*path}", get(get_app_lite)) .route("/secret_of/{*path}", get(get_secret_id)) .route( @@ -134,6 +136,7 @@ pub fn unauthed_service() -> Router { .route("/delete_s3_file", delete(delete_s3_file_from_app)) .route("/download_s3_file/{*path}", get(download_s3_file_from_app)) .route("/public_app/{secret}", get(get_public_app_by_secret)) + .route("/embed_token/{secret}", get(get_app_embed_token)) .route("/public_resource/{*path}", get(get_public_resource)) .route("/get_data/v/{*id}", get(get_raw_app_data)) } @@ -300,6 +303,13 @@ pub struct Policy { pub execution_mode: ExecutionMode, pub s3_inputs: Option>, pub allowed_s3_keys: Option>, + // WIN-2006: publisher opt-in to iframe sandbox isolation (alpha). When true the + // app is isolated from each viewer's Windmill session: low-code renders in an + // opaque-origin iframe with a scoped embed token, raw renders its bundle in an + // opaque iframe. Default/absent means unsandboxed — the app runs same-origin + // with the viewer's full session, the pre-isolation behavior. + #[serde(skip_serializing_if = "Option::is_none")] + pub sandbox: Option, } #[derive(Deserialize)] @@ -348,6 +358,12 @@ async fn list_search_apps( Path(w_id): Path, Extension(user_db): Extension, ) -> JsonResult> { + // Require domain-level read: this returns every visible app's full value (code). + // The route layer treats `apps:run` as satisfying read, so without this handler + // check a scoped embed token (apps:run + apps:read:) could read all + // apps' definitions. `check_scopes` uses ScopeDefinition::includes, where run + // does NOT include read, so it correctly denies such tokens. + check_scopes(&authed, || "apps:read".to_string())?; #[cfg(feature = "enterprise")] let n = 1000; @@ -379,6 +395,9 @@ async fn list_apps( Query(pagination): Query, Query(lq): Query, ) -> JsonResult> { + // Domain-level read (see list_search_apps): keeps a scoped embed token, whose + // `apps:run` only satisfies read at the route layer, from listing all apps. + check_scopes(&authed, || "apps:read".to_string())?; let (per_page, offset) = paginate(pagination); let mut sqlb = SqlBuilder::select_from("app") @@ -553,6 +572,7 @@ async fn list_apps( async fn get_raw_app_data( Path((w_id, secret_with_ext)): Path<(String, String)>, + Query(query): Query>, Extension(db): Extension, ) -> Result { #[cfg(all(feature = "enterprise", feature = "parquet"))] @@ -575,13 +595,52 @@ async fn get_raw_app_data( .await?; let file_type = splitted.next().unwrap_or(""); + + // Sandboxed wrapper document that hosts the bundle. Served from a real URL + // (not blob:/srcdoc) so we can attach `CSP: sandbox` as a response header, + // which forces an opaque origin even on direct navigation — a raw-app + // bundle can then never reach the authenticated Windmill origin (WIN-2006). + // The `.js`/`.css` are loaded as same-path subresources by this document. + if file_type == "html" { + // ALWAYS served with `CSP: sandbox`, which forces an opaque origin even on + // direct top-level navigation — so this real-origin URL can never be used + // to run a raw-app bundle with the viewer's session (WIN-2006). The + // unsandboxed (default) render is NOT applied here: it is handled entirely + // on the viewer side, which builds its own same-origin wrapper. Relaxing + // this header from a policy flag would let anyone with the share secret + // hand a logged-in victim a same-origin URL that runs the bundle with + // their session — so the standalone document stays sandboxed no matter how + // it is reached. + let html = raw_app_wrapper_html(secret_id); + let mut builder = Response::builder() + .header(http::header::CONTENT_TYPE, "text/html; charset=utf-8") + .header("X-Content-Type-Options", "nosniff") + .header("Cross-Origin-Resource-Policy", "cross-origin") + .header( + http::header::CONTENT_SECURITY_POLICY, + "sandbox allow-scripts allow-forms allow-popups \ + allow-popups-to-escape-sandbox allow-downloads allow-modals \ + allow-top-navigation", + ); + // When the public app page is embedded in a cross-origin-isolated page + // (`wm_coep` opt-in, COEP `require-corp`), this nested wrapper document + // must itself assert COEP to be allowed to load. Opt-in only — COEP + // restricts the bundle's own subresources to CORP'd/same-origin ones + // (e.g. external images would break), so it must not be always-on. The + // viewer propagates the flag from the page URL (see RawAppPreview). + if query.contains_key("wm_coep") { + builder = builder.header("Cross-Origin-Embedder-Policy", "require-corp"); + } + return Ok(builder.body(Body::from(html)).unwrap()); + } + let file_type = if file_type == "css" { "css" } else if file_type == "js" { "js" } else { return Err(Error::BadRequest( - "Invalid file type, only .css and .js are supported".to_string(), + "Invalid file type, only .css, .js and .html are supported".to_string(), )); }; // tracing::info!("file_type: {}", file_type); @@ -632,20 +691,128 @@ async fn get_raw_app_data( if let Some(body) = body { // let stream = tokio_util::io::ReaderStream::new(file); - let res = Response::builder().header( - http::header::CONTENT_TYPE, - if file_type == "css" { - "text/css" - } else { - "text/javascript" - }, - ); + let res = Response::builder() + .header( + http::header::CONTENT_TYPE, + if file_type == "css" { + "text/css" + } else { + "text/javascript" + }, + ) + // nosniff + CORP so the bundle loads correctly as a subresource of + // the opaque, sandboxed wrapper (incl. under a cross-origin-isolated + // / COEP `require-corp` embedder). + .header("X-Content-Type-Options", "nosniff") + .header("Cross-Origin-Resource-Policy", "cross-origin"); Ok(res.body(body).unwrap()) } else { return Err(Error::NotFound("File not found".to_string())); } } +/// HTML wrapper that hosts a raw-app bundle inside a sandboxed, opaque-origin +/// iframe. Served by [`get_raw_app_data`] for the `.html` "file type". It loads +/// the bundle `.js`/`.css` as same-path subresources, shims web storage (which +/// an opaque origin disallows), and waits for the embedder to hand it the user +/// context via `postMessage` before evaluating the bundle — so the bundle never +/// receives a credential and `window.ctx` is set synchronously when it runs. +fn raw_app_wrapper_html(secret: &str) -> String { + const TEMPLATE: &str = r##" + + + +App + + + + +
+ + +"##; + TEMPLATE.replace("__SECRET__", secret) +} + // async fn get_app_version( // authed: ApiAuthed, // Extension(user_db): Extension, @@ -937,6 +1104,17 @@ async fn get_public_app_by_secret( let mut app = not_found_if_none(app_o, "App", id.to_string())?; + // Confine the app embed token (the only credential handed to untrusted app JS, + // carrying the viewer's identity + `apps:read:`) to the app the secret + // resolves to: without this, app JS could reuse the viewer's identity to read any + // app it can see by secret via the RLS check below. Scoped to embed tokens only — + // other callers (anonymous, cookie, plain external JWT) keep their existing access. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:read:{}", app.path))?; + } + } + let policy = serde_json::from_str::(app.policy.0.get()).map_err(to_anyhow)?; if !matches!(policy.execution_mode, ExecutionMode::Anonymous) { @@ -971,6 +1149,300 @@ async fn get_public_app_by_secret( Ok(Json(app)) } +/// Scopes granted to a short-lived "app embed token". This is the token the +/// app-embedder page hands the (opaque-origin) app iframe at startup so the app +/// never receives the viewer's session cookie. Instead of restricting which +/// routes a *domain* may hit, we restrict which routes the *token* may hit, so +/// that even a malicious or compromised app document can only reach the +/// endpoints an app legitimately needs. The `app_embed` sentinel turns each of +/// these into a strict route allowlist (`app_embed_route_denied`): +/// - `jobs:read` → by-id job poll/cancel only; enumeration, counts, exports, +/// and `job_signature`/`resume_urls` are denied, and by-id +/// reads are confined to the app's own runs. +/// - `app_embed` → sentinel tagging this as an app embed token (grants nothing). +/// - `resources:run` → resource metadata only (pickers, type schemas), never values. +/// - `users:read` → `users/whoami` only. +/// - `folders:read` → `folders/listnames` only. +/// Plus two path-scoped scopes minted per app (see `mint_app_embed_token`): +/// - `apps:read:` → the app's own definition (`apps/get/p/`); no +/// `apps:write`, so management routes are unreachable. +/// - `apps:run:` → run THIS app's components (`execute_component`, which +/// re-checks the path); `apps_u/*` public-serving routes. +pub const APP_EMBED_SCOPES: [&str; 5] = [ + "jobs:read", + windmill_api_auth::scopes::APP_EMBED_SENTINEL, + "resources:run", + "users:read", + "folders:read", +]; + +/// How long an app embed token stays valid. The embedder re-mints on demand +/// (e.g. after a `401` from the iframe) so this can stay short. +const APP_EMBED_TOKEN_VALIDITY_HOURS: i64 = 12; + +#[derive(Serialize)] +pub struct EmbedTokenResponse { + /// Narrowly-scoped token for the iframe. `None` for fully anonymous access + /// (the iframe then calls the public endpoints anonymously). + pub token: Option, + pub expiration: Option>, + /// WIN-2006: raw apps render single-iframe (the bundle is already isolated in + /// its own opaque iframe), so the viewer skips the opaque-viewer indirection + /// and the embed token entirely — it loads the app with the page credential. + #[serde(default)] + pub raw_app: bool, + /// WIN-2006: publisher opted this app into sandbox isolation. When false the + /// viewer runs the app same-origin with its full session (the default, + /// pre-isolation behavior). + #[serde(default)] + pub sandbox: bool, + /// WIN-2006: the resolved app path. The embedder uses it (together with + /// `workspace_id`) to scope the app's backing `localStorage` per app (so + /// sandboxed apps don't share one store). Not a new disclosure — the viewer + /// already receives `path` when it loads the app (e.g. `get_public_app_by_secret`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub app_path: Option, + /// WIN-2006: the resolved workspace. Pairs with `app_path` for the per-app + /// `localStorage` key so two apps at the same path in different workspaces don't + /// share a store. For custom-path apps the viewer can't derive this itself. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub workspace_id: Option, +} + +/// Mint a short-lived, narrowly-scoped embed token for `app_path` when a caller +/// is authenticated. When `opt_authed` is `None` (anonymous access to an +/// anonymous app) no token is minted and the iframe relies on the public +/// endpoints. +/// +/// The CALLER MUST verify the viewer's access to `app_path` before calling: this +/// mints a token on behalf of `opt_authed` unconditionally (DB access remains +/// gated by the viewer's own RLS, but the token's existence is not access-checked +/// here). All current call sites (`get_app_embed_token`, +/// `get_app_embed_token_for_path`, and the EE custom-path variant) do this. +/// +/// Scope confinement IS enforced here: the minted scopes must be within the +/// caller's own (`ensure_scopes_within_caller`), so a scope-restricted bearer +/// token cannot bootstrap a broader-scoped embed token. For the normal caller — +/// an unscoped browser session — this is a no-op and the mint is purely +/// narrowing. +pub async fn mint_app_embed_token( + db: &DB, + w_id: &str, + app_path: &str, + opt_authed: Option<&ApiAuthed>, +) -> Result { + let token_and_exp = if let Some(authed) = opt_authed { + // An app embed token represents untrusted app JS in the sandboxed iframe; it + // must never reach this mint path to renew itself. The 12h expiry is the + // blast-radius cap on a leaked embed token, and `ensure_scopes_within_caller` + // below would pass a same-scoped renewal (the requested scopes equal the + // caller's own), making the credential indefinitely self-renewable. Refresh + // minting is the trusted embedder session/JWT's job. + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + return Err(Error::NotAuthorized( + "App embed tokens cannot mint or renew embed tokens".to_string(), + )); + } + let expiration = + chrono::Utc::now() + chrono::Duration::hours(APP_EMBED_TOKEN_VALIDITY_HOURS); + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // Path-scoped read so the app can fetch its OWN definition (apps/get/p, + // which the in-workspace sandboxed viewer uses) — but no other app's. The + // public viewer fetches via apps_u/public_app and doesn't rely on this. + scopes.push(format!("apps:read:{app_path}")); + // Path-scoped run (NOT unqualified `apps:run`) so the token can only execute + // THIS app's components: `execute_component` re-checks `apps:run:` for + // the requested app, so the token can't drive another app's runnables. + scopes.push(format!("apps:run:{app_path}")); + // A scope-restricted caller token must not bootstrap a broader-scoped + // embed token (`create_token_internal` deliberately does not check this + // itself). No-op for unscoped sessions — the normal embed flow. + ensure_scopes_within_caller(authed, Some(&scopes))?; + let token_config = NewToken::new( + Some(format!("embed_app:{app_path}")), + Some(expiration), + None, + Some(scopes), + Some(w_id.to_string()), + // Never let an embed token gain write capability the caller's own + // session lacks. + Some(authed.read_only), + ); + let mut tx = db.begin().await?; + let token = create_token_internal(&mut *tx, db, authed, token_config).await?; + tx.commit().await?; + Some((token, expiration)) + } else { + None + }; + + Ok(EmbedTokenResponse { + token: token_and_exp.as_ref().map(|(t, _)| t.clone()), + expiration: token_and_exp.map(|(_, e)| e), + raw_app: false, + sandbox: false, + app_path: Some(app_path.to_string()), + workspace_id: Some(w_id.to_string()), + }) +} + +/// Issue an embed token for a public app addressed by its (secret) share id. +/// Mirrors the access check in [`get_public_app_by_secret`]: anonymous apps are +/// reachable without auth, otherwise the caller must be logged in and have read +/// access to the app. +async fn get_app_embed_token( + OptAuthed(opt_authed): OptAuthed, + Extension(user_db): Extension, + Extension(db): Extension, + Path((w_id, secret)): Path<(String, String)>, +) -> JsonResult { + let id = get_id_from_secret(&db, &w_id, secret, None).await?; + + let app = sqlx::query!( + "SELECT a.path, a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app + FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] + WHERE a.id = $1 AND a.workspace_id = $2", + id, + &w_id + ) + .fetch_optional(&db) + .await?; + let app = not_found_if_none(app, "App", id.to_string())?; + let raw_app = app.raw_app; + let policy_str = app + .policy + .ok_or_else(|| Error::internal_err("App policy missing".to_string()))?; + // Lenient field-level read instead of a strict `Policy` parse: a legacy app + // whose stored policy predates newer required fields must still resolve to + // its (unsandboxed) render here rather than erroring out of the viewer. + let policy = parse_embed_policy(&policy_str)?; + + let authed_for_token = if policy.anonymous_execution { + // Anonymous app: still mint a scoped token if the viewer happens to be + // logged in (so the app sees their identity), otherwise stay anonymous. + opt_authed + } else { + let authed = opt_authed.ok_or_else(|| { + Error::NotAuthorized( + "App visibility does not allow public access and you are not logged in".to_string(), + ) + })?; + let mut tx = user_db.begin(&authed).await?; + let is_visible = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM app WHERE id = $1 AND workspace_id = $2)", + id, + &w_id + ) + .fetch_one(&mut *tx) + .await?; + tx.commit().await?; + if !is_visible.unwrap_or(false) { + return Err(Error::NotAuthorized( + "App visibility does not allow public access and you are logged in but you have no read-access to that app".to_string(), + )); + } + Some(authed) + }; + + // The token is only consumed by the sandboxed low-code render. Raw apps + // render single-iframe with the page credential (WIN-2006 Variant A), and + // unsandboxed apps render same-origin with the viewer's own session — minting + // for those would write a useless token row per view and, worse, could fail + // the whole render for a scope-restricted caller (`ensure_scopes_within_caller`) + // even though no token is needed. The access check above still gates + // visibility in every case. + let mut resp = if raw_app || !policy.sandbox { + EmbedTokenResponse { + token: None, + expiration: None, + raw_app, + sandbox: policy.sandbox, + app_path: None, + workspace_id: None, + } + } else { + mint_app_embed_token(&db, &w_id, &app.path, authed_for_token.as_ref()).await? + }; + resp.raw_app = raw_app; + resp.sandbox = policy.sandbox; + resp.app_path = Some(app.path); + resp.workspace_id = Some(w_id.to_string()); + Ok(Json(resp)) +} + +/// Minimal, lenient view of an app policy for the embed-token endpoints +/// (WIN-2006). Reads only the fields the sandbox decision needs, via +/// `serde_json::Value`, so a legacy policy that no longer satisfies the strict +/// [`Policy`] struct (e.g. `triggerables_v2` entries predating now-required +/// fields) still renders instead of failing the viewer with "Not found". +/// A missing/unknown `execution_mode` is treated as NOT anonymous — the +/// strictest access interpretation. +pub struct EmbedPolicyView { + pub anonymous_execution: bool, + pub sandbox: bool, +} + +pub fn parse_embed_policy(policy_str: &str) -> Result { + let v: serde_json::Value = serde_json::from_str(policy_str).map_err(to_anyhow)?; + Ok(EmbedPolicyView { + anonymous_execution: v.get("execution_mode").and_then(|m| m.as_str()) == Some("anonymous"), + sandbox: v.get("sandbox").and_then(|b| b.as_bool()).unwrap_or(false), + }) +} + +/// Authenticated, path-based embed token for the in-workspace app viewer +/// (WIN-2006). Mirrors [`get_app_embed_token`] but keyed by app path and gated by +/// the caller's read access (RLS), so the logged-in `/apps/get` viewer can render +/// the app sandboxed — isolated from the member's full session — using the same +/// scoped token. Raw apps get no token (single-iframe with the page credential). +async fn get_app_embed_token_for_path( + authed: ApiAuthed, + Extension(user_db): Extension, + Extension(db): Extension, + Path((w_id, path)): Path<(String, StripPath)>, +) -> JsonResult { + let path = path.to_path(); + check_scopes(&authed, || format!("apps:read:{}", path))?; + // RLS: the caller must have read access to this app, otherwise it's not found. + let mut tx = user_db.begin(&authed).await?; + let app = sqlx::query!( + "SELECT a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app + FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] + WHERE a.path = $1 AND a.workspace_id = $2", + path, + &w_id + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + let app = not_found_if_none(app, "App", path)?; + let raw_app = app.raw_app; + let policy_str = app + .policy + .ok_or_else(|| Error::internal_err("App policy missing".to_string()))?; + // Lenient parse + mint only for the sandboxed low-code render — see + // [`get_app_embed_token`] for the rationale (identical here). + let policy = parse_embed_policy(&policy_str)?; + + let mut resp = if raw_app || !policy.sandbox { + EmbedTokenResponse { + token: None, + expiration: None, + raw_app, + sandbox: policy.sandbox, + app_path: None, + workspace_id: None, + } + } else { + mint_app_embed_token(&db, &w_id, path, Some(&authed)).await? + }; + resp.raw_app = raw_app; + resp.sandbox = policy.sandbox; + resp.app_path = Some(path.to_string()); + resp.workspace_id = Some(w_id.to_string()); + Ok(Json(resp)) +} + async fn get_id_from_secret( db: &DB, w_id: &str, @@ -2285,6 +2757,17 @@ async fn execute_component( Path((w_id, path)): Path<(String, StripPath)>, Json(mut payload): Json, ) -> Result { + let path = path.to_path(); + // Authorize FIRST, before touching the payload: confine the app embed token (the + // only credential handed to untrusted app JS, carrying `apps:run:`) to + // the app it was minted for. The route layer can't path-check the apps domain, so + // enforce it here. Scoped to embed tokens only — other callers (anonymous, cookie, + // plain external JWT) keep their existing access; the run is still policy-gated. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:run:{}", path))?; + } + } // Only honor temp_script_refs for the inline-script preview path: // preview/editor mode (force_viewer_static_fields set, == `is_preview`), // raw_code present, and no deployed app_script id — i.e. `wmill app dev`. @@ -2307,7 +2790,6 @@ async fn execute_component( _ => {} }; - let path = path.to_path(); let (arc_policy, policy): (Arc, Policy); let policy_triggerables_default = Default::default(); // Preview mode means the request was issued from the editor; the editing @@ -2771,6 +3253,15 @@ async fn upload_s3_file_from_app( Query(query): Query, request: axum::extract::Request, ) -> JsonResult { + // Confine an app embed token (untrusted app JS) to uploading for its OWN app. + // The route is reachable with `apps:run` (RUN_PATH_ACTIONS), so without this a + // token minted for app A could drive app B's upload policy. Mirrors + // execute_component / download_s3_file; other callers are unaffected. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:run:{}", path.to_path()))?; + } + } let policy = if let Some(file_key_regex) = query.force_viewer_file_key_regex { // `force_viewer_*` lets the caller supply a synthetic upload policy that // bypasses the deployed app's file_key_regex / resource restrictions. @@ -2805,6 +3296,7 @@ async fn upload_s3_file_from_app( .unwrap_or_default(), }]), allowed_s3_keys: None, + sandbox: None, }) } else { let policy_o = sqlx::query_scalar!( @@ -3164,6 +3656,7 @@ async fn get_on_behalf_authed_from_app( on_behalf_of_email: None, s3_inputs: None, allowed_s3_keys: Some(force_allowed_s3_keys), + sandbox: None, } } else { // TODO: improve db query to not return uneeded fields @@ -3186,6 +3679,7 @@ async fn get_on_behalf_authed_from_app( on_behalf_of_email: None, s3_inputs: None, allowed_s3_keys: None, + sandbox: None, }) }; @@ -3229,34 +3723,46 @@ async fn check_if_allowed_to_access_s3_file_from_app( return Err(Error::InternalErr( "Internal error: signature validation is not supported in open source mode".to_string(), )); - } else if opt_authed.is_some() { + } else if opt_authed.as_ref().is_some_and(|authed| { + !windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) + }) { + // A normal logged-in caller (editor / full session) may fetch any file they + // can reach. An app embed token also carries an identity but represents + // untrusted app JS, so it falls through to the allowlist below instead of + // this bypass — otherwise the app could read arbitrary S3 keys the + // viewer/on-behalf identity can see, beyond its own declared keys/outputs. Ok(()) } else { - let allowed = policy - .allowed_s3_keys + // Anonymous viewer, or an app embed token: confine to the app's declared S3 + // keys, or files produced by THIS app's own component runs. The producing + // identity is the embed viewer for a token, else `anonymous`. + let creator = opt_authed .as_ref() - .unwrap() - .iter() - .any(|key| key.s3_path == file_query.s3 && key.storage == file_query.storage) - || { - sqlx::query_scalar!( - r#"SELECT EXISTS ( + .map(|authed| authed.username.clone()) + .unwrap_or_else(|| "anonymous".to_string()); + let allowed = policy.allowed_s3_keys.as_ref().is_some_and(|keys| { + keys.iter() + .any(|key| key.s3_path == file_query.s3 && key.storage == file_query.storage) + }) || { + sqlx::query_scalar!( + r#"SELECT EXISTS ( SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id) WHERE j.workspace_id = $2 AND (j.kind = 'appscript' OR j.kind = 'preview') - AND j.created_by = 'anonymous' + AND j.created_by = $4 AND c.started_at > now() - interval '3 hours' AND j.runnable_path LIKE $3 || '/%' AND c.result @> ('{"s3":"' || $1 || '"}')::jsonb )"#, - file_query.s3, - w_id, - path, - ) - .fetch_one(db) - .await? - .unwrap_or(false) - }; + file_query.s3, + w_id, + path, + creator, + ) + .fetch_one(db) + .await? + .unwrap_or(false) + }; if !allowed { Err(Error::BadRequest("File restricted".to_string())) @@ -3295,6 +3801,15 @@ async fn download_s3_file_from_app( let path = path.to_path(); + // Authorize the app path first: a scoped caller (notably an app embed token, + // which carries `apps:read:`) may only download files for the app it + // was minted for — otherwise it could read another app's S3 files via that app's + // on-behalf policy. Unscoped sessions / anonymous callers pass through (the + // latter still gated by the policy allowlist in `check_if_allowed_...`). + if let Some(authed) = opt_authed.as_ref() { + check_scopes(authed, || format!("apps:read:{}", path))?; + } + let force_viewer_allowed_s3_keys = if let Some(force_viewer_allowed_s3_keys) = query.force_viewer_allowed_s3_keys.clone() { @@ -3571,3 +4086,253 @@ async fn build_args( job_id, )) } + +#[cfg(test)] +mod embed_token_tests { + use super::APP_EMBED_SCOPES; + use windmill_api_auth::scopes::check_scopes_for_route; + + /// The embed token must reach exactly the endpoints an app needs and nothing + /// else. This locks the allow/deny matrix that confines a malicious or + /// compromised app to app-only routes (WIN-2006). + #[test] + fn embed_scopes_allow_app_routes_and_deny_the_rest() { + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // Mirror mint_app_embed_token: the per-app path-scoped read + run. + scopes.push("apps:read:u/admin/app".to_string()); + scopes.push("apps:run:u/admin/app".to_string()); + let scopes = Some(scopes.as_slice()); + + // Allowed: the routes a running app legitimately calls. + let allowed = [ + // Own definition + the public app-serving / execution endpoints. + ("/api/w/test/apps/get/p/u/admin/app", "GET"), + ("/api/w/test/apps_u/public_app/secret", "GET"), + ("/api/w/test/apps_u/get_data/v/secret.js", "GET"), + ("/api/w/test/apps_u/public_resource/f/app_themes/t", "GET"), + ("/api/w/test/apps_u/execute_component/u/admin/app", "POST"), + // S3 file upload from the app's S3 File Input component: a `run` action + // (RUN_PATH_ACTIONS) so the embed token reaches it; the handler re-checks + // `apps:run:` to confine it to this app, like execute_component. + ("/api/w/test/apps_u/upload_s3_file/u/admin/app", "POST"), + // By-id job poll routes (the JobLoader surface) stay allowed. + ("/api/w/test/jobs_u/get/some-uuid", "GET"), + ("/api/w/test/jobs_u/getupdate/some-uuid", "GET"), + ("/api/w/test/jobs_u/getupdate_sse/some-uuid", "GET"), + ("/api/w/test/jobs_u/completed/get_result/some-uuid", "GET"), + ("/api/w/test/jobs_u/completed/get_timing/some-uuid", "GET"), + // By-id cancel (POST): permitted at the route layer; the handler confines + // it to the app's own jobs (created_by == viewer). + ("/api/w/test/jobs_u/queue/cancel/some-uuid", "POST"), + ("/api/w/test/users/whoami", "GET"), + // Resource METADATA only (picker list + type schemas) — never values. + ("/api/w/test/resources/list", "GET"), + ("/api/w/test/resources/exists/u/admin/r", "GET"), + ("/api/w/test/resources/type/list", "GET"), + ("/api/w/test/folders/listnames", "GET"), + ]; + for (path, method) in allowed { + assert!( + check_scopes_for_route(scopes, path, method).is_ok(), + "embed token should allow {method} {path}" + ); + } + + // Denied: anything outside what an app needs, including app management + // (apps:write is intentionally withheld), resource VALUE reads (which can + // hold credentials), and other workspace domains. + let denied = [ + ("/api/w/test/apps/update/u/admin/app", "POST"), + ("/api/w/test/apps/delete/u/admin/app", "DELETE"), + // Workspace app inventory must NOT be reachable (Apps domain is + // default-denied for the embed sentinel; only own-def + apps_u/* allowed). + ("/api/w/test/apps/exists/u/admin/app", "GET"), + ("/api/w/test/apps/custom_path_exists/foo", "GET"), + ( + "/api/w/test/apps/list_paths_from_workspace_runnable/script/u/admin/x", + "GET", + ), + ("/api/w/test/apps/list", "GET"), + // The embed-token MINT endpoints are public app routes (`apps_u/`) but + // create credentials — denied so a captured embed token can't renew + // itself indefinitely past the 12h expiry (refresh is the embedder's job). + ("/api/w/test/apps_u/embed_token/secret", "GET"), + ("/api/w/test/apps_u/embed_token_by_custom_path/foo", "GET"), + ("/api/w/test/scripts/list", "GET"), + ("/api/w/test/variables/list", "GET"), + ("/api/w/test/resources/update/u/admin/r", "POST"), + // Resource value reads must NOT be reachable with the embed token. + ("/api/w/test/resources/get/u/admin/r", "GET"), + ("/api/w/test/resources/get_value/u/admin/r", "GET"), + ( + "/api/w/test/resources/get_value_interpolated/u/admin/r", + "GET", + ), + ("/api/w/test/resources/list_search", "GET"), + // Workspace-wide job enumeration/export must NOT be reachable — an app + // reads only jobs it launched, by id (blocked via the app_embed sentinel). + ("/api/w/test/jobs/list", "GET"), + ("/api/w/test/jobs/list_filtered_uuids", "GET"), + ("/api/w/test/jobs/completed/list", "GET"), + ("/api/w/test/jobs/completed/export", "GET"), + ("/api/w/test/jobs/queue/list", "GET"), + ("/api/w/test/jobs/queue/list_filtered_uuids", "GET"), + ("/api/w/test/jobs/queue/export", "GET"), + // Job counts (workspace-wide aggregates) and the capability-minting + // routes (signed resume/approval URLs) are NOT by-id polling — denied. + ("/api/w/test/jobs/completed/count", "GET"), + ("/api/w/test/jobs/completed/count_jobs", "GET"), + ("/api/w/test/jobs/queue/count", "GET"), + ("/api/w/test/jobs/job_signature/some-uuid/some-rid", "GET"), + ("/api/w/test/jobs/resume_urls/some-uuid/some-rid", "GET"), + // get_root_job_id has no access check in its handler and the app never + // calls it — denied so the token can't probe foreign jobs' flow lineage. + ("/api/w/test/jobs_u/get_root_job_id/some-uuid", "GET"), + // `users:read`/`folders:read` exist only for whoami/listnames — every + // other route in those domains is denied via the app_embed sentinel + // (the whole /users and /folders routers are CORS-enabled for the iframe). + ("/api/w/test/users/list", "GET"), + ("/api/w/test/users/list_usage", "GET"), + ("/api/w/test/users/username_to_email/admin", "GET"), + ("/api/w/test/folders/list", "GET"), + ("/api/w/test/folders/get/myfolder", "GET"), + ("/api/w/test/folders/getusage/myfolder", "GET"), + ]; + for (path, method) in denied { + assert!( + check_scopes_for_route(scopes, path, method).is_err(), + "embed token should deny {method} {path}" + ); + } + } + + /// `apps:run` satisfies read at the route layer, so `apps/list` / `apps/list_search` + /// pass the route check — that's why those handlers ALSO call + /// `check_scopes(apps:read)`, which uses `ScopeDefinition::includes` (where run + /// does NOT include read). Lock that: no embed scope, including the + /// dynamically-minted path-scoped read, satisfies a domain-level `apps:read`, so + /// the token cannot list all apps' definitions (their full `value`/code). + #[test] + fn embed_scopes_cannot_satisfy_domain_app_read() { + use windmill_api_auth::scopes::ScopeDefinition; + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // mint_app_embed_token also grants read scoped to the single app path: + scopes.push("apps:read:u/admin/app".to_string()); + let required = ScopeDefinition::from_scope_string("apps:read").unwrap(); + for s in &scopes { + // The `app_embed` sentinel intentionally doesn't parse as a domain:action + // scope (it grants nothing; it only drives the job-enumeration deny). + let Ok(def) = ScopeDefinition::from_scope_string(s) else { + continue; + }; + assert!( + !def.includes(&required), + "embed scope {s} must not satisfy domain-level apps:read (would leak apps/list[_search])" + ); + } + // Sanity: a genuine domain-level apps:read token does satisfy it. + assert!(ScopeDefinition::from_scope_string("apps:read") + .unwrap() + .includes(&required)); + } + + /// The token carries path-scoped `apps:run:` and `apps:read:` + /// (NOT unqualified `apps:run`). Every handler that resolves an app and acts on + /// its behalf re-checks the requested path via `ScopeDefinition::includes`, so the + /// token is confined to its OWN app: + /// - `apps:run:` — `execute_component`. + /// - `apps:read:` — `get_app` (apps/get/p), `get_public_app_by_secret`, + /// the EE custom-path `get_public_app_by_custom_path`, and + /// `download_s3_file_from_app`. + /// This blocks cross-app execution, definition reads (by secret / custom path), + /// and S3 file reads through another app's on-behalf policy. + #[test] + fn embed_run_scope_is_path_scoped_to_its_app() { + use windmill_api_auth::scopes::ScopeDefinition; + // The mint must not grant unqualified run (which would include any path). + assert!( + !APP_EMBED_SCOPES.contains(&"apps:run"), + "embed scopes must not include unqualified apps:run" + ); + for action in ["run", "read"] { + let own = + ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/app")).unwrap(); + assert!( + own.includes( + &ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/app")) + .unwrap() + ), + "apps:{action} must grant its own app" + ); + assert!( + !own.includes( + &ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/other")) + .unwrap() + ), + "apps:{action} must NOT grant another app (cross-app)" + ); + } + } + + /// `mint_app_embed_token` guards its `create_token_internal` call with + /// `ensure_scopes_within_caller`, so a scope-restricted bearer token cannot + /// bootstrap the broader embed-scope set. Lock that boundary on the exact + /// scope vec the mint builds: rejected for a path-scoped caller, no-op for + /// the unscoped browser session that is the normal embed flow. + #[test] + fn embed_token_mint_is_scope_bounded() { + use windmill_api_auth::{ensure_scopes_within_caller, ApiAuthed}; + + // Same scope set mint_app_embed_token assembles for an app. + let mut minted: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + minted.push("apps:read:u/admin/app".to_string()); + + // A caller restricted to a single app read must not widen to the full + // embed set (apps:run, jobs:read, resources:read, ...). + let restricted = ApiAuthed { + scopes: Some(vec!["apps:read:u/admin/app".to_string()]), + ..Default::default() + }; + assert!( + ensure_scopes_within_caller(&restricted, Some(&minted)).is_err(), + "a path-scoped caller must not mint the broader embed-scope set" + ); + + // An unscoped session (the normal embed flow) passes — the mint only + // narrows. + let unscoped = ApiAuthed { scopes: None, ..Default::default() }; + assert!(ensure_scopes_within_caller(&unscoped, Some(&minted)).is_ok()); + } + + /// The embed-token endpoints must keep working for legacy apps whose stored + /// policy no longer satisfies the strict `Policy` struct (pre-dating + /// now-required fields): `parse_embed_policy` reads only the sandbox-decision + /// fields, leniently, and treats a missing/unknown `execution_mode` as NOT + /// anonymous (the strictest access interpretation). + #[test] + fn embed_policy_parse_is_lenient() { + use super::parse_embed_policy; + + // Quirky legacy policy: triggerables_v2 entry missing required fields, + // no execution_mode at all — must still parse, and absent `sandbox` + // resolves to the unsandboxed default. + let p = parse_embed_policy(r#"{"triggerables_v2": {"x": {}}}"#).unwrap(); + assert!(!p.sandbox); + assert!( + !p.anonymous_execution, + "missing execution_mode must not grant anonymous access" + ); + + // Normal policies map field-for-field. + let p = parse_embed_policy(r#"{"execution_mode": "anonymous", "sandbox": true}"#).unwrap(); + assert!(p.anonymous_execution); + assert!(p.sandbox); + + // Unknown execution_mode value: lenient parse, but not anonymous. + let p = parse_embed_policy(r#"{"execution_mode": "weird"}"#).unwrap(); + assert!(!p.anonymous_execution); + + // Invalid JSON still errors. + assert!(parse_embed_policy("not json").is_err()); + } +} diff --git a/backend/windmill-api/src/jobs.rs b/backend/windmill-api/src/jobs.rs index 30dc64d8e2..15442bd815 100644 --- a/backend/windmill-api/src/jobs.rs +++ b/backend/windmill-api/src/jobs.rs @@ -513,6 +513,26 @@ async fn cancel_job_api( Path((w_id, id)): Path<(String, Uuid)>, Json(CancelJob { reason }): Json, ) -> error::Result { + // App embed tokens (the sandboxed app iframe) may cancel ONLY jobs they launched + // — their app's component runs, stamped created_by == viewer. cancel_job_api has + // no other per-job ownership check, so without this an embed token (which carries + // the viewer's identity) could cancel any job by id. NotFound (not 403) so the + // untrusted app can't probe job existence. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + let created_by = sqlx::query_scalar!( + "SELECT created_by FROM v2_job WHERE id = $1 AND workspace_id = $2", + id, + &w_id + ) + .fetch_optional(&db) + .await?; + if created_by.as_deref() != Some(authed.username.as_str()) { + return Err(Error::NotFound(format!("Job {id} not found"))); + } + } + } + let tx = db.begin().await?; let audit_author: AuditAuthor = match opt_authed.as_ref() { @@ -1007,6 +1027,17 @@ async fn require_job_read_access( return Ok(()); } + // App embed tokens (the sandboxed app iframe) carry the viewer's identity so the + // app can read its own component runs — which are stamped `created_by == viewer` + // and so already returned above. They must NOT inherit the viewer's *broader* + // job access (share links, folder ACLs, admin RLS): user-authored app JS holds + // this token, and letting it reach any job merely visible to the viewer would + // expose unrelated runs' results/logs. Stop at the launched-by-viewer grant. + // NotFound (not PermissionDenied) so the untrusted app can't probe job existence. + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + return Err(Error::NotFound(format!("Job {job_id} not found"))); + } + // `username_override` is derived from the token *label* (`username_override_from_label`), // which is fully user-controlled with no uniqueness/ownership check (webhook-/http-/ // email-/ws- trigger tokens, `ephemeral-script-end-user-*`, and the generic `label-*` diff --git a/backend/windmill-api/src/lib.rs b/backend/windmill-api/src/lib.rs index 2d31712e88..19b7ddfa6f 100644 --- a/backend/windmill-api/src/lib.rs +++ b/backend/windmill-api/src/lib.rs @@ -546,7 +546,15 @@ pub async fn run_server( Router::new() // Reordered alphabetically .nest("/acls", granular_acls::workspaced_service()) - .nest("/apps", apps::workspaced_service(request_size_limit * 5)) + // CORS so the opaque-origin in-workspace app viewer (WIN-2006, + // sandboxed /apps/get) can read the app definition by path + // (apps/get/p, apps/embed_token/p) with a scoped embed token. + // Bearer-token-only (no cookies), consistent with the other + // workspaced services the iframe calls. + .nest( + "/apps", + apps::workspaced_service(request_size_limit * 5).layer(cors.clone()), + ) .nest("/assets", windmill_api_assets::workspaced_service()) .nest("/audit", audit::workspaced_service()) .nest("/capture", capture::workspaced_service()) @@ -566,7 +574,13 @@ pub async fn run_server( "/flow_conversations", windmill_api_flow_conversations::workspaced_service(), ) - .nest("/folders", folders::workspaced_service()) + // CORS so an opaque-origin app iframe (WIN-2006 embed, + // no separate domain) can read folders/listnames with a + // scoped embed token. Consistent with apps_u/jobs_u cors. + .nest( + "/folders", + folders::workspaced_service().layer(cors.clone()), + ) .nest("/folders_history", folder_history::workspaced_service()) .nest("/groups", groups::workspaced_service()) .nest("/groups_history", group_history::workspaced_service()) @@ -616,14 +630,23 @@ pub async fn run_server( path_autocomplete::workspaced_service(), ) .nest("/raw_apps", raw_apps::workspaced_service()) - .nest("/resources", resources::workspaced_service()) + // CORS so the opaque-origin app iframe can read + // resources/list, resources/type/* with a scoped token. + .nest( + "/resources", + resources::workspaced_service().layer(cors.clone()), + ) .nest("/shared_ui", workspace_shared_ui::workspaced_service()) .nest("/schedules", windmill_api_schedule::workspaced_service()) .nest("/scripts", scripts::workspaced_service()) .nest("/trash", trash::workspaced_service()) .nest( "/users", - users::workspaced_service().layer(Extension(argon2.clone())), + // CORS so the opaque-origin app iframe can read + // users/whoami with a scoped embed token. + users::workspaced_service() + .layer(Extension(argon2.clone())) + .layer(cors.clone()), ) .nest("/variables", variables::workspaced_service()) .nest("/volumes", volumes_oss::workspaced_service()) @@ -729,7 +752,11 @@ pub async fn run_server( .nest("/apps_u", { #[cfg(feature = "enterprise")] { - apps_oss::global_unauthed_service() + // CORS so the opaque-origin app viewer (WIN-2006 embed, no + // separate domain) can load a custom-path public app via + // public_app_by_custom_path cross-origin. Consistent with + // the workspaced /w/{workspace_id}/apps_u mount below. + apps_oss::global_unauthed_service().layer(cors.clone()) } #[cfg(not(feature = "enterprise"))] diff --git a/frontend/src/app.html b/frontend/src/app.html index 718800105a..aad967fd91 100644 --- a/frontend/src/app.html +++ b/frontend/src/app.html @@ -1,4 +1,4 @@ - + @@ -23,6 +23,106 @@ } + %sveltekit.head% @@ -56,7 +156,7 @@ /> { + if (WINDMILL_RESERVED_QUERY_PARAMS.has(k)) reserved.push(k) + }) + reserved.forEach((k) => params.delete(k)) + const qs = params.toString() + const url = window.location.pathname + (qs ? `?${qs}` : '') + window.location.hash + // `/app_embed/{workspace}/` is the opaque in-workspace viewer route + // (WIN-2006) — same app-path suffix as `/apps/get/`. + const processedUrl = url + .replace('/apps/edit/', '') + .replace('/apps/get/', '') + .replace(/^\/app_embed\/[^/]+\//, '') return processedUrl } diff --git a/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte b/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte index ba17f6c7c3..8e068274b8 100644 --- a/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte +++ b/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte @@ -86,7 +86,10 @@ } const resolvedConfig = $state( - initConfig(components['dbexplorercomponent'].initialData.configuration, untrack(() => configuration)) + initConfig( + components['dbexplorercomponent'].initialData.configuration, + untrack(() => configuration) + ) ) let timeoutInput: number | undefined = undefined @@ -180,18 +183,22 @@ ) } - let outputs = initOutput($worldStore, untrack(() => id), { - selectedRowIndex: 0, - selectedRow: {}, - selectedRows: [] as any[], - result: [] as any[], - inputs: {}, - loading: false, - page: 0, - newChange: { row: 0, column: '', value: undefined }, - ready: undefined as boolean | undefined, - openedModalRow: {} - }) + let outputs = initOutput( + $worldStore, + untrack(() => id), + { + selectedRowIndex: 0, + selectedRow: {}, + selectedRows: [] as any[], + result: [] as any[], + inputs: {}, + loading: false, + page: 0, + newChange: { row: 0, column: '', value: undefined }, + ready: undefined as boolean | undefined, + openedModalRow: {} + } + ) let lastResource: string | undefined = undefined @@ -260,9 +267,7 @@ resolvedConfig.type, { table: { - selectOptions: dbSchemas - ? await getTablesByResource(dbSchemas, dbtype, dbPath, $workspaceStore!) - : [], + selectOptions: dbSchemas ? await getTablesByResource(dbSchemas, dbtype) : [], loading: false } } diff --git a/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts b/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts index 3ec21a3801..1d0fa63342 100644 --- a/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts +++ b/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts @@ -1,4 +1,4 @@ -import { JobService, ResourceService } from '$lib/gen' +import { JobService } from '$lib/gen' import { runScriptAndPollResult } from '$lib/components/jobs/utils' import type { DbInput } from '$lib/components/dbTypes' @@ -39,17 +39,9 @@ export async function loadTableMetaData( const ducklake = input.type === 'ducklake' ? input.ducklake : undefined const dbArg = getDatabaseArg(input) - // MySQL needs the database name for metadata queries - let databaseName: string | undefined - if (input.type === 'database' && input.resourceType === 'mysql') { - const resourceObj = (await ResourceService.getResourceValue({ - workspace, - path: input.resourcePath - })) as any - databaseName = resourceObj?.database - } - - const content = makeMetadataMarker('LOAD_TABLE_METADATA', { table, databaseName }, ducklake) + // MySQL: the metadata query resolves the database name server-side (it falls + // back to `DATABASE()`), so we don't read the resource value client-side for it. + const content = makeMetadataMarker('LOAD_TABLE_METADATA', { table }, ducklake) const job = await JobService.runScriptPreview({ workspace, @@ -106,22 +98,10 @@ export async function loadAllTablesMetaData( const dbArg = getDatabaseArg(input) const ducklake = input.type === 'ducklake' ? input.ducklake : undefined - // MySQL needs the database name for metadata queries - let databaseName: string | undefined - if (input.type === 'database' && input.resourceType === 'mysql') { - const resourceObj = (await ResourceService.getResourceValue({ - workspace, - path: input.resourcePath - })) as any - databaseName = resourceObj?.database - } - const language = getLanguageByResourceType(dbType) - const content = makeMetadataMarker( - 'LOAD_TABLE_METADATA', - { table: undefined, databaseName }, - ducklake - ) + // MySQL db name is resolved server-side via `DATABASE()` (see loadTableMetaData); + // no client-side resource-value read. + const content = makeMetadataMarker('LOAD_TABLE_METADATA', { table: undefined }, ducklake) let result = (await runScriptAndPollResult({ workspace, @@ -259,7 +239,11 @@ export async function getDbSchemas( const dbSchema = { lang: resourceTypeToLang(resourceType) as SQLSchema['lang'], schema, - publicOnly: !!schema.public || !!schema.PUBLIC || !!schema.dbo + publicOnly: !!schema.public || !!schema.PUBLIC || !!schema.dbo, + // MySQL introspection selects `DATABASE() AS default_db_name`; carry it + // so the table picker can tell the default db apart from other visible + // schemas. Other dbs don't return it (stays undefined). + defaultDb: Array.isArray(result) ? (result[0] as any)?.default_db_name : undefined } return { ...dbSchema, stringified: stringifySchema(dbSchema) } } else { @@ -283,9 +267,7 @@ export async function getDbSchemas( export async function getTablesByResource( schema: Partial>, - dbType: DbType | undefined, - dbPath: string, - workspace: string + dbType: DbType | undefined ): Promise { const s = Object.values(schema)?.[0] switch (dbType) { @@ -301,14 +283,15 @@ export async function getTablesByResource( return paths } case 'mysql': { - const resourceObj = (await ResourceService.getResourceValue({ - workspace, - path: dbPath.split('$res:')[1] - })) as any + // MySQL introspection lists DATABASE() plus any other visible non-system + // schemas. Show the default db's tables unprefixed and the rest as + // `db.table` — matching the pre-removal behavior (which matched the + // resource's `database`); `defaultDb` is the connection's DATABASE(). + const defaultDb = s && 'defaultDb' in s ? s.defaultDb : undefined const paths: string[] = [] for (const key in s?.schema) { for (const subKey in s.schema[key]) { - if (key === resourceObj?.database) { + if (key === defaultDb) { paths.push(`${subKey}`) } else { paths.push(`${key}.${subKey}`) diff --git a/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte b/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte index dd294be6af..c7c4e7d160 100644 --- a/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte +++ b/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte @@ -3,7 +3,7 @@ import type { AppInput } from '../../inputType' import type { Output } from '../../rx' import type { AppViewerContext, ListContext } from '../../types' - import { isScriptByNameDefined, isScriptByPathDefined } from '../../utils' + import { appNavigateSameWindow, isScriptByNameDefined, isScriptByPathDefined } from '../../utils' import NonRunnableComponent from './NonRunnableComponent.svelte' import RunnableComponent from './RunnableComponent.svelte' import { sendUserToast } from '$lib/toast' @@ -261,7 +261,9 @@ if (newTab) { window.open(gotoUrl, '_blank') } else { - window.location.href = gotoUrl + // Top-level load; inside the opaque viewer iframe this targets the + // top page (pre-sandbox behavior) instead of the cookieless frame. + appNavigateSameWindow(gotoUrl) } break diff --git a/frontend/src/lib/components/apps/components/helpers/eval.ts b/frontend/src/lib/components/apps/components/helpers/eval.ts index 86523b2959..bacfb65b2d 100644 --- a/frontend/src/lib/components/apps/components/helpers/eval.ts +++ b/frontend/src/lib/components/apps/components/helpers/eval.ts @@ -2,6 +2,8 @@ import type { World } from '../../rx' import { sendUserToast } from '$lib/toast' import { waitJob } from '$lib/components/waitJob' import { base } from '$lib/base' +import { appNavigateSameWindow } from '../../utils' +import { OpenAPI } from '$lib/gen/core/OpenAPI' export function computeGlobalContext( world: World | undefined, @@ -200,7 +202,9 @@ export async function eval_like( } window.open(x, '_blank') } else { - window.location.href = x + // Top-level load; inside the opaque viewer iframe this targets the + // top page (pre-sandbox behavior) instead of the cookieless frame. + appNavigateSameWindow(x) } }, (id, index) => { @@ -292,10 +296,30 @@ export async function eval_like( if (typeof input === 'object' && input.s3) { const workspaceId = ((context ?? {}) as any).ctx?.workspace - const s3href = `${base}/api/w/${workspaceId}/job_helpers/download_s3_file?file_key=${encodeURIComponent( - input?.s3 ?? '' - )}${input?.storage ? `&storage=${input.storage}` : ''}` - downloadFile(s3href, filename || input.s3) + const appPath = ((context ?? {}) as any).ctx?.app_path + let inSandbox = false + try { + inSandbox = + window.parent !== window && + new URLSearchParams(window.location.search).get('wm_embed') === '1' + } catch (_) {} + if (inSandbox && appPath && typeof OpenAPI.TOKEN === 'string' && OpenAPI.TOKEN) { + // Sandboxed viewer: the opaque iframe carries no cookie, so the + // cookie-authed job_helpers download fails. Route through the + // app-policy-confined apps_u endpoint with the embed token in the + // query (like the image/file components), scoped to this app's path. + const params = new URLSearchParams() + params.append('s3', input.s3 ?? '') + if (input.storage) params.append('storage', input.storage) + params.append('token', OpenAPI.TOKEN) + const s3href = `${base}/api/w/${workspaceId}/apps_u/download_s3_file/${appPath}?${params.toString()}` + downloadFile(s3href, filename || input.s3) + } else { + const s3href = `${base}/api/w/${workspaceId}/job_helpers/download_s3_file?file_key=${encodeURIComponent( + input?.s3 ?? '' + )}${input?.storage ? `&storage=${input.storage}` : ''}` + downloadFile(s3href, filename || input.s3) + } } else if (typeof input === 'string') { if (input.startsWith('data:')) { downloadFile(input, filename) diff --git a/frontend/src/lib/components/apps/editor/AppEditorHeader.svelte b/frontend/src/lib/components/apps/editor/AppEditorHeader.svelte index ca4ae77be4..a03aa9e9eb 100644 --- a/frontend/src/lib/components/apps/editor/AppEditorHeader.svelte +++ b/frontend/src/lib/components/apps/editor/AppEditorHeader.svelte @@ -395,14 +395,16 @@ } } - async function setPublishState() { + async function setPublishState(message?: string) { policy = await updatePolicy($app, policy) await AppService.updateApp({ workspace: $workspaceStore!, path: $appPath, requestBody: { policy } }) - if (policy.execution_mode == 'anonymous') { + if (message) { + sendUserToast(message) + } else if (policy.execution_mode == 'anonymous') { sendUserToast('App require no login to be accessed') } else { sendUserToast('App require login and read-access') diff --git a/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte b/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte index c344114404..0d5fc0c44a 100644 --- a/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte +++ b/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte @@ -1,5 +1,6 @@ + + { + refresh = requestTokenRefresh + loadApp() + }} +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + + +{#if canWriteApp && !hideEditBtn} +
+ +
+{/if} diff --git a/frontend/src/lib/components/apps/editor/PublicApp.svelte b/frontend/src/lib/components/apps/editor/PublicApp.svelte index fef6ba8fa3..92cde8788b 100644 --- a/frontend/src/lib/components/apps/editor/PublicApp.svelte +++ b/frontend/src/lib/components/apps/editor/PublicApp.svelte @@ -7,8 +7,13 @@ import { isCloudHosted } from '$lib/cloud' import { Alert, Skeleton } from '$lib/components/common' import { WindmillIcon } from '$lib/components/icons' - import { onMount, setContext } from 'svelte' - import { IS_APP_PUBLIC_CONTEXT_KEY, type EditorBreakpoint } from '../types' + import { getContext, onMount, setContext } from 'svelte' + import { + EMBED_NAV_CONTEXT_KEY, + IS_APP_PUBLIC_CONTEXT_KEY, + type EditorBreakpoint, + type EmbedNav + } from '../types' import { UserService, type AppWithLastVersion, type GlobalWhoamiResponse } from '$lib/gen' import { urlParamsToObject } from '$lib/utils' import { goto } from '$app/navigation' @@ -24,7 +29,9 @@ jwtError, onLoginSuccess, app, - workspace + workspace, + inWorkspace = false, + hideRefreshBar = false }: { notExists: boolean noPermission: boolean @@ -32,12 +39,27 @@ onLoginSuccess: () => void app: (AppWithLastVersion & { value: any; workspace_id?: string }) | undefined workspace: string | undefined + /** + * In-workspace rendering (`/apps/get`, `/app_embed`): keep exact parity + * with the pre-sandbox member viewer — no "Powered by Windmill" badge, no + * user overlay, no HTML-result approval gate, column flex wrapper. + */ + inWorkspace?: boolean + hideRefreshBar?: boolean } = $props() // Use workspace from props or from app.workspace_id (for custom path responses) let effectiveWorkspace = $derived(workspace ?? app?.workspace_id) - setContext(IS_APP_PUBLIC_CONTEXT_KEY, true) + // HTML results from runnables only need viewer approval on the public + // surfaces (untrusted distribution); the in-workspace viewer never gated them. + setContext(IS_APP_PUBLIC_CONTEXT_KEY, !inWorkspace) + + // WIN-2006: inside the opaque viewer iframe, navigations to other routes + // (navbar "app" items) must happen on the TOP page — the iframe is cookieless, + // so navigating it would just show a login screen. PublicAppFrame provides the + // relay; outside the opaque viewer this is undefined and goto works directly. + const embedNav = getContext(EMBED_NAV_CONTEXT_KEY) const breakpoint = writable('lg') @@ -71,27 +93,29 @@ }) -
- Powered by   Windmill -
+ Powered by   Windmill +
-{#snippet userInfo(child)} -
{child}
-{/snippet} + {#snippet userInfo(child)} +
{child}
+ {/snippet} -
{#if $userStore} - {@render userInfo($userStore.username)} - {:else if globalUser} - {@render userInfo(globalUser.email)} - {:else}{/if} -
+
{#if $userStore} + {@render userInfo($userStore.username)} + {:else if globalUser} + {@render userInfo(globalUser.email)} + {:else}{/if} +
+{/if} {#if notExists}
goto(path)} - gotoFn={(path, opt) => goto(path, opt)} + gotoFn={(path, opt) => (embedNav ? embedNav.navigateTop(path) : goto(path, opt))} />
{/if} diff --git a/frontend/src/lib/components/apps/editor/PublicAppFrame.svelte b/frontend/src/lib/components/apps/editor/PublicAppFrame.svelte new file mode 100644 index 0000000000..43dc937651 --- /dev/null +++ b/frontend/src/lib/components/apps/editor/PublicAppFrame.svelte @@ -0,0 +1,427 @@ + + +{#if isViewer} + {#if viewerReady} + {@render viewer()} + {:else if viewerOrphaned} +
+ + This is a Windmill app viewer and must be loaded by Windmill. If you embedded it in your own + page, use the app's public URL without the wm_embed parameter. + +
+ {:else} + + {/if} +{:else if status === 'loading'} + +{:else if status === 'notExists'} +
+ + There was an error loading the app, is the url correct? + Go to Windmill + +
+{:else if status === 'noPermission'} + +
This app requires read access
+
+ initEmbedder()} + popup + rd={page.url.pathname + page.url.search + page.url.hash} + /> +
+{:else if unsandboxed} + + {@render viewer()} +{:else if isRaw} + + {@render viewer()} +{:else} + + +{/if} diff --git a/frontend/src/lib/components/apps/editor/appPolicy.ts b/frontend/src/lib/components/apps/editor/appPolicy.ts index 8c54926f1b..b6e35bd65e 100644 --- a/frontend/src/lib/components/apps/editor/appPolicy.ts +++ b/frontend/src/lib/components/apps/editor/appPolicy.ts @@ -180,12 +180,13 @@ export async function updatePolicy(app: App, currentPolicy: Policy | undefined): }) .filter(Boolean) as { s3_path: string; storage?: string | undefined }[] - return { + const next = { ...(currentPolicy ?? {}), allowed_s3_keys: s3FileKeys, s3_inputs, triggerables_v2: ntriggerables } + return next } export async function processRunnable( diff --git a/frontend/src/lib/components/apps/types.ts b/frontend/src/lib/components/apps/types.ts index e4e030de67..322f9ea969 100644 --- a/frontend/src/lib/components/apps/types.ts +++ b/frontend/src/lib/components/apps/types.ts @@ -370,6 +370,12 @@ export type EditorBreakpoint = 'sm' | 'lg' export const IS_APP_PUBLIC_CONTEXT_KEY = 'isAppPublicContext' as const +// Set by PublicAppFrame in opaque-viewer mode (WIN-2006). Lets the app relay +// top-level navigations (e.g. navbar links to another app) to the embedder, +// since navigating inside the opaque iframe would load the SPA cookieless. +export const EMBED_NAV_CONTEXT_KEY = 'appEmbedNav' as const +export type EmbedNav = { navigateTop: (href: string) => void } + type ComponentID = string export type ContextPanelContext = { diff --git a/frontend/src/lib/components/apps/utils.ts b/frontend/src/lib/components/apps/utils.ts index 85bdd774eb..2dfdd42e6b 100644 --- a/frontend/src/lib/components/apps/utils.ts +++ b/frontend/src/lib/components/apps/utils.ts @@ -20,6 +20,31 @@ import type { } from './types' import { allItems, BG_PREFIX } from './editor/appUtilsCore' +/** + * Same-window navigation for app code (frontend-script `goto`, button + * `onSuccess: gotoUrl`). Inside the opaque viewer iframe (WIN-2006, + * `wm_embed=1`), navigating the current window would load the target inside + * the cookieless frame — so the navigation is relayed to the embedder page, + * which navigates itself (`wm_embed_navigate` in PublicAppFrame). That matches + * the pre-sandbox behavior exactly: the app used to run ON the embedder page, + * including when that page is itself inside a third-party iframe (where the + * embedder — not the third party's top — was what `window.location` changed). + * Outside the opaque viewer it keeps navigating the current window as before. + */ +export function appNavigateSameWindow(url: string) { + try { + const params = new URLSearchParams(window.location.search) + if (window.parent !== window && params.get('wm_embed') === '1') { + window.parent.postMessage( + { type: 'wm_embed_navigate', href: url }, + params.get('wm_embedder_origin') ?? '*' + ) + return + } + } catch (_) {} + window.location.href = url +} + // `migrateApp` moved to its own light module so non-editor callers can reuse it // without pulling the whole `apps/utils` graph; re-exported here for existing // `from '../utils'` importers. diff --git a/frontend/src/lib/components/common/table/RawAppRow.svelte b/frontend/src/lib/components/common/table/RawAppRow.svelte index e759a1d9fe..3f74356506 100644 --- a/frontend/src/lib/components/common/table/RawAppRow.svelte +++ b/frontend/src/lib/components/common/table/RawAppRow.svelte @@ -33,7 +33,7 @@ editor: boolean workspace: string + /** + * Restrict waitJob/getJob/streamJob to job ids launched by this app + * instance (WIN-2006): a SANDBOXED bundle must not read arbitrary + * workspace jobs through the credentialed bridge. Off for unsandboxed + * renders (the default, and editor preview) — there the bundle holds + * the same credential as the bridge, so gating adds nothing and would + * only break unsandboxed apps that poll persisted or runnable-returned + * job ids. + */ + gateJobIds?: boolean } let { @@ -24,10 +34,18 @@ jobs = $bindable([]), jobsById = $bindable({}), editor, - workspace + workspace, + gateJobIds = true }: Props = $props() + // Job ids launched by this app instance — see `gateJobIds`. + const launchedJobs = new Set() + let listener = async (event) => { + // Only accept messages from the bundle iframe (opaque origin) so other + // frames/extensions can't drive the runnable bridge (WIN-2006). Reject + // unconditionally until the iframe is bound — never process a message from + // an unknown source. if (!iframe || event.source !== iframe.contentWindow) return const data = event.data @@ -115,6 +133,7 @@ }, undefined ) + launchedJobs.add(uuid) let job: JobById = { component: runnable_id, created_at: Date.now(), job: uuid } if (event.data.type == 'backendAsync') { let result = uuid @@ -134,14 +153,29 @@ console.error('No runnable found for', runnable_id) } } else if (event.data.type == 'waitJob') { + if (gateJobIds && !launchedJobs.has(data.jobId)) { + respond({ result: { message: 'Unknown job' }, error: true }) + return + } await respondWithResult(data.jobId) } else if (event.data.type == 'getJob') { + if (gateJobIds && !launchedJobs.has(data.jobId)) { + respond({ result: { message: 'Unknown job' }, error: true }) + return + } const job = await JobService.getJob({ workspace, id: data.jobId }) respond({ result: job }) } else if (event.data.type == 'streamJob') { // Stream job results using SSE const jobId = data.jobId const reqId = data.reqId + if (gateJobIds && !launchedJobs.has(jobId)) { + iframe?.contentWindow?.postMessage( + { type: 'streamJobRes', reqId, error: true, result: { message: 'Unknown job' } }, + '*' + ) + return + } const params = new URLSearchParams() params.set('fast', 'true') params.set('only_result', 'true') diff --git a/frontend/src/lib/components/raw_apps/RawAppEditor.svelte b/frontend/src/lib/components/raw_apps/RawAppEditor.svelte index 07e7c7bb29..1af802c7fe 100644 --- a/frontend/src/lib/components/raw_apps/RawAppEditor.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppEditor.svelte @@ -1496,6 +1496,7 @@ bind:jobsById {runnables} {path} + gateJobIds={false} />
{ - initialHash = window.location.hash || '' + // WIN-2006: unless the publisher opted into sandbox isolation, run the bundle + // same-origin with full access (the default); otherwise the opaque-origin sandbox. + const unsandboxedCtx = getContext<{ value: boolean }>('IS_APP_UNSANDBOXED') + let unsandboxed = $derived(unsandboxedCtx?.value ?? false) + // Unsandboxed (the default) must match the pre-isolation viewer exactly: NO + // sandbox attribute (a same-origin blob with full session — an attribute would + // only break leftover features like unsandboxed popups for OAuth flows, while + // adding no isolation). The sandboxed path keeps the restrictive attribute; the + // wrapper document's `CSP: sandbox` response header enforces the opaque origin + // regardless. + let sandboxAttr = $derived( + unsandboxed + ? undefined + : 'allow-scripts allow-forms allow-popups allow-popups-to-escape-sandbox allow-downloads allow-modals allow-top-navigation' + ) + + // WIN-2006: source of the bundle iframe. + // - DEFAULT (isolated): a real API URL serving a sandboxed, opaque-origin + // document (`CSP: sandbox` response header + the iframe sandbox attribute), + // so a malicious bundle can never reach the authenticated Windmill origin + // (no cookie, no window.parent, no token). Root-relative so it resolves + // against the real host even when this component itself runs inside an opaque + // viewer (where `location.origin` is "null"). Context is handed over via + // postMessage — never baked into the document, never a credential. + // - UNSANDBOXED (the default — publisher did not opt into isolation): a + // client-built blob: wrapper (same-origin with the SPA) loaded with `allow-same-origin`, + // so relative `fetch('/api/...')` and the session cookie work. The backend + // `.html` is ALWAYS sandboxed, so we must build the same-origin wrapper here + // rather than relax a real-origin endpoint a victim could be linked to. + let iframeSrc = $derived.by(() => { + if (!secret || typeof window === 'undefined') return undefined + if (unsandboxed) { + // untrack(user) so userStore refreshes don't regenerate the blob URL and + // reload the iframe (losing state); ctx is only needed for initial render. + // Always pass the wrapper object — pre-sandbox bundles rely on + // `window.ctx.workspace` even for anonymous viewers (ctx.ctx undefined). + const u = untrack(() => user) + const html = unsandboxedRawAppHtml( + workspace, + secret, + { ctx: u, workspace }, + window.location.origin, + window.location.hash || '' + ) + return URL.createObjectURL(new Blob([html], { type: 'text/html' })) + } + // `wm_coep` (embed-in-cross-origin-isolated-page opt-in) must be propagated + // to the wrapper document: under a COEP `require-corp` embedder, a nested + // document is only allowed to load if it asserts COEP itself, so the + // backend adds the header when the flag is present. + const coep = new URLSearchParams(window.location.search).has('wm_coep') ? '?wm_coep=1' : '' + return `/api/w/${workspace}/apps_u/get_data/v/${secret}.html${coep}` }) - // Use blob URL instead of srcDoc to give the iframe a proper origin. - // srcDoc iframes have "null" origin which breaks URL constructor in routers. - // untrack(user) so that userStore refreshes don't regenerate the blob URL - // and cause the iframe to fully reload (losing all state). - // The user context is only needed for initial render. - let blobUrl = $derived.by(() => { - if (!secret) return undefined - const u = untrack(() => user) - const baseUrl = typeof window !== 'undefined' ? window.location.origin : '' - const html = htmlContent(workspace, secret, { ctx: u, workspace }, baseUrl, initialHash) - const blob = new Blob([html], { type: 'text/html' }) - return URL.createObjectURL(blob) - }) - - // Cleanup blob URL when it changes or component unmounts + // Revoke blob: URLs (unsandboxed path) when they change or on unmount. $effect(() => { - const url = blobUrl + const url = iframeSrc return () => { - if (url) URL.revokeObjectURL(url) + if (url && url.startsWith('blob:')) URL.revokeObjectURL(url) + } + }) + + // Persistence for the bundle's (opaque-origin) localStorage, backed by a store + // scoped PER APP (keyed by workspace + app path) so one sandboxed app can't read + // or clobber another's (even two apps at the same path in different workspaces). On a real origin (workspace viewer, public page — even when + // that page sits inside someone else's iframe) it reads/writes real localStorage + // directly. Only inside an opaque frame (the Windmill embed viewer), where Web + // Storage throws, does it relay per-key ops up to the embedder, the persistence + // authority. `framed` therefore probes storage rather than just `window.parent`: + // an externally-embedded public page is framed too, but its parent is not the + // Windmill embedder and would never answer the relay (leaving the bundle without + // ctx). The snapshot is handed to the bundle before it evaluates so its + // localStorage is hydrated synchronously. + const SHARED_LS_KEY = `wm_apps_localstorage:${workspace}:${path}` + function storageAccessible(): boolean { + try { + localStorage.getItem(SHARED_LS_KEY) + return true + } catch (_) { + return false + } + } + const framed = typeof window !== 'undefined' && window.parent !== window && !storageAccessible() + let bundleStorage: Record | undefined = undefined + let pendingReady = false + + function readDirect(): Record { + try { + return JSON.parse(localStorage.getItem(SHARED_LS_KEY) || '{}') + } catch (_) { + return {} + } + } + + function applyDirectOp(d: any) { + try { + const s = readDirect() + if (d.op === 'set') s[d.key] = String(d.value) + else if (d.op === 'remove') delete s[d.key] + else if (d.op === 'clear') for (const k in s) delete s[k] + localStorage.setItem(SHARED_LS_KEY, JSON.stringify(s)) + } catch (_) {} + } + + function respondCtx() { + iframe?.contentWindow?.postMessage( + { + type: 'windmill:ctx', + // Same shape as the unsandboxed wrapper: always the object, so + // `window.ctx.workspace` works for anonymous viewers too. + ctx: { ctx: user, workspace }, + initialHash, + storage: { local: bundleStorage ?? {}, session: {} } + }, + '*' + ) + } + + onMount(() => { + initialHash = window.location.hash || '' + if (framed) { + // Pre-fetch the shared store from the embedder. + try { + window.parent.postMessage({ type: 'wm_ls_req' }, '*') + } catch (_) {} + // If the parent never answers (it isn't the Windmill embedder, e.g. an + // opaque context created by a third party), don't hold the bundle's ctx + // hostage: proceed with empty storage. Must beat the backend wrapper's + // own 1.5s no-ctx fallback. + const fallback = setTimeout(() => { + if (bundleStorage === undefined) { + bundleStorage = {} + if (pendingReady) { + pendingReady = false + respondCtx() + } + } + }, 750) + return () => clearTimeout(fallback) } }) - // Listen for hash changes from iframe and update parent URL $effect(() => { function handleMessage(event: MessageEvent) { - console.log('[Parent] Received message:', event.data) - if (event.data?.type === 'windmill:hashchange') { - const newHash = event.data.hash || '' - console.log('[Parent] Updating hash to:', newHash) - // Update parent URL without triggering navigation + const data = event.data + // Shared-store hydration from the embedder (public mode only). + if (framed && event.source === window.parent && data?.type === 'wm_ls_hydrate') { + bundleStorage = data.data || {} + if (pendingReady) { + pendingReady = false + respondCtx() + } + return + } + // Everything else must come from the bundle iframe. + if (event.source !== iframe?.contentWindow) return + if (data?.type === 'windmill:ready') { + // Hand the bundle its context + shared storage before it evaluates. + if (!framed) { + bundleStorage = readDirect() + respondCtx() + } else if (bundleStorage !== undefined) { + respondCtx() + } else { + pendingReady = true + } + } else if (data?.type === 'wm_ls_op') { + // The bundle mutated localStorage — apply it to the shared store. + if (!framed) { + applyDirectOp(data) + } else { + try { + window.parent.postMessage( + { type: 'wm_ls_op', op: data.op, key: data.key, value: data.value }, + '*' + ) + } catch (_) {} + } + } else if (data?.type === 'windmill:hashchange') { + // Keep the parent URL hash in sync for shareable URLs. + const newHash = data.hash || '' if (window.location.hash !== newHash) { history.replaceState(null, '', newHash || window.location.pathname) } @@ -65,13 +212,29 @@ }) - + -{#if blobUrl} +{#if iframeSrc} + + {/if} diff --git a/frontend/src/lib/components/raw_apps/rawAppPolicy.ts b/frontend/src/lib/components/raw_apps/rawAppPolicy.ts index f33c6e7eb6..16f588c993 100644 --- a/frontend/src/lib/components/raw_apps/rawAppPolicy.ts +++ b/frontend/src/lib/components/raw_apps/rawAppPolicy.ts @@ -19,10 +19,11 @@ export async function updateRawAppPolicy( ) ).filter((entry): entry is [string, TriggerableV2] => entry != null) const triggerables_v2 = Object.fromEntries(entries) - return { + const next: Policy = { ...currentPolicy, triggerables_v2 } + return next } type RunnableWithInlineScript = RunnableWithFields & { diff --git a/frontend/src/lib/components/raw_apps/utils.ts b/frontend/src/lib/components/raw_apps/utils.ts index 9558697708..c0fba48e18 100644 --- a/frontend/src/lib/components/raw_apps/utils.ts +++ b/frontend/src/lib/components/raw_apps/utils.ts @@ -141,65 +141,50 @@ export function formatAppRunsForChat(runs: RawAppRunSummary[]): string { return JSON.stringify(runs, null, 2) } -export function htmlContent( +// The sandboxed (isolated) raw-app wrapper is generated server-side and served as +// a sandboxed, opaque-origin document (see `get_raw_app_data` in the backend +// `apps.rs`, WIN-2006) — a blob: URL cannot carry the `CSP: sandbox` response +// header that enforces isolation, so the wrapper must come from the backend. +// +// The function below is used ONLY for the unsandboxed path (the default — the +// publisher did not opt into sandbox isolation). It is loaded as a blob: URL — +// same-origin with the SPA — so, with `allow-same-origin`, the bundle runs with +// the viewer's full session. Crucially this is an in-memory blob, not a +// real-origin endpoint, so it is not a URL an attacker can navigate a logged-in +// victim to in order to gain isolation-bypassing access — the backend `.html` +// document stays sandboxed whenever the publisher did opt in. +export function unsandboxedRawAppHtml( workspace: string, - secret: string | undefined, + secret: string, ctx: any, - baseUrl: string = '', - initialHash: string = '' + baseUrl: string, + initialHash: string ) { return ` - App Preview + App diff --git a/frontend/src/lib/stores.ts b/frontend/src/lib/stores.ts index 3a55e7b31c..3e3ab817eb 100644 --- a/frontend/src/lib/stores.ts +++ b/frontend/src/lib/stores.ts @@ -186,6 +186,10 @@ export interface SQLSchema { schema: SQLBaseSchema publicOnly: boolean | undefined stringified: string + /** MySQL only: the connection's default database (`DATABASE()`), surfaced by the + * introspection script. Lets the table picker render the default db's tables + * unprefixed even when the connection can also see other (non-system) schemas. */ + defaultDb?: string } export interface GraphqlSchema { diff --git a/frontend/src/lib/utils.ts b/frontend/src/lib/utils.ts index 9e14e7ab9a..d96cd51b75 100644 --- a/frontend/src/lib/utils.ts +++ b/frontend/src/lib/utils.ts @@ -1190,8 +1190,10 @@ export function isCodeInjection(expr: string | undefined): boolean { // app logic via the `query` context. Only params we actually own are listed // here — the `wm_` prefix is a naming convention, not a reserved namespace, so // we don't strip it wholesale (that would break apps reading their own `wm_*` -// params). `wm_coep` is a transport flag for cross-origin isolation headers. -export const WINDMILL_RESERVED_QUERY_PARAMS = new Set(['wm_coep']) +// params). `wm_coep` is a transport flag for cross-origin isolation headers; +// `wm_embed`/`wm_embedder_origin` are the opaque app viewer transport params +// (see PublicAppFrame). +export const WINDMILL_RESERVED_QUERY_PARAMS = new Set(['wm_coep', 'wm_embed', 'wm_embedder_origin']) export function urlParamsToObject( params: URLSearchParams, diff --git a/frontend/src/routes/(root)/(logged)/+layout.svelte b/frontend/src/routes/(root)/(logged)/+layout.svelte index 4f6004ce2a..e91ad87583 100644 --- a/frontend/src/routes/(root)/(logged)/+layout.svelte +++ b/frontend/src/routes/(root)/(logged)/+layout.svelte @@ -164,8 +164,9 @@ const toPath = navigation.to?.url.pathname if (toPath && (toPath.startsWith('/apps_raw/add') || toPath.startsWith('/apps_raw/edit'))) { const currentPath = navigation.from?.url.pathname - // Reload if we're not on an apps_raw path, or if we're on /apps/get_raw/ (viewing a raw app) - // The /apps/get_raw/ path doesn't have cross-origin isolation headers, so we need to reload + // Reload if we're not on an apps_raw path, or if we're on the raw app viewer + // (/apps_raw/get/): the viewer doesn't have cross-origin isolation headers, so + // we need a full reload to fetch them for the editor. if (!currentPath?.startsWith('/apps_raw/') || currentPath?.startsWith('/apps_raw/get/')) { navigation.cancel() window.location.href = navigation.to!.url.href diff --git a/frontend/src/routes/(root)/(logged)/apps/get/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps/get/[...path]/+page.svelte index 460b2957e0..0af3cdeded 100644 --- a/frontend/src/routes/(root)/(logged)/apps/get/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps/get/[...path]/+page.svelte @@ -1,91 +1,30 @@ -{#if app} - {#key app} -
- { - goto(path) - }} - gotoFn={(path, opt) => { - goto(path, opt) - }} - /> - {#if can_write && !hideEditBtn} -
- -
- {/if} -
+ +{#if workspace && path} + + {#key `${workspace}/${path}`} + {/key} {:else} diff --git a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.js b/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.js deleted file mode 100644 index 5b680f2fd0..0000000000 --- a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.js +++ /dev/null @@ -1,5 +0,0 @@ -export function load({ params }) { - return { - stuff: { title: `App ${params.path}` } - } -} diff --git a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte index db7b7da001..dbba9ce790 100644 --- a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte @@ -1,41 +1,17 @@ - -
- -{#if !loaded} - -{/if} diff --git a/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte index 9fdf4c5c7d..990e74ed8c 100644 --- a/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte @@ -1,65 +1,26 @@ -
- {#if !$workspaceStore || !$userStore || !app} - - {:else} - - {/if} - {#if can_write && !hideEditBtn} -
- -
- {/if} -
+{#if workspace && path} + + {#key `${workspace}/${path}`} + + {/key} +{:else} + +{/if} diff --git a/frontend/src/routes/a/[...path]/+page.svelte b/frontend/src/routes/a/[...path]/+page.svelte index 062f7ae8de..3456c6e4f8 100644 --- a/frontend/src/routes/a/[...path]/+page.svelte +++ b/frontend/src/routes/a/[...path]/+page.svelte @@ -3,14 +3,15 @@ import { AppService, OpenAPI, type AppWithLastVersion } from '$lib/gen' import { userStore, workspaceStore } from '$lib/stores' + import { sendUserToast } from '$lib/toast' - import { setContext } from 'svelte' import { setLicense } from '$lib/enterpriseUtils' import { getUserExt } from '$lib/user' - import { sendUserToast } from '$lib/toast' import { page } from '$app/state' + import { base } from '$lib/base' import PublicApp from '$lib/components/apps/editor/PublicApp.svelte' + import PublicAppFrame from '$lib/components/apps/editor/PublicAppFrame.svelte' let app: (AppWithLastVersion & { value: any }) | undefined = $state(undefined) let notExists = $state(false) @@ -44,16 +45,44 @@ } } - let workspace: string | undefined = $state(undefined) - async function loadApp() { - const parsedCustomPath = parseCustomPath(page.params.path ?? '') + const parsedCustomPath = parseCustomPath(page.params.path ?? '') + // URL for the opaque viewer iframe: the custom-path URL WITHOUT the trailing + // JWT segment. The JWT is a viewer credential (broader and longer-lived than + // the scoped embed token) consumed here on the embedder side only — it must + // never appear in the iframe's own location, where app-authored code could + // read it. Captured once (not reactively): the embedder mirrors the app's + // hash/query back onto this page's URL, and re-deriving the src from it would + // reload the app on its every navigation. + const viewerUrl = `${base}/a/${parsedCustomPath.path}${page.url.search}${page.url.hash}` + + let workspace: string | undefined = $state(undefined) + let refresh: (() => void) | undefined + + // Embedder side: validate access (main session cookie or shared JWT) and mint + // a scoped embed token for the opaque iframe (WIN-2006). + async function fetchEmbedToken(): Promise<{ token?: string }> { if (parsedCustomPath.jwt) { - const token = 'jwt_ext_' + parsedCustomPath.jwt - OpenAPI.TOKEN = token - setContext<{ token?: string }>('AuthToken', { token }) - jwtError = false + OpenAPI.TOKEN = 'jwt_ext_' + parsedCustomPath.jwt } + const headers: Record = {} + if (typeof OpenAPI.TOKEN === 'string' && OpenAPI.TOKEN) { + headers['Authorization'] = `Bearer ${OpenAPI.TOKEN}` + } + const res = await fetch( + `${OpenAPI.BASE}/apps_u/embed_token_by_custom_path/${parsedCustomPath.path}`, + { headers } + ) + if (!res.ok) { + const err: any = new Error('Failed to fetch embed token') + err.status = res.status + throw err + } + return await res.json() + } + + // Viewer side: load the app + user using the embed token handed to the iframe. + async function loadApp() { try { app = await AppService.getPublicAppByCustomPath({ customPath: parsedCustomPath.path @@ -62,9 +91,13 @@ workspaceStore.set(app.workspace_id) noPermission = false notExists = false + jwtError = false try { userStore.set(await getUserExt(app.workspace_id)) + // A JWT in the custom path that fails to resolve a user is surfaced as a + // toast (matches the pre-sandbox custom-path viewer) rather than silently + // falling through to anonymous. if (!$userStore && parsedCustomPath.jwt) { jwtError = true sendUserToast('Could not authentify user with jwt token', true) @@ -74,7 +107,8 @@ } } catch (e) { if (e.status == 401) { - noPermission = true + // Embed token missing/expired — ask the embedder for a fresh one. + refresh?.() } else { notExists = true } @@ -83,17 +117,25 @@ if (BROWSER) { setLicense() - loadApp() } - { + { + refresh = requestTokenRefresh loadApp() }} -> +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte b/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte new file mode 100644 index 0000000000..d0af10b433 --- /dev/null +++ b/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte @@ -0,0 +1,99 @@ + + + { + refresh = requestTokenRefresh + loadApp() + }} +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js b/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js deleted file mode 100644 index 42a8b51427..0000000000 --- a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js +++ /dev/null @@ -1,5 +0,0 @@ -export function load({ params }) { - return { - stuff: { title: `Public App` } - } -} diff --git a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte b/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte deleted file mode 100644 index 38563cfbd9..0000000000 --- a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte +++ /dev/null @@ -1,11 +0,0 @@ - - - diff --git a/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte b/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte index c661aa48c6..089b95c97d 100644 --- a/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte +++ b/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte @@ -4,21 +4,20 @@ import { AppService, OpenAPI, type AppWithLastVersion } from '$lib/gen' import { userStore } from '$lib/stores' - import { setContext } from 'svelte' import { setLicense } from '$lib/enterpriseUtils' import { getUserExt } from '$lib/user' - import { sendUserToast } from '$lib/toast' import { page } from '$app/state' + import { base } from '$lib/base' import PublicApp from '$lib/components/apps/editor/PublicApp.svelte' + import PublicAppFrame from '$lib/components/apps/editor/PublicAppFrame.svelte' let app: (AppWithLastVersion & { value: any }) | undefined = $state(undefined) let notExists = $state(false) let noPermission = $state(false) - let jwtError = $state(false) - function parseSecret(secret: string): { secret: string; jwt: string } { + function parseSecret(secret: string): { secret: string; jwt: string | undefined } { const parts = secret.split('/') return { secret: parts[0], @@ -27,18 +26,59 @@ } const parsedSecret = parseSecret(page.params.secret ?? '') + const workspace = page.params.workspace ?? '' + // URL for the opaque viewer iframe: the share URL WITHOUT the trailing JWT + // segment. The JWT is a viewer credential (broader and longer-lived than the + // scoped embed token) consumed here on the embedder side only — it must never + // appear in the iframe's own location, where app-authored code could read it. + // Captured once (not reactively): the embedder mirrors the app's hash/query + // back onto this page's URL, and re-deriving the src from it would reload the + // app on its every navigation. + const viewerUrl = `${base}/public/${workspace}/${parsedSecret.secret}${page.url.search}${page.url.hash}` + + let refresh: (() => void) | undefined + + // Embedder side: validate access (using the main session cookie or the shared + // JWT) and mint a scoped embed token for the opaque iframe (WIN-2006). + async function fetchEmbedToken(): Promise<{ token?: string }> { + if (parsedSecret.jwt) { + OpenAPI.TOKEN = 'jwt_ext_' + parsedSecret.jwt + } + const headers: Record = {} + if (typeof OpenAPI.TOKEN === 'string' && OpenAPI.TOKEN) { + headers['Authorization'] = `Bearer ${OpenAPI.TOKEN}` + } + const res = await fetch( + `${OpenAPI.BASE}/w/${workspace}/apps_u/embed_token/${parsedSecret.secret}`, + { headers } + ) + if (!res.ok) { + const err: any = new Error('Failed to fetch embed token') + err.status = res.status + throw err + } + return await res.json() + } + + // Viewer side: load the app + user using the embed token handed to the iframe. async function loadApp() { + try { + userStore.set(await getUserExt(workspace)) + } catch (e) { + console.warn('Anonymous user') + } try { app = await AppService.getPublicAppBySecret({ - workspace: page.params.workspace ?? '', + workspace, path: parsedSecret.secret }) noPermission = false notExists = false } catch (e) { if (e.status == 401) { - noPermission = true + // Embed token missing/expired — ask the embedder for a fresh one. + refresh?.() } else { notExists = true } @@ -47,42 +87,25 @@ if (BROWSER) { setLicense() - loadAll() - } - - function loadAll() { - console.log('loadAll') - loadUser().then(() => { - loadApp() - }) - } - - async function loadUser() { - if (parsedSecret.jwt) { - const token = 'jwt_ext_' + parsedSecret.jwt - OpenAPI.TOKEN = token - setContext<{ token?: string }>('AuthToken', { token }) - jwtError = false - } - try { - userStore.set(await getUserExt(page.params.workspace ?? '')) - if (!$userStore && parsedSecret.jwt) { - jwtError = true - sendUserToast('Could not authentify user with jwt token', true) - } - } catch (e) { - console.warn('Anonymous user') - } } - { - loadAll() + { + refresh = requestTokenRefresh + loadApp() }} -> +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + From 8dea38383f884f59b2956c39f1424005a21265bd Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 10:13:29 +0200 Subject: [PATCH 023/117] fix: prevent silent audit-partition outage via monitor watchdog + alert (#9729) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The monitor loop runs ~25 periodic tasks under a single join!, so any one stuck on a non-DB await (statement_timeout only bounds DB statements) freezes the whole loop indefinitely — silently halting audit-partition creation. Once the missing partition's date is reached, audit inserts fail; because login writes its audit row in the same transaction, that poisons the login tx and locks every user out. - Wrap monitor_db in a 600s timeout (> statement_timeout) so a stuck task can no longer freeze the loop; report a critical error and continue on timeout. - After creating partitions, verify the lookahead window is actually covered and raise a critical alert naming any missing partitions, turning a silent latent outage into an early page. Co-authored-by: Claude Opus 4.8 (1M context) --- backend/src/main.rs | 50 +++++++++++++++++++------ backend/src/monitor.rs | 85 ++++++++++++++++++++++++++++++------------ 2 files changed, 100 insertions(+), 35 deletions(-) diff --git a/backend/src/main.rs b/backend/src/main.rs index 57cc02598a..94b8e2f4f6 100644 --- a/backend/src/main.rs +++ b/backend/src/main.rs @@ -1446,19 +1446,45 @@ Windmill Community Edition {GIT_VERSION} } else { None }; - monitor_db( - &conn, - &base_internal_url, - server_mode, - worker_mode, - false, - tx.clone(), - Some(MonitorIteration { - rd_shift, - iter: monitor_iteration, - }), + // Hard cap on a single monitor pass. monitor_db runs all its + // periodic tasks under one join!, so a single task stuck on a + // non-DB await (statement_timeout only bounds DB statements) + // would otherwise freeze the whole loop indefinitely — silently + // stopping critical maintenance like audit-partition creation. + // Larger than statement_timeout (5min) so a slow-but-progressing + // statement is never killed prematurely. + const MONITOR_DB_TIMEOUT: Duration = Duration::from_secs(600); + let monitor_timed_out = tokio::time::timeout( + MONITOR_DB_TIMEOUT, + monitor_db( + &conn, + &base_internal_url, + server_mode, + worker_mode, + false, + tx.clone(), + Some(MonitorIteration { + rd_shift, + iter: monitor_iteration, + }), + ), ) - .await; + .await + .is_err(); + if monitor_timed_out { + windmill_common::utils::report_critical_error( + format!( + "monitor task did not finish within {}s and was aborted; \ + a background maintenance task is likely stuck. \ + Continuing to the next iteration.", + MONITOR_DB_TIMEOUT.as_secs() + ), + db.clone(), + None, + None, + ) + .await; + } monitor_iteration += 1; if let Some(handle) = warn_handle { handle.abort(); diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index 728d445456..02e3c12bd5 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -4436,11 +4436,19 @@ async fn audit_log_retention_days() -> i64 { } } +/// Number of days ahead (including today) for which an audit partition must +/// always exist. A missing partition in this window means audit inserts fail +/// once that date is reached — and because some callers (notably login) write +/// the audit row in the same transaction as their own work, that failure +/// poisons the whole transaction, so a missing partition is a hard outage, not +/// just a dropped audit row. +const AUDIT_PARTITION_LOOKAHEAD_DAYS: i64 = 3; + async fn manage_audit_partitions(db: &DB, retention_days: i64) { let today = chrono::Utc::now().date_naive(); - // Create partitions for today and the next 3 days - for days_ahead in 0..=3i64 { + // Create partitions for today and the next few days + for days_ahead in 0..=AUDIT_PARTITION_LOOKAHEAD_DAYS { let date = today + chrono::Duration::days(days_ahead); let next_date = date + chrono::Duration::days(1); let partition_name = format!("audit_{}", date.format("%Y%m%d")); @@ -4456,9 +4464,6 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) { } } - // Drop expired partitions - let cutoff_date = today - chrono::Duration::days(retention_days); - let partitions = sqlx::query_scalar::<_, String>( "SELECT c.relname::text \ FROM pg_inherits i \ @@ -4468,28 +4473,62 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) { .fetch_all(db) .await; - match partitions { - Ok(partitions) => { - for partition_name in partitions { - if let Some(date_str) = partition_name.strip_prefix("audit_") { - if let Ok(date) = chrono::NaiveDate::parse_from_str(date_str, "%Y%m%d") { - if date < cutoff_date { - let quoted_name = - format!("\"{}\"", partition_name.replace('"', "\"\"")); - let sql = format!("DROP TABLE IF EXISTS {quoted_name}"); - match sqlx::query(&sql).execute(db).await { - Ok(_) => tracing::info!( - "Dropped expired audit partition {partition_name}" - ), - Err(e) => tracing::error!( - "Error dropping audit partition {partition_name}: {e:?}" - ), - } + let partitions = match partitions { + Ok(partitions) => partitions, + Err(e) => { + tracing::error!("Error listing audit partitions: {e:?}"); + return; + } + }; + + // Verify the lookahead window is actually covered. If a create above failed + // (or this loop has not run for several days), alert loudly instead of + // letting it surface days later as failed audit inserts and broken logins. + let existing: std::collections::HashSet<&str> = partitions.iter().map(|s| s.as_str()).collect(); + let missing: Vec = (0..=AUDIT_PARTITION_LOOKAHEAD_DAYS) + .map(|days_ahead| { + format!( + "audit_{}", + (today + chrono::Duration::days(days_ahead)).format("%Y%m%d") + ) + }) + .filter(|name| !existing.contains(name.as_str())) + .collect(); + if !missing.is_empty() { + report_critical_error( + format!( + "Audit log partitions missing after maintenance run: {}. \ + Audit inserts will fail once these dates are reached, which also \ + breaks logins (the login audit row shares the login transaction). \ + Check for earlier 'Error creating audit partition' logs and verify \ + the audit-partition maintenance loop is still running.", + missing.join(", ") + ), + db.clone(), + None, + None, + ) + .await; + } + + // Drop expired partitions + let cutoff_date = today - chrono::Duration::days(retention_days); + for partition_name in &partitions { + if let Some(date_str) = partition_name.strip_prefix("audit_") { + if let Ok(date) = chrono::NaiveDate::parse_from_str(date_str, "%Y%m%d") { + if date < cutoff_date { + let quoted_name = format!("\"{}\"", partition_name.replace('"', "\"\"")); + let sql = format!("DROP TABLE IF EXISTS {quoted_name}"); + match sqlx::query(&sql).execute(db).await { + Ok(_) => { + tracing::info!("Dropped expired audit partition {partition_name}") } + Err(e) => tracing::error!( + "Error dropping audit partition {partition_name}: {e:?}" + ), } } } } - Err(e) => tracing::error!("Error listing audit partitions: {e:?}"), } } From 6d9486510933af9109f52011d93b13847dbdbb39 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 10:14:40 +0200 Subject: [PATCH 024/117] fix: optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries (#9727) The job_perms/job_result_stream_v2 orphan cleanups in monitor_db used `NOT IN` anti-joins, `RETURNING job_id` + `fetch_all` (loading every deleted UUID into memory) and no batch limit. On high-throughput instances these tables can accumulate tens of millions of orphaned rows, so a single execution ran for ~298s; because monitor_db awaits each iteration, the cleanup ran effectively continuously, saturating DB I/O and starving audit partition creation. Rewrite both deletes as bounded `NOT EXISTS` anti-joins selecting `ctid` with a LIMIT 100000, executed via `.execute()` (using rows_affected instead of fetch_all). Each run is now fast and bounded, while the 30s monitor cadence is preserved so the tables keep draining promptly. Fixes WIN-2088 Co-authored-by: Claude Opus 4.8 (1M context) --- ...c8abbcca4705814a553a62a115f4c241359da.json | 12 ++++++ ...5383bc7fe142c515dac3edd47991839485e51.json | 20 --------- ...42b8ad59b9129f032c6b918c27426ab304f2b.json | 20 --------- ...1ae5b1ab2609ab93fca1aa3097e5c8f5ec05f.json | 12 ++++++ backend/src/monitor.rs | 42 +++++++++++-------- 5 files changed, 48 insertions(+), 58 deletions(-) create mode 100644 backend/.sqlx/query-0bf9e39caa84ca689e55022de1cc8abbcca4705814a553a62a115f4c241359da.json delete mode 100644 backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json delete mode 100644 backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json create mode 100644 backend/.sqlx/query-d026dbde66c6921ed838a9b92d21ae5b1ab2609ab93fca1aa3097e5c8f5ec05f.json diff --git a/backend/.sqlx/query-0bf9e39caa84ca689e55022de1cc8abbcca4705814a553a62a115f4c241359da.json b/backend/.sqlx/query-0bf9e39caa84ca689e55022de1cc8abbcca4705814a553a62a115f4c241359da.json new file mode 100644 index 0000000000..d5345f38bf --- /dev/null +++ b/backend/.sqlx/query-0bf9e39caa84ca689e55022de1cc8abbcca4705814a553a62a115f4c241359da.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM job_perms\n WHERE ctid IN (\n SELECT jp.ctid FROM job_perms jp\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)\n LIMIT 100000\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "0bf9e39caa84ca689e55022de1cc8abbcca4705814a553a62a115f4c241359da" +} diff --git a/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json b/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json deleted file mode 100644 index 2f62420f87..0000000000 --- a/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM job_result_stream_v2\n WHERE job_id NOT IN (SELECT id FROM v2_job_queue)\n AND job_id NOT IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at > NOW() - INTERVAL '60 seconds'\n )\n RETURNING job_id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "job_id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51" -} diff --git a/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json b/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json deleted file mode 100644 index a68387f905..0000000000 --- a/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM job_perms\nWHERE job_id NOT IN (SELECT id FROM v2_job_queue)\nRETURNING job_id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "job_id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b" -} diff --git a/backend/.sqlx/query-d026dbde66c6921ed838a9b92d21ae5b1ab2609ab93fca1aa3097e5c8f5ec05f.json b/backend/.sqlx/query-d026dbde66c6921ed838a9b92d21ae5b1ab2609ab93fca1aa3097e5c8f5ec05f.json new file mode 100644 index 0000000000..27f8117b7a --- /dev/null +++ b/backend/.sqlx/query-d026dbde66c6921ed838a9b92d21ae5b1ab2609ab93fca1aa3097e5c8f5ec05f.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM job_result_stream_v2\n WHERE ctid IN (\n SELECT jrs.ctid FROM job_result_stream_v2 jrs\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id)\n AND NOT EXISTS (\n SELECT 1 FROM v2_job_completed c\n WHERE c.id = jrs.job_id\n AND c.completed_at > NOW() - INTERVAL '60 seconds'\n )\n LIMIT 100000\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "d026dbde66c6921ed838a9b92d21ae5b1ab2609ab93fca1aa3097e5c8f5ec05f" +} diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index 02e3c12bd5..68b9d5c867 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -4369,16 +4369,20 @@ RETURNING key,job_id } async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> { - let result = sqlx::query_scalar!( + let result = sqlx::query!( "DELETE FROM job_perms -WHERE job_id NOT IN (SELECT id FROM v2_job_queue) -RETURNING job_id" + WHERE ctid IN ( + SELECT jp.ctid FROM job_perms jp + WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id) + LIMIT 100000 + )" ) - .fetch_all(db) + .execute(db) .await?; - if !result.is_empty() { - tracing::info!("Cleaned up {} orphaned job_perms rows", result.len()); + let count = result.rows_affected(); + if count > 0 { + tracing::info!("Cleaned up {count} orphaned job_perms rows"); } Ok(()) } @@ -4386,21 +4390,23 @@ RETURNING job_id" async fn cleanup_job_result_stream_orphaned_jobs(db: &DB) -> error::Result<()> { let result = sqlx::query!( "DELETE FROM job_result_stream_v2 - WHERE job_id NOT IN (SELECT id FROM v2_job_queue) - AND job_id NOT IN ( - SELECT id FROM v2_job_completed - WHERE completed_at > NOW() - INTERVAL '60 seconds' - ) - RETURNING job_id", + WHERE ctid IN ( + SELECT jrs.ctid FROM job_result_stream_v2 jrs + WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id) + AND NOT EXISTS ( + SELECT 1 FROM v2_job_completed c + WHERE c.id = jrs.job_id + AND c.completed_at > NOW() - INTERVAL '60 seconds' + ) + LIMIT 100000 + )", ) - .fetch_all(db) + .execute(db) .await?; - if result.len() > 0 { - tracing::info!( - "Cleaned up {} orphaned job_result_stream_v2 rows", - result.len() - ); + let count = result.rows_affected(); + if count > 0 { + tracing::info!("Cleaned up {count} orphaned job_result_stream_v2 rows"); } Ok(()) } From fa3596885bf2d7ee8859f295e820ee362c756911 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 11:13:04 +0200 Subject: [PATCH 025/117] fix: allow SQL args in managed // materialize scripts (#9733) Co-authored-by: Claude Opus 4.8 (1M context) --- backend/windmill-api-scripts/src/scripts.rs | 26 ++--------- .../windmill-worker/src/duckdb_executor.rs | 44 ++++++++++++++++++- 2 files changed, 46 insertions(+), 24 deletions(-) diff --git a/backend/windmill-api-scripts/src/scripts.rs b/backend/windmill-api-scripts/src/scripts.rs index ea54b83c31..7b53fb44c9 100644 --- a/backend/windmill-api-scripts/src/scripts.rs +++ b/backend/windmill-api-scripts/src/scripts.rs @@ -1284,28 +1284,10 @@ async fn create_script_internal<'c>( if let Err(e) = windmill_parser::sql_materialize::classify_wrap(&ns.content) { return Err(Error::BadRequest(e.message())); } - // Managed materialize strips line comments when it wraps the SELECT, - // so a `-- $name (TYPE)` declaration is lost while its `$name` - // reference survives in the embedded SELECT — it would run unbound. - // Managed materialize takes no SQL args (the partition is supplied by - // the engine, not bound). Reject declared args with a clear error. - if let Ok(sig) = windmill_parser_sql::parse_duckdb_sig(&ns.content) { - if !sig.args.is_empty() { - let names = sig - .args - .iter() - .map(|a| format!("${}", a.name)) - .collect::>() - .join(", "); - return Err(Error::BadRequest(format!( - "managed `// materialize` cannot take SQL arguments ({names}): wrapping your \ - SELECT drops the `-- $arg` declarations, so they would run unbound. The \ - partition is supplied by the engine — reference its value with the \ - `{{partition}}` token, or use `// materialize manual` to write the DDL (and \ - bind args) yourself." - ))); - } - } + // SQL args are supported: managed materialize strips line comments + // (including `-- $name (type)` declarations) when it wraps the SELECT, + // but the executor parses the signature from the un-wrapped script, so + // `$name` references in the SELECT stay bound at run time. } // `key=` (merge) and `append` are mutually exclusive reconciliation // strategies; append (INSERT-only) wins. Surface the conflict rather diff --git a/backend/windmill-worker/src/duckdb_executor.rs b/backend/windmill-worker/src/duckdb_executor.rs index 3b6a07f702..fc558b9c1c 100644 --- a/backend/windmill-worker/src/duckdb_executor.rs +++ b/backend/windmill-worker/src/duckdb_executor.rs @@ -255,6 +255,14 @@ pub async fn do_duckdb( } else { None }; + // Parse the signature from the ORIGINAL script: managed materialize wraps + // the trailing SELECT and strips line comments, which drops the + // `-- $name (type)` arg declarations while their `$name` references + // survive in the embedded SELECT. Parsing args here (pre-wrap) keeps them + // declared so they are still bound — and s3object args translated to + // `s3://` URIs — at run time. + let sig = parse_duckdb_sig(query)?.args; + let materialized_query; let query: &str = match &materialize { Some((Some(rewritten), _)) => { @@ -263,8 +271,6 @@ pub async fn do_duckdb( } _ => query, }; - - let sig = parse_duckdb_sig(query)?.args; let mut job_args = build_args_values(job, client, conn).await?; let reserved_variables = @@ -1106,6 +1112,40 @@ mod tests { ); } + // Managed `// materialize` may take SQL args (e.g. an s3object uploaded on + // the run form). The wrap strips line comments — including the + // `-- $name (type)` declarations — so the executor parses the signature from + // the original script (done above, before the rewrite) while the `$name` + // references survive inside the wrapped SELECT. This pins both halves of that + // contract so a regression that drops either is caught. + #[test] + fn materialize_preserves_sql_args() { + let script = "-- materialize ducklake://main/rows\n\ + -- $file (s3object)\n\ + SELECT * FROM read_json_auto($file)"; + + // The signature is recoverable from the original (un-wrapped) script. + let sig = parse_duckdb_sig(script).expect("sig parses").args; + let file_arg = sig + .iter() + .find(|a| a.name == "file") + .expect("`$file` declared"); + assert_eq!(file_arg.otyp.as_deref(), Some("s3object")); + + // The wrapped query still references `$file`, so the parsed sig binds it. + let (rewritten, _) = build_materialized_query(script, None) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("managed mode rewrites the query"); + assert!( + rewritten.contains("$file"), + "wrapped query must keep the `$file` reference, got:\n{rewritten}" + ); + // The declaration comment is gone (wrap strips line comments) — which is + // exactly why the sig must come from the original, not the rewrite. + assert!(!rewritten.contains("-- $file")); + } + // Tests for parse_attach_db_resource function #[test] fn test_parse_attach_db_resource_postgres_res_prefix() { From 75bafabeeec76cf6da33eef41f588e37071df011 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 11:23:50 +0200 Subject: [PATCH 026/117] perf(monitor): hash active-root exclusion in retention delete (WIN-2088) (#9732) * perf(monitor): hash active-root exclusion in retention delete The expired-job retention delete (delete_expired_jobs_batch) excluded jobs belonging to still-active root flows with `COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)`. That ScalarArrayOp is evaluated per candidate row as a linear scan of $3, so cost grows with the number of active root jobs. Express the exclusion as `NOT IN (SELECT u FROM unnest($3) u WHERE u IS NOT NULL)` instead. The subquery form lets Postgres build a one-time hashed SubPlan and apply it as a filter on the ordered index scan, giving O(1) membership per candidate while preserving the `ORDER BY completed_at ASC LIMIT` early termination. The `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids). Measured on a 2M-row synthetic v2_job_completed (batch LIMIT 20000, 5-run min): active roots | != ALL (before) | NOT IN hashed (after) -------------|-----------------|---------------------- 100 | 108 ms | 104 ms 1000 | 168 ms | 105 ms 10000 | 719 ms | 131 ms Both forms return identical row sets (verified via EXCEPT, 0 diff). Neutral at small active-root counts, ~5.5x faster when many flows are active. Relates to WIN-2088 Co-Authored-By: Claude Opus 4.8 (1M context) * perf(monitor): apply hashed active-root exclusion to log_cleanup mirror windmill-api-settings/log_cleanup.rs::delete_expired_jobs_batch carries a byte-identical copy of the retention delete and shared its prepared-query cache. Updating only monitor.rs removed that shared cache entry and broke the SQLX_OFFLINE build of the mirror. Apply the same NOT IN (hashed SubPlan) rewrite so both copies converge on one cached query and the mirror gets the same speedup. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- ...6b03366b2bd117ceb6bc152d2abcd45850ff6aecff9.json} | 4 ++-- backend/src/monitor.rs | 12 ++++++++++-- backend/windmill-api-settings/src/log_cleanup.rs | 6 +++++- 3 files changed, 17 insertions(+), 5 deletions(-) rename backend/.sqlx/{query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json => query-fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9.json} (65%) diff --git a/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json b/backend/.sqlx/query-fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9.json similarity index 65% rename from backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json rename to backend/.sqlx/query-fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9.json index f63afbc986..62ec17f0f8 100644 --- a/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json +++ b/backend/.sqlx/query-fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id", + "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id", "describe": { "columns": [ { @@ -20,5 +20,5 @@ false ] }, - "hash": "45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f" + "hash": "fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9" } diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index 68b9d5c867..9827040358 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -1532,14 +1532,22 @@ async fn delete_expired_jobs_batch( .await?; // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas - // ORDER BY completed_at ensures we delete oldest jobs first + // ORDER BY completed_at ensures we delete oldest jobs first. + // Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than + // `!= ALL($3)`: the subquery form lets the planner build a one-time hashed + // SubPlan and apply it as a filter on the ordered index scan, giving O(1) + // membership per candidate instead of a per-row linear array scan (which + // degrades sharply when many root jobs are active). The `u IS NOT NULL` guard + // sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids). let deleted_jobs: Vec = sqlx::query_scalar!( "DELETE FROM v2_job_completed WHERE id IN ( SELECT jc.id FROM v2_job_completed jc LEFT JOIN v2_job j ON j.id = jc.id WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval - AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3) + AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( + SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL + ) ORDER BY jc.completed_at ASC LIMIT $2 FOR UPDATE OF jc SKIP LOCKED diff --git a/backend/windmill-api-settings/src/log_cleanup.rs b/backend/windmill-api-settings/src/log_cleanup.rs index 762a40e69d..2264e27037 100644 --- a/backend/windmill-api-settings/src/log_cleanup.rs +++ b/backend/windmill-api-settings/src/log_cleanup.rs @@ -395,13 +395,17 @@ async fn delete_expired_jobs_batch( .fetch_all(&mut *tx) .await?; + // Active-root exclusion via NOT IN (hashed SubPlan) instead of `!= ALL($3)`; + // see backend/src/monitor.rs::delete_expired_jobs_batch for the rationale. let deleted_jobs: Vec = sqlx::query_scalar!( "DELETE FROM v2_job_completed WHERE id IN ( SELECT jc.id FROM v2_job_completed jc LEFT JOIN v2_job j ON j.id = jc.id WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval - AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3) + AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( + SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL + ) ORDER BY jc.completed_at ASC LIMIT $2 FOR UPDATE OF jc SKIP LOCKED From 31d9215e5a61f19662cc87be8147007e1d47ebb6 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 11:24:36 +0200 Subject: [PATCH 027/117] fix: bound orphan-cleanup drain rate with capped multi-batch loop (#9730) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to #9727. The orphan cleanups (cleanup_job_perms_orphaned and cleanup_job_result_stream_orphaned_jobs) deleted at most one 100k batch per monitor iteration. Each statement stays short and lock-light, but a single batch per ~30s cycle caps the drain rate at ~100k/30s, so a large one-time backlog (tens of millions of rows) takes ~hours to clear. Loop the batched delete up to ORPHAN_CLEANUP_MAX_BATCHES (10) times per cycle, stopping early once a batch deletes fewer than ORPHAN_CLEANUP_BATCH_SIZE rows. Each DELETE remains bounded (≤100k, short locks, no long single statement), while per-cycle throughput rises to ~1M rows so backlogs drain ~10x faster. The per-cycle cap keeps monitor_db responsive. Co-authored-by: Claude Opus 4.8 (1M context) --- ...c8abbcca4705814a553a62a115f4c241359da.json | 12 --- ...9201c2bffb0f0f7419b0b598b2786bdb326ab.json | 12 +++ ...1ae5b1ab2609ab93fca1aa3097e5c8f5ec05f.json | 12 --- ...84daaed616dc1ac609a6ca81bcd446e4dc230.json | 12 +++ backend/src/monitor.rs | 84 ++++++++++++------- 5 files changed, 76 insertions(+), 56 deletions(-) delete mode 100644 backend/.sqlx/query-0bf9e39caa84ca689e55022de1cc8abbcca4705814a553a62a115f4c241359da.json create mode 100644 backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json delete mode 100644 backend/.sqlx/query-d026dbde66c6921ed838a9b92d21ae5b1ab2609ab93fca1aa3097e5c8f5ec05f.json create mode 100644 backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json diff --git a/backend/.sqlx/query-0bf9e39caa84ca689e55022de1cc8abbcca4705814a553a62a115f4c241359da.json b/backend/.sqlx/query-0bf9e39caa84ca689e55022de1cc8abbcca4705814a553a62a115f4c241359da.json deleted file mode 100644 index d5345f38bf..0000000000 --- a/backend/.sqlx/query-0bf9e39caa84ca689e55022de1cc8abbcca4705814a553a62a115f4c241359da.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM job_perms\n WHERE ctid IN (\n SELECT jp.ctid FROM job_perms jp\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)\n LIMIT 100000\n )", - "describe": { - "columns": [], - "parameters": { - "Left": [] - }, - "nullable": [] - }, - "hash": "0bf9e39caa84ca689e55022de1cc8abbcca4705814a553a62a115f4c241359da" -} diff --git a/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json b/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json new file mode 100644 index 0000000000..36ea3d7fb2 --- /dev/null +++ b/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM job_perms\n WHERE ctid IN (\n SELECT jp.ctid FROM job_perms jp\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)\n LIMIT 100000\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab" +} diff --git a/backend/.sqlx/query-d026dbde66c6921ed838a9b92d21ae5b1ab2609ab93fca1aa3097e5c8f5ec05f.json b/backend/.sqlx/query-d026dbde66c6921ed838a9b92d21ae5b1ab2609ab93fca1aa3097e5c8f5ec05f.json deleted file mode 100644 index 27f8117b7a..0000000000 --- a/backend/.sqlx/query-d026dbde66c6921ed838a9b92d21ae5b1ab2609ab93fca1aa3097e5c8f5ec05f.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM job_result_stream_v2\n WHERE ctid IN (\n SELECT jrs.ctid FROM job_result_stream_v2 jrs\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id)\n AND NOT EXISTS (\n SELECT 1 FROM v2_job_completed c\n WHERE c.id = jrs.job_id\n AND c.completed_at > NOW() - INTERVAL '60 seconds'\n )\n LIMIT 100000\n )", - "describe": { - "columns": [], - "parameters": { - "Left": [] - }, - "nullable": [] - }, - "hash": "d026dbde66c6921ed838a9b92d21ae5b1ab2609ab93fca1aa3097e5c8f5ec05f" -} diff --git a/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json b/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json new file mode 100644 index 0000000000..4364da5dd0 --- /dev/null +++ b/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM job_result_stream_v2\n WHERE ctid IN (\n SELECT jrs.ctid FROM job_result_stream_v2 jrs\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id)\n AND NOT EXISTS (\n SELECT 1 FROM v2_job_completed c\n WHERE c.id = jrs.job_id\n AND c.completed_at > NOW() - INTERVAL '60 seconds'\n )\n LIMIT 100000\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230" +} diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index 9827040358..f04646120e 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -4376,45 +4376,65 @@ RETURNING key,job_id Ok(()) } -async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> { - let result = sqlx::query!( - "DELETE FROM job_perms - WHERE ctid IN ( - SELECT jp.ctid FROM job_perms jp - WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id) - LIMIT 100000 - )" - ) - .execute(db) - .await?; +// Per-statement cap keeps each delete short and lock-light; the per-cycle batch +// cap bounds total work per monitor iteration so monitor_db stays responsive. +// A large backlog drains across several iterations rather than one long delete. +const ORPHAN_CLEANUP_BATCH_SIZE: u64 = 100_000; +const ORPHAN_CLEANUP_MAX_BATCHES: usize = 10; - let count = result.rows_affected(); - if count > 0 { - tracing::info!("Cleaned up {count} orphaned job_perms rows"); +async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> { + let mut total: u64 = 0; + for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES { + let count = sqlx::query!( + "DELETE FROM job_perms + WHERE ctid IN ( + SELECT jp.ctid FROM job_perms jp + WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id) + LIMIT 100000 + )" + ) + .execute(db) + .await? + .rows_affected(); + total += count; + if count < ORPHAN_CLEANUP_BATCH_SIZE { + break; + } + } + + if total > 0 { + tracing::info!("Cleaned up {total} orphaned job_perms rows"); } Ok(()) } async fn cleanup_job_result_stream_orphaned_jobs(db: &DB) -> error::Result<()> { - let result = sqlx::query!( - "DELETE FROM job_result_stream_v2 - WHERE ctid IN ( - SELECT jrs.ctid FROM job_result_stream_v2 jrs - WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id) - AND NOT EXISTS ( - SELECT 1 FROM v2_job_completed c - WHERE c.id = jrs.job_id - AND c.completed_at > NOW() - INTERVAL '60 seconds' - ) - LIMIT 100000 - )", - ) - .execute(db) - .await?; + let mut total: u64 = 0; + for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES { + let count = sqlx::query!( + "DELETE FROM job_result_stream_v2 + WHERE ctid IN ( + SELECT jrs.ctid FROM job_result_stream_v2 jrs + WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id) + AND NOT EXISTS ( + SELECT 1 FROM v2_job_completed c + WHERE c.id = jrs.job_id + AND c.completed_at > NOW() - INTERVAL '60 seconds' + ) + LIMIT 100000 + )", + ) + .execute(db) + .await? + .rows_affected(); + total += count; + if count < ORPHAN_CLEANUP_BATCH_SIZE { + break; + } + } - let count = result.rows_affected(); - if count > 0 { - tracing::info!("Cleaned up {count} orphaned job_result_stream_v2 rows"); + if total > 0 { + tracing::info!("Cleaned up {total} orphaned job_result_stream_v2 rows"); } Ok(()) } From c644311eca4bcaf4b68058cf1d5d79d4078aee1a Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 11:25:23 +0200 Subject: [PATCH 028/117] fix(ext-jwt): reject external JWT auth for non-existent workspaces (#9723) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(ext-jwt): reject external JWT auth for non-existent workspaces External JWTs are validated (not generated) on our side and never revoked by us. The usage-tracking upsert into unique_ext_jwt_token ran unconditionally, so a token carrying a workspace_id whose workspace no longer exists kept refreshing its row on every presentation — surfacing as a "new token" in the superadmin external-JWT view. Gate jwt_ext_auth on the requested workspace existing (EE companion). When it does not, auth fails (token is unusable) and no usage row is written. The check is existence-only and intentionally ignores the soft-delete flag, so deleted-then-restored workspaces keep working. Bumps ee-repo-ref.txt to the EE companion commit. Co-Authored-By: Claude Opus 4.8 (1M context) * perf(ext-jwt): cache workspace-existence lookups in jwt_ext_auth Bumps ee-repo-ref.txt to the EE companion commit that caches the workspace-existence check added in the previous commit, so a token aimed at a missing workspace no longer hits the DB on every request (auth failures aren't cached upstream). Co-Authored-By: Claude Opus 4.8 (1M context) * chore: update ee-repo-ref to ac1f6f666f36141cb6ba6f8eaa614821a90464ad This commit updates the EE repository reference after PR #626 was merged in windmill-ee-private. Previous ee-repo-ref: e23fa03ec16909c127e8ecf0855595911c29512d New ee-repo-ref: ac1f6f666f36141cb6ba6f8eaa614821a90464ad Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: windmill-internal-app[bot] --- backend/ee-repo-ref.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index 40ad91749d..a4fea858de 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -b0cb761bf9852974e571b2978032d310cc998517 +ac1f6f666f36141cb6ba6f8eaa614821a90464ad From 723a65920fe75a471f9025fe2e16adb284037660 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 12:10:15 +0200 Subject: [PATCH 029/117] chore(main): release 1.737.0 (#9728) * chore(main): release 1.737.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> --- CHANGELOG.md | 21 +++ backend/Cargo.lock | 156 +++++++++--------- backend/Cargo.toml | 4 +- .../parsers/windmill-parser-wasm/Cargo.lock | 48 +++--- .../parsers/windmill-parser-wasm/Cargo.toml | 2 +- backend/windmill-api/openapi.yaml | 2 +- benchmarks/lib.ts | 2 +- cli/src/core/constants.ts | 2 +- frontend/package-lock.json | 4 +- frontend/package.json | 2 +- lsp/Pipfile | 2 +- openflow.openapi.yaml | 2 +- .../WindmillClient/WindmillClient.psd1 | 2 +- python-client/wmill/pyproject.toml | 2 +- typescript-client/jsr.json | 2 +- typescript-client/package.json | 2 +- version.txt | 2 +- 17 files changed, 139 insertions(+), 118 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 47e91fa1a0..df77488710 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,26 @@ # Changelog +## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23) + + +### Features + +* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904)) + + +### Bug Fixes + +* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911)) +* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6)) +* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a)) +* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39)) +* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd)) + + +### Performance Improvements + +* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011)) + ## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23) diff --git a/backend/Cargo.lock b/backend/Cargo.lock index de575d4aa1..7d47bbcdeb 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -13735,7 +13735,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-nats", @@ -13817,7 +13817,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.736.0" +version = "1.737.0" dependencies = [ "async-stream", "async-trait", @@ -13850,7 +13850,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -13863,7 +13863,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "argon2", @@ -14001,7 +14001,7 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14024,7 +14024,7 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14037,7 +14037,7 @@ dependencies = [ [[package]] name = "windmill-api-auth" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14063,7 +14063,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.736.0" +version = "1.737.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14073,7 +14073,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14090,7 +14090,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14112,7 +14112,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14135,7 +14135,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14151,7 +14151,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14172,7 +14172,7 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14193,7 +14193,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14207,7 +14207,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-nats", @@ -14242,7 +14242,7 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14267,7 +14267,7 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "flate2", @@ -14285,7 +14285,7 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14307,7 +14307,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14327,7 +14327,7 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14364,7 +14364,7 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14392,7 +14392,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.736.0" +version = "1.737.0" dependencies = [ "lazy_static", "serde", @@ -14404,7 +14404,7 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.736.0" +version = "1.737.0" dependencies = [ "argon2", "axum 0.8.9", @@ -14429,7 +14429,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14443,7 +14443,7 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.736.0" +version = "1.737.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14476,7 +14476,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.736.0" +version = "1.737.0" dependencies = [ "chrono", "lazy_static", @@ -14490,7 +14490,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14509,7 +14509,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.736.0" +version = "1.737.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -14611,7 +14611,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.736.0" +version = "1.737.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -14630,7 +14630,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.736.0" +version = "1.737.0" dependencies = [ "regex", "serde", @@ -14645,7 +14645,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -14669,7 +14669,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "futures", @@ -14686,7 +14686,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.736.0" +version = "1.737.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -14702,7 +14702,7 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -14723,7 +14723,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -14754,7 +14754,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "arc-swap", @@ -14779,7 +14779,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-stream", @@ -14813,7 +14813,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "futures", @@ -14831,7 +14831,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.736.0" +version = "1.737.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -14840,7 +14840,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "lazy_static", @@ -14852,7 +14852,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde_json", @@ -14864,7 +14864,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "gosyn", @@ -14876,7 +14876,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "lazy_static", @@ -14888,7 +14888,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde_json", @@ -14900,7 +14900,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "nu-parser", @@ -14911,7 +14911,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14922,7 +14922,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14934,7 +14934,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "rustpython-ast", @@ -14945,7 +14945,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-recursion", @@ -14967,7 +14967,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde_json", @@ -14979,7 +14979,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "lazy_static", @@ -14993,7 +14993,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15010,7 +15010,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "lazy_static", @@ -15023,7 +15023,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde", @@ -15035,7 +15035,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "lazy_static", @@ -15053,7 +15053,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15069,7 +15069,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15085,7 +15085,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde", @@ -15096,7 +15096,7 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-recursion", @@ -15134,7 +15134,7 @@ dependencies = [ [[package]] name = "windmill-runtime-nativets" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "const_format", @@ -15173,7 +15173,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.736.0" +version = "1.737.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15184,7 +15184,7 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-recursion", @@ -15216,7 +15216,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15240,7 +15240,7 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15273,7 +15273,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15306,7 +15306,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15326,7 +15326,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15360,7 +15360,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15396,7 +15396,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15419,7 +15419,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15443,7 +15443,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-nats", @@ -15467,7 +15467,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15502,7 +15502,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15530,7 +15530,7 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-trait", @@ -15555,7 +15555,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "bitflags 2.13.0", @@ -15574,7 +15574,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-once-cell", @@ -15684,7 +15684,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.736.0" +version = "1.737.0" dependencies = [ "bytes", "futures", diff --git a/backend/Cargo.toml b/backend/Cargo.toml index db53f33726..bce80a8c0d 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.736.0" +version = "1.737.0" authors.workspace = true edition.workspace = true @@ -87,7 +87,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.736.0" +version = "1.737.0" authors = ["Ruben Fiszel "] edition = "2021" diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index d7f194b2d1..9d20e86a8a 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.736.0" +version = "1.737.0" dependencies = [ "aho-corasick", "anyhow", @@ -6272,7 +6272,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.736.0" +version = "1.737.0" dependencies = [ "proc-macro2", "quote", @@ -6284,7 +6284,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.736.0" +version = "1.737.0" dependencies = [ "convert_case", "serde", @@ -6293,7 +6293,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "lazy_static", @@ -6305,7 +6305,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde_json", @@ -6317,7 +6317,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "gosyn", @@ -6329,7 +6329,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "lazy_static", @@ -6341,7 +6341,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde_json", @@ -6353,7 +6353,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "nu-parser", @@ -6364,7 +6364,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6375,7 +6375,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6387,7 +6387,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6398,7 +6398,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "async-recursion", @@ -6420,7 +6420,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde_json", @@ -6432,7 +6432,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "lazy_static", @@ -6446,7 +6446,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "convert_case", @@ -6463,7 +6463,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "lazy_static", @@ -6476,7 +6476,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde", @@ -6488,7 +6488,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "lazy_static", @@ -6506,7 +6506,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6522,7 +6522,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6538,7 +6538,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6570,7 +6570,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "serde", @@ -6581,7 +6581,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.736.0" +version = "1.737.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index a17cd25e9c..046249b27e 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.736.0" +version = "1.737.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index 102efc765d..f200cdb902 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: - version: 1.736.0 + version: 1.737.0 title: Windmill API contact: diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index 9c39887880..2d928acce8 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.736.0"; +export const VERSION = "v1.737.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ diff --git a/cli/src/core/constants.ts b/cli/src/core/constants.ts index d80f1b123d..7402929aee 100644 --- a/cli/src/core/constants.ts +++ b/cli/src/core/constants.ts @@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork"; // (e.g. utils.ts) can read it without importing main.ts and creating a circular // dependency (main → workspace → utils → main) that triggers a TDZ. // Re-exported from main.ts for backwards compatibility. -export const VERSION = "1.736.0"; +export const VERSION = "1.737.0"; diff --git a/frontend/package-lock.json b/frontend/package-lock.json index c1ab7d0fc3..8302448dfb 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "@windmill-labs/components", - "version": "1.736.0", + "version": "1.737.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@windmill-labs/components", - "version": "1.736.0", + "version": "1.737.0", "hasInstallScript": true, "license": "AGPL-3.0", "dependencies": { diff --git a/frontend/package.json b/frontend/package.json index 94a27efcad..b080ab058a 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "@windmill-labs/components", - "version": "1.736.0", + "version": "1.737.0", "scripts": { "dev": "vite dev", "dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev", diff --git a/lsp/Pipfile b/lsp/Pipfile index d6dde9126a..eb25523cf7 100644 --- a/lsp/Pipfile +++ b/lsp/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -wmill = ">=1.736.0" +wmill = ">=1.737.0" sendgrid = "*" mysql-connector-python = "*" pymongo = "*" diff --git a/openflow.openapi.yaml b/openflow.openapi.yaml index 7fb41c38e8..c64c32a1be 100644 --- a/openflow.openapi.yaml +++ b/openflow.openapi.yaml @@ -1,7 +1,7 @@ openapi: '3.0.3' info: - version: 1.736.0 + version: 1.737.0 title: OpenFlow Spec contact: name: Ruben Fiszel diff --git a/powershell-client/WindmillClient/WindmillClient.psd1 b/powershell-client/WindmillClient/WindmillClient.psd1 index 9583aa5711..75288a8267 100644 --- a/powershell-client/WindmillClient/WindmillClient.psd1 +++ b/powershell-client/WindmillClient/WindmillClient.psd1 @@ -12,7 +12,7 @@ RootModule = 'WindmillClient.psm1' # Version number of this module. - ModuleVersion = '1.736.0' + ModuleVersion = '1.737.0' # Supported PSEditions # CompatiblePSEditions = @() diff --git a/python-client/wmill/pyproject.toml b/python-client/wmill/pyproject.toml index bf1a825f18..abcb7296f7 100644 --- a/python-client/wmill/pyproject.toml +++ b/python-client/wmill/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "wmill" -version = "1.736.0" +version = "1.737.0" description = "A client library for accessing Windmill server wrapping the Windmill client API" license = "Apache-2.0" homepage = "https://windmill.dev" diff --git a/typescript-client/jsr.json b/typescript-client/jsr.json index 27b20dd666..1d7043d5f7 100644 --- a/typescript-client/jsr.json +++ b/typescript-client/jsr.json @@ -1,6 +1,6 @@ { "name": "@windmill/windmill", - "version": "1.736.0", + "version": "1.737.0", "exports": "./src/index.ts", "publish": { "exclude": ["!src", "./s3Types.ts", "./sqlUtils.ts", "./client.ts"] diff --git a/typescript-client/package.json b/typescript-client/package.json index 197d39772a..983c066936 100644 --- a/typescript-client/package.json +++ b/typescript-client/package.json @@ -1,7 +1,7 @@ { "name": "windmill-client", "description": "Windmill SDK client for browsers and Node.js", - "version": "1.736.0", + "version": "1.737.0", "author": "Ruben Fiszel", "license": "Apache 2.0", "homepage": "https://github.com/windmill-labs/windmill/tree/main/typescript-client#readme", diff --git a/version.txt b/version.txt index 20bc4b92da..63bd46762f 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.736.0 +1.737.0 From ba4b368706e95e22f346a10e5fe145b0795ac3f6 Mon Sep 17 00:00:00 2001 From: hugocasa Date: Tue, 23 Jun 2026 12:45:36 +0200 Subject: [PATCH 030/117] fix: prevent variable push from corrupting is_secret variables (#9705) * fix: prevent variable push from corrupting is_secret variables Co-Authored-By: Claude Opus 4.8 (1M context) * test(cli): unit-test looksLikeWorkspaceCiphertext shape detection Co-Authored-By: Claude Opus 4.8 (1M context) * fix(cli): scope is_secret downgrade to single-file push, not sync push Co-Authored-By: Claude Opus 4.8 (1M context) * fix(cli): warn when variable push stores a secret value as already-encrypted Co-Authored-By: Claude Opus 4.8 (1M context) * fix(cli): route workspace-resolution and auth diagnostics to stderr Co-Authored-By: Claude Opus 4.8 (1M context) * docs(cli): rephrase comments to describe current behavior, not history Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- backend/Cargo.lock | 2 + backend/windmill-store/Cargo.toml | 4 + backend/windmill-store/src/variables.rs | 102 +++++++++++++++++- cli/src/commands/variable/variable.ts | 78 +++++++++++++- cli/src/core/auth.ts | 10 +- cli/src/core/context.ts | 58 +++++----- cli/src/core/log.ts | 15 +++ cli/src/core/login.ts | 8 +- .../variable_ciphertext_shape_unit.test.ts | 42 ++++++++ cli/test/variable_resource_push.test.ts | 97 +++++++++++++++++ 10 files changed, 372 insertions(+), 44 deletions(-) create mode 100644 cli/test/variable_ciphertext_shape_unit.test.ts diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 7d47bbcdeb..834c636f0a 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -15189,12 +15189,14 @@ dependencies = [ "anyhow", "async-recursion", "axum 0.8.9", + "base64 0.22.1", "chrono", "futures", "hex", "http 1.4.2", "hyper 1.10.1", "lazy_static", + "magic-crypt", "quick_cache", "reqwest 0.13.1", "serde", diff --git a/backend/windmill-store/Cargo.toml b/backend/windmill-store/Cargo.toml index b3aca5e669..fdd82a7e5d 100644 --- a/backend/windmill-store/Cargo.toml +++ b/backend/windmill-store/Cargo.toml @@ -53,3 +53,7 @@ futures.workspace = true chrono.workspace = true reqwest.workspace = true anyhow.workspace = true +base64.workspace = true + +[dev-dependencies] +magic-crypt.workspace = true diff --git a/backend/windmill-store/src/variables.rs b/backend/windmill-store/src/variables.rs index d5a906392d..55047ec0c2 100644 --- a/backend/windmill-store/src/variables.rs +++ b/backend/windmill-store/src/variables.rs @@ -14,8 +14,8 @@ use windmill_common::db::DB; use windmill_common::workspaces::{check_deploy_rules, RuleCheckResult}; use crate::secret_backend_ext::{ - delete_secret_from_backend, get_secret_value, is_vault_stored_value, rename_vault_secret, - store_secret_value, + delete_secret_from_backend, get_secret_value, is_external_stored_value, is_vault_stored_value, + rename_vault_secret, store_secret_value, }; use windmill_common::utils::{escape_ilike_pattern, BulkDeleteRequest}; use windmill_common::webhook::{WebhookMessage, WebhookShared}; @@ -25,6 +25,7 @@ use axum::{ routing::{delete, get, post}, Json, Router, }; +use base64::{engine::general_purpose::STANDARD, Engine as _}; use futures::future::try_join_all; use hyper::StatusCode; use serde_json::Value; @@ -535,6 +536,35 @@ async fn check_path_conflict(db: &DB, w_id: &str, path: &str) -> Result<()> { return Ok(()); } +/// Reject a secret value flagged as already-encrypted (`already_encrypted=true`) +/// that is not actually workspace-key ciphertext — e.g. plaintext mistakenly +/// pushed as encrypted. Storing plaintext in the encrypted `value` column +/// silently bricks the variable: every later read fails to decrypt it. +/// +/// The check is purely structural and never decrypts, so it cannot act as a +/// decryption/padding oracle for a caller who can write but not read secrets. +/// `encrypt` (AES-256-CBC) always yields standard base64 decoding to a non-zero +/// multiple of the 16-byte block size; anything else cannot be our ciphertext. +/// Values stored by an external backend ($vault:/$aws_sm:/$azure_kv: markers) +/// are not workspace ciphertext and are passed through untouched. +fn validate_already_encrypted_secret(path: &str, value: &str) -> Result<()> { + if is_external_stored_value(value) { + return Ok(()); + } + let looks_like_ciphertext = STANDARD + .decode(value) + .map(|bytes| !bytes.is_empty() && bytes.len() % 16 == 0) + .unwrap_or(false); + if !looks_like_ciphertext { + return Err(Error::BadRequest(format!( + "Variable {path} was sent as already-encrypted (already_encrypted=true) but its \ + value is not valid workspace-encrypted ciphertext. To push a plaintext secret, \ + send it without already_encrypted (CLI: use --plain-secrets) so it gets encrypted." + ))); + } + Ok(()) +} + async fn create_variable( authed: ApiAuthed, Extension(db): Extension, @@ -585,6 +615,11 @@ async fn create_variable( // Use secret backend for encryption (supports both DB and Vault) store_secret_value(&db, &w_id, &variable.path, &plain).await? } else { + if variable.is_secret { + // already_encrypted == true: value is stored verbatim, so it must be + // ciphertext and not plaintext mislabeled as encrypted. + validate_already_encrypted_secret(&variable.path, &variable.value)?; + } variable.value }; @@ -1082,6 +1117,11 @@ async fn update_variable( // Store at target_path (new path if renaming, otherwise current path) store_secret_value(&db, &w_id, target_path, &plain).await? } else { + if is_secret { + // already_encrypted == true: value is stored verbatim, so it must + // be ciphertext and not plaintext mislabeled as encrypted. + validate_already_encrypted_secret(target_path, &nvalue)?; + } nvalue }; sqlb.set_str("value", &value); @@ -1513,3 +1553,61 @@ pub async fn get_value_internal<'a>( Ok(r) } + +#[cfg(test)] +mod tests { + use super::*; + use magic_crypt::MagicCryptTrait; + + #[test] + fn accepts_real_workspace_ciphertext() { + // The exact shape produced by `encrypt` (AES-256-CBC, base64). + let mc = magic_crypt::new_magic_crypt!("a-test-workspace-key", 256); + for plain in [ + "", + "original-secret", + "some: plaintext\n", + "a".repeat(500).as_str(), + ] { + let ciphertext = mc.encrypt_str_to_base64(plain); + assert!( + validate_already_encrypted_secret("f/x/cfg", &ciphertext).is_ok(), + "should accept genuine ciphertext for plaintext {plain:?}: {ciphertext}" + ); + } + } + + #[test] + fn rejects_plaintext_mislabeled_as_encrypted() { + // Plaintext mislabeled as encrypted: storing it verbatim would make the + // variable undecryptable on every read, so it must be rejected. + for plaintext in [ + "some: plaintext\n", + "original-secret", + "hunter2", + "{\"a\": 1}", + "not base64!!", + " leading-space", + ] { + assert!( + validate_already_encrypted_secret("f/x/cfg", plaintext).is_err(), + "should reject plaintext mislabeled as encrypted: {plaintext:?}" + ); + } + } + + #[test] + fn rejects_empty_and_non_block_aligned() { + // Valid base64 but not a whole number of AES blocks -> cannot be our ciphertext. + assert!(validate_already_encrypted_secret("p", "").is_err()); + assert!(validate_already_encrypted_secret("p", "dGVzdA==").is_err()); // "test" -> 4 bytes + } + + #[test] + fn passes_through_external_backend_markers() { + // External secret backends store $-prefixed markers, not workspace ciphertext. + for marker in ["$vault:f/x/cfg", "$aws_sm:f/x/cfg", "$azure_kv:f/x/cfg"] { + assert!(validate_already_encrypted_secret("f/x/cfg", marker).is_ok()); + } + } +} diff --git a/cli/src/commands/variable/variable.ts b/cli/src/commands/variable/variable.ts index dec35fc4f0..2d885c9dd5 100644 --- a/cli/src/commands/variable/variable.ts +++ b/cli/src/commands/variable/variable.ts @@ -99,6 +99,31 @@ export interface VariableFile { is_oauth?: boolean; } +/** + * Whether `value` has the structural shape of a workspace-encrypted secret + * (the form produced by `sync pull` without --plain-secrets), as opposed to a + * plaintext value a user authored by hand. + * + * Mirrors the server guard (windmill-store/src/variables.rs): workspace + * ciphertext (AES-256-CBC, base64) is standard base64 decoding to a non-zero + * multiple of the 16-byte block size. External secret-backend markers + * ($vault:/$aws_sm:/$azure_kv:) are stored verbatim too, so they count as + * already-encrypted. This is a shape check only — it never decrypts. + */ +export function looksLikeWorkspaceCiphertext(value: string): boolean { + if ( + value.startsWith("$vault:") || + value.startsWith("$aws_sm:") || + value.startsWith("$azure_kv:") + ) { + return true; + } + if (value.length === 0 || value.length % 4 !== 0) return false; + if (!/^[A-Za-z0-9+/]+={0,2}$/.test(value)) return false; + const decodedLen = Buffer.from(value, "base64").length; + return decodedLen > 0 && decodedLen % 16 === 0; +} + export async function pushVariable( workspace: string, remotePath: string, @@ -106,6 +131,11 @@ export async function pushVariable( localVariable: VariableFile, plainSecrets: boolean, wsSpecific?: boolean, + // Whether a secret->non-secret downgrade may be applied. Only an authoritative + // single-file `variable push` sets this. Bulk `sync push` leaves it false: a + // pulled secret's spec value is ciphertext, and demoting it would store that + // ciphertext verbatim as a visible non-secret value. + allowSecretDowngrade: boolean = false, ): Promise { remotePath = removeType(remotePath, "variable"); log.debug(`Processing local variable ${remotePath}`); @@ -130,14 +160,26 @@ export async function pushVariable( log.debug(`Variable ${remotePath} is not up-to-date, updating`); + // Apply is_secret only when it differs from the remote (the value is always + // sent, so the server allows the flag change). Upgrades (non-secret->secret) + // always apply; downgrades only when explicitly allowed (single-file push) — + // see allowSecretDowngrade. `undefined` leaves the flag untouched. + let nextIsSecret: boolean | undefined = undefined; + if (localVariable.is_secret !== variable.is_secret) { + if (localVariable.is_secret) { + nextIsSecret = true; + } else if (allowSecretDowngrade) { + nextIsSecret = false; + } + } + await wmill.updateVariable({ workspace, path: remotePath.replaceAll(SEP, "/"), alreadyEncrypted: !plainSecrets, requestBody: { ...localVariable, - is_secret: - localVariable.is_secret && !variable.is_secret ? true : undefined, + is_secret: nextIsSecret, ...(wsSpecific !== undefined ? { ws_specific: wsSpecific } : {}), }, }); @@ -174,12 +216,40 @@ async function push( log.info(colors.bold.yellow("Pushing variable...")); + const local = parseFromFile(filePath) as VariableFile; + + // A secret value in a single-file push is authored by the user and is + // therefore plaintext that must be encrypted server-side — unless it has the + // shape of workspace ciphertext (a value round-tripped from `sync pull`). + // Pushing plaintext as already-encrypted would brick the variable. An explicit + // --plain-secrets always forces the plaintext (encrypt) path. + let plainSecrets = opts.plainSecrets ?? false; + if (opts.plainSecrets === undefined && local.is_secret) { + if (!looksLikeWorkspaceCiphertext(local.value)) { + log.info( + colors.yellow( + "Secret value is not in encrypted form; pushing as plaintext to be encrypted server-side (pass --plain-secrets to silence)." + ) + ); + plainSecrets = true; + } else { + // The value has the shape of workspace ciphertext, so it's stored as-is. + // A plaintext secret that coincidentally looks like ciphertext (e.g. a + // base64 token) would be stored unreadable, so surface the assumption. + log.warn( + "Secret value looks already-encrypted; pushing it as-is. If it is a plaintext secret, re-run with --plain-secrets so it gets encrypted." + ); + } + } + await pushVariable( workspace.workspaceId, remotePath, undefined, - parseFromFile(filePath), - opts.plainSecrets ?? false + local, + plainSecrets, + undefined, + true // single-file push is authoritative: allow secret->non-secret downgrade ); log.info(colors.bold.underline.green(`Variable ${remotePath} pushed`)); } diff --git a/cli/src/core/auth.ts b/cli/src/core/auth.ts index b3496d3cb8..bee70aef28 100644 --- a/cli/src/core/auth.ts +++ b/cli/src/core/auth.ts @@ -104,25 +104,25 @@ export async function requireLogin( // 403 means the token authenticated but lacks scope — re-issuing // won't help. Keep this distinct from the 401 message so the user // doesn't waste time reproducing the token. - log.info(colors.red( + log.infoStderr(colors.red( `Permission denied: the token is valid but lacks the required scope.${bodyStr ? `\n${bodyStr}` : ""}` )); } else if (status === 401) { - log.info(colors.red( + log.infoStderr(colors.red( `Could not authenticate with the provided credentials. Please check your --token and --base-url and try again.${bodyStr ? `\n${bodyStr}` : ""}` )); } else { - log.info(colors.red( + log.infoStderr(colors.red( `Request failed (${status ?? "unknown"}): ${bodyStr}` )); } return process.exit(1); } - log.info(colors.red("Could not authenticate with the provided credentials. Please check your --token and --base-url and try again.")); + log.infoStderr(colors.red("Could not authenticate with the provided credentials. Please check your --token and --base-url and try again.")); return process.exit(1); } - log.info( + log.infoStderr( "! Could not reach API given existing credentials. Attempting to reauth..." ); const newToken = await loginInteractive(workspace.remote); diff --git a/cli/src/core/context.ts b/cli/src/core/context.ts index a4ec705473..929a3728e1 100644 --- a/cli/src/core/context.ts +++ b/cli/src/core/context.ts @@ -57,7 +57,7 @@ async function selectFromMultipleProfiles( (p) => p.name === lastUsedProfileName ); if (lastUsedProfile) { - log.info( + log.infoStderr( colors.green( `Using last used profile '${lastUsedProfile.name}' for ${context}` ) @@ -69,7 +69,7 @@ async function selectFromMultipleProfiles( // No last used or it no longer exists - prompt for selection if (!!!process.stdin.isTTY || !!!process.stdout.isTTY) { const selectedProfile = profiles[0]; - log.info( + log.infoStderr( colors.yellow( `Multiple profiles found for ${context}. Using first available profile: '${selectedProfile.name}'` ) @@ -87,7 +87,7 @@ async function selectFromMultipleProfiles( return selectedProfile; } - log.info( + log.infoStderr( colors.yellow(`\nMultiple workspace profiles found for ${context}:`) ); @@ -125,14 +125,14 @@ async function createWorkspaceProfileInteractively( ): Promise { // Log appropriate message based on context if (!context.isForked) { - log.info( + log.infoStderr( colors.yellow( `\nNo workspace profile found for branch '${context.rawBranch}'\n` + `(${normalizedBaseUrl}, ${workspaceId})` ) ); } else { - log.info( + log.infoStderr( colors.yellow( `\nNo workspace profile was found for this forked workspace\n` + `(${normalizedBaseUrl}, ${workspaceId})` @@ -141,7 +141,7 @@ async function createWorkspaceProfileInteractively( } if (!!!process.stdin.isTTY || !!!process.stdout.isTTY) { - log.info( + log.infoStderr( "Not a TTY, cannot create profile interactively. Use 'wmill workspace add' first." ); return undefined; @@ -187,12 +187,12 @@ async function createWorkspaceProfileInteractively( opts.configDir ); - log.info( + log.infoStderr( colors.green( `✓ Created profile '${profileName}' for ${workspaceId} on ${normalizedBaseUrl}` ) ); - log.info(colors.green(`✓ Profile '${profileName}' is now active`)); + log.infoStderr(colors.green(`✓ Profile '${profileName}' is now active`)); return newWorkspace; } @@ -244,7 +244,7 @@ async function tryResolveWorkspace( `workspace '${opts.workspace}'`, opts.configDir ); - log.info( + log.infoStderr( colors.green( `Using workspace profile '${selected.name}' for workspace '${opts.workspace}' (${workspaceId} on ${normalizedBaseUrl})` ) @@ -254,7 +254,7 @@ async function tryResolveWorkspace( } // No matching profile — offer to create one - log.info( + log.infoStderr( `No profile found for workspace '${opts.workspace}' (${workspaceId} on ${normalizedBaseUrl})` ); const ws = await createWorkspaceProfileInteractively( @@ -309,7 +309,7 @@ export async function tryResolveBranchWorkspace( wsEntry = config.workspaces?.[workspaceNameOverride] as WorkspaceEntryConfig | undefined; if (wsEntry) { wsName = workspaceNameOverride; - log.info(`Using workspace override: ${workspaceNameOverride}`); + log.infoStderr(`Using workspace override: ${workspaceNameOverride}`); } } else { // Only try branch-based resolution if in a Git repository @@ -328,7 +328,7 @@ export async function tryResolveBranchWorkspace( const branchToLookup = originalBranchIfForked ?? rawBranch; if (originalBranchIfForked) { - log.info( + log.infoStderr( `Using original branch \`${originalBranchIfForked}\` for finding workspace from workspaces section in wmill.yaml` ); } @@ -346,7 +346,7 @@ export async function tryResolveBranchWorkspace( if (!wsEntry.baseUrl) { if (workspaceNameOverride) { // User explicitly asked for this workspace but it has no baseUrl - log.warn( + log.warnStderr( `⚠️ Workspace '${wsName}' has no baseUrl configured. Cannot resolve a profile.\n` + ` Add baseUrl to workspace '${wsName}' in wmill.yaml, or use --base-url flag.` ); @@ -370,7 +370,7 @@ export async function tryResolveBranchWorkspace( reason = `matched current git branch '${rawBranch}'`; } - log.info( + log.infoStderr( `Using workspace '${wsName}' (${reason}) → ${workspaceId} on ${baseUrl}` ); @@ -406,7 +406,7 @@ export async function tryResolveBranchWorkspace( if (matchingProfiles.length === 1) { selectedProfile = matchingProfiles[0]; - log.info( + log.infoStderr( colors.green( `Using workspace profile '${selectedProfile.name}' for workspace '${wsName}' with workspace id \`${workspaceId}\`` ) @@ -424,7 +424,7 @@ export async function tryResolveBranchWorkspace( (p) => p.name === lastUsedName ); if (lastUsedProfile) { - log.info( + log.infoStderr( colors.green( `Using workspace profile '${lastUsedProfile.name}' for workspace '${wsName}' (last used)` ) @@ -449,7 +449,7 @@ export async function tryResolveBranchWorkspace( opts.configDir ); - log.info( + log.infoStderr( colors.green( `Using workspace profile '${selectedProfile.name}' for workspace '${wsName}'` ) @@ -459,7 +459,7 @@ export async function tryResolveBranchWorkspace( if (workspaceIdIfForked) { selectedProfile.name = `${selectedProfile.name}/${workspaceIdIfForked}`; selectedProfile.workspaceId = workspaceIdIfForked; - log.info( + log.infoStderr( `Using fork workspace \`${workspaceIdIfForked}\` (parent: \`${workspaceId}\`) from branch \`${rawBranch}\`` ); } @@ -480,7 +480,7 @@ export async function resolveWorkspace( try { normalizedBaseUrl = new URL(opts.baseUrl).toString(); } catch (error) { - log.info(colors.red(`Invalid base URL: ${opts.baseUrl}`)); + log.infoStderr(colors.red(`Invalid base URL: ${opts.baseUrl}`)); return process.exit(-1); } @@ -514,7 +514,7 @@ export async function resolveWorkspace( if (existingWorkspace) { if (existingWorkspace.remote !== normalizedBaseUrl) { - log.info( + log.infoStderr( colors.red( `Base URL mismatch: --base-url is ${normalizedBaseUrl} but workspace profile "${opts.workspace}" uses ${existingWorkspace.remote}` ) @@ -535,7 +535,7 @@ export async function resolveWorkspace( token: opts.token, }; } else { - log.info( + log.infoStderr( colors.red( "If you specify a base URL with --base-url, you must also specify a workspace (--workspace) and token (--token)." ) @@ -555,7 +555,7 @@ export async function resolveWorkspace( if (workspaceNameOverride || opts.workspace || !branch || !branch.startsWith(WM_FORK_PREFIX)) { return workspace; } else { - log.info( + log.infoStderr( `Found an active workspace \`${workspace.name}\` but the branch name indicates this is a forked workspace. Ignoring active workspace and trying to resolve the correct workspace from the branch name \`${branch}\`. Use --workspace to override.` ); } @@ -572,9 +572,9 @@ export async function resolveWorkspace( if (suggestions.length > 0) { msg += ` Did you mean: ${suggestions.map((s) => `"${s.name}"`).join(", ")}?`; } - log.info(colors.red.bold(msg)); + log.infoStderr(colors.red.bold(msg)); if (profiles.length > 0) { - log.info("\nAvailable workspaces:"); + log.infoStderr("\nAvailable workspaces:"); new Table() .header(["name", "remote", "workspace id"]) .padding(2) @@ -620,12 +620,12 @@ export async function resolveWorkspace( if (wsNames.length === 1) { pickedWsName = wsNames[0]; - log.info( + log.infoStderr( `Auto-selected workspace '${pickedWsName}' (only workspace in config).\n` + `Use --workspace to override or 'wmill workspace bind' to add more workspaces.` ); } else if (process.stdin.isTTY) { - log.info( + log.infoStderr( `Multiple workspaces configured but none matched the current context.\n` + `Configured workspaces:\n${wsListStr}\n` + `Use --workspace to skip this prompt.` @@ -675,7 +675,7 @@ export async function resolveWorkspace( try { normalizedBaseUrl = new URL(envBaseUrl).toString(); } catch { - log.info(colors.red(`Invalid BASE_INTERNAL_URL: ${envBaseUrl}`)); + log.infoStderr(colors.red(`Invalid BASE_INTERNAL_URL: ${envBaseUrl}`)); return process.exit(-1); } log.debug( @@ -691,7 +691,7 @@ export async function resolveWorkspace( return ws; } - log.info(colors.red.bold("No workspace given and no default set. Run 'wmill workspace add' to configure one.")); + log.infoStderr(colors.red.bold("No workspace given and no default set. Run 'wmill workspace add' to configure one.")); return process.exit(-1); } @@ -746,7 +746,7 @@ export async function tryResolveVersion( export function validatePath(path: string): boolean { if (!(path.startsWith("g") || path.startsWith("u") || path.startsWith("f"))) { - log.info( + log.infoStderr( colors.red( "Given remote path looks invalid. Remote paths are typically of the form //..." ) diff --git a/cli/src/core/log.ts b/cli/src/core/log.ts index 034e13e3e1..e78450597f 100644 --- a/cli/src/core/log.ts +++ b/cli/src/core/log.ts @@ -21,11 +21,26 @@ export function info(msg: unknown) { console.log(`\x1b[34m${String(msg)}\x1b[39m`); } +// Like `info` but written to stderr, for diagnostics (e.g. the workspace-profile +// banner printed on every command) that must not pollute stdout when a command's +// data output is piped or redirected (e.g. `wmill variable get path > file`). +export function infoStderr(msg: unknown) { + if (silentMode) return; + console.error(`\x1b[34m${String(msg)}\x1b[39m`); +} + export function warn(msg: unknown) { if (silentMode) return; console.log(`\x1b[33m${String(msg)}\x1b[39m`); } +// Like `warn` but written to stderr; see `infoStderr` for why diagnostics must +// not land on stdout. +export function warnStderr(msg: unknown) { + if (silentMode) return; + console.error(`\x1b[33m${String(msg)}\x1b[39m`); +} + export function error(msg: unknown) { console.error(`\x1b[31m${String(msg)}\x1b[39m`); } diff --git a/cli/src/core/login.ts b/cli/src/core/login.ts index f6034b3add..e6a06e8383 100644 --- a/cli/src/core/login.ts +++ b/cli/src/core/login.ts @@ -10,7 +10,7 @@ import * as http from "node:http"; export async function loginInteractive(remote: string) { let token: string | undefined; if (!process.stdin.isTTY) { - log.info("Not a TTY, can't login interactively."); + log.infoStderr("Not a TTY, can't login interactively."); return undefined; } if ( @@ -55,7 +55,7 @@ export async function browserLogin( const port = await getPort.default({ port: env }); if (port == undefined) { - log.info(colors.red.underline("failed to aquire port")); + log.infoStderr(colors.red.underline("failed to aquire port")); return undefined; } @@ -79,7 +79,7 @@ export async function browserLogin( }); const url = `${baseUrl}user/cli?port=${port}`; - log.info(`Login by going to ${url}`); + log.infoStderr(`Login by going to ${url}`); try { open.default(url).catch((error) => { @@ -88,7 +88,7 @@ export async function browserLogin( ); }); - log.info("Opened browser for you"); + log.infoStderr("Opened browser for you"); } catch (error) { console.error( `Failed to open browser, please navigate to ${url}, error: ${error}` diff --git a/cli/test/variable_ciphertext_shape_unit.test.ts b/cli/test/variable_ciphertext_shape_unit.test.ts new file mode 100644 index 0000000000..0113929164 --- /dev/null +++ b/cli/test/variable_ciphertext_shape_unit.test.ts @@ -0,0 +1,42 @@ +import { expect, test } from "bun:test"; + +import { looksLikeWorkspaceCiphertext } from "../src/commands/variable/variable.ts"; + +// ============================================================================= +// looksLikeWorkspaceCiphertext drives whether single-file `variable push` treats +// a secret's value as already-encrypted (store verbatim) or as plaintext to be +// encrypted server-side. It must agree with the server guard +// (validate_already_encrypted_secret in windmill-store/src/variables.rs): a value +// is "ciphertext shaped" iff it is an external-backend marker, or standard base64 +// decoding to a non-zero multiple of the AES block size (16 bytes). +// ============================================================================= + +test("treats workspace-ciphertext-shaped values as already-encrypted", () => { + const ciphertextShaped = [ + "MpYeXnSBBF7dzI6K8J89xQ==", // real magic_crypt output: 16 bytes + Buffer.alloc(16, 7).toString("base64"), // 16 bytes + Buffer.alloc(32, 7).toString("base64"), // 32 bytes + "$vault:f/x/cfg", + "$aws_sm:f/x/cfg", + "$azure_kv:f/x/cfg", + ]; + for (const value of ciphertextShaped) { + expect(looksLikeWorkspaceCiphertext(value)).toBe(true); + } +}); + +test("treats hand-authored plaintext as NOT already-encrypted", () => { + const plaintext = [ + "some: plaintext\n", // space, colon, newline + "original-secret", // hyphen, not length % 4 + "hunter2", + '{"a": 1}', + "", // empty + "dGVzdA==", // valid base64 but decodes to 4 bytes (not % 16) + Buffer.alloc(17, 7).toString("base64"), // 17 bytes (not % 16) + "$omething-plain", // starts with $ but is not a real backend marker + ]; + for (const value of plaintext) { + expect(looksLikeWorkspaceCiphertext(value)).toBe(false); + } +}); diff --git a/cli/test/variable_resource_push.test.ts b/cli/test/variable_resource_push.test.ts index e8b0dc9e26..e6bc84c326 100644 --- a/cli/test/variable_resource_push.test.ts +++ b/cli/test/variable_resource_push.test.ts @@ -219,6 +219,103 @@ describe("variable", () => { }); }); + test("push encrypts a plaintext secret value (no --plain-secrets) and round-trips", async () => { + await withTestBackend(async (backend, tempDir) => { + await setupWorkspaceProfile(backend); + + const uniqueId = Date.now(); + const varPath = `f/test/sec_push_${uniqueId}`; + + // Existing secret variable (server-encrypted). + const createResp = await backend.apiRequest!( + `/api/w/${backend.workspace}/variables/create`, + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + path: varPath, + value: "original-secret", + is_secret: true, + description: "", + }), + } + ); + expect(createResp.status).toBeLessThan(300); + await createResp.text(); + + // A hand-authored spec file: plaintext value, is_secret: true. Pushing it + // without --plain-secrets must encrypt the value server-side, not store the + // plaintext verbatim as ciphertext (which would make every read fail). + const specPath = join(tempDir, "v.yaml"); + await writeFile( + specPath, + `value: |\n some: plaintext\nis_secret: true\ndescription: ""\n`, + "utf-8" + ); + + const pushResult = await backend.runCLICommand( + ["variable", "push", specPath, varPath], + tempDir + ); + expect(pushResult.code).toEqual(0); + + // The value must decrypt cleanly to the pushed plaintext. + const apiResp = await backend.apiRequest!( + `/api/w/${backend.workspace}/variables/get/${varPath}?decrypt_secret=true` + ); + expect(apiResp.status).toEqual(200); + const varData = await apiResp.json(); + expect(varData.is_secret).toBe(true); + expect(varData.value).toBe("some: plaintext\n"); + }); + }); + + test("push flips is_secret from true to false", async () => { + await withTestBackend(async (backend, tempDir) => { + await setupWorkspaceProfile(backend); + + const uniqueId = Date.now(); + const varPath = `f/test/sec_down_${uniqueId}`; + + const createResp = await backend.apiRequest!( + `/api/w/${backend.workspace}/variables/create`, + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + path: varPath, + value: "original-secret", + is_secret: true, + description: "", + }), + } + ); + expect(createResp.status).toBeLessThan(300); + await createResp.text(); + + const specPath = join(tempDir, "v_down.yaml"); + await writeFile( + specPath, + `value: "now-public"\nis_secret: false\ndescription: ""\n`, + "utf-8" + ); + + const pushResult = await backend.runCLICommand( + ["variable", "push", specPath, varPath], + tempDir + ); + expect(pushResult.code).toEqual(0); + + const apiResp = await backend.apiRequest!( + `/api/w/${backend.workspace}/variables/get/${varPath}?decrypt_secret=true` + ); + expect(apiResp.status).toEqual(200); + const varData = await apiResp.json(); + expect(varData.is_secret).toBe(false); + expect(varData.value).toBe("now-public"); + }); + }); + test("pull retrieves variables into local files", async () => { await withTestBackend(async (backend, tempDir) => { await setupWorkspaceProfile(backend); From 984ea728d98649b66b1cae899bdab9af3176caa7 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 14:35:37 +0200 Subject: [PATCH 031/117] fix: pipeline annotation false-positives from body comments (#9736) * fix: reject pipeline `# tag` annotation false-positives on regular comments `parse_pipeline_annotations` treats any comment line starting with `# tag ` as a worker-tag annotation. In Python scripts, ordinary English comments beginning with "# tag ..." were misinterpreted: values over 50 chars failed the `script.tag` INSERT (varchar(50)), and shorter ones silently overrode the script's worker tag. Worker tags are single-word identifiers (e.g. `heavy`, `gpu`), so reject any candidate that contains whitespace or exceeds 50 characters. Mirror the same validation in the TS parity parser and add regression tests on both sides. Fixes WIN-2090 Co-Authored-By: Claude Opus 4.8 (1M context) * fix: restrict pipeline annotation scan to the leading comment header The root cause of the `# tag` false-positive is broader than the `tag` keyword: `parse_pipeline_annotations` scanned every comment line in the whole file, so any body comment matching an annotation grammar (`on`, `freshness`, `tag`, `retry`, ...) was misinterpreted. The `tag` case was the most visible because an over-length value crashed the `script.tag` INSERT (varchar(50)). Windmill's other comment-directive parsers (BashAnnotations::sandbox_image, ssh_target) already scan only the leading comment header and stop at the first line of real code. Align parse_pipeline_annotations (and its TS mirror) with that convention: skip blank lines, break on the first non-comment line. This eliminates body-comment false-positives for every annotation, not just `tag`. The `tag` whitespace/length guard from the previous commit is kept as defense for prose that sits in the header itself. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .../windmill-parser/src/asset_parser.rs | 78 +++++++++++++++++-- .../parsePipelineAnnotations.test.ts | 32 ++++++++ .../AssetGraph/parsePipelineAnnotations.ts | 12 ++- 3 files changed, 115 insertions(+), 7 deletions(-) diff --git a/backend/parsers/windmill-parser/src/asset_parser.rs b/backend/parsers/windmill-parser/src/asset_parser.rs index 61e30a343f..e4a90c638e 100644 --- a/backend/parsers/windmill-parser/src/asset_parser.rs +++ b/backend/parsers/windmill-parser/src/asset_parser.rs @@ -487,9 +487,13 @@ fn parse_kv_opts(s: &str) -> BTreeMap { out } -// Scan raw source for pipeline annotations. Language-agnostic: any line -// whose first non-whitespace tokens are a comment prefix (`//`, `#`, or -// `--`) followed by one of the recognized keywords: +// Scan the leading comment header for pipeline annotations. Only the +// contiguous block of comment lines at the top of the file is considered +// (blank lines tolerated, scan stops at the first line of actual code) so +// that ordinary comments in the body can't false-positive as annotations. +// Language-agnostic: any header line whose first non-whitespace tokens are +// a comment prefix (`//`, `#`, or `--`) followed by one of the recognized +// keywords: // - `pipeline` → opt-in marker (must be alone on the line) // - `on ` → asset / native trigger edge (including // the marker-only `on schedule` form) @@ -527,6 +531,9 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { for raw_line in code.lines() { let line = raw_line.trim_start(); + if line.is_empty() { + continue; + } let rest = if let Some(r) = line.strip_prefix("//") { r } else if let Some(r) = line.strip_prefix("--") { @@ -534,7 +541,11 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { } else if let Some(r) = line.strip_prefix('#') { r } else { - continue; + // Annotations live in the leading comment header. Stop at the first + // line of actual code so comments inside the body (e.g. a regular + // `# tag ...` prose comment) can't false-positive as annotations. + // Mirrors BashAnnotations::sandbox_image / ssh_target. + break; }; let rest = rest.trim_start(); @@ -584,7 +595,14 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { if let Some(after_kw) = consume_keyword(rest, "tag") { let name = after_kw.trim(); - if !name.is_empty() && out.tag.is_none() { + // Worker tags are single-word identifiers (e.g. `heavy`, `gpu`). + // A value with whitespace or beyond the `script.tag` column width + // is almost certainly a regular comment starting with "# tag ...". + if !name.is_empty() + && !name.contains(char::is_whitespace) + && name.len() <= 50 + && out.tag.is_none() + { out.tag = Some(name.to_string()); } continue; @@ -1074,6 +1092,56 @@ mod pipeline_annotation_tests { assert!(out.tag.is_none()); } + #[test] + fn tag_with_whitespace_is_skipped() { + // A regular English comment starting with "# tag " must not be + // mistaken for a worker-tag annotation (worker tags are single words). + let out = + parse_pipeline_annotations("# tag this function so we remember to refactor it later"); + assert!(out.tag.is_none()); + } + + #[test] + fn tag_too_long_is_skipped() { + let long = "x".repeat(51); + let out = parse_pipeline_annotations(&format!("// tag {long}")); + assert!(out.tag.is_none()); + } + + #[test] + fn annotations_in_body_are_ignored() { + // Only the leading comment header is scanned. A regular `# tag ...` + // prose comment buried in the body — the WIN-2090 false-positive that + // crashed the `script.tag` INSERT — must not be treated as an + // annotation once real code has started. + let code = concat!( + "import pandas as pd\n", + "\n", + "def main():\n", + " # tag each row with its source so downstream steps can filter\n", + " # on s3://should/not/parse\n", + " return pd.DataFrame()\n", + ); + let out = parse_pipeline_annotations(code); + assert!(out.tag.is_none()); + assert!(out.triggers.is_empty()); + } + + #[test] + fn header_allows_blank_lines_before_code() { + // Blank lines (e.g. after a shebang) don't end the header; the first + // line of real code does. + let code = concat!( + "#!/usr/bin/env python\n", + "\n", + "# tag heavy\n", + "import os\n", + "# tag light\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!(out.tag.as_deref(), Some("heavy")); + } + #[test] fn retry_count_only() { let out = parse_pipeline_annotations("// retry 3"); diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts index f73819b959..15b12b959d 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts @@ -25,6 +25,38 @@ describe('parsePipelineAnnotations: tag', () => { const out = parsePipelineAnnotations('// tagged heavy') expect(out.tag).toBeUndefined() }) + + it('skips a tag value containing whitespace (regular comment false-positive)', () => { + const out = parsePipelineAnnotations('# tag this function so we remember to refactor it later') + expect(out.tag).toBeUndefined() + }) + + it('skips a tag value longer than 50 chars', () => { + const out = parsePipelineAnnotations('// tag ' + 'x'.repeat(51)) + expect(out.tag).toBeUndefined() + }) +}) + +describe('parsePipelineAnnotations: header scan', () => { + it('ignores annotations in the body once code has started', () => { + const code = [ + 'import pandas as pd', + '', + 'def main():', + ' # tag each row with its source so downstream steps can filter', + ' # on s3://should/not/parse', + ' return pd.DataFrame()' + ].join('\n') + const out = parsePipelineAnnotations(code) + expect(out.tag).toBeUndefined() + expect(out.triggerAssets).toHaveLength(0) + }) + + it('tolerates blank lines before code but stops at the first code line', () => { + const code = ['#!/usr/bin/env python', '', '# tag heavy', 'import os', '# tag light'].join('\n') + const out = parsePipelineAnnotations(code) + expect(out.tag).toBe('heavy') + }) }) describe('parsePipelineAnnotations: retry', () => { diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts index 700513e39a..67aa551a56 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts @@ -291,8 +291,13 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { } for (const rawLine of code.split('\n')) { + // Annotations live in the leading comment header: skip blank lines but + // stop at the first line of actual code, so comments inside the body + // (e.g. a regular `# tag ...` prose comment) can't false-positive. + // Mirrors the Rust parse_pipeline_annotations header scan. + if (rawLine.trim() === '') continue const rest = stripCommentPrefix(rawLine) - if (rest === undefined) continue + if (rest === undefined) break const inner = rest.trimStart() const afterPipeline = consumeKeyword(inner, 'pipeline') @@ -323,7 +328,10 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { const afterTag = consumeKeyword(inner, 'tag') if (afterTag !== undefined) { const name = afterTag.trim() - if (name && !out.tag) { + // Worker tags are single-word identifiers; a value with whitespace + // or beyond the script.tag column width is almost certainly a + // regular comment starting with "# tag ...". + if (name && !out.tag && !/\s/.test(name) && name.length <= 50) { out.tag = name } continue From 11d0e65f3af9a048bc1921bbdd3d676a07483a57 Mon Sep 17 00:00:00 2001 From: Diego Imbert <70353967+diegoimbert@users.noreply.github.com> Date: Tue, 23 Jun 2026 16:31:41 +0200 Subject: [PATCH 032/117] fix(frontend): preserve editor content when closing instance settings drawer (#9740) Closing the Instance settings drawer cleared the underlying script editor. On unmount, SuperadminSettingsInner.removeHash() stripped the `#superadmin-settings` hash with a SvelteKit `goto()`, and that navigation re-fired the script editor page's path-reactive `$effect`, reloading the script and wiping unsaved editor content. Use `replaceState` to drop the hash without a navigation (matching the existing RunForm.svelte pattern), guarded against router-teardown throws. Co-authored-by: Claude Opus 4.8 (1M context) --- .../lib/components/SuperadminSettingsInner.svelte | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/frontend/src/lib/components/SuperadminSettingsInner.svelte b/frontend/src/lib/components/SuperadminSettingsInner.svelte index 700a5df2d7..e74154b406 100644 --- a/frontend/src/lib/components/SuperadminSettingsInner.svelte +++ b/frontend/src/lib/components/SuperadminSettingsInner.svelte @@ -10,7 +10,7 @@ import { base } from '$lib/base' import SearchItems from './SearchItems.svelte' import { page } from '$app/state' - import { goto as gotoUrl } from '$app/navigation' + import { replaceState } from '$app/navigation' import Version from './Version.svelte' import Uptodate from './Uptodate.svelte' import InstanceSettings from './InstanceSettings.svelte' @@ -70,7 +70,16 @@ const index = page.url.href.lastIndexOf('#') if (index === -1) return const hashRemoved = page.url.href.slice(0, index) - gotoUrl(hashRemoved) + // Strip the drawer's URL hash without a SvelteKit navigation: a `goto` + // here re-fires path-reactive effects on the underlying page (e.g. the + // script editor's load effect), wiping unsaved editor content. + try { + replaceState(hashRemoved, page.state) + } catch (e) { + // replaceState throws if the router isn't initialized yet — possible + // when onDestroy runs during router teardown. + console.error(e) + } } onDestroy(() => { From fc797a35fe7885630c81453df0fc94769e73873a Mon Sep 17 00:00:00 2001 From: Diego Imbert <70353967+diegoimbert@users.noreply.github.com> Date: Tue, 23 Jun 2026 17:23:06 +0200 Subject: [PATCH 033/117] fix(ai-chat): Fix incorrect editor edits from ai chat #1 (#9741) --- .../components/copilot/chat/monaco-adapter.ts | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/frontend/src/lib/components/copilot/chat/monaco-adapter.ts b/frontend/src/lib/components/copilot/chat/monaco-adapter.ts index dfe3eb7565..1600d8fe9a 100644 --- a/frontend/src/lib/components/copilot/chat/monaco-adapter.ts +++ b/frontend/src/lib/components/copilot/chat/monaco-adapter.ts @@ -129,9 +129,17 @@ export class AIChatEditorHandler { const deletedChange = group.changes[0] const addedChange = group.changes[1] if (deletedChange.type === 'deleted' && addedChange.type === 'added_block') { - applyChange(this.editor, deletedChange) - addedChange.position.afterLineNumber = deletedChange.range.startLine - 1 - applyChange(this.editor, addedChange) + this.editor.executeEdits('chat', [ + { + range: { + startLineNumber: deletedChange.range.startLine, + startColumn: 1, + endLineNumber: deletedChange.range.endLine + 1, + endColumn: 0 + }, + text: addedChange.value + '\n' + } + ]) } else { throw new Error('Invalid group') } @@ -284,7 +292,7 @@ export class AIChatEditorHandler { }) if (!opts?.applyAll) { - ;({ collection, ids } = await displayVisualChanges( + ; ({ collection, ids } = await displayVisualChanges( 'editor-windmill-chat-style', this.editor, changes, From 6dfccd9d88d5df94609da49320e5704341be0b74 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 17:52:22 +0000 Subject: [PATCH 034/117] frontend improvements --- .../src/lib/components/ScriptBuilder.svelte | 4 +-- .../src/lib/components/ScriptEditor.svelte | 27 +++++++++++++------ 2 files changed, 20 insertions(+), 11 deletions(-) diff --git a/frontend/src/lib/components/ScriptBuilder.svelte b/frontend/src/lib/components/ScriptBuilder.svelte index 063a17f9d1..7599bae9e5 100644 --- a/frontend/src/lib/components/ScriptBuilder.svelte +++ b/frontend/src/lib/components/ScriptBuilder.svelte @@ -952,9 +952,7 @@ } }) }) - $effect(() => { - readFieldsRecursively(script) - }) + // Mirror the draft triggers (held in a separate `triggersState` $state) // back into `script.draft_triggers` so the UserDraft autosave — which // deep-tracks `script` — picks them up. Pre-PR ScriptBuilder ran its own diff --git a/frontend/src/lib/components/ScriptEditor.svelte b/frontend/src/lib/components/ScriptEditor.svelte index bd5fdc63cf..c75dcaf30d 100644 --- a/frontend/src/lib/components/ScriptEditor.svelte +++ b/frontend/src/lib/components/ScriptEditor.svelte @@ -1591,18 +1591,29 @@ let error = $derived(getError(testJob)) $effect(() => { - const options: ScriptOptions = { - code, - lang: lang as ScriptLang, - error, - args: args ?? {}, - path, + ;[ + editor, lastSavedCode, lastDeployedCode, diffMode, - workflowAsCode: workflowAsCodeAiContext - } + workflowAsCodeAiContext, + args, + error, + lang, + path + ] untrack(() => { + const options: ScriptOptions = { + code, + lang: lang as ScriptLang, + error, + args: args ?? {}, + path, + lastSavedCode, + lastDeployedCode, + diffMode, + workflowAsCode: workflowAsCodeAiContext + } aiChatManager.scriptEditorOptions = options aiChatManager.scriptEditorApplyCode = async (code: string, opts?: ReviewChangesOpts) => { hideDiffMode() From 29c67ced97bf2919584986f9d9eceb4337c34ad9 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 22:45:02 +0200 Subject: [PATCH 035/117] =?UTF-8?q?fix(frontend):=20debounce=20external=20?= =?UTF-8?q?code=E2=86=92Monaco=20sync=20in=20Editor=20(#9743)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(frontend): debounce external code→Monaco sync in Editor Make the external `code` prop → Monaco sync always-on and 500ms debounced, replacing the opt-in `syncExternalCode` prop. Removes the prop from the two inline rawscript call sites in FlowModuleComponent. Includes temporary debug scaffolding (A→B executeEdits button and console logs) for diagnosing successive-edit behavior. Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(frontend): share alignCodeWithEditor + bump debounce to 800ms Extract the full-range executeEdits sync into alignCodeWithEditor() and reuse it from both setCode and the debounced external-code effect. Bump the external-sync debounce 500ms -> 800ms. ScriptEditor now calls editor.setCode when syncing external code in. Co-Authored-By: Claude Opus 4.8 (1M context) * nits * nits * Fix AI not seeing latest code * remvoe debug button * nit types * nits * Nits * Check timeoutModel is undefined * fix(frontend): suppress editor echo in external code sync to prevent typing clobber * fix(frontend): cancel pending keystroke debounce in setCode to prevent clobber * fix(frontend): preserve pending external code write in updateCode * Revert "fix(frontend): preserve pending external code write in updateCode" This reverts commit 731d877730651d2169df1438ee4000b0c9efd791. --------- Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: Diego Imbert --- frontend/src/lib/components/Editor.svelte | 113 ++++++++---------- .../src/lib/components/ScriptEditor.svelte | 3 +- .../lib/components/copilot/chat/AIChat.svelte | 3 +- .../copilot/chat/AIChatManager.svelte.ts | 5 +- .../copilot/chat/ContextManager.svelte.ts | 20 ++-- .../copilot/chat/script/CodeDisplay.svelte | 2 +- .../flows/content/FlowModuleComponent.svelte | 2 - 7 files changed, 70 insertions(+), 78 deletions(-) diff --git a/frontend/src/lib/components/Editor.svelte b/frontend/src/lib/components/Editor.svelte index 476869900d..1966074a92 100644 --- a/frontend/src/lib/components/Editor.svelte +++ b/frontend/src/lib/components/Editor.svelte @@ -158,11 +158,6 @@ preparedAssetsSqlQueries?: InferAssetsSqlQueryDetails[] | undefined // To execute preview scripts with the right worker group customTag?: string - // Opt-in: reflect external `code` prop mutations back into Monaco (see - // the effect below). One-way `code={...}` callers that need live - // external updates — e.g. the inline flow rawscript — set this. Off by - // default so every other caller's behavior is unchanged. - syncExternalCode?: boolean } let { @@ -195,8 +190,7 @@ enablePreprocessorSnippet = false, rawAppRunnableKey = undefined, preparedAssetsSqlQueries, - customTag, - syncExternalCode = false + customTag }: Props = $props() $effect.pre(() => { @@ -375,23 +369,12 @@ code = ncode } - if (noHistory) { - editor?.setValue(ncode) - } else { - if (editor?.getModel()) { - // editor.setValue(ncode) - editor.pushUndoStop() - - editor.executeEdits('set', [ - { - range: editor.getModel()!.getFullModelRange(), // full range - text: ncode - } - ]) - - editor.pushUndoStop() - } - } + // setCode is an authoritative overwrite (reset, AI apply, module switch). + // Cancel any in-flight keystroke debounce first: otherwise alignCodeWithEditor + // skips on the `timeoutModel` guard (leaving Monaco stale), and the pending + // updateCode later reads the old buffer and writes it back over `ncode`. + cancelPendingChanges() + alignCodeWithEditor(!noHistory) // Dispatch change immediately when code actually changed. This ensures // callers like the Reset button and copilot trigger on:change handlers. // The debounced onDidChangeModelContent handler will no-op since code @@ -425,6 +408,7 @@ return } code = ncode + lastEditorCode = ncode dispatch('change', ncode) } @@ -436,12 +420,19 @@ * see it. Clears the chain state so the next keystroke after this * flush is a fresh leading fire. */ export function flushPendingChanges(): void { + cancelPendingChanges() + updateCode() + } + + /** Discard any in-flight keystroke debounce without materializing it, so a + * deferred updateCode can't fire later. Resets chain state to a fresh leading + * fire on the next keystroke. */ + function cancelPendingChanges(): void { if (timeoutModel !== undefined) { clearTimeout(timeoutModel) timeoutModel = undefined } changeChainStart = undefined - updateCode() } export function append(code: string): void { @@ -1901,29 +1892,6 @@ lang = scriptLangToEditorLang(scriptLang) }) - // Opt-in (syncExternalCode): reflect external `code` prop mutations into - // Monaco's model. Parents that pass `code={...}` one-way (no bind) — e.g. - // the inline rawscript in the flow editor — otherwise mutate the prop - // without Monaco ever showing the change (the AI chat editing a flow - // module's content in a session is the motivating case). Gated off by - // default: Editor is sensitive and most callers either bind:code (and - // carry their own external-sync) or treat code as init-only, so a blanket - // setValue would risk clobbering them. The `getValue() !== code` guard - // keeps the caret intact when the change originated from typing inside - // Monaco (which round-trips code back via `$bindable`, re-firing this - // effect with `code === getValue()`). - let lastExternalCodeSync = code - $effect(() => { - if (!syncExternalCode) return - if (code === lastExternalCodeSync) return - lastExternalCodeSync = code - if (!editor) return - untrack(() => { - if (editor!.getValue() !== code) { - editor!.setValue(code ?? '') - } - }) - }) $effect(() => { filePath = computePath(path) }) @@ -2011,25 +1979,50 @@ }) }) - // External `code` prop changes should flow into the Monaco editor. The - // `untrack` block reads/writes Monaco without subscribing — only the - // prop read above is tracked — so the editor's own change handler - // (`updateCode`) re-running with the same value short-circuits and we - // don't loop. - $effect(() => { - const next = code ?? '' + let applyExternalCode = useDebounce(() => alignCodeWithEditor(true), 800) + + // Last `code` value the editor itself produced or aligned to. Used to tell an + // echo (the bindable changed because the user typed — Monaco is already + // ahead) from a genuine external write. Without this, a typing burst longer + // than the debounce window would sync the lagging `code` back over newer + // keystrokes. Must be kept in step with every editor↔`code` sync point. + let lastEditorCode = code + + function alignCodeWithEditor(history: boolean) { const ed = editor if (!ed) return - untrack(() => { - if (ed.getValue() === next) return - const model = ed.getModel() - if (!model) return + const next = code ?? '' + const value = ed.getValue() + const model = ed.getModel() + // Some keystrokes are still being debounced, don't overwrite them. + // When the debounce is done, updateCode will be called and the code will be aligned with the editor. + if (timeoutModel !== undefined) return + if (!model) return + lastEditorCode = next + if (value === next) return + if (history) { ed.pushUndoStop() ed.executeEdits('external', [{ range: model.getFullModelRange(), text: next }]) ed.pushUndoStop() + } else { + ed.setValue(next) + } + } + + // External `code` prop changes should flow into the Monaco editor. Skip + // echoes: when `code` matches what the editor last produced (`updateCode`) + // or aligned to, the change came from the editor itself, so syncing back + // would clobber input typed since. Only genuine external writes — where + // `code` diverges from `lastEditorCode` — schedule a sync. The `untrack` + // block reads/writes Monaco without subscribing, so we don't loop. + $effect(() => { + ;[code, editor] + if (!editor) return + untrack(() => { + if (code === lastEditorCode) return + applyExternalCode() }) }) - let isTsWorkerInitialized = resource([() => lang, () => initialized], async () => { if (lang !== 'typescript' || !initialized) return false // Use the stable model URI (computed once at mount), not filePath which changes on rename diff --git a/frontend/src/lib/components/ScriptEditor.svelte b/frontend/src/lib/components/ScriptEditor.svelte index c75dcaf30d..8e31f35c1c 100644 --- a/frontend/src/lib/components/ScriptEditor.svelte +++ b/frontend/src/lib/components/ScriptEditor.svelte @@ -268,6 +268,7 @@ if (activeModuleTab === null && code !== lastSyncedCode) { editorCode = code lastSyncedCode = code + editor?.setCode(editorCode) // immediate sync, don't wait for the 800ms debounce untrack(() => inferSchema(code)) } }) @@ -1604,7 +1605,7 @@ ] untrack(() => { const options: ScriptOptions = { - code, + getCode: () => code, lang: lang as ScriptLang, error, args: args ?? {}, diff --git a/frontend/src/lib/components/copilot/chat/AIChat.svelte b/frontend/src/lib/components/copilot/chat/AIChat.svelte index add2c7b44f..17578d6e0e 100644 --- a/frontend/src/lib/components/copilot/chat/AIChat.svelte +++ b/frontend/src/lib/components/copilot/chat/AIChat.svelte @@ -163,7 +163,8 @@ {headerLeft} hasDiff={aiChatManager.scriptEditorOptions && !!aiChatManager.scriptEditorOptions.lastDeployedCode && - aiChatManager.scriptEditorOptions.lastDeployedCode !== aiChatManager.scriptEditorOptions.code} + aiChatManager.scriptEditorOptions.lastDeployedCode !== + aiChatManager.scriptEditorOptions.getCode()} diffMode={aiChatManager.scriptEditorOptions?.diffMode ?? false} {disabled} {disabledMessage} diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts index 9bd54e6a33..e11b61d733 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts @@ -772,7 +772,7 @@ export class AIChatManager { this.helpers = { getScriptOptions: () => { return { - code: this.scriptEditorOptions?.code ?? '', + code: this.scriptEditorOptions?.getCode() ?? '', lang: lang, path: this.scriptEditorOptions?.path ?? '', args: this.scriptEditorOptions?.args ?? {} @@ -1922,14 +1922,13 @@ export class AIChatManager { lastDeployedCode: undefined, lastSavedCode: undefined } - return { args: moduleState?.previewArgs ?? {}, error: moduleState && !moduleState.previewSuccess ? getStringError(moduleState.previewResult) : undefined, - code: module.value.content, + getCode: () => module.value.type === 'rawscript' ? module.value.content : '', lang: module.value.language, path: module.id, ...editorRelated diff --git a/frontend/src/lib/components/copilot/chat/ContextManager.svelte.ts b/frontend/src/lib/components/copilot/chat/ContextManager.svelte.ts index 4216b6cfa0..1e0765784a 100644 --- a/frontend/src/lib/components/copilot/chat/ContextManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/ContextManager.svelte.ts @@ -11,7 +11,7 @@ import type { ExtendedOpenFlow } from '$lib/components/flows/types' export interface ScriptOptions { lang: ScriptLang | 'bunnative' - code: string + getCode: () => string error: string | undefined args: Record path: string | undefined @@ -192,7 +192,7 @@ export default class ContextManager { { type: 'code', title: this.getContextCodePath(scriptOptions) ?? '', - content: scriptOptions.code, + content: scriptOptions.getCode(), lang: scriptOptions.lang } ] @@ -209,22 +209,22 @@ export default class ContextManager { } } - if (scriptOptions.lastSavedCode && scriptOptions.lastSavedCode !== scriptOptions.code) { + if (scriptOptions.lastSavedCode && scriptOptions.lastSavedCode !== scriptOptions.getCode()) { newAvailableContext.push({ type: 'diff', title: 'diff_with_last_saved_draft', // can't use spaces in the title, because it will break the word match in the context text area hightlighting logic content: scriptOptions.lastSavedCode ?? '', - diff: diffLines(scriptOptions.lastSavedCode ?? '', scriptOptions.code), + diff: diffLines(scriptOptions.lastSavedCode ?? '', scriptOptions.getCode()), lang: scriptOptions.lang }) } - if (scriptOptions.lastDeployedCode && scriptOptions.lastDeployedCode !== scriptOptions.code) { + if (scriptOptions.lastDeployedCode && scriptOptions.lastDeployedCode !== scriptOptions.getCode()) { newAvailableContext.push({ type: 'diff', title: 'diff_with_last_deployed_version', content: scriptOptions.lastDeployedCode ?? '', - diff: diffLines(scriptOptions.lastDeployedCode ?? '', scriptOptions.code), + diff: diffLines(scriptOptions.lastDeployedCode ?? '', scriptOptions.getCode()), lang: scriptOptions.lang }) } @@ -251,7 +251,7 @@ export default class ContextManager { { type: 'code', title: this.getContextCodePath(scriptOptions) ?? '', - content: scriptOptions.code, + content: scriptOptions.getCode(), lang: scriptOptions.lang, deletable: false }, @@ -277,7 +277,7 @@ export default class ContextManager { newSelectedContext = newSelectedContext .filter( (c) => - (c.type === 'code_piece' && scriptOptions.code.includes(c.content)) || + (c.type === 'code_piece' && scriptOptions.getCode().includes(c.content)) || c.type === 'code' || // Workspace references are user-picked via @-mention and not in // availableContext; preserve so badges survive editor refreshes. @@ -289,7 +289,7 @@ export default class ContextManager { if (c.type === 'code') { return { ...c, - content: scriptOptions.code, + content: scriptOptions.getCode(), title: this.getContextCodePath(scriptOptions) } } @@ -403,7 +403,7 @@ export default class ContextManager { type: 'diff' as const, title: 'diff_with_last_deployed_version', content: this.scriptOptions.lastDeployedCode ?? '', - diff: diffLines(this.scriptOptions.lastDeployedCode ?? '', this.scriptOptions.code), + diff: diffLines(this.scriptOptions.lastDeployedCode ?? '', this.scriptOptions.getCode()), lang: this.scriptOptions.lang } ] diff --git a/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte b/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte index 3709eda53c..dc1e5e3594 100644 --- a/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte +++ b/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte @@ -90,7 +90,7 @@ if ( aiChatManager.mode !== AIMode.SCRIPT || !aiChatManager.scriptEditorApplyCode || - code === aiChatManager.scriptEditorOptions?.code + code === aiChatManager.scriptEditorOptions?.getCode() ) { return false } diff --git a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte index 1bf578a081..46a5c78d32 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte @@ -867,7 +867,6 @@ bind:this={editor} class="h-full relative" code={flowModule.value.content} - syncExternalCode scriptLang={flowModule?.value?.language} automaticLayout={true} cmdEnterAction={async () => { @@ -931,7 +930,6 @@ bind:this={editor} class="h-full relative" code={flowModule.value.content} - syncExternalCode scriptLang={flowModule?.value?.language} automaticLayout={true} cmdEnterAction={async () => { From e90b2be8fade1eb78cd685890291f5a4553a6a10 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 22:49:55 +0200 Subject: [PATCH 036/117] perf(monitor): skip protected prefix in retention delete via cross-batch watermark (WIN-2088) (#9744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The expired-job retention loop re-scanned the same oldest rows on every batch. When the oldest completed jobs are undeletable (children of a still-active root flow), the ORDER BY completed_at ASC scan walked that protected prefix on each of the up-to-20 batches, doing a v2_job PK lookup per row — quadratic in prefix size (measured ~9s/batch, ~180s/cleanup-cycle on a 1.5M-row prefix). Carry a completed_at watermark (max deleted) across batches and re-apply it as completed_at >= floor so each batch resumes past the already-processed prefix. Also skip the v2_job join entirely when no old root flow is active (the common case), since nothing is protected then. Measured: subsequent batches 9000ms -> 159ms; empty-set path 154 -> 36ms. The watermark only ever skips rows the current run already deleted, was protecting, or skip-locked — all deferred to the next run, identical to the unbounded scan's row set (verified: union of batched deletes == single delete, 0 diff). Mirrored in windmill-api-settings log_cleanup. Co-authored-by: Claude Opus 4.8 (1M context) --- ...41a9e9f2d5a56e12a9d6805ca58ff40f61614.json | 30 +++++ ...fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json | 31 +++++ ...6b2bd117ceb6bc152d2abcd45850ff6aecff9.json | 24 ---- backend/src/monitor.rs | 116 +++++++++++++----- .../windmill-api-settings/src/log_cleanup.rs | 85 +++++++++---- 5 files changed, 202 insertions(+), 84 deletions(-) create mode 100644 backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json create mode 100644 backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json delete mode 100644 backend/.sqlx/query-fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9.json diff --git a/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json b/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json new file mode 100644 index 0000000000..24e387a783 --- /dev/null +++ b/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($3::timestamptz IS NULL OR completed_at >= $3)\n ORDER BY completed_at ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING id, completed_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "completed_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Int8", + "Int8", + "Timestamptz" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614" +} diff --git a/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json b/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json new file mode 100644 index 0000000000..b2e218e511 --- /dev/null +++ b/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json @@ -0,0 +1,31 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id, completed_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "completed_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Int8", + "Int8", + "UuidArray", + "Timestamptz" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75" +} diff --git a/backend/.sqlx/query-fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9.json b/backend/.sqlx/query-fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9.json deleted file mode 100644 index 62ec17f0f8..0000000000 --- a/backend/.sqlx/query-fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Int8", - "Int8", - "UuidArray" - ] - }, - "nullable": [ - false - ] - }, - "hash": "fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9" -} diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index f04646120e..fe3e3e4601 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -1324,6 +1324,9 @@ pub async fn delete_expired_items(db: &DB) -> () { let cleanup_start = Instant::now(); let mut total_deleted = 0u64; let mut batch_num = 0i32; + // Watermark carried across batches so each one resumes after the rows the previous batch + // already processed instead of re-scanning the (potentially undeletable) oldest prefix. + let mut completed_at_floor: Option> = None; // Process batches until no more expired jobs or max batches reached loop { @@ -1336,14 +1339,17 @@ pub async fn delete_expired_items(db: &DB) -> () { } // Each batch runs in its own transaction to avoid long-running locks - let batch_result = delete_expired_jobs_batch(db, job_retention_secs, batch_size).await; + let batch_result = + delete_expired_jobs_batch(db, job_retention_secs, batch_size, completed_at_floor) + .await; match batch_result { - Ok(deleted_count) => { + Ok((deleted_count, max_completed_at)) => { if deleted_count == 0 { // No more expired jobs to delete break; } + completed_at_floor = max_completed_at.or(completed_at_floor); total_deleted += deleted_count as u64; batch_num += 1; } @@ -1510,12 +1516,20 @@ pub async fn check_expiring_tokens(db: &DB) { /// Delete a batch of expired jobs with LIMIT and SKIP LOCKED for high-scale environments. /// Uses a single transaction per batch to minimize lock duration. -/// Returns the number of jobs deleted in this batch. +/// +/// `completed_at_floor` is the watermark from the previous batch in the same cleanup run (the +/// max `completed_at` it deleted); pass `None` for the first batch. It is re-applied as +/// `completed_at >= floor` so the scan resumes past the rows already processed instead of +/// re-walking them (see the inline comment on the DELETE for why this matters). +/// +/// Returns `(jobs deleted in this batch, max completed_at deleted)`. The caller feeds the +/// returned watermark back in as `completed_at_floor` for the next batch. async fn delete_expired_jobs_batch( db: &DB, job_retention_secs: i64, batch_size: i64, -) -> error::Result { + completed_at_floor: Option>, +) -> error::Result<(usize, Option>)> { let mut tx = db.begin().await?; // Fetch active ROOT job IDs that started before the retention period. We only care about @@ -1531,34 +1545,70 @@ async fn delete_expired_jobs_batch( .fetch_all(&mut *tx) .await?; - // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas - // ORDER BY completed_at ensures we delete oldest jobs first. - // Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than - // `!= ALL($3)`: the subquery form lets the planner build a one-time hashed - // SubPlan and apply it as a filter on the ordered index scan, giving O(1) - // membership per candidate instead of a per-row linear array scan (which - // degrades sharply when many root jobs are active). The `u IS NOT NULL` guard - // sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids). - let deleted_jobs: Vec = sqlx::query_scalar!( - "DELETE FROM v2_job_completed - WHERE id IN ( - SELECT jc.id FROM v2_job_completed jc - LEFT JOIN v2_job j ON j.id = jc.id - WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval - AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( - SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL - ) - ORDER BY jc.completed_at ASC - LIMIT $2 - FOR UPDATE OF jc SKIP LOCKED - ) - RETURNING id", - job_retention_secs, - batch_size, - &active_root_job_ids - ) - .fetch_all(&mut *tx) - .await?; + // `completed_at_floor` is a watermark carried across batches within a cleanup run: it is the + // max(completed_at) deleted by the previous batch. Re-applying it as `completed_at >= floor` + // lets each batch resume after the rows the previous batch already processed instead of + // re-scanning them. This matters when the oldest rows are undeletable (children of a + // still-active root flow): without the floor the `ORDER BY completed_at ASC` scan walks that + // same protected prefix on every batch, turning a cleanup run quadratic in prefix size. + // Floor only ever skips rows the current run already deleted, was protecting, or skip-locked — + // all correctly deferred to the next run, identical to the unbounded scan's semantics. + // + // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas; ORDER BY completed_at + // deletes oldest jobs first. + let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() { + // Common case: no old root flow is still running, so nothing is protected and the + // v2_job join (a PK lookup per candidate) is pure overhead — skip it entirely. + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT id FROM v2_job_completed + WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($3::timestamptz IS NULL OR completed_at >= $3) + ORDER BY completed_at ASC + LIMIT $2 + FOR UPDATE SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + } else { + // Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than `!= ALL($3)`: + // the subquery form lets the planner build a one-time hashed SubPlan and apply it as a + // filter on the ordered index scan, giving O(1) membership per candidate instead of a + // per-row linear array scan (which degrades sharply when many root jobs are active). The + // `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids). + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT jc.id FROM v2_job_completed jc + LEFT JOIN v2_job j ON j.id = jc.id + WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($4::timestamptz IS NULL OR jc.completed_at >= $4) + AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( + SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL + ) + ORDER BY jc.completed_at ASC + LIMIT $2 + FOR UPDATE OF jc SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + &active_root_job_ids, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + }; let deleted_count = deleted_jobs.len(); @@ -1618,7 +1668,7 @@ async fn delete_expired_jobs_batch( tx.commit().await?; - Ok(deleted_count) + Ok((deleted_count, max_completed_at)) } async fn delete_log_files_from_disk_and_store( diff --git a/backend/windmill-api-settings/src/log_cleanup.rs b/backend/windmill-api-settings/src/log_cleanup.rs index 2264e27037..1e71029874 100644 --- a/backend/windmill-api-settings/src/log_cleanup.rs +++ b/backend/windmill-api-settings/src/log_cleanup.rs @@ -339,13 +339,15 @@ async fn cleanup_job_logs( return Ok(()); } + let mut completed_at_floor: Option> = None; loop { - let (deleted_count, rel_paths) = - delete_expired_jobs_batch(db, retention_secs, JOB_BATCH).await?; + let (deleted_count, rel_paths, max_completed_at) = + delete_expired_jobs_batch(db, retention_secs, JOB_BATCH, completed_at_floor).await?; if deleted_count == 0 { break; } + completed_at_floor = max_completed_at.or(completed_at_floor); let s3_paths: Vec = rel_paths .iter() @@ -382,7 +384,8 @@ async fn delete_expired_jobs_batch( db: &DB, job_retention_secs: i64, batch_size: i64, -) -> error::Result<(usize, Vec)> { + completed_at_floor: Option>, +) -> error::Result<(usize, Vec, Option>)> { let mut tx = db.begin().await?; let active_root_job_ids: Vec = sqlx::query_scalar!( @@ -395,33 +398,61 @@ async fn delete_expired_jobs_batch( .fetch_all(&mut *tx) .await?; - // Active-root exclusion via NOT IN (hashed SubPlan) instead of `!= ALL($3)`; - // see backend/src/monitor.rs::delete_expired_jobs_batch for the rationale. - let deleted_jobs: Vec = sqlx::query_scalar!( - "DELETE FROM v2_job_completed - WHERE id IN ( - SELECT jc.id FROM v2_job_completed jc - LEFT JOIN v2_job j ON j.id = jc.id - WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval - AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( - SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL - ) - ORDER BY jc.completed_at ASC - LIMIT $2 - FOR UPDATE OF jc SKIP LOCKED - ) - RETURNING id", - job_retention_secs, - batch_size, - &active_root_job_ids - ) - .fetch_all(&mut *tx) - .await?; + // `completed_at_floor` carries a watermark across batches so each one resumes after the rows + // the previous batch processed instead of re-scanning the (potentially undeletable) oldest + // prefix; the empty-active-roots branch skips the v2_job join entirely. See + // backend/src/monitor.rs::delete_expired_jobs_batch for the full rationale. + let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() { + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT id FROM v2_job_completed + WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($3::timestamptz IS NULL OR completed_at >= $3) + ORDER BY completed_at ASC + LIMIT $2 + FOR UPDATE SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + } else { + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT jc.id FROM v2_job_completed jc + LEFT JOIN v2_job j ON j.id = jc.id + WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($4::timestamptz IS NULL OR jc.completed_at >= $4) + AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( + SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL + ) + ORDER BY jc.completed_at ASC + LIMIT $2 + FOR UPDATE OF jc SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + &active_root_job_ids, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + }; let deleted_count = deleted_jobs.len(); if deleted_count == 0 { tx.commit().await?; - return Ok((0, Vec::new())); + return Ok((0, Vec::new(), max_completed_at)); } if let Err(e) = sqlx::query!( @@ -471,7 +502,7 @@ async fn delete_expired_jobs_batch( tx.commit().await?; - Ok((deleted_count, log_paths)) + Ok((deleted_count, log_paths, max_completed_at)) } /// Scan S3 under the `logs/` prefix for orphan log files and delete them. From 24446e80093ade349f7fbf65063d2d1cb5551c1e Mon Sep 17 00:00:00 2001 From: hugocasa Date: Tue, 23 Jun 2026 22:50:40 +0200 Subject: [PATCH 037/117] fix: allow object storage test for non-super-admins, harden on cloud (#9739) * fix: allow non-super-admin object storage test, harden SSRF surface on cloud Co-Authored-By: Claude Opus 4.8 (1M context) * fix: validate effective object storage host to close region/bucket SSRF bypass Co-Authored-By: Claude Opus 4.8 (1M context) * fix: validate gcs_base_url/token_uri in GCS service account key to close SSRF bypass Co-Authored-By: Claude Opus 4.8 (1M context) * fix: match url scheme case-insensitively in object storage host validation Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- backend/windmill-api-settings/src/lib.rs | 426 +++++++++++++++++++++-- 1 file changed, 391 insertions(+), 35 deletions(-) diff --git a/backend/windmill-api-settings/src/lib.rs b/backend/windmill-api-settings/src/lib.rs index 7a278afc49..3f7fed4489 100644 --- a/backend/windmill-api-settings/src/lib.rs +++ b/backend/windmill-api-settings/src/lib.rs @@ -256,52 +256,300 @@ pub async fn test_s3_bucket( use bytes::Bytes; use futures::StreamExt; - require_super_admin(&db, &authed.email).await?; + // The probe executes on the API server itself. On multi-tenant Cloud that is a shared control + // plane, so we constrain untrusted callers to remove the SSRF / credential-exfiltration / + // local-filesystem surface (see validate_object_storage_test). On self-hosted instances the + // object store usually lives on the local/private network and all authenticated users are + // trusted, so testing there stays unrestricted. Super admins keep the unrestricted path too. + let is_super_admin = is_super_admin_email(&db, &authed.email).await?; + let restrict = !is_super_admin && *CLOUD_HOSTED; + if restrict { + validate_object_storage_test(&test_s3_bucket).await?; + } let client = build_object_store_from_settings(test_s3_bucket, Some(&db)) .await? .store; - let mut list = client.list(Some( - &windmill_object_store::object_store_reexports::Path::from("".to_string()), - )); - let first_file = list.next().await; - if first_file.is_some() { - if let Err(e) = first_file.as_ref().unwrap() { - tracing::error!("error listing bucket: {e:#}"); - error::Error::internal_err(format!("Failed to list files in blob storage: {e:#}")); + let run = async { + let mut list = client.list(Some( + &windmill_object_store::object_store_reexports::Path::from("".to_string()), + )); + let first_file = list.next().await; + if first_file.is_some() { + if let Err(e) = first_file.as_ref().unwrap() { + tracing::error!("error listing bucket: {e:#}"); + error::Error::internal_err(format!("Failed to list files in blob storage: {e:#}")); + } + tracing::info!("Listed files: {:?}", first_file.unwrap()); + } else { + tracing::info!("No files in blob storage"); } - tracing::info!("Listed files: {:?}", first_file.unwrap()); + + let path = windmill_object_store::object_store_reexports::Path::from(format!( + "/test-s3-bucket-{uuid}", + uuid = uuid::Uuid::new_v4() + )); + tracing::info!("Testing blob storage at path: {path}"); + client + .put( + &path, + windmill_object_store::object_store_reexports::PutPayload::from_static(b"hello"), + ) + .await + .map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?; + let content = client + .get(&path) + .await + .map_err(to_anyhow)? + .bytes() + .await + .map_err(to_anyhow)?; + if content != Bytes::from_static(b"hello") { + return Err(error::Error::internal_err( + "Failed to read back from blob storage".to_string(), + )); + } + client.delete(&path).await.map_err(to_anyhow)?; + Ok::("Tested blob storage successfully".to_string()) + }; + + if restrict { + // The object-store client is built with timeouts disabled, so a malicious endpoint could + // otherwise hold the API server connection open indefinitely. + tokio::time::timeout(Duration::from_secs(15), run) + .await + .map_err(|_| { + error::Error::internal_err("Object storage connectivity test timed out".to_string()) + })? } else { - tracing::info!("No files in blob storage"); + run.await + } +} + +// Hardening for the object-storage connectivity test by an untrusted (non-super-admin) caller on +// Cloud. The probe runs on the shared API server, so without these constraints an authenticated +// user could coerce the server into connecting to arbitrary internal endpoints (SSRF), signing +// requests with the instance role (credential exfiltration), or reading/writing the server's local +// disk (filesystem object store). +#[cfg(feature = "parquet")] +async fn validate_object_storage_test(settings: &ObjectSettings) -> error::Result<()> { + fn non_empty(opt: &Option) -> bool { + opt.as_ref().is_some_and(|s| !s.is_empty()) } - let path = windmill_object_store::object_store_reexports::Path::from(format!( - "/test-s3-bucket-{uuid}", - uuid = uuid::Uuid::new_v4() - )); - tracing::info!("Testing blob storage at path: {path}"); - client - .put( - &path, - windmill_object_store::object_store_reexports::PutPayload::from_static(b"hello"), - ) + // Reject backends that rely on the server's identity or local filesystem, require explicit + // credentials for the rest (so the server never falls back to its own ambient credentials), and + // resolve the host the client will actually connect to. We derive the *effective* endpoint here + // — mirroring build_*_from_settings: the region/account-derived default and the virtual-hosted + // bucket prefix — rather than only validating a caller-supplied `endpoint`, so caller-controlled + // `region`/`account_name`/`bucket` cannot smuggle an internal host past the check (e.g. an empty + // endpoint with region = "@169.254.169.254/" otherwise resolves to the cloud metadata service). + let effective_endpoint: Option = match settings { + ObjectSettings::Filesystem(_) => { + return Err(error::Error::NotAuthorized( + "Testing a local filesystem object store requires a super admin".to_string(), + )); + } + ObjectSettings::AwsOidc(_) => { + return Err(error::Error::NotAuthorized( + "Testing OIDC-based object storage requires a super admin".to_string(), + )); + } + ObjectSettings::S3(s3) => { + if !(non_empty(&s3.access_key) && non_empty(&s3.secret_key)) { + return Err(error::Error::NotAuthorized( + "Testing S3 storage without explicit credentials requires a super admin" + .to_string(), + )); + } + let region = s3 + .region + .clone() + .filter(|r| !r.is_empty()) + .or_else(|| std::env::var("AWS_REGION").ok().filter(|r| !r.is_empty())) + .unwrap_or_else(|| "us-east-1".to_string()); + let raw_endpoint = s3 + .endpoint + .clone() + .filter(|e| !e.is_empty()) + .or_else(|| std::env::var("S3_ENDPOINT").ok().filter(|e| !e.is_empty())) + .unwrap_or_else(|| format!("s3.{region}.amazonaws.com")); + Some(windmill_object_store::render_endpoint( + raw_endpoint, + !s3.allow_http.unwrap_or(true), + s3.port, + s3.path_style, + s3.bucket.clone().unwrap_or_default(), + )) + } + ObjectSettings::Azure(azure) => { + if !non_empty(&azure.access_key) { + return Err(error::Error::NotAuthorized( + "Testing Azure storage without an explicit access key requires a super admin" + .to_string(), + )); + } + Some( + azure + .endpoint + .clone() + .filter(|e| !e.is_empty()) + .unwrap_or_else(|| format!("{}.blob.core.windows.net", azure.account_name)), + ) + } + ObjectSettings::Gcs(gcs) => { + if gcs.service_account_key.is_empty() { + return Err(error::Error::NotAuthorized( + "Testing GCS storage without a service account key requires a super admin" + .to_string(), + )); + } + // The service-account-key JSON can override the data-plane URL (`gcs_base_url`) and the + // OAuth token endpoint (`token_uri`); the GCS client connects to whatever they point at. + // Validate every http(s) URL embedded in the key. When none override it, the host stays + // the public storage.googleapis.com, so no further check is needed. + if let Ok(serde_json::Value::Object(map)) = + serde_json::from_str::(&gcs.service_account_key) + { + for value in map.values() { + if let Some(url) = value.as_str() { + // Match how the URL parser reads the value: leading whitespace/control is + // ignored and the scheme is case-insensitive. + let url = + url.trim_start_matches(|c: char| c.is_whitespace() || c.is_control()); + if strip_http_scheme(url).is_some() { + validate_public_endpoint(url).await?; + } + } + } + } + None + } + }; + + // Block non-public network targets (internal services, cloud metadata, loopback, ...). + if let Some(endpoint) = effective_endpoint { + validate_public_endpoint(&endpoint).await?; + } + Ok(()) +} + +#[cfg(feature = "parquet")] +async fn validate_public_endpoint(endpoint: &str) -> error::Result<()> { + let host = extract_host(endpoint).ok_or_else(|| { + error::Error::BadRequest(format!("Invalid object storage endpoint: {endpoint}")) + })?; + + let addrs: Vec = tokio::net::lookup_host((host.as_str(), 443u16)) .await - .map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?; - let content = client - .get(&path) - .await - .map_err(to_anyhow)? - .bytes() - .await - .map_err(to_anyhow)?; - if content != Bytes::from_static(b"hello") { - return Err(error::Error::internal_err( - "Failed to read back from blob storage".to_string(), - )); + .map_err(|e| { + error::Error::BadRequest(format!( + "Could not resolve object storage endpoint '{host}': {e}" + )) + })? + .collect(); + + if addrs.is_empty() { + return Err(error::Error::BadRequest(format!( + "Could not resolve object storage endpoint '{host}'" + ))); + } + + // Reject if any resolved address is non-public, which also defeats the simplest DNS-rebinding + // attempts (a name resolving to both a public and a private address). + for addr in addrs { + if is_forbidden_ip(addr.ip()) { + return Err(error::Error::NotAuthorized( + "Testing object storage at a private, loopback, or link-local endpoint requires a super admin" + .to_string(), + )); + } + } + Ok(()) +} + +// Strip a leading `http://`/`https://` scheme case-insensitively (URL schemes are +// case-insensitive), returning the remainder when one was present. +#[cfg(feature = "parquet")] +fn strip_http_scheme(s: &str) -> Option<&str> { + for scheme in ["https://", "http://"] { + let b = scheme.as_bytes(); + if s.len() >= b.len() && s.as_bytes()[..b.len()].eq_ignore_ascii_case(b) { + return Some(&s[b.len()..]); + } + } + None +} + +#[cfg(feature = "parquet")] +fn extract_host(endpoint: &str) -> Option { + let mut s = endpoint.trim(); + if let Some(rest) = strip_http_scheme(s) { + s = rest; + } + s = s.split(['/', '?', '#', '\\']).next().unwrap_or(s); + if let Some((_, rest)) = s.rsplit_once('@') { + s = rest; + } + let host = if let Some(rest) = s.strip_prefix('[') { + // IPv6 literal, e.g. [::1]:9000 + rest.split(']').next().unwrap_or(rest) + } else { + // host or host:port + s.split(':').next().unwrap_or(s) + } + .trim(); + if host.is_empty() { + None + } else { + Some(host.to_string()) + } +} + +#[cfg(feature = "parquet")] +fn is_forbidden_ip(ip: std::net::IpAddr) -> bool { + use std::net::{IpAddr, Ipv4Addr}; + match ip { + IpAddr::V4(v4) => { + v4.is_loopback() + || v4.is_private() + || v4.is_link_local() // 169.254.0.0/16, incl. the cloud metadata endpoint + || v4.is_unspecified() + || v4.is_broadcast() + || v4.is_documentation() + || v4.is_multicast() + || v4.octets()[0] == 0 // 0.0.0.0/8 + || (v4.octets()[0] == 100 && (v4.octets()[1] & 0xc0) == 64) // 100.64.0.0/10 CGNAT + } + IpAddr::V6(v6) => { + // Any IPv4 embedded in an IPv6 address (IPv4-mapped ::ffff:0:0/96, IPv4-compatible + // ::/96, or NAT64 64:ff9b::/96) is re-checked against the IPv4 rules, so e.g. + // 64:ff9b::169.254.169.254 cannot route to the metadata endpoint in a NAT64 network. + let seg = v6.segments(); + let is_v4_compatible = seg[0..6] == [0, 0, 0, 0, 0, 0]; + let is_nat64 = seg[0] == 0x0064 && seg[1] == 0xff9b && seg[2..6] == [0, 0, 0, 0]; + if let Some(v4) = v6.to_ipv4_mapped() { + return is_forbidden_ip(IpAddr::V4(v4)); + } + if is_v4_compatible || is_nat64 { + let embedded = Ipv4Addr::new( + (seg[6] >> 8) as u8, + (seg[6] & 0xff) as u8, + (seg[7] >> 8) as u8, + (seg[7] & 0xff) as u8, + ); + if is_forbidden_ip(IpAddr::V4(embedded)) { + return true; + } + } + v6.is_loopback() + || v6.is_unspecified() + || v6.is_multicast() + || (seg[0] & 0xfe00) == 0xfc00 // fc00::/7 unique local + || (seg[0] & 0xffc0) == 0xfe80 // fe80::/10 link-local + } } - client.delete(&path).await.map_err(to_anyhow)?; - Ok("Tested blob storage successfully".to_string()) } #[cfg(feature = "parquet")] @@ -1861,3 +2109,111 @@ mod tests { ); } } + +#[cfg(all(test, feature = "parquet"))] +mod object_storage_test_hardening { + use super::{extract_host, is_forbidden_ip, validate_object_storage_test}; + use std::net::IpAddr; + use windmill_object_store::ObjectSettings; + + // IP literals (not hostnames) keep validate_public_endpoint deterministic — `lookup_host` + // parses them without any network round-trip. + fn gcs_settings(gcs_base_url: &str) -> ObjectSettings { + serde_json::from_value(serde_json::json!({ + "type": "Gcs", + "bucket": "b", + "serviceAccountKey": { "gcs_base_url": gcs_base_url, "client_email": "x@y.z" } + })) + .unwrap() + } + + #[tokio::test] + async fn rejects_gcs_internal_base_url() { + // gcs_base_url in the service-account key must not smuggle an internal host past the check, + // including via a mixed-case scheme (URL schemes are case-insensitive). + for url in [ + "http://169.254.169.254", + "HTTP://169.254.169.254", + "Https://10.0.0.5", + ] { + assert!( + validate_object_storage_test(&gcs_settings(url)) + .await + .is_err(), + "{url} should be rejected" + ); + } + } + + #[tokio::test] + async fn allows_gcs_public_base_url() { + assert!( + validate_object_storage_test(&gcs_settings("https://8.8.8.8")) + .await + .is_ok() + ); + } + + fn ip(s: &str) -> IpAddr { + s.parse().unwrap() + } + + #[test] + fn forbids_internal_ips() { + for s in [ + "127.0.0.1", // loopback + "169.254.169.254", // cloud metadata (link-local) + "10.0.0.5", // private + "172.16.3.4", // private + "192.168.1.10", // private + "0.0.0.0", // unspecified + "100.64.0.1", // CGNAT + "::1", // IPv6 loopback + "fe80::1", // IPv6 link-local + "fc00::1", // IPv6 unique local + "::ffff:127.0.0.1", // IPv4-mapped loopback + "::ffff:169.254.169.254", // IPv4-mapped metadata + "::169.254.169.254", // IPv4-compatible metadata + "64:ff9b::169.254.169.254", // NAT64-embedded metadata + "64:ff9b::a9fe:a9fe", // NAT64-embedded metadata (hex form) + ] { + assert!(is_forbidden_ip(ip(s)), "{s} should be forbidden"); + } + } + + #[test] + fn allows_public_ips() { + for s in ["8.8.8.8", "1.1.1.1", "52.95.110.1", "2606:4700:4700::1111"] { + assert!(!is_forbidden_ip(ip(s)), "{s} should be allowed"); + } + } + + #[test] + fn extracts_host_from_endpoint() { + let cases = [ + ("s3.amazonaws.com", Some("s3.amazonaws.com")), + ("https://minio.internal:9000", Some("minio.internal")), + ("http://10.0.0.5:9000/bucket", Some("10.0.0.5")), + ("user:pass@host.example:443", Some("host.example")), + ("[::1]:9000", Some("::1")), + ("https://[fe80::1]/x", Some("fe80::1")), + ("", None), + // Injection via region/bucket interpolation into the default endpoint string: the + // userinfo `@` and the path `/` must not hide the real authority from the host check. + ( + "https://s3.@169.254.169.254/.amazonaws.com", + Some("169.254.169.254"), + ), + ( + "https://@169.254.169.254/mybucket.s3.amazonaws.com", + Some("169.254.169.254"), + ), + ("s3.#@169.254.169.254/x.amazonaws.com", Some("s3.")), + // Scheme is case-insensitive. + ("HTTP://169.254.169.254", Some("169.254.169.254")), + ]; + for (input, expected) in cases { + assert_eq!(extract_host(input).as_deref(), expected, "input: {input}"); + } + } +} From 9793d01575415963a89609a1baf2cd64f0d050cc Mon Sep 17 00:00:00 2001 From: hugocasa Date: Tue, 23 Jun 2026 22:53:12 +0200 Subject: [PATCH 038/117] feat: add resource and infrastructure telemetry (#9737) * feat(telemetry): disclose resource and infra usage stats When minimal telemetry is disabled, the stats payload now includes resource counts (workspaces, scripts per language, flows, workflows as code, low-code and raw apps) and, on EE only, infrastructure info (container runtime, database size, max connections, RDS detection). Update the telemetry disclosure in instance settings accordingly: resource counts are listed for both CE and EE; infra info is shown only on EE since it is collected only there. Bump the EE ref and add the sqlx cache for the new queries. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(telemetry): expand EE infra disclosure and add sysinfo dep Disclose the expanded EE infrastructure telemetry (deployment mode, host OS/arch/CPU/memory, filesystem space, Postgres version and connection counts, object storage backend, sandboxing and retention settings) in instance settings. Add sysinfo as a windmill-common dependency for host memory and filesystem stats, bump the EE ref, and add the sqlx cache for the new queries. Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(telemetry): focus EE infra disclosure on wrapping platform Drop the single-server host details (OS, arch, CPU, memory, filesystem) and tuning config from the EE infra disclosure, and revert the sysinfo dependency they required. Reflect managed-database-provider detection in place of the RDS flag. Bump the EE ref and update the sqlx cache for the revised queries. Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(telemetry): drop deployment mode and worker count from disclosure Remove deployment mode and worker count from the EE infra disclosure to match the backend, and bump the EE ref. They reflect only the node sending telemetry, not the deployment topology. Co-Authored-By: Claude Opus 4.8 (1M context) * chore: update ee-repo-ref to 6d3301507db50818f1683dac3941d3e0cf1152a7 This commit updates the EE repository reference after PR #627 was merged in windmill-ee-private. Previous ee-repo-ref: d30e7d18d14992598a97356d0ed13f7d5d585115 New ee-repo-ref: 6d3301507db50818f1683dac3941d3e0cf1152a7 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: windmill-internal-app[bot] Co-authored-by: Ruben Fiszel --- ...ad042144e656adcca56a4936a2eb196d3f48c.json | 20 +++++++ ...c74ac5a589542d3e76b1e756a82fb19d49ee8.json | 58 +++++++++++++++++++ ...abf590e5e94898954711d57a602e6fd8a2f84.json | 20 +++++++ ...a2bc7822d263605a445f1f4a76e58e76a3e79.json | 20 +++++++ ...c8e75790b7c3dd7845359019ff645a1f7c8bf.json | 20 +++++++ ...73a85256c0a3e5a701cc13c944e66f6402617.json | 20 +++++++ ...299bf8290c3b969b576cb185c8b5b0abb0265.json | 20 +++++++ ...cbe0b55cc39d64f73a3c90c250ea670f9cdee.json | 20 +++++++ ...fe58b8a53d071cdc83eca3092194b4a9c9174.json | 26 +++++++++ ...5b86575c1be761de158ad664670e981524cbf.json | 20 +++++++ ...99f681f5630c024ae9a437a301f149636b0db.json | 38 ++++++++++++ backend/ee-repo-ref.txt | 2 +- .../lib/components/InstanceSettings.svelte | 12 ++++ 13 files changed, 295 insertions(+), 1 deletion(-) create mode 100644 backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json create mode 100644 backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json create mode 100644 backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json create mode 100644 backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json create mode 100644 backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json create mode 100644 backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json create mode 100644 backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json create mode 100644 backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json create mode 100644 backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json create mode 100644 backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json create mode 100644 backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json diff --git a/backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json b/backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json new file mode 100644 index 0000000000..173fe9d05c --- /dev/null +++ b/backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT SUM(pg_database_size(datname))::BIGINT AS \"v!\" FROM pg_database", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c" +} diff --git a/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json b/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json new file mode 100644 index 0000000000..cef272c219 --- /dev/null +++ b/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json @@ -0,0 +1,58 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT language AS \"language!: _\", COUNT(*)::BIGINT AS \"count!\"\n FROM script\n WHERE archived = false AND deleted = false AND kind = 'script'\n AND (auto_kind IS NULL OR auto_kind <> 'wac')\n GROUP BY language\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "language!: _", + "type_info": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby", + "rlang" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + null + ] + }, + "hash": "11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8" +} diff --git a/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json b/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json new file mode 100644 index 0000000000..eabe7f9bf6 --- /dev/null +++ b/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::INT AS \"v!\" FROM pg_stat_activity", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84" +} diff --git a/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json b/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json new file mode 100644 index 0000000000..8d9c34ed2b --- /dev/null +++ b/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM script WHERE archived = false AND deleted = false AND auto_kind = 'wac'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79" +} diff --git a/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json b/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json new file mode 100644 index 0000000000..43f527b3aa --- /dev/null +++ b/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_database_size(current_database())::BIGINT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf" +} diff --git a/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json b/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json new file mode 100644 index 0000000000..e99432f11f --- /dev/null +++ b/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT current_setting('max_connections')::INT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617" +} diff --git a/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json b/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json new file mode 100644 index 0000000000..5bd3cf80a7 --- /dev/null +++ b/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM flow WHERE archived = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265" +} diff --git a/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json b/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json new file mode 100644 index 0000000000..711970ff32 --- /dev/null +++ b/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT current_setting('server_version_num')::INT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee" +} diff --git a/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json b/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json new file mode 100644 index 0000000000..20c5c58c40 --- /dev/null +++ b/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n COUNT(*) FILTER (WHERE av.raw_app = false)::BIGINT AS \"low_code!\",\n COUNT(*) FILTER (WHERE av.raw_app = true)::BIGINT AS \"raw!\"\n FROM app a\n JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "low_code!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "raw!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null + ] + }, + "hash": "60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174" +} diff --git a/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json b/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json new file mode 100644 index 0000000000..0f75ec3883 --- /dev/null +++ b/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM workspace WHERE deleted = false AND id NOT LIKE 'wm-fork%'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf" +} diff --git a/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json b/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json new file mode 100644 index 0000000000..a604c85831 --- /dev/null +++ b/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json @@ -0,0 +1,38 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n EXISTS(SELECT 1 FROM pg_proc WHERE proname = 'aurora_version') AS \"aurora!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'rds_superuser') AS \"rds!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'cloudsqlsuperuser') AS \"cloudsql!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname IN ('azure_pg_admin', 'azuresu')) AS \"azure!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "aurora!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "rds!", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "cloudsql!", + "type_info": "Bool" + }, + { + "ordinal": 3, + "name": "azure!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db" +} diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index a4fea858de..e56f6790f9 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -ac1f6f666f36141cb6ba6f8eaa614821a90464ad +6d3301507db50818f1683dac3941d3e0cf1152a7 diff --git a/frontend/src/lib/components/InstanceSettings.svelte b/frontend/src/lib/components/InstanceSettings.svelte index 1e1416295a..b3d0ee1446 100644 --- a/frontend/src/lib/components/InstanceSettings.svelte +++ b/frontend/src/lib/components/InstanceSettings.svelte @@ -1063,6 +1063,14 @@
  • AI chat usage (provider, model, mode, session count, message count — last 30 days)
    • +
  • resource counts (workspaces, scripts per language, flows, workflows as code, low-code + apps, raw apps)
    • +
  • infrastructure info (container runtime, managed database provider, database version, + size and cluster size, max and active connections, object storage backend)

    • For air-gapped instances, you can download the telemetry data and send it manually.
    @@ -1101,6 +1109,10 @@
  • AI chat usage (provider, model, mode, session count, message count — last 30 days)
    • +
  • resource counts (workspaces, scripts per language, flows, workflows as code, low-code + apps, raw apps)
    • {/if} From cbf54d4eb432638e27f67c4c8b879cbcc0291da3 Mon Sep 17 00:00:00 2001 From: hugocasa Date: Tue, 23 Jun 2026 22:54:07 +0200 Subject: [PATCH 039/117] fix: preserve fork parent linkage on workspace id change (#9716) Co-authored-by: Claude Opus 4.8 (1M context) --- ...f0f1075b878f9916145929b3cd3b1a53b777e.json | 15 ++++++++++ ...22ba3d670cc36bdbc6451f29b8f22f8cff688.json | 28 +++++++++++++++++++ ...b775225dcb430026cebe15ba4994ac636514d.json | 17 +++++++++++ .../tests/workspaces.rs | 14 ++++++++++ .../src/deployment_requests.rs | 27 ++++++++++++------ .../src/workspaces_extra.rs | 27 ++++++++++++++++-- .../lib/components/ForkWorkspaceBanner.svelte | 6 +++- 7 files changed, 122 insertions(+), 12 deletions(-) create mode 100644 backend/.sqlx/query-40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e.json create mode 100644 backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json create mode 100644 backend/.sqlx/query-a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d.json diff --git a/backend/.sqlx/query-40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e.json b/backend/.sqlx/query-40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e.json new file mode 100644 index 0000000000..6fd7d38f69 --- /dev/null +++ b/backend/.sqlx/query-40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e" +} diff --git a/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json b/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json new file mode 100644 index 0000000000..a6ab79cd7c --- /dev/null +++ b/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\"\n FROM workspace f\n JOIN workspace p ON p.id = f.parent_workspace_id\n WHERE f.id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "deleted!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688" +} diff --git a/backend/.sqlx/query-a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d.json b/backend/.sqlx/query-a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d.json new file mode 100644 index 0000000000..819928ddc1 --- /dev/null +++ b/backend/.sqlx/query-a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id)\n SELECT $1, $2, owner, false, premium,\n CASE WHEN $4 THEN parent_workspace_id ELSE NULL END\n FROM workspace WHERE id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d" +} diff --git a/backend/windmill-api-integration-tests/tests/workspaces.rs b/backend/windmill-api-integration-tests/tests/workspaces.rs index fcbdeea8ba..92f179c39b 100644 --- a/backend/windmill-api-integration-tests/tests/workspaces.rs +++ b/backend/windmill-api-integration-tests/tests/workspaces.rs @@ -646,6 +646,20 @@ async fn test_workspace_endpoints(db: Pool) -> anyhow::Result<()> { .unwrap(); assert_eq!(resp.json::().await?, true); + // Regression: changing a fork's workspace id must preserve its parent + // linkage. Dropping it leaves a wm-fork- workspace with no parent — a + // "fork of nothing" that can no longer be compared or merged. + let parent: Option = + sqlx::query_scalar("SELECT parent_workspace_id FROM workspace WHERE id = $1") + .bind("wm-fork-renamed") + .fetch_one(&db) + .await?; + assert_eq!( + parent.as_deref(), + Some("new-test-ws"), + "renamed fork must keep its parent_workspace_id" + ); + // --- create_fork over an existing (active) workspace id: clear 400, not a raw SQL 500 --- let resp = authed(client().post(format!("{new_ws_base}/create_fork"))) .json(&json!({ diff --git a/backend/windmill-api-workspaces/src/deployment_requests.rs b/backend/windmill-api-workspaces/src/deployment_requests.rs index 7782382073..a279f82d6c 100644 --- a/backend/windmill-api-workspaces/src/deployment_requests.rs +++ b/backend/windmill-api-workspaces/src/deployment_requests.rs @@ -717,16 +717,27 @@ async fn create_deployment_request_comment( // ---- helpers ------------------------------------------------------------ async fn parent_of_fork(db: &DB, w_id: &str) -> Result { - sqlx::query_scalar!( - "SELECT parent_workspace_id FROM workspace WHERE id = $1", + // Resolve the fork's parent and require it to still exist and be active. A + // parent that is archived (soft-deleted) can no longer be accessed, so a + // diff or deployment request against it targets an unreachable workspace. + let parent = sqlx::query!( + "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\" + FROM workspace f + JOIN workspace p ON p.id = f.parent_workspace_id + WHERE f.id = $1", w_id, ) .fetch_optional(db) - .await? - .flatten() - .ok_or_else(|| { - Error::BadRequest(format!( + .await?; + + match parent { + None => Err(Error::BadRequest(format!( "workspace {w_id} is not a fork (no parent_workspace_id)" - )) - }) + ))), + Some(p) if p.deleted => Err(Error::BadRequest(format!( + "parent workspace {} of fork {w_id} is archived", + p.id + ))), + Some(p) => Ok(p.id), + } } diff --git a/backend/windmill-api-workspaces/src/workspaces_extra.rs b/backend/windmill-api-workspaces/src/workspaces_extra.rs index aa64514b4c..f1dfc4677c 100644 --- a/backend/windmill-api-workspaces/src/workspaces_extra.rs +++ b/backend/windmill-api-workspaces/src/workspaces_extra.rs @@ -65,13 +65,22 @@ pub(crate) async fn change_workspace_id( old_id, rw.new_id ); - // Create new workspace with new id and name + // Create new workspace with new id and name. A fork that keeps a wm-fork- + // id must carry its parent_workspace_id over, otherwise it becomes a + // parentless "fork of nothing" with no source to compare or merge against. + // A non-fork target id means the workspace is being promoted out of a fork, + // so the parent pointer is intentionally cleared. info!("Creating new workspace row"); + let new_is_fork = rw.new_id.starts_with(WM_FORK_PREFIX); sqlx::query!( - "INSERT INTO workspace SELECT $1, $2, owner, false, premium FROM workspace WHERE id = $3", + "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id) + SELECT $1, $2, owner, false, premium, + CASE WHEN $4 THEN parent_workspace_id ELSE NULL END + FROM workspace WHERE id = $3", &rw.new_id, &rw.new_name, - &old_id + &old_id, + new_is_fork ) .execute(&mut *tx) .await?; @@ -347,6 +356,18 @@ pub(crate) async fn change_workspace_id( .execute(&mut *tx) .await?; + // Re-parent child forks: any fork whose parent_workspace_id was the old id + // must follow the renamed parent to the new id, otherwise it is left + // pointing at the soft-deleted old shell (whose data has moved here). + info!("Re-parenting child forks to the new workspace id"); + sqlx::query!( + "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2", + &rw.new_id, + &old_id + ) + .execute(&mut *tx) + .await?; + info!("Updating workspace_protection_rule table"); sqlx::query!( "UPDATE workspace_protection_rule SET workspace_id = $1 WHERE workspace_id = $2", diff --git a/frontend/src/lib/components/ForkWorkspaceBanner.svelte b/frontend/src/lib/components/ForkWorkspaceBanner.svelte index 56418c1988..76740163ed 100644 --- a/frontend/src/lib/components/ForkWorkspaceBanner.svelte +++ b/frontend/src/lib/components/ForkWorkspaceBanner.svelte @@ -12,10 +12,14 @@ let comparison: WorkspaceComparison | undefined = $state(undefined) let error: string | undefined = $state(undefined) - let isFork = $derived($workspaceStore?.startsWith('wm-fork-') ?? false) let currentWorkspaceData = $derived($userWorkspaces.find((w) => w.id === $workspaceStore)) let parentWorkspaceId = $derived(currentWorkspaceData?.parent_workspace_id) let parentWorkspaceData = $derived($userWorkspaces.find((w) => w.id === parentWorkspaceId)) + // A fork must have a parent to compare/merge against. Treating the wm-fork- + // prefix alone as "is a fork" renders a parentless "Fork of ()" banner when + // the parent linkage was dropped (e.g. by a workspace id change), so require + // both, matching the forks/compare page. + let isFork = $derived(($workspaceStore?.startsWith('wm-fork-') ?? false) && !!parentWorkspaceId) // Drafts in this fork. When the fork is otherwise in sync with its parent, a // user with only pending drafts should still get the draft CTA (mirrors the From cfb9f1dbc23110ecf8f91bb3c8c81fc6e35dc09b Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 23:02:10 +0200 Subject: [PATCH 040/117] feat: render mermaid diagrams in chat code blocks (#9738) * feat: render mermaid diagrams in chat code blocks Co-Authored-By: Claude Opus 4.8 (1M context) * fix: guard mermaid render against out-of-order async and transient streaming failures Co-Authored-By: Claude Opus 4.8 (1M context) * fix: only show mermaid diagram while it matches current source Addresses Codex review: keeping the last good SVG through parse failures left a stale, mismatched diagram on screen when the source changed to something invalid. Tie the rendered SVG to the source that produced it and only display it while it still matches the current code, falling back to the raw source otherwise. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- frontend/package-lock.json | 765 ++++++++++++++++-- frontend/package.json | 1 + .../copilot/chat/script/CodeDisplay.svelte | 23 +- .../copilot/chat/script/MermaidDisplay.svelte | 62 ++ 4 files changed, 766 insertions(+), 85 deletions(-) create mode 100644 frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 8302448dfb..24af2dc2ed 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -52,6 +52,7 @@ "lru-cache": "^11.1.0", "lucide-svelte": "^0.540.0", "mdast-util-find-and-replace": "^3.0.2", + "mermaid": "^11.15.0", "minimatch": "^10.0.1", "monaco-editor": "npm:@codingame/monaco-vscode-editor-api@=25.0.0", "monaco-languageclient": "10.6.0", @@ -179,6 +180,28 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/@antfu/install-pkg": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@antfu/install-pkg/-/install-pkg-1.1.0.tgz", + "integrity": "sha512-MGQsmw10ZyI+EJo45CdSER4zEb+p31LpDAFp2Z3gkSd1yqVZGi0Ebx++YTEMonJy4oChEMLsxZ64j8FH6sSqtQ==", + "license": "MIT", + "dependencies": { + "package-manager-detector": "^1.3.0", + "tinyexec": "^1.0.1" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/@antfu/install-pkg/node_modules/tinyexec": { + "version": "1.2.4", + "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-1.2.4.tgz", + "integrity": "sha512-SHf/r48b7vOrjve9PxJo3MN5v5yuyjHvdUcrQffT3WXMUfnGmHDVbC4k3sHJaJTgZCwpUplIaAo5ANtMyp3YHg==", + "license": "MIT", + "engines": { + "node": ">=18" + } + }, "node_modules/@anthropic-ai/sdk": { "version": "0.60.0", "resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.60.0.tgz", @@ -298,6 +321,18 @@ "node": ">=6.9.0" } }, + "node_modules/@braintree/sanitize-url": { + "version": "7.1.2", + "resolved": "https://registry.npmjs.org/@braintree/sanitize-url/-/sanitize-url-7.1.2.tgz", + "integrity": "sha512-jigsZK+sMF/cuiB7sERuo9V7N9jx+dhmHHnQyDSVdpZwVutaBu7WvNYqMDLSgFgfB30n452TP3vjDAvFC973mA==", + "license": "MIT" + }, + "node_modules/@chevrotain/types": { + "version": "11.1.2", + "resolved": "https://registry.npmjs.org/@chevrotain/types/-/types-11.1.2.tgz", + "integrity": "sha512-U+HFai5+zmJCkK86QsaJtoITlboZHBqrVketcO2ROv865xfCMSFpELQoz1GkX5GzME8pTa+3kbKrZHQtI0gdbw==", + "license": "Apache-2.0" + }, "node_modules/@codingame/monaco-vscode-api": { "version": "25.0.0", "resolved": "https://registry.npmjs.org/@codingame/monaco-vscode-api/-/monaco-vscode-api-25.0.0.tgz", @@ -843,7 +878,6 @@ "version": "1.10.0", "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.10.0.tgz", "integrity": "sha512-yq6OkJ4p82CAfPl0u9mQebQHKPJkY7WrIuk205cTYnYe+k2Z8YBh11FrbRG/H6ihirqcacOgl2BIO8oyMQLeXw==", - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -855,7 +889,6 @@ "version": "1.10.0", "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz", "integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==", - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -866,7 +899,6 @@ "version": "1.2.1", "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.1.tgz", "integrity": "sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w==", - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1081,6 +1113,23 @@ "dev": true, "license": "BSD-3-Clause" }, + "node_modules/@iconify/types": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/@iconify/types/-/types-2.0.0.tgz", + "integrity": "sha512-+wluvCrRhXrhyOmRDJ3q8mux9JkKy5SJ/v8ol2tu4FVjyYvtEzkc/3pK15ET6RKg4b4w4BmTk1+gsCUhf21Ykg==", + "license": "MIT" + }, + "node_modules/@iconify/utils": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/@iconify/utils/-/utils-3.1.3.tgz", + "integrity": "sha512-LPKOXPn/zV+zis1oOfGWogaXVpqUybF3ZS6SCZIsz8vg0ivVp9+fVqyYB7xq0aiST/VhUQYGO1qo6uoYSiEJqw==", + "license": "MIT", + "dependencies": { + "@antfu/install-pkg": "^1.1.0", + "@iconify/types": "^2.0.0", + "import-meta-resolve": "^4.2.0" + } + }, "node_modules/@internationalized/date": { "version": "3.10.0", "resolved": "https://registry.npmjs.org/@internationalized/date/-/date-3.10.0.tgz", @@ -1352,11 +1401,19 @@ "svelte": "^3.0.0 || ^4.0.0 || ^5.0.0-next.118" } }, + "node_modules/@mermaid-js/parser": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@mermaid-js/parser/-/parser-1.1.1.tgz", + "integrity": "sha512-VuHdsYMK1bT6X2JbcAaWAhugTRvRBRyuZgd+c22swUeI9g/ntaxF7CY7dYarhZovofCbUNO0G7JesfmNtjYOCw==", + "license": "MIT", + "dependencies": { + "@chevrotain/types": "~11.1.1" + } + }, "node_modules/@napi-rs/wasm-runtime": { "version": "1.1.4", "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.4.tgz", "integrity": "sha512-3NQNNgA1YSlJb/kMH1ildASP9HW7/7kYnRI2szWJaofaS1hWmbGI4H+d3+22aGzXXN9IJ+n+GiFVcGipJP18ow==", - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1505,7 +1562,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1522,7 +1578,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1539,7 +1594,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1556,7 +1610,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1573,7 +1626,6 @@ "cpu": [ "arm" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1590,7 +1642,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1607,7 +1658,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1624,7 +1674,6 @@ "cpu": [ "ppc64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1641,7 +1690,6 @@ "cpu": [ "s390x" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1658,7 +1706,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1675,7 +1722,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1692,7 +1738,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1709,7 +1754,6 @@ "cpu": [ "wasm32" ], - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1728,7 +1772,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1745,7 +1788,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -2051,7 +2093,6 @@ "version": "0.10.2", "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.2.tgz", "integrity": "sha512-RoBvJ2X0wuKlWFIjrwffGw1IqZHKQqzIchKaadZZfnNpsAYp2mM0h36JtPCjNDAHGgYez/15uMBpfGwchhiMgg==", - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -2080,7 +2121,6 @@ "version": "7.4.3", "resolved": "https://registry.npmjs.org/@types/d3/-/d3-7.4.3.tgz", "integrity": "sha512-lZXZ9ckh5R8uiFVt8ogUNf+pIrK4EsWrx2Np75WvF/eTpJ0FMHNhjXk8CKEx/+gpHbNQyJWehbFaTvqmHWB3ww==", - "dev": true, "license": "MIT", "dependencies": { "@types/d3-array": "*", @@ -2119,14 +2159,12 @@ "version": "3.2.2", "resolved": "https://registry.npmjs.org/@types/d3-array/-/d3-array-3.2.2.tgz", "integrity": "sha512-hOLWVbm7uRza0BYXpIIW5pxfrKe0W+D5lrFiAEYR+pb6w3N2SwSMaJbXdUfSEv+dT4MfHBLtn5js0LAWaO6otw==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-axis": { "version": "3.0.6", "resolved": "https://registry.npmjs.org/@types/d3-axis/-/d3-axis-3.0.6.tgz", "integrity": "sha512-pYeijfZuBd87T0hGn0FO1vQ/cgLk6E1ALJjfkC0oJ8cbwkZl3TpgS8bVBLZN+2jjGgg38epgxb2zmoGtSfvgMw==", - "dev": true, "license": "MIT", "dependencies": { "@types/d3-selection": "*" @@ -2136,7 +2174,6 @@ "version": "3.0.6", "resolved": "https://registry.npmjs.org/@types/d3-brush/-/d3-brush-3.0.6.tgz", "integrity": "sha512-nH60IZNNxEcrh6L1ZSMNA28rj27ut/2ZmI3r96Zd+1jrZD++zD3LsMIjWlvg4AYrHn/Pqz4CF3veCxGjtbqt7A==", - "dev": true, "license": "MIT", "dependencies": { "@types/d3-selection": "*" @@ -2146,7 +2183,6 @@ "version": "3.0.6", "resolved": "https://registry.npmjs.org/@types/d3-chord/-/d3-chord-3.0.6.tgz", "integrity": "sha512-LFYWWd8nwfwEmTZG9PfQxd17HbNPksHBiJHaKuY1XeqscXacsS2tyoo6OdRsjf+NQYeB6XrNL3a25E3gH69lcg==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-color": { @@ -2159,7 +2195,6 @@ "version": "3.0.6", "resolved": "https://registry.npmjs.org/@types/d3-contour/-/d3-contour-3.0.6.tgz", "integrity": "sha512-BjzLgXGnCWjUSYGfH1cpdo41/hgdWETu4YxpezoztawmqsvCeep+8QGfiY6YbDvfgHz/DkjeIkkZVJavB4a3rg==", - "dev": true, "license": "MIT", "dependencies": { "@types/d3-array": "*", @@ -2170,14 +2205,12 @@ "version": "6.0.4", "resolved": "https://registry.npmjs.org/@types/d3-delaunay/-/d3-delaunay-6.0.4.tgz", "integrity": "sha512-ZMaSKu4THYCU6sV64Lhg6qjf1orxBthaC161plr5KuPHo3CNm8DTHiLw/5Eq2b6TsNP0W0iJrUOFscY6Q450Hw==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-dispatch": { "version": "3.0.7", "resolved": "https://registry.npmjs.org/@types/d3-dispatch/-/d3-dispatch-3.0.7.tgz", "integrity": "sha512-5o9OIAdKkhN1QItV2oqaE5KMIiXAvDWBDPrD85e58Qlz1c1kI/J0NcqbEG88CoTwJrYe7ntUCVfeUl2UJKbWgA==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-drag": { @@ -2193,21 +2226,18 @@ "version": "3.0.7", "resolved": "https://registry.npmjs.org/@types/d3-dsv/-/d3-dsv-3.0.7.tgz", "integrity": "sha512-n6QBF9/+XASqcKK6waudgL0pf/S5XHPPI8APyMLLUHd8NqouBGLsU8MgtO7NINGtPBtk9Kko/W4ea0oAspwh9g==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-ease": { "version": "3.0.2", "resolved": "https://registry.npmjs.org/@types/d3-ease/-/d3-ease-3.0.2.tgz", "integrity": "sha512-NcV1JjO5oDzoK26oMzbILE6HW7uVXOHLQvHshBUW4UMdZGfiY6v5BeQwh9a9tCzv+CeefZQHJt5SRgK154RtiA==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-fetch": { "version": "3.0.7", "resolved": "https://registry.npmjs.org/@types/d3-fetch/-/d3-fetch-3.0.7.tgz", "integrity": "sha512-fTAfNmxSb9SOWNB9IoG5c8Hg6R+AzUHDRlsXsDZsNp6sxAEOP0tkP3gKkNSO/qmHPoBFTxNrjDprVHDQDvo5aA==", - "dev": true, "license": "MIT", "dependencies": { "@types/d3-dsv": "*" @@ -2217,21 +2247,18 @@ "version": "3.0.10", "resolved": "https://registry.npmjs.org/@types/d3-force/-/d3-force-3.0.10.tgz", "integrity": "sha512-ZYeSaCF3p73RdOKcjj+swRlZfnYpK1EbaDiYICEEp5Q6sUiqFaFQ9qgoshp5CzIyyb/yD09kD9o2zEltCexlgw==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-format": { "version": "3.0.4", "resolved": "https://registry.npmjs.org/@types/d3-format/-/d3-format-3.0.4.tgz", "integrity": "sha512-fALi2aI6shfg7vM5KiR1wNJnZ7r6UuggVqtDA+xiEdPZQwy/trcQaHnwShLuLdta2rTymCNpxYTiMZX/e09F4g==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-geo": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/@types/d3-geo/-/d3-geo-3.1.0.tgz", "integrity": "sha512-856sckF0oP/diXtS4jNsiQw/UuK5fQG8l/a9VVLeSouf1/PPbBE1i1W852zVwKwYCBkFJJB7nCFTbk6UMEXBOQ==", - "dev": true, "license": "MIT", "dependencies": { "@types/geojson": "*" @@ -2241,7 +2268,6 @@ "version": "3.1.7", "resolved": "https://registry.npmjs.org/@types/d3-hierarchy/-/d3-hierarchy-3.1.7.tgz", "integrity": "sha512-tJFtNoYBtRtkNysX1Xq4sxtjK8YgoWUNpIiUee0/jHGRwqvzYxkq0hGVbbOGSz+JgFxxRu4K8nb3YpG3CMARtg==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-interpolate": { @@ -2257,35 +2283,30 @@ "version": "3.1.1", "resolved": "https://registry.npmjs.org/@types/d3-path/-/d3-path-3.1.1.tgz", "integrity": "sha512-VMZBYyQvbGmWyWVea0EHs/BwLgxc+MKi1zLDCONksozI4YJMcTt8ZEuIR4Sb1MMTE8MMW49v0IwI5+b7RmfWlg==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-polygon": { "version": "3.0.2", "resolved": "https://registry.npmjs.org/@types/d3-polygon/-/d3-polygon-3.0.2.tgz", "integrity": "sha512-ZuWOtMaHCkN9xoeEMr1ubW2nGWsp4nIql+OPQRstu4ypeZ+zk3YKqQT0CXVe/PYqrKpZAi+J9mTs05TKwjXSRA==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-quadtree": { "version": "3.0.6", "resolved": "https://registry.npmjs.org/@types/d3-quadtree/-/d3-quadtree-3.0.6.tgz", "integrity": "sha512-oUzyO1/Zm6rsxKRHA1vH0NEDG58HrT5icx/azi9MF1TWdtttWl0UIUsjEQBBh+SIkrpd21ZjEv7ptxWys1ncsg==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-random": { "version": "3.0.3", "resolved": "https://registry.npmjs.org/@types/d3-random/-/d3-random-3.0.3.tgz", "integrity": "sha512-Imagg1vJ3y76Y2ea0871wpabqp613+8/r0mCLEBfdtqC7xMSfj9idOnmBYyMoULfHePJyxMAw3nWhJxzc+LFwQ==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-scale": { "version": "4.0.9", "resolved": "https://registry.npmjs.org/@types/d3-scale/-/d3-scale-4.0.9.tgz", "integrity": "sha512-dLmtwB8zkAeO/juAMfnV+sItKjlsw2lKdZVVy6LRr0cBmegxSABiLEpGVmSJJ8O08i4+sGR6qQtb6WtuwJdvVw==", - "dev": true, "license": "MIT", "dependencies": { "@types/d3-time": "*" @@ -2295,7 +2316,6 @@ "version": "3.1.0", "resolved": "https://registry.npmjs.org/@types/d3-scale-chromatic/-/d3-scale-chromatic-3.1.0.tgz", "integrity": "sha512-iWMJgwkK7yTRmWqRB5plb1kadXyQ5Sj8V/zYlFGMUBbIPKQScw+Dku9cAAMgJG+z5GYDoMjWGLVOvjghDEFnKQ==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-selection": { @@ -2308,7 +2328,6 @@ "version": "3.1.7", "resolved": "https://registry.npmjs.org/@types/d3-shape/-/d3-shape-3.1.7.tgz", "integrity": "sha512-VLvUQ33C+3J+8p+Daf+nYSOsjB4GXp19/S/aGo60m9h1v6XaxjiT82lKVWJCfzhtuZ3yD7i/TPeC/fuKLLOSmg==", - "dev": true, "license": "MIT", "dependencies": { "@types/d3-path": "*" @@ -2318,21 +2337,18 @@ "version": "3.0.4", "resolved": "https://registry.npmjs.org/@types/d3-time/-/d3-time-3.0.4.tgz", "integrity": "sha512-yuzZug1nkAAaBlBBikKZTgzCeA+k1uy4ZFwWANOfKw5z5LRhV0gNA7gNkKm7HoK+HRN0wX3EkxGk0fpbWhmB7g==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-time-format": { "version": "4.0.3", "resolved": "https://registry.npmjs.org/@types/d3-time-format/-/d3-time-format-4.0.3.tgz", "integrity": "sha512-5xg9rC+wWL8kdDj153qZcsJ0FWiFt0J5RB6LYUNZjwSnesfblqrI/bJ1wBdJ8OQfncgbJG5+2F+qfqnqyzYxyg==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-timer": { "version": "3.0.2", "resolved": "https://registry.npmjs.org/@types/d3-timer/-/d3-timer-3.0.2.tgz", "integrity": "sha512-Ps3T8E8dZDam6fUyNiMkekK3XUsaUEik+idO9/YjPtfj2qruF8tFBXS7XhtE4iIXBLxhmLjP3SXpLhVf21I9Lw==", - "dev": true, "license": "MIT" }, "node_modules/@types/d3-transition": { @@ -2387,7 +2403,6 @@ "version": "7946.0.16", "resolved": "https://registry.npmjs.org/@types/geojson/-/geojson-7946.0.16.tgz", "integrity": "sha512-6C8nqWur3j98U6+lXDfTUWIfgvZU+EumvpHKcYjujKH7woYyLj2sUmff0tRhrqM7BohUw7Pz3ZB1jj2gW9Fvmg==", - "dev": true, "license": "MIT" }, "node_modules/@types/hammerjs": { @@ -2795,6 +2810,16 @@ "integrity": "sha512-WmoN8qaIAo7WTYWbAZuG8PYEhn5fkz7dZrqTBZ7dtt//lL2Gwms1IcnQ5yHqjDfX8Ft5j4YzDM23f87zBfDe9g==", "license": "ISC" }, + "node_modules/@upsetjs/venn.js": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/@upsetjs/venn.js/-/venn.js-2.0.0.tgz", + "integrity": "sha512-WbBhLrooyePuQ1VZxrJjtLvTc4NVfpOyKx0sKqioq9bX1C1m7Jgykkn8gLrtwumBioXIqam8DLxp88Adbue6Hw==", + "license": "MIT", + "optionalDependencies": { + "d3-selection": "^3.0.0", + "d3-transition": "^3.0.1" + } + }, "node_modules/@vitest/expect": { "version": "4.1.0", "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.0.tgz", @@ -4084,6 +4109,15 @@ "integrity": "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ==", "license": "MIT" }, + "node_modules/cose-base": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/cose-base/-/cose-base-1.0.3.tgz", + "integrity": "sha512-s9whTXInMSgAp/NVXVNuVxVKzGH2qck3aQlVHxDCdAEPgtMKwc4Wq6/QKhgdEdgbLSi9rBTAcPoRa6JpiG4ksg==", + "license": "MIT", + "dependencies": { + "layout-base": "^1.0.0" + } + }, "node_modules/cosmiconfig": { "version": "8.3.6", "resolved": "https://registry.npmjs.org/cosmiconfig/-/cosmiconfig-8.3.6.tgz", @@ -4338,11 +4372,99 @@ "dev": true, "license": "CC0-1.0" }, + "node_modules/cytoscape": { + "version": "3.34.0", + "resolved": "https://registry.npmjs.org/cytoscape/-/cytoscape-3.34.0.tgz", + "integrity": "sha512-62rNSrioXw93uliKFBwjukeQyeWwH2PqDrTac31r2P6464u3AUvTk0xS4LVvT251g7IgkFunrI48ZEZGjywSOg==", + "license": "MIT", + "engines": { + "node": ">=0.10" + } + }, + "node_modules/cytoscape-cose-bilkent": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/cytoscape-cose-bilkent/-/cytoscape-cose-bilkent-4.1.0.tgz", + "integrity": "sha512-wgQlVIUJF13Quxiv5e1gstZ08rnZj2XaLHGoFMYXz7SkNfCDOOteKBE6SYRfA9WxxI/iBc3ajfDoc6hb/MRAHQ==", + "license": "MIT", + "dependencies": { + "cose-base": "^1.0.0" + }, + "peerDependencies": { + "cytoscape": "^3.2.0" + } + }, + "node_modules/cytoscape-fcose": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/cytoscape-fcose/-/cytoscape-fcose-2.2.0.tgz", + "integrity": "sha512-ki1/VuRIHFCzxWNrsshHYPs6L7TvLu3DL+TyIGEsRcvVERmxokbf5Gdk7mFxZnTdiGtnA4cfSmjZJMviqSuZrQ==", + "license": "MIT", + "dependencies": { + "cose-base": "^2.2.0" + }, + "peerDependencies": { + "cytoscape": "^3.2.0" + } + }, + "node_modules/cytoscape-fcose/node_modules/cose-base": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/cose-base/-/cose-base-2.2.0.tgz", + "integrity": "sha512-AzlgcsCbUMymkADOJtQm3wO9S3ltPfYOFD5033keQn9NJzIbtnZj+UdBJe7DYml/8TdbtHJW3j58SOnKhWY/5g==", + "license": "MIT", + "dependencies": { + "layout-base": "^2.0.0" + } + }, + "node_modules/cytoscape-fcose/node_modules/layout-base": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/layout-base/-/layout-base-2.0.1.tgz", + "integrity": "sha512-dp3s92+uNI1hWIpPGH3jK2kxE2lMjdXdr+DH8ynZHpd6PUlH6x6cbuXnoMmiNumznqaNO31xu9e79F0uuZ0JFg==", + "license": "MIT" + }, + "node_modules/d3": { + "version": "7.9.0", + "resolved": "https://registry.npmjs.org/d3/-/d3-7.9.0.tgz", + "integrity": "sha512-e1U46jVP+w7Iut8Jt8ri1YsPOvFpg46k+K8TpCb0P+zjCkjkPnV7WzfDJzMHy1LnA+wj5pLT1wjO901gLXeEhA==", + "license": "ISC", + "dependencies": { + "d3-array": "3", + "d3-axis": "3", + "d3-brush": "3", + "d3-chord": "3", + "d3-color": "3", + "d3-contour": "4", + "d3-delaunay": "6", + "d3-dispatch": "3", + "d3-drag": "3", + "d3-dsv": "3", + "d3-ease": "3", + "d3-fetch": "3", + "d3-force": "3", + "d3-format": "3", + "d3-geo": "3", + "d3-hierarchy": "3", + "d3-interpolate": "3", + "d3-path": "3", + "d3-polygon": "3", + "d3-quadtree": "3", + "d3-random": "3", + "d3-scale": "4", + "d3-scale-chromatic": "3", + "d3-selection": "3", + "d3-shape": "3", + "d3-time": "3", + "d3-time-format": "4", + "d3-timer": "3", + "d3-transition": "3", + "d3-zoom": "3" + }, + "engines": { + "node": ">=12" + } + }, "node_modules/d3-array": { "version": "3.2.4", "resolved": "https://registry.npmjs.org/d3-array/-/d3-array-3.2.4.tgz", "integrity": "sha512-tdQAmyA18i4J7wprpYq8ClcxZy3SC31QMeByyCFyRt7BVHdREQZ5lpzoe5mFEYZUWe+oq8HBvk9JjpibyEV4Jg==", - "dev": true, "license": "ISC", "dependencies": { "internmap": "1 - 2" @@ -4351,6 +4473,43 @@ "node": ">=12" } }, + "node_modules/d3-axis": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/d3-axis/-/d3-axis-3.0.0.tgz", + "integrity": "sha512-IH5tgjV4jE/GhHkRV0HiVYPDtvfjHQlQfJHs0usq7M30XcSBvOotpmH1IgkcXsO/5gEQZD43B//fc7SRT5S+xw==", + "license": "ISC", + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-brush": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/d3-brush/-/d3-brush-3.0.0.tgz", + "integrity": "sha512-ALnjWlVYkXsVIGlOsuWH1+3udkYFI48Ljihfnh8FZPF2QS9o+PzGLBslO0PjzVoHLZ2KCVgAM8NVkXPJB2aNnQ==", + "license": "ISC", + "dependencies": { + "d3-dispatch": "1 - 3", + "d3-drag": "2 - 3", + "d3-interpolate": "1 - 3", + "d3-selection": "3", + "d3-transition": "3" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-chord": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/d3-chord/-/d3-chord-3.0.1.tgz", + "integrity": "sha512-VE5S6TNa+j8msksl7HwjxMHDM2yNK3XCkusIlpX5kwauBfXuyLAtNg9jCp/iHH61tgI4sb6R/EIMWCqEIdjT/g==", + "license": "ISC", + "dependencies": { + "d3-path": "1 - 3" + }, + "engines": { + "node": ">=12" + } + }, "node_modules/d3-color": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/d3-color/-/d3-color-3.1.0.tgz", @@ -4360,6 +4519,18 @@ "node": ">=12" } }, + "node_modules/d3-contour": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/d3-contour/-/d3-contour-4.0.2.tgz", + "integrity": "sha512-4EzFTRIikzs47RGmdxbeUvLWtGedDUNkTcmzoeyg4sP/dvCexO47AaQL7VKy/gul85TOxw+IBgA8US2xwbToNA==", + "license": "ISC", + "dependencies": { + "d3-array": "^3.2.0" + }, + "engines": { + "node": ">=12" + } + }, "node_modules/d3-dag": { "version": "0.11.5", "resolved": "https://registry.npmjs.org/d3-dag/-/d3-dag-0.11.5.tgz", @@ -4373,6 +4544,18 @@ "quadprog": "^1.6.1" } }, + "node_modules/d3-delaunay": { + "version": "6.0.4", + "resolved": "https://registry.npmjs.org/d3-delaunay/-/d3-delaunay-6.0.4.tgz", + "integrity": "sha512-mdjtIZ1XLAM8bm/hx3WwjfHt6Sggek7qH043O8KEjDXN40xi3vx/6pYSVTwLjEgiXQTbvaouWKynLBiUZ6SK6A==", + "license": "ISC", + "dependencies": { + "delaunator": "5" + }, + "engines": { + "node": ">=12" + } + }, "node_modules/d3-dispatch": { "version": "3.0.1", "resolved": "https://registry.npmjs.org/d3-dispatch/-/d3-dispatch-3.0.1.tgz", @@ -4395,6 +4578,40 @@ "node": ">=12" } }, + "node_modules/d3-dsv": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/d3-dsv/-/d3-dsv-3.0.1.tgz", + "integrity": "sha512-UG6OvdI5afDIFP9w4G0mNq50dSOsXHJaRE8arAS5o9ApWnIElp8GZw1Dun8vP8OyHOZ/QJUKUJwxiiCCnUwm+Q==", + "license": "ISC", + "dependencies": { + "commander": "7", + "iconv-lite": "0.6", + "rw": "1" + }, + "bin": { + "csv2json": "bin/dsv2json.js", + "csv2tsv": "bin/dsv2dsv.js", + "dsv2dsv": "bin/dsv2dsv.js", + "dsv2json": "bin/dsv2json.js", + "json2csv": "bin/json2dsv.js", + "json2dsv": "bin/json2dsv.js", + "json2tsv": "bin/json2dsv.js", + "tsv2csv": "bin/dsv2dsv.js", + "tsv2json": "bin/dsv2json.js" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-dsv/node_modules/commander": { + "version": "7.2.0", + "resolved": "https://registry.npmjs.org/commander/-/commander-7.2.0.tgz", + "integrity": "sha512-QrWXB+ZQSVPmIWIhtEO9H+gwHaMGYiF5ChvoJ+K9ZGHG/sVsa6yiesAD1GC/x46sET00Xlwo1u49RVVVzvcSkw==", + "license": "MIT", + "engines": { + "node": ">= 10" + } + }, "node_modules/d3-ease": { "version": "3.0.1", "resolved": "https://registry.npmjs.org/d3-ease/-/d3-ease-3.0.1.tgz", @@ -4404,6 +4621,62 @@ "node": ">=12" } }, + "node_modules/d3-fetch": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/d3-fetch/-/d3-fetch-3.0.1.tgz", + "integrity": "sha512-kpkQIM20n3oLVBKGg6oHrUchHM3xODkTzjMoj7aWQFq5QEM+R6E4WkzT5+tojDY7yjez8KgCBRoj4aEr99Fdqw==", + "license": "ISC", + "dependencies": { + "d3-dsv": "1 - 3" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-force": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/d3-force/-/d3-force-3.0.0.tgz", + "integrity": "sha512-zxV/SsA+U4yte8051P4ECydjD/S+qeYtnaIyAs9tgHCqfguma/aAQDjo85A9Z6EKhBirHRJHXIgJUlffT4wdLg==", + "license": "ISC", + "dependencies": { + "d3-dispatch": "1 - 3", + "d3-quadtree": "1 - 3", + "d3-timer": "1 - 3" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-format": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/d3-format/-/d3-format-3.1.2.tgz", + "integrity": "sha512-AJDdYOdnyRDV5b6ArilzCPPwc1ejkHcoyFarqlPqT7zRYjhavcT3uSrqcMvsgh2CgoPbK3RCwyHaVyxYcP2Arg==", + "license": "ISC", + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-geo": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/d3-geo/-/d3-geo-3.1.1.tgz", + "integrity": "sha512-637ln3gXKXOwhalDzinUgY83KzNWZRKbYubaG+fGVuc/dxO64RRljtCTnf5ecMyE1RIdtqpkVcq0IbtU2S8j2Q==", + "license": "ISC", + "dependencies": { + "d3-array": "2.5.0 - 3" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-hierarchy": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/d3-hierarchy/-/d3-hierarchy-3.1.2.tgz", + "integrity": "sha512-FX/9frcub54beBdugHjDCdikxThEqjnR93Qt7PvQTOHxyiNCAlvMrHhclk3cD5VeAaq9fxmfRp+CnWw9rEMBuA==", + "license": "ISC", + "engines": { + "node": ">=12" + } + }, "node_modules/d3-interpolate": { "version": "3.0.1", "resolved": "https://registry.npmjs.org/d3-interpolate/-/d3-interpolate-3.0.1.tgz", @@ -4416,6 +4689,111 @@ "node": ">=12" } }, + "node_modules/d3-path": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/d3-path/-/d3-path-3.1.0.tgz", + "integrity": "sha512-p3KP5HCf/bvjBSSKuXid6Zqijx7wIfNW+J/maPs+iwR35at5JCbLUT0LzF1cnjbCHWhqzQTIN2Jpe8pRebIEFQ==", + "license": "ISC", + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-polygon": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/d3-polygon/-/d3-polygon-3.0.1.tgz", + "integrity": "sha512-3vbA7vXYwfe1SYhED++fPUQlWSYTTGmFmQiany/gdbiWgU/iEyQzyymwL9SkJjFFuCS4902BSzewVGsHHmHtXg==", + "license": "ISC", + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-quadtree": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/d3-quadtree/-/d3-quadtree-3.0.1.tgz", + "integrity": "sha512-04xDrxQTDTCFwP5H6hRhsRcb9xxv2RzkcsygFzmkSIOJy3PeRJP7sNk3VRIbKXcog561P9oU0/rVH6vDROAgUw==", + "license": "ISC", + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-random": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/d3-random/-/d3-random-3.0.1.tgz", + "integrity": "sha512-FXMe9GfxTxqd5D6jFsQ+DJ8BJS4E/fT5mqqdjovykEB2oFbTMDVdg1MGFxfQW+FBOGoB++k8swBrgwSHT1cUXQ==", + "license": "ISC", + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-sankey": { + "version": "0.12.3", + "resolved": "https://registry.npmjs.org/d3-sankey/-/d3-sankey-0.12.3.tgz", + "integrity": "sha512-nQhsBRmM19Ax5xEIPLMY9ZmJ/cDvd1BG3UVvt5h3WRxKg5zGRbvnteTyWAbzeSvlh3tW7ZEmq4VwR5mB3tutmQ==", + "license": "BSD-3-Clause", + "dependencies": { + "d3-array": "1 - 2", + "d3-shape": "^1.2.0" + } + }, + "node_modules/d3-sankey/node_modules/d3-array": { + "version": "2.12.1", + "resolved": "https://registry.npmjs.org/d3-array/-/d3-array-2.12.1.tgz", + "integrity": "sha512-B0ErZK/66mHtEsR1TkPEEkwdy+WDesimkM5gpZr5Dsg54BiTA5RXtYW5qTLIAcekaS9xfZrzBLF/OAkB3Qn1YQ==", + "license": "BSD-3-Clause", + "dependencies": { + "internmap": "^1.0.0" + } + }, + "node_modules/d3-sankey/node_modules/d3-path": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/d3-path/-/d3-path-1.0.9.tgz", + "integrity": "sha512-VLaYcn81dtHVTjEHd8B+pbe9yHWpXKZUC87PzoFmsFrJqgFwDe/qxfp5MlfsfM1V5E/iVt0MmEbWQ7FVIXh/bg==", + "license": "BSD-3-Clause" + }, + "node_modules/d3-sankey/node_modules/d3-shape": { + "version": "1.3.7", + "resolved": "https://registry.npmjs.org/d3-shape/-/d3-shape-1.3.7.tgz", + "integrity": "sha512-EUkvKjqPFUAZyOlhY5gzCxCeI0Aep04LwIRpsZ/mLFelJiUfnK56jo5JMDSE7yyP2kLSb6LtF+S5chMk7uqPqw==", + "license": "BSD-3-Clause", + "dependencies": { + "d3-path": "1" + } + }, + "node_modules/d3-sankey/node_modules/internmap": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/internmap/-/internmap-1.0.1.tgz", + "integrity": "sha512-lDB5YccMydFBtasVtxnZ3MRBHuaoE8GKsppq+EchKL2U4nK/DmEpPHNH8MZe5HkMtpSiTSOZwfN0tzYjO/lJEw==", + "license": "ISC" + }, + "node_modules/d3-scale": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/d3-scale/-/d3-scale-4.0.2.tgz", + "integrity": "sha512-GZW464g1SH7ag3Y7hXjf8RoUuAFIqklOAq3MRl4OaWabTFJY9PN/E1YklhXLh+OQ3fM9yS2nOkCoS+WLZ6kvxQ==", + "license": "ISC", + "dependencies": { + "d3-array": "2.10.0 - 3", + "d3-format": "1 - 3", + "d3-interpolate": "1.2.0 - 3", + "d3-time": "2.1.1 - 3", + "d3-time-format": "2 - 4" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-scale-chromatic": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/d3-scale-chromatic/-/d3-scale-chromatic-3.1.0.tgz", + "integrity": "sha512-A3s5PWiZ9YCXFye1o246KoscMWqf8BsD9eRiJ3He7C9OBaxKhAd5TFCdEx/7VbKtxxTsu//1mMJFrEt572cEyQ==", + "license": "ISC", + "dependencies": { + "d3-color": "1 - 3", + "d3-interpolate": "1 - 3" + }, + "engines": { + "node": ">=12" + } + }, "node_modules/d3-selection": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/d3-selection/-/d3-selection-3.0.0.tgz", @@ -4425,6 +4803,42 @@ "node": ">=12" } }, + "node_modules/d3-shape": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/d3-shape/-/d3-shape-3.2.0.tgz", + "integrity": "sha512-SaLBuwGm3MOViRq2ABk3eLoxwZELpH6zhl3FbAoJ7Vm1gofKx6El1Ib5z23NUEhF9AsGl7y+dzLe5Cw2AArGTA==", + "license": "ISC", + "dependencies": { + "d3-path": "^3.1.0" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-time": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/d3-time/-/d3-time-3.1.0.tgz", + "integrity": "sha512-VqKjzBLejbSMT4IgbmVgDjpkYrNWUYJnbCGo874u7MMKIWsILRX+OpX/gTk8MqjpT1A/c6HY2dCA77ZN0lkQ2Q==", + "license": "ISC", + "dependencies": { + "d3-array": "2 - 3" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/d3-time-format": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/d3-time-format/-/d3-time-format-4.1.0.tgz", + "integrity": "sha512-dJxPBlzC7NugB2PDLwo9Q8JiTR3M3e4/XANkreKSUxF8vvXKqm1Yfq4Q5dl8budlunRVlUUaDUgFt7eA8D6NLg==", + "license": "ISC", + "dependencies": { + "d3-time": "1 - 3" + }, + "engines": { + "node": ">=12" + } + }, "node_modules/d3-timer": { "version": "3.0.1", "resolved": "https://registry.npmjs.org/d3-timer/-/d3-timer-3.0.1.tgz", @@ -4469,6 +4883,16 @@ "node": ">=12" } }, + "node_modules/dagre-d3-es": { + "version": "7.0.14", + "resolved": "https://registry.npmjs.org/dagre-d3-es/-/dagre-d3-es-7.0.14.tgz", + "integrity": "sha512-P4rFMVq9ESWqmOgK+dlXvOtLwYg0i7u0HBGJER0LZDJT2VHIPAMZ/riPxqJceWMStH5+E61QxFra9kIS3AqdMg==", + "license": "MIT", + "dependencies": { + "d3": "^7.9.0", + "lodash-es": "^4.17.21" + } + }, "node_modules/date-fns": { "version": "2.30.0", "resolved": "https://registry.npmjs.org/date-fns/-/date-fns-2.30.0.tgz", @@ -4485,6 +4909,12 @@ "url": "https://opencollective.com/date-fns" } }, + "node_modules/dayjs": { + "version": "1.11.21", + "resolved": "https://registry.npmjs.org/dayjs/-/dayjs-1.11.21.tgz", + "integrity": "sha512-98IT+HOahAisibz/yjKbzuOBwYcjJ7BCLPzARyHiyEBmRz4fatF+KPJszEHXsGYjUG234aH/cOjW1wwTbKUZlA==", + "license": "MIT" + }, "node_modules/debug": { "version": "4.4.3", "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", @@ -4695,6 +5125,15 @@ "dev": true, "license": "MIT" }, + "node_modules/delaunator": { + "version": "5.1.0", + "resolved": "https://registry.npmjs.org/delaunator/-/delaunator-5.1.0.tgz", + "integrity": "sha512-AGrQ4QSgssa1NGmWmLPqN5NY2KajF5MqxetNEO+o0n3ZwZZeTmt7bBnvzHWrmkZFxGgr4HdyFgelzgi06otLuQ==", + "license": "ISC", + "dependencies": { + "robust-predicates": "^3.0.2" + } + }, "node_modules/delayed-stream": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz", @@ -5094,6 +5533,16 @@ "node": ">= 0.4" } }, + "node_modules/es-toolkit": { + "version": "1.48.1", + "resolved": "https://registry.npmjs.org/es-toolkit/-/es-toolkit-1.48.1.tgz", + "integrity": "sha512-wfnXlwd5I75eXRtdD2vuEs50xHHESECDsGD7yiQnfFVNoa5522NwXEbmgo98LfiukSQHs+mBM7/YG3qKJB9/mQ==", + "license": "MIT", + "workspaces": [ + "docs", + "benchmarks" + ] + }, "node_modules/escalade": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", @@ -6161,6 +6610,12 @@ "node": "^12.22.0 || ^14.16.0 || ^16.0.0 || >=17.0.0" } }, + "node_modules/hachure-fill": { + "version": "0.5.2", + "resolved": "https://registry.npmjs.org/hachure-fill/-/hachure-fill-0.5.2.tgz", + "integrity": "sha512-3GKBOn+m2LX9iq+JC1064cSFprJY4jL1jCXTcpnfER5HYE2l/4EfWSGzkPa/ZDBmYI0ZOEj5VHV/eKnPGkHuOg==", + "license": "MIT" + }, "node_modules/hammerjs": { "version": "2.0.8", "resolved": "https://registry.npmjs.org/hammerjs/-/hammerjs-2.0.8.tgz", @@ -6489,6 +6944,18 @@ "url": "https://github.com/sponsors/wooorm" } }, + "node_modules/iconv-lite": { + "version": "0.6.3", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz", + "integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==", + "license": "MIT", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3.0.0" + }, + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/idb": { "version": "8.0.3", "resolved": "https://registry.npmjs.org/idb/-/idb-8.0.3.tgz", @@ -6560,6 +7027,16 @@ "node": ">=8" } }, + "node_modules/import-meta-resolve": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/import-meta-resolve/-/import-meta-resolve-4.2.0.tgz", + "integrity": "sha512-Iqv2fzaTQN28s/FwZAoFq0ZSs/7hMAHJVX+w8PZl3cY19Pxk6jFFalxQoIfW2826i/fDLXv8IiEZRIT0lDuWcg==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/wooorm" + } + }, "node_modules/imurmurhash": { "version": "0.1.4", "resolved": "https://registry.npmjs.org/imurmurhash/-/imurmurhash-0.1.4.tgz", @@ -6620,7 +7097,6 @@ "version": "2.0.3", "resolved": "https://registry.npmjs.org/internmap/-/internmap-2.0.3.tgz", "integrity": "sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==", - "dev": true, "license": "ISC", "engines": { "node": ">=12" @@ -6852,7 +7328,7 @@ "version": "1.21.7", "resolved": "https://registry.npmjs.org/jiti/-/jiti-1.21.7.tgz", "integrity": "sha512-/imKNG4EbWNrVjoNC/1H5/9GFy+tqjGBHCaSsN+P2RnPqjsLmv6UD3Ej+Kj8nBWaRAwyk7kK5ZUc+OEatnTR3A==", - "dev": true, + "devOptional": true, "license": "MIT", "bin": { "jiti": "bin/jiti.js" @@ -7058,6 +7534,31 @@ "safe-buffer": "~5.1.0" } }, + "node_modules/katex": { + "version": "0.16.47", + "resolved": "https://registry.npmjs.org/katex/-/katex-0.16.47.tgz", + "integrity": "sha512-Eeo8Ys1doU1z+x8AZsPpQu+p/QcZBI5PeOo7QGQdy2x2m0MU/hYagBbGOmXwr5KVbEfVuWv9LpnQWeehogurjg==", + "funding": [ + "https://opencollective.com/katex", + "https://github.com/sponsors/katex" + ], + "license": "MIT", + "dependencies": { + "commander": "^8.3.0" + }, + "bin": { + "katex": "cli.js" + } + }, + "node_modules/katex/node_modules/commander": { + "version": "8.3.0", + "resolved": "https://registry.npmjs.org/commander/-/commander-8.3.0.tgz", + "integrity": "sha512-OkTL9umf+He2DZkUq8f8J9of7yL6RJKI24dVITBmNfZBmri9zYZQrKkuXiKhyfPSu8tUhnVBB1iKXevvnlR4Ww==", + "license": "MIT", + "engines": { + "node": ">= 12" + } + }, "node_modules/keyv": { "version": "4.5.4", "resolved": "https://registry.npmjs.org/keyv/-/keyv-4.5.4.tgz", @@ -7068,6 +7569,11 @@ "json-buffer": "3.0.1" } }, + "node_modules/khroma": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/khroma/-/khroma-2.1.0.tgz", + "integrity": "sha512-Ls993zuzfayK269Svk9hzpeGUKob/sIgZzyHYdjQoAdQetRKpOLj+k/QQQ/6Qi0Yz65mlROrfd+Ev+1+7dz9Kw==" + }, "node_modules/kind-of": { "version": "6.0.3", "resolved": "https://registry.npmjs.org/kind-of/-/kind-of-6.0.3.tgz", @@ -7096,6 +7602,12 @@ "dev": true, "license": "MIT" }, + "node_modules/layout-base": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/layout-base/-/layout-base-1.0.2.tgz", + "integrity": "sha512-8h2oVEZNktL4BH2JCOI90iD1yXwL6iNW7KcCKT2QZgQJR2vbqDsldCTPRU9NifTCqHZci57XvQQ15YTu+sTYPg==", + "license": "MIT" + }, "node_modules/lerc": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/lerc/-/lerc-3.0.0.tgz", @@ -7351,7 +7863,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7372,7 +7883,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7393,7 +7903,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7414,7 +7923,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7435,7 +7943,6 @@ "cpu": [ "arm" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7456,7 +7963,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7477,7 +7983,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7498,7 +8003,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7519,7 +8023,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7540,7 +8043,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7561,7 +8063,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7623,6 +8124,12 @@ "integrity": "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==", "license": "MIT" }, + "node_modules/lodash-es": { + "version": "4.18.1", + "resolved": "https://registry.npmjs.org/lodash-es/-/lodash-es-4.18.1.tgz", + "integrity": "sha512-J8xewKD/Gk22OZbhpOVSwcs60zhd95ESDwezOFuA3/099925PdHJ7OFHNTGtajL3AlZkykD32HykiMo+BIBI8A==", + "license": "MIT" + }, "node_modules/lodash.clonedeep": { "version": "4.5.0", "resolved": "https://registry.npmjs.org/lodash.clonedeep/-/lodash.clonedeep-4.5.0.tgz", @@ -8075,6 +8582,47 @@ "node": ">= 8" } }, + "node_modules/mermaid": { + "version": "11.15.0", + "resolved": "https://registry.npmjs.org/mermaid/-/mermaid-11.15.0.tgz", + "integrity": "sha512-pTMbcf3rWdtLiYGpmoTjHEpeY8seiy6sR+9nD7LOs8KfUbHE4lOUAprTRqRAcWSQ6MQpdX+YEsxShtGsINtPtw==", + "license": "MIT", + "dependencies": { + "@braintree/sanitize-url": "^7.1.1", + "@iconify/utils": "^3.0.2", + "@mermaid-js/parser": "^1.1.1", + "@types/d3": "^7.4.3", + "@upsetjs/venn.js": "^2.0.0", + "cytoscape": "^3.33.1", + "cytoscape-cose-bilkent": "^4.1.0", + "cytoscape-fcose": "^2.2.0", + "d3": "^7.9.0", + "d3-sankey": "^0.12.3", + "dagre-d3-es": "7.0.14", + "dayjs": "^1.11.19", + "dompurify": "^3.3.1", + "es-toolkit": "^1.45.1", + "katex": "^0.16.25", + "khroma": "^2.1.0", + "marked": "^16.3.0", + "roughjs": "^4.6.6", + "stylis": "^4.3.6", + "ts-dedent": "^2.2.0", + "uuid": "^11.1.0 || ^12 || ^13 || ^14.0.0" + } + }, + "node_modules/mermaid/node_modules/marked": { + "version": "16.4.2", + "resolved": "https://registry.npmjs.org/marked/-/marked-16.4.2.tgz", + "integrity": "sha512-TI3V8YYWvkVf3KJe1dRkpnjs68JUPyEa5vjKrp1XEEJUAOaQc+Qj+L1qWbPd0SJuAdQkFU0h73sXXqwDYxsiDA==", + "license": "MIT", + "bin": { + "marked": "bin/marked.js" + }, + "engines": { + "node": ">= 20" + } + }, "node_modules/methods": { "version": "1.1.2", "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz", @@ -9351,6 +9899,12 @@ "dev": true, "license": "BlueOak-1.0.0" }, + "node_modules/package-manager-detector": { + "version": "1.6.0", + "resolved": "https://registry.npmjs.org/package-manager-detector/-/package-manager-detector-1.6.0.tgz", + "integrity": "sha512-61A5ThoTiDG/C8s8UMZwSorAGwMJ0ERVGj2OjoW5pAalsNOg15+iQiPzrLJ4jhZ1HJzmC2PIHT2oEiH3R5fzNA==", + "license": "MIT" + }, "node_modules/pako": { "version": "2.1.0", "resolved": "https://registry.npmjs.org/pako/-/pako-2.1.0.tgz", @@ -9432,6 +9986,12 @@ "dev": true, "license": "MIT" }, + "node_modules/path-data-parser": { + "version": "0.1.0", + "resolved": "https://registry.npmjs.org/path-data-parser/-/path-data-parser-0.1.0.tgz", + "integrity": "sha512-NOnmBpt5Y2RWbuv0LMzsayp3lVylAHLPUTut412ZA3l+C4uw4ZVkQbjShYCQ8TCpUMdPapr4YjUqLYD6v68j+w==", + "license": "MIT" + }, "node_modules/path-exists": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/path-exists/-/path-exists-4.0.0.tgz", @@ -9679,6 +10239,22 @@ "node": ">=4" } }, + "node_modules/points-on-curve": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/points-on-curve/-/points-on-curve-0.2.0.tgz", + "integrity": "sha512-0mYKnYYe9ZcqMCWhUjItv/oHjvgEsfKvnUTg8sAtnHr3GVy7rGkXCb6d5cSyqrWqL4k81b9CPg3urd+T7aop3A==", + "license": "MIT" + }, + "node_modules/points-on-path": { + "version": "0.2.1", + "resolved": "https://registry.npmjs.org/points-on-path/-/points-on-path-0.2.1.tgz", + "integrity": "sha512-25ClnWWuw7JbWZcgqY/gJ4FQWadKxGWk+3kR/7kD0tCaDtPPMj7oHu2ToLaVhfpnHrZzYby2w6tUA0eOIuUg8g==", + "license": "MIT", + "dependencies": { + "path-data-parser": "0.1.0", + "points-on-curve": "0.2.0" + } + }, "node_modules/postcss": { "version": "8.5.14", "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.14.tgz", @@ -11083,6 +11659,12 @@ "url": "https://github.com/sponsors/isaacs" } }, + "node_modules/robust-predicates": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/robust-predicates/-/robust-predicates-3.0.3.tgz", + "integrity": "sha512-NS3levdsRIUOmiJ8FZWCP7LG3QpJyrs/TE0Zpf1yvZu8cAJJ6QMW92H1c7kWpdIHo8RvmLxN/o2JXTKHp74lUA==", + "license": "Unlicense" + }, "node_modules/rolldown": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.1.tgz", @@ -11117,6 +11699,18 @@ "@rolldown/binding-win32-x64-msvc": "1.0.1" } }, + "node_modules/roughjs": { + "version": "4.6.6", + "resolved": "https://registry.npmjs.org/roughjs/-/roughjs-4.6.6.tgz", + "integrity": "sha512-ZUz/69+SYpFN/g/lUlo2FXcIjRkSu3nDarreVdGGndHEBJ6cXPdKguS8JGxwj5HA5xIbVKSmLgr5b3AWxtRfvQ==", + "license": "MIT", + "dependencies": { + "hachure-fill": "^0.5.2", + "path-data-parser": "^0.1.0", + "points-on-curve": "^0.2.0", + "points-on-path": "^0.2.1" + } + }, "node_modules/run-parallel": { "version": "1.2.0", "resolved": "https://registry.npmjs.org/run-parallel/-/run-parallel-1.2.0.tgz", @@ -11208,6 +11802,12 @@ ], "license": "MIT" }, + "node_modules/safer-buffer": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", + "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", + "license": "MIT" + }, "node_modules/scule": { "version": "1.3.0", "resolved": "https://registry.npmjs.org/scule/-/scule-1.3.0.tgz", @@ -11887,6 +12487,12 @@ "node": ">=8" } }, + "node_modules/stylis": { + "version": "4.4.0", + "resolved": "https://registry.npmjs.org/stylis/-/stylis-4.4.0.tgz", + "integrity": "sha512-5Z9ZpRzfuH6l/UAvCPAPUo3665Nk2wLaZU3x+TLHKVzIz33+sbJqbtrYoC3KD4/uVOr2Zp+L0LySezP9OHV9yA==", + "license": "MIT" + }, "node_modules/sucrase": { "version": "3.35.0", "resolved": "https://registry.npmjs.org/sucrase/-/sucrase-3.35.0.tgz", @@ -12144,21 +12750,6 @@ } } }, - "node_modules/svelte-check/node_modules/picomatch": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", - "integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/jonschlinkert" - } - }, "node_modules/svelte-eslint-parser": { "version": "0.43.0", "resolved": "https://registry.npmjs.org/svelte-eslint-parser/-/svelte-eslint-parser-0.43.0.tgz", @@ -12800,6 +13391,15 @@ "url": "https://github.com/sponsors/wooorm" } }, + "node_modules/ts-dedent": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/ts-dedent/-/ts-dedent-2.3.0.tgz", + "integrity": "sha512-JfJeIHke7y2egdGGgRAvpCwYFUsHlM2gPcrVOxFkznt/4uzQ7HFmvE63iFHVLBJNDuyDOQgijDK/tXH/f6Msjg==", + "license": "MIT", + "engines": { + "node": ">=6.10" + } + }, "node_modules/ts-interface-checker": { "version": "0.1.13", "resolved": "https://registry.npmjs.org/ts-interface-checker/-/ts-interface-checker-0.1.13.tgz", @@ -12889,7 +13489,7 @@ "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", - "dev": true, + "devOptional": true, "license": "Apache-2.0", "bin": { "tsc": "bin/tsc", @@ -13108,6 +13708,19 @@ "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", "license": "MIT" }, + "node_modules/uuid": { + "version": "14.0.1", + "resolved": "https://registry.npmjs.org/uuid/-/uuid-14.0.1.tgz", + "integrity": "sha512-6ZxzVpzDXDa3bJWaHilVayA+BH/1zmxCJoVgvmqJnid/gPoKHxUrS/aC/T6LGQtNHT+XHG9fXPJB4d+IrU30Ew==", + "funding": [ + "https://github.com/sponsors/broofa", + "https://github.com/sponsors/ctavan" + ], + "license": "MIT", + "bin": { + "uuid": "dist-node/bin/uuid" + } + }, "node_modules/validate-npm-package-license": { "version": "3.0.4", "resolved": "https://registry.npmjs.org/validate-npm-package-license/-/validate-npm-package-license-3.0.4.tgz", diff --git a/frontend/package.json b/frontend/package.json index b080ab058a..aead7671b3 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -127,6 +127,7 @@ "lru-cache": "^11.1.0", "lucide-svelte": "^0.540.0", "mdast-util-find-and-replace": "^3.0.2", + "mermaid": "^11.15.0", "minimatch": "^10.0.1", "monaco-editor": "npm:@codingame/monaco-vscode-editor-api@=25.0.0", "monaco-languageclient": "10.6.0", diff --git a/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte b/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte index dc1e5e3594..d0a4cba462 100644 --- a/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte +++ b/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte @@ -17,6 +17,7 @@ import { AIMode } from '../AIChatManager.svelte' import { getAiChatManager } from '../aiChatManagerContext' import { Check, Play } from 'lucide-svelte' + import MermaidDisplay from './MermaidDisplay.svelte' const aiChatManager = getAiChatManager() @@ -108,14 +109,18 @@
    - + {#if language === 'mermaid'} + + {:else} + + {/if}
    diff --git a/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte b/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte new file mode 100644 index 0000000000..06bcb63d98 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte @@ -0,0 +1,62 @@ + + +{#if showSvg} +
    + + {@html svg} +
    +{:else} + + 
    {code}
    +{/if} From 9e4cf139b1345a9db18931fb4b8cf9d6bb646844 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 23 Jun 2026 23:08:15 +0200 Subject: [PATCH 041/117] chore(main): release 1.738.0 (#9735) * chore(main): release 1.738.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> --- CHANGELOG.md | 24 +++ backend/Cargo.lock | 156 +++++++++--------- backend/Cargo.toml | 4 +- .../parsers/windmill-parser-wasm/Cargo.lock | 48 +++--- .../parsers/windmill-parser-wasm/Cargo.toml | 2 +- backend/windmill-api/openapi.yaml | 2 +- benchmarks/lib.ts | 2 +- cli/src/core/constants.ts | 2 +- frontend/package-lock.json | 54 +++++- frontend/package.json | 2 +- lsp/Pipfile | 2 +- openflow.openapi.yaml | 2 +- .../WindmillClient/WindmillClient.psd1 | 2 +- python-client/wmill/pyproject.toml | 2 +- typescript-client/jsr.json | 2 +- typescript-client/package.json | 2 +- version.txt | 2 +- 17 files changed, 190 insertions(+), 120 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index df77488710..f715123c3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,29 @@ # Changelog +## [1.738.0](https://github.com/windmill-labs/windmill/compare/v1.737.0...v1.738.0) (2026-06-23) + + +### Features + +* add resource and infrastructure telemetry ([#9737](https://github.com/windmill-labs/windmill/issues/9737)) ([9793d01](https://github.com/windmill-labs/windmill/commit/9793d01575415963a89609a1baf2cd64f0d050cc)) +* render mermaid diagrams in chat code blocks ([#9738](https://github.com/windmill-labs/windmill/issues/9738)) ([cfb9f1d](https://github.com/windmill-labs/windmill/commit/cfb9f1dbc23110ecf8f91bb3c8c81fc6e35dc09b)) + + +### Bug Fixes + +* **ai-chat:** Fix incorrect editor edits from ai chat [#1](https://github.com/windmill-labs/windmill/issues/1) ([#9741](https://github.com/windmill-labs/windmill/issues/9741)) ([fc797a3](https://github.com/windmill-labs/windmill/commit/fc797a35fe7885630c81453df0fc94769e73873a)) +* allow object storage test for non-super-admins, harden on cloud ([#9739](https://github.com/windmill-labs/windmill/issues/9739)) ([24446e8](https://github.com/windmill-labs/windmill/commit/24446e80093ade349f7fbf65063d2d1cb5551c1e)) +* **frontend:** debounce external code→Monaco sync in Editor ([#9743](https://github.com/windmill-labs/windmill/issues/9743)) ([29c67ce](https://github.com/windmill-labs/windmill/commit/29c67ced97bf2919584986f9d9eceb4337c34ad9)) +* **frontend:** preserve editor content when closing instance settings drawer ([#9740](https://github.com/windmill-labs/windmill/issues/9740)) ([11d0e65](https://github.com/windmill-labs/windmill/commit/11d0e65f3af9a048bc1921bbdd3d676a07483a57)) +* pipeline annotation false-positives from body comments ([#9736](https://github.com/windmill-labs/windmill/issues/9736)) ([984ea72](https://github.com/windmill-labs/windmill/commit/984ea728d98649b66b1cae899bdab9af3176caa7)) +* preserve fork parent linkage on workspace id change ([#9716](https://github.com/windmill-labs/windmill/issues/9716)) ([cbf54d4](https://github.com/windmill-labs/windmill/commit/cbf54d4eb432638e27f67c4c8b879cbcc0291da3)) +* prevent variable push from corrupting is_secret variables ([#9705](https://github.com/windmill-labs/windmill/issues/9705)) ([ba4b368](https://github.com/windmill-labs/windmill/commit/ba4b368706e95e22f346a10e5fe145b0795ac3f6)) + + +### Performance Improvements + +* **monitor:** skip protected prefix in retention delete via cross-batch watermark (WIN-2088) ([#9744](https://github.com/windmill-labs/windmill/issues/9744)) ([e90b2be](https://github.com/windmill-labs/windmill/commit/e90b2be8fade1eb78cd685890291f5a4553a6a10)) + ## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23) diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 834c636f0a..1f267a94e8 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -13735,7 +13735,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-nats", @@ -13817,7 +13817,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.737.0" +version = "1.738.0" dependencies = [ "async-stream", "async-trait", @@ -13850,7 +13850,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -13863,7 +13863,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "argon2", @@ -14001,7 +14001,7 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14024,7 +14024,7 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14037,7 +14037,7 @@ dependencies = [ [[package]] name = "windmill-api-auth" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14063,7 +14063,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.737.0" +version = "1.738.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14073,7 +14073,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14090,7 +14090,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14112,7 +14112,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14135,7 +14135,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14151,7 +14151,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14172,7 +14172,7 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14193,7 +14193,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14207,7 +14207,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-nats", @@ -14242,7 +14242,7 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14267,7 +14267,7 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "flate2", @@ -14285,7 +14285,7 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14307,7 +14307,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14327,7 +14327,7 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14364,7 +14364,7 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14392,7 +14392,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.737.0" +version = "1.738.0" dependencies = [ "lazy_static", "serde", @@ -14404,7 +14404,7 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.737.0" +version = "1.738.0" dependencies = [ "argon2", "axum 0.8.9", @@ -14429,7 +14429,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14443,7 +14443,7 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.737.0" +version = "1.738.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14476,7 +14476,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.737.0" +version = "1.738.0" dependencies = [ "chrono", "lazy_static", @@ -14490,7 +14490,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14509,7 +14509,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.737.0" +version = "1.738.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -14611,7 +14611,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.737.0" +version = "1.738.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -14630,7 +14630,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.737.0" +version = "1.738.0" dependencies = [ "regex", "serde", @@ -14645,7 +14645,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -14669,7 +14669,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "futures", @@ -14686,7 +14686,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.737.0" +version = "1.738.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -14702,7 +14702,7 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -14723,7 +14723,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -14754,7 +14754,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "arc-swap", @@ -14779,7 +14779,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-stream", @@ -14813,7 +14813,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "futures", @@ -14831,7 +14831,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.737.0" +version = "1.738.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -14840,7 +14840,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "lazy_static", @@ -14852,7 +14852,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde_json", @@ -14864,7 +14864,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "gosyn", @@ -14876,7 +14876,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "lazy_static", @@ -14888,7 +14888,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde_json", @@ -14900,7 +14900,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "nu-parser", @@ -14911,7 +14911,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14922,7 +14922,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14934,7 +14934,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "rustpython-ast", @@ -14945,7 +14945,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-recursion", @@ -14967,7 +14967,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde_json", @@ -14979,7 +14979,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "lazy_static", @@ -14993,7 +14993,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15010,7 +15010,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "lazy_static", @@ -15023,7 +15023,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde", @@ -15035,7 +15035,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "lazy_static", @@ -15053,7 +15053,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15069,7 +15069,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15085,7 +15085,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde", @@ -15096,7 +15096,7 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-recursion", @@ -15134,7 +15134,7 @@ dependencies = [ [[package]] name = "windmill-runtime-nativets" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "const_format", @@ -15173,7 +15173,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.737.0" +version = "1.738.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15184,7 +15184,7 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-recursion", @@ -15218,7 +15218,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15242,7 +15242,7 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15275,7 +15275,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15308,7 +15308,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15328,7 +15328,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15362,7 +15362,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15398,7 +15398,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15421,7 +15421,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15445,7 +15445,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-nats", @@ -15469,7 +15469,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15504,7 +15504,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15532,7 +15532,7 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-trait", @@ -15557,7 +15557,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "bitflags 2.13.0", @@ -15576,7 +15576,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-once-cell", @@ -15686,7 +15686,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.737.0" +version = "1.738.0" dependencies = [ "bytes", "futures", diff --git a/backend/Cargo.toml b/backend/Cargo.toml index bce80a8c0d..7a33b9503e 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.737.0" +version = "1.738.0" authors.workspace = true edition.workspace = true @@ -87,7 +87,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.737.0" +version = "1.738.0" authors = ["Ruben Fiszel "] edition = "2021" diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index 9d20e86a8a..6039a52327 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.737.0" +version = "1.738.0" dependencies = [ "aho-corasick", "anyhow", @@ -6272,7 +6272,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.737.0" +version = "1.738.0" dependencies = [ "proc-macro2", "quote", @@ -6284,7 +6284,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.737.0" +version = "1.738.0" dependencies = [ "convert_case", "serde", @@ -6293,7 +6293,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "lazy_static", @@ -6305,7 +6305,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde_json", @@ -6317,7 +6317,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "gosyn", @@ -6329,7 +6329,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "lazy_static", @@ -6341,7 +6341,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde_json", @@ -6353,7 +6353,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "nu-parser", @@ -6364,7 +6364,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6375,7 +6375,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6387,7 +6387,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6398,7 +6398,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "async-recursion", @@ -6420,7 +6420,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde_json", @@ -6432,7 +6432,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "lazy_static", @@ -6446,7 +6446,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "convert_case", @@ -6463,7 +6463,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "lazy_static", @@ -6476,7 +6476,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde", @@ -6488,7 +6488,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "lazy_static", @@ -6506,7 +6506,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6522,7 +6522,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6538,7 +6538,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6570,7 +6570,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "serde", @@ -6581,7 +6581,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.737.0" +version = "1.738.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index 046249b27e..a341432a7e 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.737.0" +version = "1.738.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index f200cdb902..152c512b46 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: - version: 1.737.0 + version: 1.738.0 title: Windmill API contact: diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index 2d928acce8..ab4d83caad 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.737.0"; +export const VERSION = "v1.738.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ diff --git a/cli/src/core/constants.ts b/cli/src/core/constants.ts index 7402929aee..968a998ef8 100644 --- a/cli/src/core/constants.ts +++ b/cli/src/core/constants.ts @@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork"; // (e.g. utils.ts) can read it without importing main.ts and creating a circular // dependency (main → workspace → utils → main) that triggers a TDZ. // Re-exported from main.ts for backwards compatibility. -export const VERSION = "1.737.0"; +export const VERSION = "1.738.0"; diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 24af2dc2ed..f535e0d38d 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "@windmill-labs/components", - "version": "1.737.0", + "version": "1.738.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@windmill-labs/components", - "version": "1.737.0", + "version": "1.738.0", "hasInstallScript": true, "license": "AGPL-3.0", "dependencies": { @@ -878,6 +878,7 @@ "version": "1.10.0", "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.10.0.tgz", "integrity": "sha512-yq6OkJ4p82CAfPl0u9mQebQHKPJkY7WrIuk205cTYnYe+k2Z8YBh11FrbRG/H6ihirqcacOgl2BIO8oyMQLeXw==", + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -889,6 +890,7 @@ "version": "1.10.0", "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz", "integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==", + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -899,6 +901,7 @@ "version": "1.2.1", "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.1.tgz", "integrity": "sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w==", + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1414,6 +1417,7 @@ "version": "1.1.4", "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.4.tgz", "integrity": "sha512-3NQNNgA1YSlJb/kMH1ildASP9HW7/7kYnRI2szWJaofaS1hWmbGI4H+d3+22aGzXXN9IJ+n+GiFVcGipJP18ow==", + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1562,6 +1566,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1578,6 +1583,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1594,6 +1600,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1610,6 +1617,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1626,6 +1634,7 @@ "cpu": [ "arm" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1642,6 +1651,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1658,6 +1668,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1674,6 +1685,7 @@ "cpu": [ "ppc64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1690,6 +1702,7 @@ "cpu": [ "s390x" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1706,6 +1719,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1722,6 +1736,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1738,6 +1753,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1754,6 +1770,7 @@ "cpu": [ "wasm32" ], + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1772,6 +1789,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1788,6 +1806,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -2093,6 +2112,7 @@ "version": "0.10.2", "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.2.tgz", "integrity": "sha512-RoBvJ2X0wuKlWFIjrwffGw1IqZHKQqzIchKaadZZfnNpsAYp2mM0h36JtPCjNDAHGgYez/15uMBpfGwchhiMgg==", + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -7328,7 +7348,7 @@ "version": "1.21.7", "resolved": "https://registry.npmjs.org/jiti/-/jiti-1.21.7.tgz", "integrity": "sha512-/imKNG4EbWNrVjoNC/1H5/9GFy+tqjGBHCaSsN+P2RnPqjsLmv6UD3Ej+Kj8nBWaRAwyk7kK5ZUc+OEatnTR3A==", - "devOptional": true, + "dev": true, "license": "MIT", "bin": { "jiti": "bin/jiti.js" @@ -7863,6 +7883,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7883,6 +7904,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7903,6 +7925,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7923,6 +7946,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7943,6 +7967,7 @@ "cpu": [ "arm" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7963,6 +7988,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7983,6 +8009,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8003,6 +8030,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8023,6 +8051,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8043,6 +8072,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8063,6 +8093,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -12750,6 +12781,21 @@ } } }, + "node_modules/svelte-check/node_modules/picomatch": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", + "integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", + "dev": true, + "license": "MIT", + "optional": true, + "peer": true, + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, "node_modules/svelte-eslint-parser": { "version": "0.43.0", "resolved": "https://registry.npmjs.org/svelte-eslint-parser/-/svelte-eslint-parser-0.43.0.tgz", @@ -13489,7 +13535,7 @@ "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", - "devOptional": true, + "dev": true, "license": "Apache-2.0", "bin": { "tsc": "bin/tsc", diff --git a/frontend/package.json b/frontend/package.json index aead7671b3..5db38263b9 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "@windmill-labs/components", - "version": "1.737.0", + "version": "1.738.0", "scripts": { "dev": "vite dev", "dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev", diff --git a/lsp/Pipfile b/lsp/Pipfile index eb25523cf7..04cca1f905 100644 --- a/lsp/Pipfile +++ b/lsp/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -wmill = ">=1.737.0" +wmill = ">=1.738.0" sendgrid = "*" mysql-connector-python = "*" pymongo = "*" diff --git a/openflow.openapi.yaml b/openflow.openapi.yaml index c64c32a1be..2ef6abf30e 100644 --- a/openflow.openapi.yaml +++ b/openflow.openapi.yaml @@ -1,7 +1,7 @@ openapi: '3.0.3' info: - version: 1.737.0 + version: 1.738.0 title: OpenFlow Spec contact: name: Ruben Fiszel diff --git a/powershell-client/WindmillClient/WindmillClient.psd1 b/powershell-client/WindmillClient/WindmillClient.psd1 index 75288a8267..c7d1dfd23a 100644 --- a/powershell-client/WindmillClient/WindmillClient.psd1 +++ b/powershell-client/WindmillClient/WindmillClient.psd1 @@ -12,7 +12,7 @@ RootModule = 'WindmillClient.psm1' # Version number of this module. - ModuleVersion = '1.737.0' + ModuleVersion = '1.738.0' # Supported PSEditions # CompatiblePSEditions = @() diff --git a/python-client/wmill/pyproject.toml b/python-client/wmill/pyproject.toml index abcb7296f7..277211d2c7 100644 --- a/python-client/wmill/pyproject.toml +++ b/python-client/wmill/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "wmill" -version = "1.737.0" +version = "1.738.0" description = "A client library for accessing Windmill server wrapping the Windmill client API" license = "Apache-2.0" homepage = "https://windmill.dev" diff --git a/typescript-client/jsr.json b/typescript-client/jsr.json index 1d7043d5f7..7f0cf3f33b 100644 --- a/typescript-client/jsr.json +++ b/typescript-client/jsr.json @@ -1,6 +1,6 @@ { "name": "@windmill/windmill", - "version": "1.737.0", + "version": "1.738.0", "exports": "./src/index.ts", "publish": { "exclude": ["!src", "./s3Types.ts", "./sqlUtils.ts", "./client.ts"] diff --git a/typescript-client/package.json b/typescript-client/package.json index 983c066936..f45d9eeac6 100644 --- a/typescript-client/package.json +++ b/typescript-client/package.json @@ -1,7 +1,7 @@ { "name": "windmill-client", "description": "Windmill SDK client for browsers and Node.js", - "version": "1.737.0", + "version": "1.738.0", "author": "Ruben Fiszel", "license": "Apache 2.0", "homepage": "https://github.com/windmill-labs/windmill/tree/main/typescript-client#readme", diff --git a/version.txt b/version.txt index 63bd46762f..f64275236c 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.737.0 +1.738.0 From ae088fd032f46e9e55aa662c85b0cd86f8306f0d Mon Sep 17 00:00:00 2001 From: centdix <40307056+centdix@users.noreply.github.com> Date: Wed, 24 Jun 2026 00:31:14 +0200 Subject: [PATCH 042/117] stabilize global ai eval smoke path (#9745) --- ai_evals/cases/global.yaml | 1 + ai_evals/core/cases.test.ts | 3 +++ .../global/initial/user_admin_evals_folder.json | 8 ++++++++ .../lib/components/copilot/chat/global/core.test.ts | 12 ++++++++++++ .../src/lib/components/copilot/chat/global/core.ts | 1 + 5 files changed, 25 insertions(+) create mode 100644 ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json diff --git a/ai_evals/cases/global.yaml b/ai_evals/cases/global.yaml index 6a273c5ce8..3e935a730c 100644 --- a/ai_evals/cases/global.yaml +++ b/ai_evals/cases/global.yaml @@ -3,6 +3,7 @@ Create a draft Bun script at `f/evals/global/greet_user`. It should take a string `name` input and return `Hello, ${name}!`. Leave it as an AI draft only; do not deploy or save it. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json runtime: maxTurns: 10 validate: diff --git a/ai_evals/core/cases.test.ts b/ai_evals/core/cases.test.ts index 9955a73fa9..5d6e3245db 100644 --- a/ai_evals/core/cases.test.ts +++ b/ai_evals/core/cases.test.ts @@ -212,6 +212,9 @@ describe("loadCases", () => { }, ], }); + expect(caseEntry?.initialPath).toContain( + "ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json" + ); expect(caseEntry?.toolExpect).toMatchObject({ requiredToolsUsed: ["write_script"], forbiddenToolsUsed: ["deploy_workspace_item", "delete_workspace_item"], diff --git a/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json b/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json new file mode 100644 index 0000000000..236f091bc2 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json @@ -0,0 +1,8 @@ +{ + "user": { + "username": "admin", + "is_admin": true, + "folders": ["evals"], + "folders_read": ["evals"] + } +} diff --git a/frontend/src/lib/components/copilot/chat/global/core.test.ts b/frontend/src/lib/components/copilot/chat/global/core.test.ts index 242a78c2dd..3f91b12e33 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.test.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.test.ts @@ -2757,6 +2757,18 @@ describe('prepareGlobalSystemMessage', () => { expect(content).not.toContain('frontend AI draft store') }) + it('honors user-supplied shared folder paths without asking first', () => { + const content = prepareGlobalSystemMessage(undefined, { + user: { username: 'admin', is_admin: true, folders: ['evals'] } + }).content as string + + expect(content).toContain( + 'If the user supplies a fully qualified `f//...` path, use that exact path' + ) + expect(content).toContain('Do not ask for folder confirmation') + expect(content).toContain('substitute a `u/admin/...` path unless a tool rejects it') + }) + describe('folder guidance', () => { const guidanceOf = (user: { username: string diff --git a/frontend/src/lib/components/copilot/chat/global/core.ts b/frontend/src/lib/components/copilot/chat/global/core.ts index 462d8935df..5c1350f233 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.ts @@ -748,6 +748,7 @@ Path conventions: - A workspace path starts with one of two namespaces; its trailing may itself contain "/", so a path has three or more segments: - \`u/${username}/\` — your personal scope. Default for ad-hoc, exploratory, or scratch work. - \`f//\` — a shared folder scope; the must already exist (a bare \`f/\` with no folder segment is INVALID and will fail). +- If the user supplies a fully qualified \`f//...\` path, use that exact path; they have already chosen the folder. Do not ask for folder confirmation or substitute a \`u/${username}/...\` path unless a tool rejects it. - Default a bare name with no namespace prefix (e.g. "create a flow called myflow") to \`u/${username}/\`. Never invent an \`f//...\` path for a folder that does not exist.${folderGuidanceBlock} Rules: From 043c2c05b7678c49faca0ccb28e5f6393567ba4d Mon Sep 17 00:00:00 2001 From: hugocasa Date: Wed, 24 Jun 2026 00:38:27 +0200 Subject: [PATCH 043/117] fix: forbid superadmin job tokens from global user and token management (#9715) * fix: forbid superadmin job tokens from global user and token management Co-Authored-By: Claude Opus 4.8 (1M context) * fix: extend superadmin job token guard to offboard and export routes Apply forbid_superadmin_job_token to offboard_global_user and export_global_users, the remaining global user-management routes that were gated only by require_super_admin. Offboarding can delete a user along with their tokens, password, invites and instance-group membership, and export returns every user's password_hash, so both must be unreachable by a superadmin job token. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- backend/tests/wm_token_superadmin_guard.rs | 206 +++++++++++++++++++++ backend/windmill-api-auth/src/lib.rs | 27 +++ backend/windmill-api-users/src/users.rs | 16 +- backend/windmill-api/src/offboarding.rs | 6 +- backend/windmill-api/src/users.rs | 12 +- 5 files changed, 262 insertions(+), 5 deletions(-) create mode 100644 backend/tests/wm_token_superadmin_guard.rs diff --git a/backend/tests/wm_token_superadmin_guard.rs b/backend/tests/wm_token_superadmin_guard.rs new file mode 100644 index 0000000000..650cf384d4 --- /dev/null +++ b/backend/tests/wm_token_superadmin_guard.rs @@ -0,0 +1,206 @@ +//! A WM_TOKEN (job JWT) running as a superadmin must not be able to perform +//! global user/token management — promotion, password reset, user creation, +//! token creation/impersonation, offboarding, or exporting the user table. +//! A non-admin `wm_deployers` member can mint +//! such a token implicitly via an app/flow `on_behalf_of`, so trusting it would +//! let them establish *persistent* superadmin. A real superadmin who needs this +//! from a script must use a dedicated superadmin API token (which only a real +//! superadmin can create), not `$WM_TOKEN`. +//! +//! The fixture provides `test@windmill.dev` (instance superadmin, token +//! `SECRET_TOKEN`) and `test2@windmill.dev` (non-superadmin, `SECRET_TOKEN_2`). + +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_common::auth::create_jwt_token; +use windmill_common::db::Authed; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder { + builder.header("Authorization", format!("Bearer {}", token)) +} + +/// Mint a WM_TOKEN: an internally-signed job JWT (note the `job_id` claim) for +/// `email`, exactly as a running app/flow job is issued. +async fn wm_token(email: &str, is_admin: bool) -> String { + let authed = Authed { + email: email.to_string(), + username: "runner".to_string(), + is_admin, + is_operator: false, + groups: vec![], + folders: vec![], + scopes: None, + token_prefix: None, + }; + create_jwt_token( + authed, + "test-workspace", + 3600, + Some(uuid::Uuid::new_v4()), + Some("app".to_string()), + None, + None, + ) + .await + .expect("mint wm_token") +} + +#[sqlx::test(fixtures("preserve_on_behalf_of"))] +async fn test_wm_token_cannot_manage_superadmin_users(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + // The server decodes WM_TOKENs with the same in-process JWT secret, so + // setting it once lets us mint a valid one below. + set_jwt_secret().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/users"); + + // A superadmin-capable WM_TOKEN — the exact thing a deployer obtains via an + // app on_behalf_of pointed at a superadmin. + let sa_wm = wm_token("test@windmill.dev", true).await; + + // 1. Cannot mint a (superadmin) token. + let resp = authed(client().post(format!("{base}/tokens/create")), &sa_wm) + .json(&json!({})) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not create tokens: {}", + resp.text().await? + ); + + // 2. Cannot impersonate (mint a token as another user). + let resp = authed(client().post(format!("{base}/tokens/impersonate")), &sa_wm) + .json(&json!({ "impersonate_email": "test2@windmill.dev" })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not impersonate: {}", + resp.text().await? + ); + + // 3. Cannot promote a user to superadmin. + let resp = authed( + client().post(format!("{base}/update/test2@windmill.dev")), + &sa_wm, + ) + .json(&json!({ "is_super_admin": true })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not promote users: {}", + resp.text().await? + ); + + // 4. Cannot reset its own (the superadmin's) password. + let resp = authed(client().post(format!("{base}/setpassword")), &sa_wm) + .json(&json!({ "password": "hunter2" })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not reset passwords: {}", + resp.text().await? + ); + + // 4b. Cannot delete a user. + let resp = authed( + client().delete(format!("{base}/delete/test2@windmill.dev")), + &sa_wm, + ) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not delete users: {}", + resp.text().await? + ); + + // 4c. Cannot change a user's login type. + let resp = authed( + client().post(format!("{base}/set_login_type/test2@windmill.dev")), + &sa_wm, + ) + .json(&json!({ "login_type": "password" })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not change login type: {}", + resp.text().await? + ); + + // 4d. Cannot offboard a global user (deletes user, tokens, password, invites, + // instance-group membership and reassigns their assets). + let resp = authed( + client().post(format!("{base}/offboard/test2@windmill.dev")), + &sa_wm, + ) + .json(&json!({})) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not offboard users: {}", + resp.text().await? + ); + + // 4e. Cannot export the global user table (leaks every user's password_hash). + let resp = authed(client().get(format!("{base}/export")), &sa_wm) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not export global users: {}", + resp.text().await? + ); + + // 5. Escape hatch / no false positive: a real superadmin API token + // (SECRET_TOKEN, no job_id) can still create tokens. + let resp = authed( + client().post(format!("{base}/tokens/create")), + "SECRET_TOKEN", + ) + .json(&json!({ "label": "ci" })) + .send() + .await?; + assert_eq!( + resp.status(), + 201, + "a real superadmin token must still create tokens: {}", + resp.text().await? + ); + + // 6. No collateral: a non-superadmin WM_TOKEN can still create its own + // token — the guard only fires for superadmin-capable job tokens. + let user_wm = wm_token("test2@windmill.dev", false).await; + let resp = authed(client().post(format!("{base}/tokens/create")), &user_wm) + .json(&json!({ "label": "from-script" })) + .send() + .await?; + assert_eq!( + resp.status(), + 201, + "non-superadmin WM_TOKEN must still create its own token: {}", + resp.text().await? + ); + + Ok(()) +} diff --git a/backend/windmill-api-auth/src/lib.rs b/backend/windmill-api-auth/src/lib.rs index b9e6a748d4..77aeeba947 100644 --- a/backend/windmill-api-auth/src/lib.rs +++ b/backend/windmill-api-auth/src/lib.rs @@ -205,6 +205,33 @@ pub async fn require_super_admin(db: &DB, email: &str) -> error::Result<()> { } } +/// Forbid sensitive global user/token management when authenticated as a +/// superadmin *via a job token* (`WM_TOKEN`). +/// +/// A `WM_TOKEN`'s identity is derived from an app/flow `on_behalf_of`, which a +/// non-admin `wm_deployers` member can point at a superadmin. Trusting it for +/// these operations would let them establish *persistent* superadmin (promote a +/// user, reset a superadmin's password, mint a superadmin token, ...). `job_id` +/// is set only for `WM_TOKEN`s; regular session/API tokens have it `None`, so a +/// real superadmin who needs this from a script uses a dedicated superadmin API +/// token (which only a real superadmin can create) instead of `$WM_TOKEN`. +pub async fn forbid_superadmin_job_token( + db: &DB, + email: &str, + job_id: Option, +) -> error::Result<()> { + if job_id.is_some() && is_super_admin_email(db, email).await? { + return Err(Error::NotAuthorized( + "This operation cannot be performed with a job token ($WM_TOKEN) that runs as a \ + superadmin. If a script genuinely needs to do this, create a dedicated superadmin \ + token from the User settings drawer (the 'Tokens' section), store it as a secret, \ + and use that token explicitly instead of $WM_TOKEN." + .to_owned(), + )); + } + Ok(()) +} + pub fn check_scopes(authed: &ApiAuthed, required: F) -> error::Result<()> where F: FnOnce() -> String, diff --git a/backend/windmill-api-users/src/users.rs b/backend/windmill-api-users/src/users.rs index db9ecb118d..63e6c2b4a7 100644 --- a/backend/windmill-api-users/src/users.rs +++ b/backend/windmill-api-users/src/users.rs @@ -27,7 +27,7 @@ use axum::{ Json, Router, }; use hyper::{header::LOCATION, StatusCode}; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin, OptJobAuthed}; use windmill_common::usernames::{ generate_instance_wide_unique_username, get_instance_username_or_create_pending, }; @@ -1415,11 +1415,13 @@ async fn convert_user_to_group( async fn update_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Path(email_to_update): Path, Extension(db): Extension, Json(eu): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; let mut new_super_admin: Option = None; @@ -1581,10 +1583,12 @@ async fn update_user( async fn delete_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Path(email_to_delete): Path, Extension(db): Extension, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; sqlx::query!("DELETE FROM token WHERE email = $1", &email_to_delete) @@ -1877,9 +1881,11 @@ async fn set_login_type( Extension(db): Extension, Path(email): Path, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(et): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; sqlx::query!( @@ -2159,8 +2165,10 @@ pub async fn create_session_token<'c>( async fn create_token( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(token_config): Json, ) -> Result<(StatusCode, String)> { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; check_token_create_rate_limit(&authed.username)?; windmill_api_auth::ensure_scopes_within_caller(&authed, token_config.scopes.as_deref())?; @@ -2176,6 +2184,7 @@ async fn create_token( async fn impersonate( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(new_token): Json, ) -> Result<(StatusCode, String)> { use windmill_common::min_version::MIN_VERSION_SUPPORTS_TOKEN_HASH; @@ -2189,6 +2198,7 @@ async fn impersonate( Some(&token) }; require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; if new_token.impersonate_email.is_none() { return Err(Error::BadRequest( @@ -2707,8 +2717,10 @@ struct ExportedGlobalUser { async fn export_global_users( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, ) -> JsonResult> { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; let users = sqlx::query_as!( ExportedGlobalUser, @@ -2744,9 +2756,11 @@ async fn export_global_users() -> JsonResult { async fn overwrite_global_users( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(users): Json>, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; sqlx::query!("DELETE FROM password") .execute(&mut *tx) diff --git a/backend/windmill-api/src/offboarding.rs b/backend/windmill-api/src/offboarding.rs index 3d5342e3a5..24b6d3f10d 100644 --- a/backend/windmill-api/src/offboarding.rs +++ b/backend/windmill-api/src/offboarding.rs @@ -1,13 +1,13 @@ use std::collections::HashMap; -use crate::db::ApiAuthed; +use crate::db::{ApiAuthed, OptJobAuthed}; use crate::secret_backend_ext::rename_vault_secrets_with_prefix; use axum::{ extract::{Extension, Path}, Json, }; use serde::{Deserialize, Serialize}; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin}; use windmill_api_users::users::delete_workspace_user_internal; use windmill_audit::audit_oss::audit_log; use windmill_audit::ActionKind; @@ -483,11 +483,13 @@ pub(crate) async fn global_offboard_preview( pub(crate) async fn offboard_global_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Extension(db): Extension, Path(email): Path, Json(req): Json, ) -> Result> { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let workspaces = sqlx::query!( "SELECT workspace_id, username FROM usr WHERE email = $1", diff --git a/backend/windmill-api/src/users.rs b/backend/windmill-api/src/users.rs index 0da987e675..36b0d6c734 100644 --- a/backend/windmill-api/src/users.rs +++ b/backend/windmill-api/src/users.rs @@ -11,7 +11,7 @@ pub use windmill_api_users::users::*; use std::sync::Arc; -use crate::db::ApiAuthed; +use crate::db::{ApiAuthed, OptJobAuthed}; use crate::secret_backend_ext::rename_vault_secrets_with_prefix; use argon2::Argon2; use axum::{ @@ -21,7 +21,7 @@ use axum::{ }; use hyper::StatusCode; use serde::Deserialize; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin}; use windmill_audit::audit_oss::audit_log; use windmill_audit::ActionKind; use windmill_common::audit::AuditAuthor; @@ -71,11 +71,13 @@ pub fn make_unauthed_service() -> Router { async fn create_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Extension(db): Extension, Extension(webhook): Extension, Extension(argon2): Extension>>, Json(nu): Json, ) -> Result<(StatusCode, String)> { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; crate::users_oss::create_user(authed, db, webhook, argon2, nu).await } @@ -141,8 +143,10 @@ async fn set_password( Extension(db): Extension, Extension(argon2): Extension>>, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(ep): Json, ) -> Result { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let email = authed.email.clone(); crate::users_oss::set_password(db, argon2, authed, &email, ep).await } @@ -152,9 +156,11 @@ async fn set_password_of_user( Extension(argon2): Extension>>, Path(email): Path, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(ep): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; crate::users_oss::set_password(db, argon2, authed, &email, ep).await } @@ -165,11 +171,13 @@ struct RenameUser { async fn rename_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Path(user_email): Path, Extension(db): Extension, Json(ru): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; From 250a05f544ae397bb91af5fc83bf408cfe1c554d Mon Sep 17 00:00:00 2001 From: centdix <40307056+centdix@users.noreply.github.com> Date: Wed, 24 Jun 2026 07:53:14 +0200 Subject: [PATCH 044/117] fix(ai-chat): strip unclosed tag leaking into compaction summary (#9750) * fix(ai-chat): strip unclosed tag leaking into compaction summary Co-Authored-By: Claude Opus 4.8 (1M context) * fix(ai-chat): strip analysis before matching summary to avoid scratchpad leak Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .../copilot/chat/compactionPrompt.test.ts | 31 +++++++++++++++++++ .../copilot/chat/compactionPrompt.ts | 14 +++++++++ 2 files changed, 45 insertions(+) diff --git a/frontend/src/lib/components/copilot/chat/compactionPrompt.test.ts b/frontend/src/lib/components/copilot/chat/compactionPrompt.test.ts index 2161182594..b91aacfa5d 100644 --- a/frontend/src/lib/components/copilot/chat/compactionPrompt.test.ts +++ b/frontend/src/lib/components/copilot/chat/compactionPrompt.test.ts @@ -34,6 +34,37 @@ chronological thinking the model should not keep expect(formatCompactSummary(raw)).toBe('a\n\nb') }) + it('keeps the summary content when has no closing tag', () => { + const raw = '\n1. Primary Request and Intent: build the thing\n2. Pending Tasks: none' + const formatted = formatCompactSummary(raw) + expect(formatted).not.toContain('') + expect(formatted).toContain('Primary Request and Intent: build the thing') + expect(formatted).toContain('Pending Tasks: none') + }) + + it('drops the analysis scratchpad even when is left unclosed', () => { + const raw = + '\nchronological thinking the model should not keep\n\n\nthe real summary' + const formatted = formatCompactSummary(raw) + expect(formatted).not.toContain('chronological thinking') + expect(formatted).not.toContain('') + expect(formatted).not.toContain('') + expect(formatted).toBe('the real summary') + }) + + it('strips an orphaned closing summary tag', () => { + expect(formatCompactSummary('plain summary')).toBe('plain summary') + }) + + it('does not leak analysis scratchpad that mentions a literal tag', () => { + const raw = `scratchpad mentions before output +real summary` + const formatted = formatCompactSummary(raw) + expect(formatted).toBe('real summary') + expect(formatted).not.toContain('scratchpad') + expect(formatted).not.toContain('before output') + }) + it('strips every analysis block, not just the first, when the summary is untagged', () => { const raw = 'first\nkept one\nsecond\nkept two' const formatted = formatCompactSummary(raw) diff --git a/frontend/src/lib/components/copilot/chat/compactionPrompt.ts b/frontend/src/lib/components/copilot/chat/compactionPrompt.ts index 010a8cf8b8..f785682dac 100644 --- a/frontend/src/lib/components/copilot/chat/compactionPrompt.ts +++ b/frontend/src/lib/components/copilot/chat/compactionPrompt.ts @@ -100,13 +100,27 @@ export function getCompactionSummaryPrompt(): string { * well-formed-but-untagged summary is still usable. */ export function formatCompactSummary(raw: string): string { + // Strip the analysis scratchpad first: it precedes the summary and may itself + // mention / tokens that would otherwise be mistaken for the + // real summary boundary. let formatted = raw.replace(/[\s\S]*?<\/analysis>/gi, '') const summaryMatch = formatted.match(/([\s\S]*?)<\/summary>/i) if (summaryMatch) { formatted = (summaryMatch[1] ?? '').trim() + } else { + // A truncated response or a weaker model sometimes opens without + // closing it. The text after the opener is still the summary, so keep it + // rather than leak the bare tag. + const openIdx = formatted.search(//i) + if (openIdx !== -1) { + formatted = formatted.slice(openIdx) + } } + // An orphaned opener or closer left by either branch must never reach the user. + formatted = formatted.replace(/<\/?(?:analysis|summary)>/gi, '') + // Collapse the blank-line runs left behind by stripping the analysis block. return formatted.replace(/\n{3,}/g, '\n\n').trim() } From f1b5c43e02c846d14d5797ef74666a4a3af48ead Mon Sep 17 00:00:00 2001 From: Akira Yamazaki Date: Wed, 24 Jun 2026 13:53:43 +0800 Subject: [PATCH 045/117] chore: bump nixpkgs for uv 0.9.25 (#9749) --- flake.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/flake.lock b/flake.lock index 486771d582..44274d0a30 100644 --- a/flake.lock +++ b/flake.lock @@ -20,11 +20,11 @@ }, "nixpkgs": { "locked": { - "lastModified": 1764517877, - "narHash": "sha256-pp3uT4hHijIC8JUK5MEqeAWmParJrgBVzHLNfJDZxg4=", + "lastModified": 1768377964, + "narHash": "sha256-RU35vQnfg9NwJUviGCfMH9ChgHANoNSiRaAn4/wINT4=", "owner": "NixOS", "repo": "nixpkgs", - "rev": "2d293cbfa5a793b4c50d17c05ef9e385b90edf6c", + "rev": "cadda13afe838615fb74b0a9720905920559c535", "type": "github" }, "original": { From c017f7f8919a51292ddf01574961d1774bc1ba23 Mon Sep 17 00:00:00 2001 From: centdix <40307056+centdix@users.noreply.github.com> Date: Wed, 24 Jun 2026 07:54:11 +0200 Subject: [PATCH 046/117] fix(frontend): show AI skills settings only when global mode enabled (#9747) AI skills are only consumed by the GLOBAL chat mode's system prompt, and global mode itself is dev-gated by isGlobalAiEnabled(). Gate the workspace AI skills settings tab on the same flag so it isn't shown when the skills can't be used, and add it to gate.ts's rip-out inventory. Co-authored-by: Claude Opus 4.8 --- frontend/src/lib/components/copilot/chat/global/gate.ts | 4 ++-- .../src/lib/components/workspaceSettings/AISettings.svelte | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/frontend/src/lib/components/copilot/chat/global/gate.ts b/frontend/src/lib/components/copilot/chat/global/gate.ts index 664f06ff39..df8035de87 100644 --- a/frontend/src/lib/components/copilot/chat/global/gate.ts +++ b/frontend/src/lib/components/copilot/chat/global/gate.ts @@ -11,8 +11,8 @@ * When the mode is ready to ship to everyone, replace every call to * `isGlobalAiEnabled()` with `true` and delete this file. The references are * intentionally narrow (chat mode visibility, custom prompt settings, the - * `change_mode` tool enum, and the `/global_drafts` dev route) so the rip-out - * is a small grep. + * `change_mode` tool enum, the AI skills workspace settings tab, and the + * `/global_drafts` dev route) so the rip-out is a small grep. */ const STORAGE_KEY = 'wm_dev_global_ai' diff --git a/frontend/src/lib/components/workspaceSettings/AISettings.svelte b/frontend/src/lib/components/workspaceSettings/AISettings.svelte index b2501004ea..f44059e395 100644 --- a/frontend/src/lib/components/workspaceSettings/AISettings.svelte +++ b/frontend/src/lib/components/workspaceSettings/AISettings.svelte @@ -14,6 +14,7 @@ import TestAiKey from '../copilot/TestAIKey.svelte' import Label from '../Label.svelte' import AiSkillsSettings from './AiSkillsSettings.svelte' + import { isGlobalAiEnabled } from '../copilot/chat/global/gate' import SettingsPageHeader from '../settings/SettingsPageHeader.svelte' import ResourcePicker from '../ResourcePicker.svelte' import Toggle from '../Toggle.svelte' @@ -589,7 +590,7 @@ {/if} - {#if promptScope === 'workspace'} + {#if promptScope === 'workspace' && isGlobalAiEnabled()} {/if}
    From e98df38ac43823ee85209a4b09cd70690469302d Mon Sep 17 00:00:00 2001 From: Guilhem Date: Wed, 24 Jun 2026 09:23:14 +0200 Subject: [PATCH 047/117] feat(apps): show raw-app fork diffs as per-file tree items (#9491) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(apps): show raw-app fork diffs as per-file tree items Raw-app diffs previously rendered as one big YAML diff of the whole serialized app. This explodes a raw app into separate, independently collapsible diff items — one per file, one per runnable, and an app.yaml metadata item — that flow through the existing fork-diff list, sidebar tree, search and count via composite paths (/). Runnables render as script/flow rows (code shown in a Content tab), and files get extension-specific icons reused from the raw-app editor. Co-Authored-By: Claude Opus 4.8 (1M context) * chore: remove raw-app tree-diff plan doc from the branch The implementation plan was an authoring aid, not product documentation; drop it so it doesn't ship in the PR. Co-Authored-By: Claude Opus 4.8 (1M context) * feat: present raw app as an app-headed folder in the diff tree Co-Authored-By: Claude Opus 4.8 (1M context) * fix: narrow RawAppFileItem in diff viewer branch (fixes svelte-check) DiffRow.kind is a plain string so the kind check didn't narrow the union; assert the synthetic item. Also size-guard on the larger side's line count instead of the doubled total, and document normalizeRawApp's per-field value-wrapper precedence. Co-Authored-By: Claude Opus 4.8 (1M context) * style: single-line, lighter diff-tree rows for all item kinds Add a singleLine mode to WorkspaceItemRow (summary ?? path on one line; DRY'd via a shared body snippet) and use it for every diff-tree leaf, so scripts/flows/triggers/resources/etc. match the raw-app header. Bump rows to py-1.5, force font-normal, and split colours: items in text-primary, folders in text-secondary. Co-Authored-By: Claude Opus 4.8 (1M context) * refactor: extract pure diffTree model from WorkspaceDiffDrawer Move tree construction + keyboard-nav traversal + the folder-keying convention out of the 775-line component into a pure, generic, tested module (buildDiffTree → root/order/parentKeyOf/firstChildKeyOf). Parent and first-child come from a child→parent map built during construction, not from re-splitting a path at the call site, so a node's tree position and its nav parent can't drift — the class of bug behind the ArrowLeft regression. Deletes the forkDiffNav half-seam (its bug lived in the untested caller). 12 new unit tests cover order/parent/first-child incl. the storage-key-vs-friendly-path case. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): keep raw-app metadata flag + dedup runnables across path collisions Addresses two P2 review nits (Codex/claude): (1) rawAppDiffToItems marked metadata by matching path==='app.yaml', so when a real file is named app.yaml the reserved app.yaml~2 metadata item lost its flag/full-YAML toggle — now parseRawAppDiff tags the entry with isMetadata and the items read the flag; (2) runnable composite leaves weren't deduped against real files, so a real file at runnables/ could produce a duplicate leaf — now reserved (slash-normalized) like parseRawAppDiff. +2 tests. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): dedup /app.yaml metadata collision + disambiguate synthetic row keys Two follow-up P2s from Pi/Codex re-review of the prior fix: (1) parseRawAppDiff's collision set used raw file keys, so a real file /app.yaml (leading slash, which joinAppPath strips) still collided with the synthetic app.yaml leaf — now slash-normalized via a shared stripLeadingSlash, +test. (2) synthetic raw-app items (runnables rendered as script/flow) could share kind+path identity with a real workspace script/flow at /runnables/, causing duplicate {#each} keys and broken nav — itemKey now prefixes synthetic items (rawapp:). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): canonicalize raw-app file keys to dedup leading-slash collisions Codex P2: a file keyed /App.tsx on one side and App.tsx on the other became two entries that joinAppPath collapsed to one composite path → duplicate row key. asFileMap now strips the leading slash so both sides resolve to one file. +test. Co-Authored-By: Claude Opus 4.8 (1M context) * perf(apps): lazy-mount per-file diff editors as they scroll into view Exploding a raw app into N per-file rows mounted N Monaco DiffEditors at once (3 reviews flagged it). Each block's editor now mounts only when it scrolls within ~200px of the viewport (IntersectionObserver rooted on the scroll container), showing a light placeholder until then; mountedRows latches so it never unmounts on scroll-away. Verified: ~6 of 13 mount initially, the rest on scroll. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .../components/WorkspaceItemDiffViewer.svelte | 22 +- .../lib/components/WorkspaceItemRow.svelte | 89 ++-- .../components/common/table/RowIcon.svelte | 9 +- .../lib/components/raw_apps/FileIcon.svelte | 88 ++++ .../components/raw_apps/RawAppFileDiff.svelte | 97 +++++ .../raw_apps/rawAppDiffUtils.test.ts | 299 ++++++++++++++ .../components/raw_apps/rawAppDiffUtils.ts | 383 ++++++++++++++++++ .../sessions/WorkspaceDiffDrawer.svelte | 382 +++++++++-------- .../lib/components/sessions/diffTree.test.ts | 128 ++++++ .../src/lib/components/sessions/diffTree.ts | 159 ++++++++ .../components/sessions/forkDiffNav.test.ts | 32 -- .../lib/components/sessions/forkDiffNav.ts | 25 -- frontend/src/lib/editorLangUtils.ts | 16 + 13 files changed, 1465 insertions(+), 264 deletions(-) create mode 100644 frontend/src/lib/components/raw_apps/FileIcon.svelte create mode 100644 frontend/src/lib/components/raw_apps/RawAppFileDiff.svelte create mode 100644 frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts create mode 100644 frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts create mode 100644 frontend/src/lib/components/sessions/diffTree.test.ts create mode 100644 frontend/src/lib/components/sessions/diffTree.ts delete mode 100644 frontend/src/lib/components/sessions/forkDiffNav.test.ts delete mode 100644 frontend/src/lib/components/sessions/forkDiffNav.ts diff --git a/frontend/src/lib/components/WorkspaceItemDiffViewer.svelte b/frontend/src/lib/components/WorkspaceItemDiffViewer.svelte index 2ee01b4607..90abb0ebf2 100644 --- a/frontend/src/lib/components/WorkspaceItemDiffViewer.svelte +++ b/frontend/src/lib/components/WorkspaceItemDiffViewer.svelte @@ -4,6 +4,9 @@ Inline diff renderer for a single workspace item. Mirrors the per-kind rendering that DiffDrawer does in its body (`DiffDrawer.svelte:181-271`): - `flow` → `` (its own Graph / YAML toggle inside) +- `raw_app_file` → `` (one synthesized raw-app file item: a + single diff with a per-file size guard; the metadata item adds a full-app + YAML expand). Raw apps are exploded into these items by `rawAppDiffToItems`. - has `content` (scripts) → Tabs(Content | Metadata) with two Monaco diffs - everything else (apps, resources, variables, schedules, triggers…) → a single Monaco YAML diff over the metadata @@ -18,12 +21,15 @@ doesn't reflow the parent. import Tabs from './common/tabs/Tabs.svelte' import Tab from './common/tabs/Tab.svelte' import FlowDiffViewer from './FlowDiffViewer.svelte' + import RawAppFileDiff from './raw_apps/RawAppFileDiff.svelte' + import type { RawAppFileItem } from './raw_apps/rawAppDiffUtils' import { Loader2 } from 'lucide-svelte' import { cleanValueProperties, orderedYamlStringify, replaceFalseWithUndefined } from '$lib/utils' import { scriptLangToEditorLang } from '$lib/scripts' interface Props { - /** Any WorkspaceItemDiff['kind'] — used only to special-case `flow`. */ + /** Any WorkspaceItemDiff['kind'], plus the synthetic `raw_app_file`. + * `flow` and `raw_app_file` are special-cased. */ kind: string /** Raw value from `getItemValue(kind, path, parentWorkspace)`. Undefined * for "added" items (don't exist in the parent). */ @@ -33,9 +39,11 @@ doesn't reflow the parent. currentRaw?: unknown /** Force unified diff (Monaco renderSideBySide=false). Default false. */ inlineDiff?: boolean + /** For `raw_app_file`: the synthesized per-file diff item to render. */ + rawFile?: RawAppFileItem } - let { kind, originalRaw, currentRaw, inlineDiff = false }: Props = $props() + let { kind, originalRaw, currentRaw, inlineDiff = false, rawFile }: Props = $props() type Prepared = { lang?: string; content?: string; metadata: string } @@ -102,6 +110,16 @@ doesn't reflow the parent. {inlineDiff} /> +{:else if kind === 'raw_app_file' && rawFile} + {:else if hasContent}
    diff --git a/frontend/src/lib/components/WorkspaceItemRow.svelte b/frontend/src/lib/components/WorkspaceItemRow.svelte index fe923b12ca..6e5900594f 100644 --- a/frontend/src/lib/components/WorkspaceItemRow.svelte +++ b/frontend/src/lib/components/WorkspaceItemRow.svelte @@ -3,7 +3,9 @@ Visual row for a workspace item (script / flow / app / resource / schedule / trigger / …). Matches the leaf-row layout used by WorkspaceItemDrillPicker: RowIcon + summary line on top with mono path -beneath, or just the mono path when there's no summary. +beneath, or just the mono path when there's no summary. With `singleLine`, +both collapse to one row showing `summary ?? secondary` (summary in normal +text, the mono path as the fallback) — denser, for the diff tree. Pure presentation — the caller controls highlighting / current state via props, supplies the onclick/onmouseenter handlers, and can pass an @@ -26,6 +28,9 @@ doesn't steal focus from a sibling search input (matches the picker). /** For `kind: 'trigger'`, specifies the concrete trigger subtype. * Forwarded to RowIcon. */ triggerKind?: string + /** For `kind: 'raw_app_file'`, the file name/path — forwarded to RowIcon + * to pick an extension-specific icon. */ + iconPath?: string /** Optional summary text shown above the path. */ summary?: string /** Mono path (or any secondary identifier). When summary is empty @@ -47,6 +52,9 @@ doesn't steal focus from a sibling search input (matches the picker). /** Reserve two lines of height and vertically center the content so * summary and summary-less rows are the same height (diff viewer). */ uniformHeight?: boolean + /** Collapse to a single line showing `summary ?? secondary` (summary in + * normal text, secondary in mono) instead of stacking both. */ + singleLine?: boolean /** Extra left padding (px) for tree-view indentation. Adds to the * default `px-3` horizontal padding. */ indent?: number @@ -69,6 +77,7 @@ doesn't steal focus from a sibling search input (matches the picker). let { kind, triggerKind, + iconPath, summary, secondary, highlighted = false, @@ -82,7 +91,8 @@ doesn't steal focus from a sibling search input (matches the picker). onclick, onmouseenter, extras, - uniformHeight = false + uniformHeight = false, + singleLine = false }: Props = $props() const rootClass = $derived( @@ -96,6 +106,37 @@ doesn't steal focus from a sibling search input (matches the picker). ) +{#snippet body()} + +
    + {#if singleLine} +
    + {summary ?? secondary} +
    + {:else if summary} +
    {summary}
    +
    + {secondary} +
    + {:else} +
    + {secondary} +
    + {/if} +
    + {#if extras} +
    + {@render extras()} +
    + {/if} +{/snippet} + {#if href} - -
    - {#if summary} -
    {summary}
    -
    - {secondary} -
    - {:else} -
    - {secondary} -
    - {/if} -
    - {#if extras} -
    - {@render extras()} -
    - {/if} + {@render body()}     {:else} {/if} diff --git a/frontend/src/lib/components/common/table/RowIcon.svelte b/frontend/src/lib/components/common/table/RowIcon.svelte index fd08c77087..ec82773a4d 100644 --- a/frontend/src/lib/components/common/table/RowIcon.svelte +++ b/frontend/src/lib/components/common/table/RowIcon.svelte @@ -19,6 +19,7 @@ Unplug, Workflow } from 'lucide-svelte' + import FileIcon from '$lib/components/raw_apps/FileIcon.svelte' interface Props { kind: @@ -26,6 +27,7 @@ | 'flow' | 'app' | 'raw_app' + | 'raw_app_file' | 'resource' | 'variable' | 'resource_type' @@ -55,10 +57,13 @@ | 'data_pipeline' /** For 'trigger' kind, specifies the specific trigger type (routes, schedules, etc.) */ triggerKind?: string | undefined + /** For 'raw_app_file' kind: the file name/path, used to pick an + * extension-specific icon. */ + path?: string | undefined size?: number } - let { kind, triggerKind = undefined, size = 16 }: Props = $props() + let { kind, triggerKind = undefined, path = undefined, size = 16 }: Props = $props() // Map per-kind backend names (e.g. `kafka_trigger`) to the legacy short // names the icon switch already handles, so we don't have to duplicate cases. @@ -85,6 +90,8 @@ {:else if effectiveKind === 'app' || effectiveKind === 'raw_app'} + {:else if effectiveKind === 'raw_app_file'} + {:else if effectiveKind === 'script'} {:else if effectiveKind === 'variable'} diff --git a/frontend/src/lib/components/raw_apps/FileIcon.svelte b/frontend/src/lib/components/raw_apps/FileIcon.svelte new file mode 100644 index 0000000000..10ff3f4868 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/FileIcon.svelte @@ -0,0 +1,88 @@ + + + +{#if spec} + {@const Icon = spec.icon} + +{/if} diff --git a/frontend/src/lib/components/raw_apps/RawAppFileDiff.svelte b/frontend/src/lib/components/raw_apps/RawAppFileDiff.svelte new file mode 100644 index 0000000000..9dc8623313 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/RawAppFileDiff.svelte @@ -0,0 +1,97 @@ + + + +
    + {#if canExpandYaml} +
    + +
    + {/if} + {#if guarded} +
    + + Large file — {lineCount.toLocaleString()} lines. + + +
    + {:else} + {#await import('$lib/components/DiffEditor.svelte')} +
    + {:then Module} +
    + +
    + {/await} + {/if} +
    diff --git a/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts new file mode 100644 index 0000000000..300b5dde1e --- /dev/null +++ b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts @@ -0,0 +1,299 @@ +import { describe, expect, it } from 'vitest' +import { + parseRawAppDiff, + rawAppDiffToItems, + RAW_APP_METADATA_PATH, + type RawAppDiffEntry +} from './rawAppDiffUtils' + +function byPath(entries: RawAppDiffEntry[], path: string): RawAppDiffEntry | undefined { + return entries.find((e) => e.path === path) +} + +describe('parseRawAppDiff — files', () => { + it('detects added, removed and modified files, omits unchanged', () => { + const original = { + files: { 'index.html': '

    hi

    ', 'styles.css': 'body{}', 'gone.js': 'x' } + } + const current = { + files: { 'index.html': '

    hello

    ', 'styles.css': 'body{}', 'new.ts': 'y' } + } + const entries = parseRawAppDiff(original, current) + const paths = entries.map((e) => e.path).sort() + // styles.css unchanged → omitted + expect(paths).toEqual(['gone.js', 'index.html', 'new.ts']) + + expect(byPath(entries, 'index.html')?.status).toBe('modified') + expect(byPath(entries, 'index.html')?.lang).toBe('html') + expect(byPath(entries, 'gone.js')?.status).toBe('removed') + expect(byPath(entries, 'new.ts')?.status).toBe('added') + expect(byPath(entries, 'new.ts')?.lang).toBe('typescript') + }) + + it('carries original/current content on the right sides', () => { + const entries = parseRawAppDiff( + { files: { 'a.txt': 'old', 'b.txt': 'keep' } }, + { files: { 'a.txt': 'new', 'b.txt': 'keep' } } + ) + const a = byPath(entries, 'a.txt')! + expect(a.original).toBe('old') + expect(a.current).toBe('new') + }) +}) + +describe('parseRawAppDiff — runnables', () => { + it('emits per-runnable leaves for add/remove/modify', () => { + const original = { + runnables: { a: { path: 'u/x/a' }, b: { path: 'u/x/b' }, same: { path: 'u/x/same' } } + } + const current = { + runnables: { a: { path: 'u/x/a2' }, c: { path: 'u/x/c' }, same: { path: 'u/x/same' } } + } + const entries = parseRawAppDiff(original, current) + expect(byPath(entries, 'runnables/a')?.status).toBe('modified') + expect(byPath(entries, 'runnables/a')?.lang).toBe('yaml') + expect(byPath(entries, 'runnables/b')?.status).toBe('removed') + expect(byPath(entries, 'runnables/c')?.status).toBe('added') + // identical runnable omitted + expect(byPath(entries, 'runnables/same')).toBeUndefined() + }) +}) + +describe('parseRawAppDiff — metadata', () => { + it('collapses summary/data/policy/custom_path into one app.yaml leaf', () => { + const entries = parseRawAppDiff( + { summary: 'old summary', custom_path: 'foo' }, + { summary: 'new summary', custom_path: 'foo' } + ) + const meta = byPath(entries, RAW_APP_METADATA_PATH) + expect(meta?.status).toBe('modified') + expect(meta?.lang).toBe('yaml') + expect(meta?.original).toContain('old summary') + expect(meta?.current).toContain('new summary') + }) + + it('omits app.yaml when no metadata field changed', () => { + const entries = parseRawAppDiff( + { files: { 'a.txt': '1' }, summary: 's' }, + { files: { 'a.txt': '2' }, summary: 's' } + ) + expect(byPath(entries, RAW_APP_METADATA_PATH)).toBeUndefined() + expect(entries).toHaveLength(1) + }) +}) + +describe('parseRawAppDiff — whole app added / removed', () => { + it('marks everything added when the original side is absent', () => { + const current = { + files: { 'index.html': '

    hi

    ' }, + runnables: { a: { path: 'u/x/a' } }, + summary: 'brand new' + } + const entries = parseRawAppDiff(undefined, current) + expect(entries.every((e) => e.status === 'added')).toBe(true) + expect(byPath(entries, 'index.html')?.original).toBeUndefined() + expect(byPath(entries, RAW_APP_METADATA_PATH)?.status).toBe('added') + }) + + it('marks everything removed when the current side is absent', () => { + const original = { + files: { 'index.html': '

    hi

    ' }, + runnables: { a: { path: 'u/x/a' } }, + summary: 'going away' + } + const entries = parseRawAppDiff(original, undefined) + expect(entries.every((e) => e.status === 'removed')).toBe(true) + expect(byPath(entries, 'index.html')?.current).toBeUndefined() + }) +}) + +describe('parseRawAppDiff — collisions', () => { + it('disambiguates synthesized paths against real files', () => { + const original = { files: { 'app.yaml': 'real-old' }, summary: 'sa' } + const current = { files: { 'app.yaml': 'real-new' }, summary: 'sb' } + const entries = parseRawAppDiff(original, current) + // The real file keeps the natural path. + const realFile = byPath(entries, 'app.yaml')! + expect(realFile.original).toBe('real-old') + // The metadata leaf is pushed to a non-colliding path. + const meta = byPath(entries, 'app.yaml~2')! + expect(meta.original).toContain('sa') + expect(meta.current).toContain('sb') + // No two entries share a path. + const paths = entries.map((e) => e.path) + expect(new Set(paths).size).toBe(paths.length) + }) +}) + +describe('parseRawAppDiff — input shapes', () => { + // getItemValue returns the deployed app row: files/runnables/data live under + // `value`; summary/policy/custom_path at the top level. + it('reads files/runnables/data from the `value` wrapper (app-row shape)', () => { + const original = { + summary: 'classy', + policy: { execution_mode: 'viewer' }, + value: { files: { 'index.html': '

    a

    ' }, runnables: {}, data: {} } + } + const current = { + summary: 'classy', + policy: { execution_mode: 'viewer' }, + value: { files: { 'index.html': '

    b

    ' }, runnables: {}, data: {} } + } + const entries = parseRawAppDiff(original, current) + const file = byPath(entries, 'index.html') + expect(file?.status).toBe('modified') + expect(file?.original).toBe('

    a

    ') + expect(file?.current).toBe('

    b

    ') + }) + + it('still handles the flat draft shape (files at top level)', () => { + const entries = parseRawAppDiff({ files: { 'a.ts': 'x' } }, { files: { 'a.ts': 'y' } }) + expect(byPath(entries, 'a.ts')?.status).toBe('modified') + }) + + it('prefers `value` over a stray top-level files key', () => { + const entries = parseRawAppDiff( + { files: { 'top.ts': 'ignored' }, value: { files: { 'real.ts': 'a' } } }, + { files: { 'top.ts': 'ignored' }, value: { files: { 'real.ts': 'b' } } } + ) + expect(byPath(entries, 'real.ts')?.status).toBe('modified') + expect(byPath(entries, 'top.ts')).toBeUndefined() + }) +}) + +describe('rawAppDiffToItems', () => { + const appPath = 'u/admin/classy_app' + + it('produces composite-pathed items with status-derived exists flags', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/App.tsx': 'a', '/gone.ts': 'x' }, runnables: { r: { p: 1 } } } }, + { value: { files: { '/App.tsx': 'b', '/new.ts': 'y' }, runnables: {} } } + ) + const byPath = (p: string) => items.find((i) => i.path === p) + + const app = byPath('u/admin/classy_app/App.tsx')! + expect(app.kind).toBe('raw_app_file') + expect(app.status).toBe('modified') + expect(app.exists_in_source).toBe(true) + expect(app.exists_in_fork).toBe(true) + expect(app.appPath).toBe(appPath) + expect((app as any).lang).toBe('typescript') + + const gone = byPath('u/admin/classy_app/gone.ts')! + expect(gone.status).toBe('removed') + expect(gone.exists_in_source).toBe(true) + expect(gone.exists_in_fork).toBe(false) + + const added = byPath('u/admin/classy_app/new.ts')! + expect(added.status).toBe('added') + expect(added.exists_in_source).toBe(false) + expect(added.exists_in_fork).toBe(true) + + // Runnables render as script/flow rows, not file leaves. + const runnable = byPath('u/admin/classy_app/runnables/r')! + expect(runnable.kind).toBe('script') + expect(runnable.status).toBe('removed') + }) + + it('renders an inline-script runnable as a script row with code hoisted', () => { + const mk = (code: string) => ({ + value: { + files: {}, + runnables: { + a: { name: 'a', type: 'inline', inlineScript: { content: code, language: 'bun' } } + } + } + }) + const items = rawAppDiffToItems(appPath, mk('old code'), mk('new code')) + const r = items.find((i) => i.path === `${appPath}/runnables/a`) + expect(r?.kind).toBe('script') + expect(r?.status).toBe('modified') + // content + language hoisted to the top level for the script-style viewer + const cur = (r as any).currentRaw + expect(cur.content).toBe('new code') + expect(cur.language).toBe('bun') + expect(cur.inlineScript.content).toBeUndefined() + }) + + it('picks the flow kind for a runType:flow runnable', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: {}, runnables: {} } }, + { value: { files: {}, runnables: { f: { runType: 'flow', path: 'u/x/f' } } } } + ) + const r = items.find((i) => i.path === `${appPath}/runnables/f`) + expect(r?.kind).toBe('flow') + expect(r?.status).toBe('added') + }) + + it('flags the metadata item and attaches whole-app YAML for the expand view', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: {} } }, + { summary: 'new', value: { files: {} } } + ) + const meta = items.find((i) => i.path === `${appPath}/${RAW_APP_METADATA_PATH}`) as any + expect(meta.kind).toBe('raw_app_file') + expect(meta.isMetadata).toBe(true) + expect(meta.fullYamlOriginal).toContain('old') + expect(meta.fullYamlCurrent).toContain('new') + }) + + it('keeps the metadata flag on the right item when a real file is named app.yaml', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: { 'app.yaml': 'real-old' } } }, + { summary: 'new', value: { files: { 'app.yaml': 'real-new' } } } + ) + // Real file keeps the natural path and is NOT the metadata item. + const realFile = items.find((i) => i.path === `${appPath}/app.yaml`) as any + expect(realFile.isMetadata).toBe(false) + expect(realFile.fullYamlCurrent).toBeUndefined() + // Synthesized metadata moved to app.yaml~2 but still carries the flag + YAML. + const meta = items.find((i) => i.path === `${appPath}/app.yaml~2`) as any + expect(meta.isMetadata).toBe(true) + expect(meta.fullYamlCurrent).toContain('new') + }) + + it('keeps the metadata flag on the right item when a real file is named /app.yaml (leading slash)', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: { '/app.yaml': 'real-old' } } }, + { summary: 'new', value: { files: { '/app.yaml': 'real-new' } } } + ) + const realFile = items.find((i) => i.path === `${appPath}/app.yaml`) as any + const meta = items.find((i) => i.path === `${appPath}/app.yaml~2`) as any + expect(realFile.isMetadata).toBe(false) + expect(meta.isMetadata).toBe(true) + // distinct composite paths → distinct row keys + expect(realFile.path).not.toBe(meta.path) + }) + + it('treats a file keyed /App.tsx on one side and App.tsx on the other as one item', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/App.tsx': 'old' } } }, + { value: { files: { 'App.tsx': 'new' } } } + ) + const files = items.filter((i) => i.path === `${appPath}/App.tsx`) + expect(files).toHaveLength(1) + expect(files[0].status).toBe('modified') + expect((files[0] as any).original).toBe('old') + expect((files[0] as any).current).toBe('new') + }) + + it('dedups a runnable leaf against a real file named runnables/', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/runnables/foo': 'old' }, runnables: { foo: { p: 1 } } } }, + { value: { files: { '/runnables/foo': 'new' }, runnables: { foo: { p: 2 } } } } + ) + const realFile = items.find((i) => i.kind === 'raw_app_file')! + const runnable = items.find((i) => i.kind === 'script')! + expect(realFile.path).toBe(`${appPath}/runnables/foo`) + // Reserved away from the real file's composite path (slash-normalized). + expect(runnable.path).toBe(`${appPath}/runnables/foo~2`) + expect(runnable.path).not.toBe(realFile.path) + }) +}) diff --git a/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts new file mode 100644 index 0000000000..444e03a0a3 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts @@ -0,0 +1,383 @@ +import { extToLang } from '$lib/editorLangUtils' +import { cleanValueProperties, orderedYamlStringify, replaceFalseWithUndefined } from '$lib/utils' + +// A raw app rendered as a *folder of files* for diffing. Each entry is one +// virtual file: real `files` keep their natural path, runnables become +// `runnables/`, and the remaining app metadata collapses into a single +// `app.yaml` leaf. Only changed entries are emitted (unchanged are omitted). +export type RawAppDiffStatus = 'added' | 'removed' | 'modified' + +export interface RawAppDiffEntry { + /** Tree path. Real file path, `runnables/`, or `app.yaml`. */ + path: string + status: RawAppDiffStatus + /** Original (parent) side content. Undefined when status is `added`. */ + original?: string + /** Current (fork) side content. Undefined when status is `removed`. */ + current?: string + /** Monaco language id for the per-file diff editor. */ + lang: string + /** True for the synthesized metadata leaf. Tracked as a flag (not by matching + * `path === 'app.yaml'`) so it survives `reserveUnique` moving the leaf to + * `app.yaml~2` when a real file is literally named `app.yaml`. */ + isMetadata?: boolean +} + +// Loose shape — diff inputs come from `getItemValue` / drafts and aren't +// strictly typed, and arrive in TWO shapes: +// - the deployed app row (getAppByPath): `{ summary, policy, custom_path?, +// value: { files, runnables, data } }` — files/runnables/data nested under +// `value`. +// - the flat draft/runtime shape (RawAppDraft / RuntimeRawApp): everything at +// the top level. +// `normalizeRawApp` coalesces both. Anything not present is empty/absent. +export interface RawAppish { + value?: Record + files?: Record + runnables?: Record + summary?: unknown + data?: unknown + policy?: unknown + custom_path?: unknown + [k: string]: unknown +} + +interface NormalizedRawApp { + files: Record + runnables: Record + summary: unknown + data: unknown + policy: unknown + custom_path: unknown +} + +export const RAW_APP_METADATA_PATH = 'app.yaml' +const RUNNABLES_PREFIX = 'runnables/' +const METADATA_FIELDS = ['summary', 'data', 'policy', 'custom_path'] as const + +// Real file keys may carry a leading slash (`/App.tsx`) which `joinAppPath` +// strips. Collision reservation must compare in the stripped space, else a real +// `/app.yaml` and the synthetic `app.yaml` leaf both become `/app.yaml`. +const stripLeadingSlash = (p: string) => p.replace(/^\/+/, '') + +function isObject(v: unknown): v is Record { + return !!v && typeof v === 'object' +} + +function asFileMap(f: unknown): Record { + if (!isObject(f)) return {} + const out: Record = {} + for (const [k, v] of Object.entries(f)) { + // Canonicalize the key (strip the leading slash `joinAppPath` would strip + // anyway) so the same file keyed `/App.tsx` on one side and `App.tsx` on the + // other is treated as one file — not two leaves colliding at the same + // composite path. Coerce non-string content so the diff editor gets a string. + out[stripLeadingSlash(k)] = typeof v === 'string' ? v : String(v ?? '') + } + return out +} + +// Coalesce the two raw-app shapes (app row with a `value` wrapper vs flat draft) +// into a canonical view. Returns undefined when the whole app is absent. +// +// Per-field precedence is deliberately asymmetric and mirrors the app-row shape: +// the editor payload (`files`/`runnables`/`data`) lives under `value`, so those +// prefer `value` first; the row-level metadata (`summary`/`policy`/`custom_path`) +// lives at the top level, so those prefer `raw` first. Each falls back to the +// other side so a flat draft (everything top-level) still normalizes correctly. +function normalizeRawApp(raw: RawAppish | undefined): NormalizedRawApp | undefined { + if (!isObject(raw)) return undefined + const value = isObject(raw.value) ? raw.value : undefined + return { + files: asFileMap(value?.files ?? raw.files), + runnables: isObject(value?.runnables ?? raw.runnables) + ? ((value?.runnables ?? raw.runnables) as Record) + : {}, + summary: raw.summary ?? value?.summary, + data: value?.data ?? raw.data, + policy: raw.policy ?? value?.policy, + custom_path: raw.custom_path ?? value?.custom_path + } +} + +// The serialized metadata blob for one side, or undefined when the whole app +// is absent (so the `app.yaml` leaf reads as added/removed). +function metadataYaml(app: NormalizedRawApp | undefined): string | undefined { + if (!app) return undefined + const meta: Record = {} + for (const field of METADATA_FIELDS) { + if (app[field] !== undefined) meta[field] = app[field] + } + return orderedYamlStringify(meta) +} + +function extOf(path: string): string { + const base = path.split('/').pop() ?? path + const dot = base.lastIndexOf('.') + return dot >= 0 ? base.slice(dot + 1).toLowerCase() : '' +} + +function diffStatus(o: string | undefined, c: string | undefined): RawAppDiffStatus | undefined { + if (o === undefined && c !== undefined) return 'added' + if (o !== undefined && c === undefined) return 'removed' + if (o !== c) return 'modified' + return undefined +} + +// Reserve a synthesized path, disambiguating against already-taken paths so a +// real file literally named `app.yaml` or under `runnables/` never collides +// with a synthesized leaf. +function reserveUnique(path: string, taken: Set): string { + if (!taken.has(path)) { + taken.add(path) + return path + } + let i = 2 + while (taken.has(`${path}~${i}`)) i++ + const p = `${path}~${i}` + taken.add(p) + return p +} + +/** + * Diff two raw-app objects into a flat list of changed virtual files. Either + * side may be undefined (whole app added or removed). Consumers build a tree + * from the returned paths with `buildFileTree`. + */ +export function parseRawAppDiff( + original: RawAppish | undefined, + current: RawAppish | undefined, + opts: { includeRunnables?: boolean } = {} +): RawAppDiffEntry[] { + const { includeRunnables = true } = opts + const oApp = normalizeRawApp(original) + const cApp = normalizeRawApp(current) + const oFiles = oApp?.files ?? {} + const cFiles = cApp?.files ?? {} + const oRunnables = oApp?.runnables ?? {} + const cRunnables = cApp?.runnables ?? {} + + // Reserve every real file path (changed or not) up front so synthesized + // runnable/metadata paths can dodge collisions — slash-normalized so a real + // `/app.yaml` or `/runnables/x` is seen as colliding with the synthetic leaf. + const taken = new Set( + [...Object.keys(oFiles), ...Object.keys(cFiles)].map(stripLeadingSlash) + ) + + const entries: RawAppDiffEntry[] = [] + + // Real files. + for (const path of [...new Set([...Object.keys(oFiles), ...Object.keys(cFiles)])].sort()) { + const o = Object.prototype.hasOwnProperty.call(oFiles, path) ? oFiles[path] : undefined + const c = Object.prototype.hasOwnProperty.call(cFiles, path) ? cFiles[path] : undefined + const status = diffStatus(o, c) + if (!status) continue + entries.push({ path, status, original: o, current: c, lang: extToLang(extOf(path)) }) + } + + // Runnables → one YAML leaf each under `runnables/`. Consumers that render + // runnables as script/flow rows (rawAppDiffToItems) skip these and diff the + // runnable objects themselves. + const runnableNames = includeRunnables + ? [...new Set([...Object.keys(oRunnables), ...Object.keys(cRunnables)])].sort() + : [] + for (const name of runnableNames) { + const inO = Object.prototype.hasOwnProperty.call(oRunnables, name) + const inC = Object.prototype.hasOwnProperty.call(cRunnables, name) + const o = inO ? orderedYamlStringify(oRunnables[name]) : undefined + const c = inC ? orderedYamlStringify(cRunnables[name]) : undefined + const status = diffStatus(o, c) + if (!status) continue + entries.push({ + path: reserveUnique(`${RUNNABLES_PREFIX}${name}`, taken), + status, + original: o, + current: c, + lang: 'yaml' + }) + } + + // Remaining metadata → single `app.yaml` leaf. + const oMeta = metadataYaml(oApp) + const cMeta = metadataYaml(cApp) + const metaStatus = diffStatus(oMeta, cMeta) + if (metaStatus) { + entries.push({ + path: reserveUnique(RAW_APP_METADATA_PATH, taken), + status: metaStatus, + original: oMeta, + current: cMeta, + lang: 'yaml', + isMetadata: true + }) + } + + return entries +} + +// A raw-app file rendered as a standalone diff item, shaped like a +// WorkspaceItemDiff (so it flows through the existing list / tree / search / +// count machinery) plus the embedded diff payload. `kind: 'raw_app_file'` +// keeps it distinct from the backend kinds. The composite `path` +// (`/`) nests it under the app's folder in the tree. +export interface RawAppFileItem { + kind: 'raw_app_file' + path: string + /** Friendly composite path (`/`) for tree display only; + * `path` stays storage-keyed so a never-deployed draft still loads/edits via + * its `…/draft_` path. Defaults to `path` when no friendly path differs. */ + displayPath?: string + ahead: number + behind: number + has_changes: boolean + exists_in_source: boolean + exists_in_fork: boolean + /** Workspace path of the owning raw app (for the edit link). */ + appPath: string + status: RawAppDiffStatus + original?: string + current?: string + lang: string + /** True for the synthesized `app.yaml` metadata item. */ + isMetadata: boolean + /** Whole serialized app (both sides) — only on the metadata item, for its + * optional "expand to full YAML" view. */ + fullYamlOriginal?: string + fullYamlCurrent?: string +} + +// A raw-app runnable rendered as a script/flow item (what it actually is). It +// carries the reshaped runnable object per side so the normal script-style +// diff (Content + Metadata) renders it — inline code shows with proper syntax +// highlighting instead of a YAML blob. `kind` drives the row icon/label +// (script vs flow); the body is always rendered script-style. +export interface RawAppRunnableItem { + kind: 'script' | 'flow' + path: string + /** Friendly composite path for tree display only (see RawAppFileItem). */ + displayPath?: string + ahead: number + behind: number + has_changes: boolean + exists_in_source: boolean + exists_in_fork: boolean + appPath: string + status: RawAppDiffStatus + /** Reshaped runnable (content/language hoisted) for the diff viewer. */ + originalRaw?: unknown + currentRaw?: unknown +} + +export type RawAppSyntheticItem = RawAppFileItem | RawAppRunnableItem + +function joinAppPath(appPath: string, filePath: string): string { + // File keys may carry a leading slash (e.g. `/App.tsx`); strip it so the + // composite path has single separators and splits cleanly in the tree. + return `${appPath}/${filePath.replace(/^\/+/, '')}` +} + +// The whole serialized app for one side, matching the previous YAML escape +// hatch (cleaned + ordered). Undefined when the app is absent on that side. +function wholeAppYaml(raw: RawAppish | undefined): string | undefined { + if (!isObject(raw)) return undefined + return orderedYamlStringify(cleanValueProperties(replaceFalseWithUndefined(raw))) +} + +// Hoist an inline runnable's code + language to the top level so the +// script-style diff viewer (which reads top-level `content`/`language`) shows +// the code in the Content tab and the rest as Metadata. Path-referencing +// runnables (no inline script) just fall through to a YAML metadata diff. +function reshapeRunnable(runnable: unknown): unknown { + if (!isObject(runnable)) return runnable + const inline = isObject(runnable.inlineScript) ? runnable.inlineScript : undefined + if (!inline) return runnable + return { + ...runnable, + content: inline.content, + language: inline.language, + // Drop the now-hoisted code so it isn't duplicated in the Metadata tab. + inlineScript: { ...inline, content: undefined } + } +} + +// Runnables become script/flow rows; `runType: 'flow'` picks the flow icon. +function runnableKind(...sides: unknown[]): 'script' | 'flow' { + return sides.some((s) => isObject(s) && s.runType === 'flow') ? 'flow' : 'script' +} + +/** + * Expand a raw-app diff into standalone items for `appPath`: + * - files + the `app.yaml` metadata leaf → `RawAppFileItem`s (the metadata item + * also carries the whole-app YAML for its expand view); + * - runnables → `RawAppRunnableItem`s (script/flow rows). + */ +export function rawAppDiffToItems( + appPath: string, + original: RawAppish | undefined, + current: RawAppish | undefined, + displayAppPath: string = appPath +): RawAppSyntheticItem[] { + // Files + metadata (runnables handled separately as script/flow rows). + const fileItems = parseRawAppDiff(original, current, { includeRunnables: false }).map( + (e): RawAppFileItem => ({ + kind: 'raw_app_file', + path: joinAppPath(appPath, e.path), + displayPath: joinAppPath(displayAppPath, e.path), + ahead: 0, + behind: 0, + has_changes: true, + exists_in_source: e.status !== 'added', + exists_in_fork: e.status !== 'removed', + appPath, + status: e.status, + original: e.original, + current: e.current, + lang: e.lang, + isMetadata: e.isMetadata ?? false + }) + ) + const metaItem = fileItems.find((i) => i.isMetadata) + if (metaItem) { + metaItem.fullYamlOriginal = wholeAppYaml(original) + metaItem.fullYamlCurrent = wholeAppYaml(current) + } + + // Runnables → script/flow rows, diffed on their object value. + const oApp = normalizeRawApp(original) + const cApp = normalizeRawApp(current) + const oRun = oApp?.runnables ?? {} + const cRun = cApp?.runnables ?? {} + // Reserve each runnable's composite leaf against the real file paths, mirroring + // parseRawAppDiff, so a real file literally named `runnables/` can't yield + // a second leaf at the same composite path. Normalize the leading slash (which + // joinAppPath strips) so `/runnables/x` and `runnables/x` are seen as equal. + const taken = new Set( + [...Object.keys(oApp?.files ?? {}), ...Object.keys(cApp?.files ?? {})].map(stripLeadingSlash) + ) + const runnableItems: RawAppRunnableItem[] = [] + for (const name of [...new Set([...Object.keys(oRun), ...Object.keys(cRun)])].sort()) { + const inO = Object.prototype.hasOwnProperty.call(oRun, name) + const inC = Object.prototype.hasOwnProperty.call(cRun, name) + const oStr = inO ? orderedYamlStringify(oRun[name]) : undefined + const cStr = inC ? orderedYamlStringify(cRun[name]) : undefined + const status = diffStatus(oStr, cStr) + if (!status) continue + const rel = reserveUnique(`${RUNNABLES_PREFIX}${name}`, taken) + runnableItems.push({ + kind: runnableKind(oRun[name], cRun[name]), + path: joinAppPath(appPath, rel), + displayPath: joinAppPath(displayAppPath, rel), + ahead: 0, + behind: 0, + has_changes: true, + exists_in_source: status !== 'added', + exists_in_fork: status !== 'removed', + appPath, + status, + originalRaw: inO ? reshapeRunnable(oRun[name]) : undefined, + currentRaw: inC ? reshapeRunnable(cRun[name]) : undefined + }) + } + + return [...fileItems, ...runnableItems] +} diff --git a/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte b/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte index 472ea1515b..c4099edd91 100644 --- a/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte +++ b/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte @@ -19,7 +19,7 @@ + +{#snippet skillIcon(_leaf: DrillLeaf)} + +{/snippet} + +
    + +
    diff --git a/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte b/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte index 070df935e5..bc920be013 100644 --- a/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte +++ b/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte @@ -2,6 +2,8 @@ import autosize from '$lib/autosize' import { tick } from 'svelte' import type { ContextElement } from './context' + import { AIMode } from './AIChatManager.svelte' + import ChatCommandPicker from './ChatCommandPicker.svelte' import ChatContextPicker from './ChatContextPicker.svelte' import Portal from '$lib/components/Portal.svelte' import { zIndexes } from '$lib/zIndexes' @@ -68,11 +70,22 @@ let showContextTooltip = $state(false) let contextTooltipWord = $state('') + let showCommandTooltip = $state(false) + let commandTooltipWord = $state('') let textarea = $state(undefined) let tooltipElement = $state(undefined) let chatContextPicker: ChatContextPicker | undefined = $state() + let chatCommandPicker: ChatCommandPicker | undefined = $state() + let commandSkillsRefreshInFlight = false - // Virtual reference anchored at the `@` that opened the mention (not the + const commandSkills = $derived( + aiChatManager.mode === AIMode.GLOBAL && aiChatManager.isSessionChat + ? aiChatManager.globalSkills + : [] + ) + const activeTooltipWord = $derived(showContextTooltip ? contextTooltipWord : commandTooltipWord) + + // Virtual reference anchored at the trigger that opened the picker (not the // caret), so the picker stays put while the user types the query. // svelte-floating-ui's `createVirtualElement` takes a raw ClientRect and // wraps it in a function internally — re-`update()` on each anchor move. @@ -526,19 +539,33 @@ showContextTooltip = false } + function refreshCommandSkills() { + if (commandSkillsRefreshInFlight) return + commandSkillsRefreshInFlight = true + void aiChatManager.refreshGlobalSkills().finally(() => { + commandSkillsRefreshInFlight = false + }) + } + + function getCommandFilter(text: string): string | undefined { + if (aiChatManager.mode !== AIMode.GLOBAL || !aiChatManager.isSessionChat) return undefined + const match = /^\/([a-z0-9-]*)$/.exec(text) + return match?.[1] + } + function updateAnchorRect() { if (!textarea) return + const triggerWord = activeTooltipWord + if (!triggerWord) return try { - // Index of the `@` that started the current mention. handleInput - // only opens the picker when `contextTooltipWord` (= `@xxx`) is the - // LAST whitespace-separated word in `value`, so the `@` always sits - // at `value.length - contextTooltipWord.length`. - const atIndex = value.length - contextTooltipWord.length - const coords = getCaretCoordinates(textarea, atIndex) + // Inline `@` anchors to the last word; slash commands only open when + // `/...` is the whole input, so the trigger sits at index 0. + const triggerIndex = triggerWord.startsWith('/') ? 0 : value.length - triggerWord.length + const coords = getCaretCoordinates(textarea, triggerIndex) const rect = textarea.getBoundingClientRect() // getCaretCoordinates returns content-relative coords; subtract the - // textarea's own scroll so the anchor tracks the `@` once the input is - // capped (max-height) and scrolls internally. + // textarea's own scroll so the anchor tracks the trigger once the input + // is capped (max-height) and scrolls internally. anchorRect = new DOMRect( rect.left + coords.left - textarea.scrollLeft, rect.top + coords.top - textarea.scrollTop, @@ -558,6 +585,19 @@ function handleInput(e: Event) { textarea = e.target as HTMLTextAreaElement + const commandFilter = getCommandFilter(value) + if (commandFilter !== undefined) { + const wasShowing = showCommandTooltip + showCommandTooltip = true + commandTooltipWord = `/${commandFilter}` + showContextTooltip = false + contextTooltipWord = '' + if (!wasShowing) refreshCommandSkills() + return + } + showCommandTooltip = false + commandTooltipWord = '' + const words = value.split(/\s+/) const lastWord = words[words.length - 1] @@ -574,6 +614,12 @@ } } + function handleCommandSelection(skill: { name: string }) { + value = `/${skill.name} ` + showCommandTooltip = false + setTimeout(() => textarea?.focus(), 0) + } + function handleKeyDown(e: KeyboardEvent) { // Pass to parent first if provided if (onKeyDown) { @@ -585,6 +631,22 @@ return } + if (showCommandTooltip) { + if ( + e.key === 'ArrowDown' || + e.key === 'ArrowUp' || + e.key === 'Enter' || + e.key === 'Tab' || + e.key === 'Escape' + ) { + chatCommandPicker?.handleKeydown(e) + } + if (e.key === 'Enter') { + e.preventDefault() + } + return + } + if (showContextTooltip) { // Forward navigation keys to the picker so the textarea-focused // user can drive it. The picker preventDefault/stopPropagation's @@ -622,11 +684,11 @@ } $effect(() => { - // Re-track on every value change. The `@` position can shift when the - // user adds/deletes text BEFORE it (line wrap, etc.); the picker should - // follow. floating-ui's autoUpdate only fires on scroll/resize. + // Re-track on every value change. The trigger position can shift when + // the user adds/deletes text before it (line wrap, etc.); the picker + // should follow. floating-ui's autoUpdate only fires on scroll/resize. void value - if (showContextTooltip) updateAnchorRect() + if (showContextTooltip || showCommandTooltip) updateAnchorRect() }) $effect(() => { @@ -700,9 +762,9 @@ ondragstart={handlePasteDragStart} onscroll={(e) => { scrollTop = e.currentTarget.scrollTop - // Keep the `@` picker pinned to its anchor while the input scrolls + // Keep the picker pinned to its anchor while the input scrolls // internally (autoUpdate can't observe a virtual ref's scroll). - if (showContextTooltip) updateAnchorRect() + if (showContextTooltip || showCommandTooltip) updateAnchorRect() }} onblur={() => { setTimeout(() => { @@ -711,6 +773,7 @@ return } showContextTooltip = false + showCommandTooltip = false }, 200) }} {placeholder} @@ -724,7 +787,7 @@ >
    -{#if showContextTooltip} +{#if showContextTooltip || showCommandTooltip}
    - { - handleContextSelection(element) - }} - onSelectWorkspaceItem={(element) => { - onAddContext(element) - updateInstructionsWithContext(element) - showContextTooltip = false - setTimeout(() => textarea?.focus(), 0) - }} - externalFilter={contextTooltipWord.slice(1)} - autoFocus={false} - setShowing={(showing) => { - showContextTooltip = showing - }} - onSelectFile={(name) => { - // Replace the in-progress `@word` with the chosen mention (bracketed if the - // filename has spaces, so the highlighter captures it whole). - const index = value.lastIndexOf('@') - value = (index !== -1 ? value.substring(0, index) : value) + `${formatMention(name)} ` - showContextTooltip = false - setTimeout(() => textarea?.focus(), 0) - }} - /> + {#if showCommandTooltip} + { + showCommandTooltip = showing + }} + /> + {:else} + { + handleContextSelection(element) + }} + onSelectWorkspaceItem={(element) => { + onAddContext(element) + updateInstructionsWithContext(element) + showContextTooltip = false + setTimeout(() => textarea?.focus(), 0) + }} + externalFilter={contextTooltipWord.slice(1)} + autoFocus={false} + setShowing={(showing) => { + showContextTooltip = showing + }} + onSelectFile={(name) => { + // Replace the in-progress `@word` with the chosen mention (bracketed if the + // filename has spaces, so the highlighter captures it whole). + const index = value.lastIndexOf('@') + value = (index !== -1 ? value.substring(0, index) : value) + `${formatMention(name)} ` + showContextTooltip = false + setTimeout(() => textarea?.focus(), 0) + }} + /> + {/if}
    {/if} From fada673bb40b7c1d4b05e18375107d89d31208d0 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Wed, 24 Jun 2026 13:58:08 +0200 Subject: [PATCH 053/117] chore: bump uv to 0.11.24 in images and CI (#9759) Co-authored-by: Claude Opus 4.8 (1M context) --- .github/DockerfileBackendTests | 2 +- .github/workflows/backend-test-windows.yml | 2 +- .github/workflows/backend-test.yml | 2 +- Dockerfile | 2 +- docker/DockerfileSlim | 2 +- docker/DockerfileSlimEe | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/DockerfileBackendTests b/.github/DockerfileBackendTests index 88275f204b..9b29f72a64 100644 --- a/.github/DockerfileBackendTests +++ b/.github/DockerfileBackendTests @@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin" ENV GO_PATH=/usr/local/go/bin/go # UV -RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv +RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv ENV TZ=Etc/UTC diff --git a/.github/workflows/backend-test-windows.yml b/.github/workflows/backend-test-windows.yml index 1c73e5d429..dd8764f6e8 100644 --- a/.github/workflows/backend-test-windows.yml +++ b/.github/workflows/backend-test-windows.yml @@ -74,7 +74,7 @@ jobs: - uses: astral-sh/setup-uv@v6.2.1 with: - version: "0.9.25" + version: "0.11.24" - uses: shivammathur/setup-php@v2 with: diff --git a/.github/workflows/backend-test.yml b/.github/workflows/backend-test.yml index 8f1f15447c..ffaaf74c9e 100644 --- a/.github/workflows/backend-test.yml +++ b/.github/workflows/backend-test.yml @@ -62,7 +62,7 @@ jobs: node-version: "20" - uses: astral-sh/setup-uv@v6.2.1 with: - version: "0.9.25" + version: "0.11.24" - uses: shivammathur/setup-php@v2 with: php-version: "8.3" diff --git a/Dockerfile b/Dockerfile index d327c9b394..3b9714c8a0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -233,7 +233,7 @@ ENV PATH="${PATH}:/usr/local/go/bin" ENV GO_PATH=/usr/local/go/bin/go # Install UV -RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv +RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv # Preinstall python runtimes to temp build location (will copy with world-writable perms later) # --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run diff --git a/docker/DockerfileSlim b/docker/DockerfileSlim index aca18aea22..ef3fb1261f 100644 --- a/docker/DockerfileSlim +++ b/docker/DockerfileSlim @@ -54,7 +54,7 @@ RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmo ENV TZ=Etc/UTC # Install UV -RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv +RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv # Preinstall python runtime to temp location (will copy with world-writable perms later) # --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run diff --git a/docker/DockerfileSlimEe b/docker/DockerfileSlimEe index b8e5c06a01..3e36d67a12 100644 --- a/docker/DockerfileSlimEe +++ b/docker/DockerfileSlimEe @@ -54,7 +54,7 @@ RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmo ENV TZ=Etc/UTC # Install UV -RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv +RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv # Preinstall python runtime to temp location (will copy with world-writable perms later) # --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run From 3d48ba7738c3d3356539b5fc44a871f6b7f9d548 Mon Sep 17 00:00:00 2001 From: Guilhem Date: Wed, 24 Jun 2026 13:58:31 +0200 Subject: [PATCH 054/117] feat(frontend): add filter submenu to collapsed AI sessions popover (#9757) Co-authored-by: Claude Opus 4.8 (1M context) --- .../lib/components/meltComponents/Menu.svelte | 19 +++++- .../sessions/SessionFilterMenu.svelte | 62 +++++++++++++++++++ .../components/sessions/SessionPicker.svelte | 22 +++++-- 3 files changed, 96 insertions(+), 7 deletions(-) create mode 100644 frontend/src/lib/components/sessions/SessionFilterMenu.svelte diff --git a/frontend/src/lib/components/meltComponents/Menu.svelte b/frontend/src/lib/components/meltComponents/Menu.svelte index 879df7bc2c..e97a3c4c15 100644 --- a/frontend/src/lib/components/meltComponents/Menu.svelte +++ b/frontend/src/lib/components/meltComponents/Menu.svelte @@ -24,6 +24,11 @@ menuClass?: string open?: boolean renderContent?: boolean + // Move the scroll/overflow onto an inner wrapper instead of the melt element. The + // melt element is the fixed-positioned containing block for any submenu, so overflow + // on it clips submenus that open to the side. Opt in only when using a submenu — the + // default keeps the existing single-element markup untouched for every other menu. + submenuSafe?: boolean classNames?: string triggr?: import('svelte').Snippet<[any]> children?: import('svelte').Snippet<[any]> @@ -42,6 +47,7 @@ menuClass = '', open = $bindable(false), renderContent = false, + submenuSafe = false, class: classNames = '', triggr, children @@ -60,6 +66,7 @@ //Melt const { elements: { trigger, menu: menuElement, item }, + builders, states } = menu @@ -106,15 +113,21 @@ use:melt={$menuElement} data-menu class={twMerge( - 'z-[6000] border w-56 origin-top-right rounded-md shadow-md focus:outline-none overflow-y-auto', + 'z-[6000] border w-56 origin-top-right rounded-md shadow-md focus:outline-none', + // Default: scroll on the melt element. submenuSafe moves it to the inner + // wrapper so a side-opening submenu isn't clipped by this element's overflow. + submenuSafe ? '' : 'overflow-y-auto', lightMode ? 'bg-surface-inverse' : 'bg-surface', invisible ? 'opacity-0' : '', menuClass )} onclick={bubble('click')} > -
    - {@render children?.({ item, open })} +
    + {@render children?.({ item, open, builders })}
    {/if} diff --git a/frontend/src/lib/components/sessions/SessionFilterMenu.svelte b/frontend/src/lib/components/sessions/SessionFilterMenu.svelte new file mode 100644 index 0000000000..d026eb7cf6 --- /dev/null +++ b/frontend/src/lib/components/sessions/SessionFilterMenu.svelte @@ -0,0 +1,62 @@ + + + + +{#if $subOpen} +
    + +
    + + {#if archivedCount > 0} + + {archivedCount} archived session{archivedCount === 1 ? '' : 's'} + + {/if} +
    +
    +{/if} diff --git a/frontend/src/lib/components/sessions/SessionPicker.svelte b/frontend/src/lib/components/sessions/SessionPicker.svelte index 55683b4d96..936943e3ae 100644 --- a/frontend/src/lib/components/sessions/SessionPicker.svelte +++ b/frontend/src/lib/components/sessions/SessionPicker.svelte @@ -41,6 +41,7 @@ removeSession } from './sessionRuntime.svelte' import SessionStatusDot from './SessionStatusDot.svelte' + import SessionFilterMenu from './SessionFilterMenu.svelte' import { Menu, Menubar, MenuItem } from '$lib/components/meltComponents' import MenuButton from '$lib/components/sidebar/MenuButton.svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' @@ -326,7 +327,7 @@
    {#snippet children({ createMenu })} - + {#snippet triggr({ trigger })}
    {/snippet} - {#snippet children({ item })} + {#snippet children({ item, builders })}
    @@ -354,6 +355,15 @@ New session
    + {#if archivedCount > 0 || showArchived.val} +
    + +
    + {/if}
    {#each visibleSessions as session (session.id)} {@const runtime = getRuntime(session.id)} @@ -363,7 +373,11 @@ {@const unread = unreadFor(session)} {@const draft = hasDraft(session)} activate(session)} {item} > @@ -490,7 +504,7 @@ class={twMerge( 'flex flex-row items-center group rounded', isSelected ? 'bg-surface-hover text-primary' : 'hover:bg-surface-hover', - session.archived ? 'italic opacity-60' : '' + session.archived ? 'opacity-60' : '' )} > {#if isEditing} From 288318ac269714fc03b15622dbb86b1c28268a36 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Wed, 24 Jun 2026 14:14:52 +0200 Subject: [PATCH 055/117] fix(apps): realign legacy raw-app drafts to raw_app draft kind (#9761) Co-authored-by: Claude Opus 4.8 (1M context) --- ...realign_legacy_raw_app_draft_kind.down.sql | 3 ++ ...9_realign_legacy_raw_app_draft_kind.up.sql | 35 +++++++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql create mode 100644 backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql diff --git a/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql new file mode 100644 index 0000000000..3c69d46fc8 --- /dev/null +++ b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql @@ -0,0 +1,3 @@ +-- Irreversible data backfill: once a raw app's draft is retyped to 'raw_app' it +-- is indistinguishable from one saved as 'raw_app' by the per-kind code, so the +-- original typ='app' state cannot be reconstructed. No-op on revert. diff --git a/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql new file mode 100644 index 0000000000..d4b068a80c --- /dev/null +++ b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql @@ -0,0 +1,35 @@ +-- The pre-per-user `DRAFT_TYPE` enum had only ('script','flow','app'): a raw +-- app's draft was therefore stored as typ='app'. The new model splits app vs +-- raw_app into distinct draft kinds chosen from the deployed app's `raw_app` +-- flag, so a raw app's pre-migration draft is invisible to the per-kind lookups +-- (editor overlay, migrate-legacy, get-for-user), which all query typ='raw_app'. +-- Realign every such draft (any owner, including the legacy NULL-email row) to +-- 'raw_app' when the deployed app at that path is a raw app. + +-- Drop, don't retype, a stale 'app' row when a 'raw_app' draft already exists +-- for the same owner (the newer 'raw_app' row, saved with the per-kind code, is +-- authoritative) — retyping would collide on the draft_pkey_with_user / +-- draft_pkey_legacy partial unique indexes over (workspace_id, path, typ, email). +DELETE FROM draft d +USING app a +JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] +WHERE d.typ = 'app' + AND a.workspace_id = d.workspace_id + AND a.path = d.path + AND av.raw_app IS TRUE + AND EXISTS ( + SELECT 1 FROM draft d2 + WHERE d2.workspace_id = d.workspace_id + AND d2.path = d.path + AND d2.typ = 'raw_app' + AND d2.email IS NOT DISTINCT FROM d.email + ); + +UPDATE draft d +SET typ = 'raw_app' +FROM app a +JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] +WHERE d.typ = 'app' + AND a.workspace_id = d.workspace_id + AND a.path = d.path + AND av.raw_app IS TRUE; From 42c5e7a3fc9b74256de8806ec0d7b62e8bbf029c Mon Sep 17 00:00:00 2001 From: Guilhem Date: Wed, 24 Jun 2026 14:41:18 +0200 Subject: [PATCH 056/117] feat: scope AI sessions per workspace root with lifecycle reconcile (#9734) * feat: scope AI sessions per workspace family with lifecycle reconcile Co-Authored-By: Claude Opus 4.8 (1M context) * refactor: centralize session reconcile trigger + extract pure lifecycle decision Co-Authored-By: Claude Opus 4.8 (1M context) * perf: remove unused workspace family index * refactor: scope sessions by workspace root id, drop family_id column Co-Authored-By: Claude Opus 4.8 (1M context) * fix(sessions): preserve user-archived sessions when archiving their workspace archiveSessionsForWorkspace tagged every session archivedByWorkspace, including ones the user had already archived by hand, so a later workspace unarchive auto-restored them. Skip already-archived sessions so only workspace-archived ones are tagged, matching decideSessionLifecycle. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: archived-session banner with unarchive, suppress workspace-gone banner while archived Co-Authored-By: Claude Opus 4.8 (1M context) * fix: re-root sub-fork sessions on reconcile when an ancestor is deleted Co-Authored-By: Claude Opus 4.8 (1M context) * feat: group AI sessions by workspace family with show-all-workspaces filter Co-Authored-By: Claude Opus 4.8 (1M context) * chore: revert unrelated AIProviderPicker cosmetic changes Co-Authored-By: Claude Opus 4.8 (1M context) * fix: hide per-session unarchive when workspace is gone, show move/discard instead Co-Authored-By: Claude Opus 4.8 (1M context) * fix: GC attached files on lifecycle delete + reconcile on sidebar fork delete Addresses Codex review: deleteSessionsForWorkspace/reconcile delete now GC linked files (deleteItemsForSession), matching deleteSession; sidebar deleteFork now reconciles so surviving child forks re-root off the deleted ancestor. Also de-flaked post-rehydrate reads in the IndexedDB tests via vi.waitFor. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: don't strand user if post-delete reconcile throws; refresh stale warmSessions comment Addresses auto-review P2s: wrap reconcileAfterWorkspaceChange in deleteFork so the parent switch + navigation always runs even on reconcile failure; correct the warmSessions comment which no longer holds under 'Show all workspaces'. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: don't fail/strand fork archive+delete when client session cleanup throws Addresses cubic P1/P2 on forks/compare: the workspace archive/delete is authoritative; wrap the best-effort session cleanup + reconcile so a local IndexedDB failure neither falsely reports failure nor blocks navigation away from the gone fork. Mirrors the SidebarContent fix. Co-Authored-By: Claude Opus 4.8 (1M context) * docs: drop drafting-history aside from reconcileAfterWorkspaceChange comment Addresses auto-review P2: keep the refresh-before-reconcile invariant, drop the 'which they did inconsistently' narration per AGENTS.md (comments record constraints, not drafting history). Co-Authored-By: Claude Opus 4.8 (1M context) * fix: clean up sessions on fork-id reuse + make all workspace-mutation cleanup best-effort Addresses Codex P1s: (1) CreateWorkspaceInner 'permanently delete existing fork' (id-reuse) now drops local sessions for that id so they don't resurface on the recreated fork; (2) workspace_settings archive/delete and SidebarContent child-delete loop + main delete now treat post-mutation session cleanup as best-effort, so a local IndexedDB failure can't strand the user or abort remaining deletes (matching the compare-page fix). Co-Authored-By: Claude Opus 4.8 (1M context) * fix: make fork-reuse session cleanup fire-and-forget (non-blocking) Addresses cubic P2: don't await the best-effort cleanup so a slow IndexedDB op can't block the delete/reuse flow. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: drop previous user's transient drafts on user change Addresses Pi P1: hydrateSessions preserved transient (unsent) drafts across user changes, so user A's draft + its pending fork/workspace state bled into user B's list and got reused by createSession. onUserChange now drops transients when the email changes; reconcile (intra-user) still preserves them. Regression test added. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- ...a72e61719fae68aa4b42730f9076f3bd97441.json | 29 ++ .../windmill-api-workspaces/src/workspaces.rs | 42 ++ backend/windmill-api/openapi.yaml | 33 ++ .../sessions/SessionFilterMenu.svelte | 39 +- .../components/sessions/SessionForkBar.svelte | 14 +- .../components/sessions/SessionPicker.svelte | 478 ++++++++++-------- .../components/sessions/SessionWrapper.svelte | 78 ++- .../sessions/sessionScope.svelte.ts | 36 +- .../sessions/sessionState.svelte.ts | 329 ++++++++++-- .../components/sessions/sessionState.test.ts | 42 +- .../sessions/sessionStateIndexedDb.test.ts | 186 ++++++- .../components/sidebar/SidebarContent.svelte | 36 +- .../components/workspace/WorkspaceCard.svelte | 3 + .../CreateWorkspaceInner.svelte | 9 + .../(logged)/forks/compare/+page.svelte | 36 +- .../(root)/(logged)/sessions/+page.svelte | 36 +- .../(logged)/workspace_settings/+page.svelte | 142 ++++-- 17 files changed, 1176 insertions(+), 392 deletions(-) create mode 100644 backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json diff --git a/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json b/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json new file mode 100644 index 0000000000..484dd23e2a --- /dev/null +++ b/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT req.id AS \"id!\",\n (CASE\n WHEN usr.email IS NULL THEN 'deleted'\n WHEN workspace.deleted THEN 'archived'\n ELSE 'active'\n END) AS \"status!\"\n FROM unnest($1::text[]) AS req(id)\n LEFT JOIN workspace ON workspace.id = req.id\n LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "status!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "TextArray", + "Text" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441" +} diff --git a/backend/windmill-api-workspaces/src/workspaces.rs b/backend/windmill-api-workspaces/src/workspaces.rs index 6387bb27c8..a356d7ee88 100644 --- a/backend/windmill-api-workspaces/src/workspaces.rs +++ b/backend/windmill-api-workspaces/src/workspaces.rs @@ -197,6 +197,7 @@ pub fn global_service() -> Router { .route("/list_as_superadmin", get(list_workspaces_as_super_admin)) .route("/list", get(list_workspaces)) .route("/users", get(user_workspaces)) + .route("/session_workspace_status", post(session_workspace_status)) .route("/create", post(create_workspace)) .route("/create_fork", post(deprecated_create_workspace_fork)) .route("/exists", post(exists_workspace)) @@ -3623,6 +3624,47 @@ async fn user_workspaces( Ok(Json(WorkspaceList { email, workspaces })) } +#[derive(Deserialize)] +struct SessionWorkspaceStatusRequest { + workspace_ids: Vec, +} + +/// Reconciliation support for client-side AI sessions, which the backend cannot touch +/// directly. The client posts the workspace ids its sessions reference and uses the +/// per-id status to keep sessions in sync with workspace lifecycle: `deleted` (no row / +/// no access → unresolvable) drops the sessions, `archived` (soft-deleted, still a +/// member) archives them, `active` restores ones previously archived-by-workspace. +/// Archived and hard-deleted workspaces are absent from `user_workspaces`, so this is the +/// only way the client learns about a change made while it was away or on another device. +async fn session_workspace_status( + Extension(db): Extension, + ApiAuthed { email, .. }: ApiAuthed, + Json(req): Json, +) -> JsonResult> { + if req.workspace_ids.len() > 1000 { + return Err(Error::BadRequest( + "Too many workspace ids (max 1000)".to_string(), + )); + } + let rows = sqlx::query!( + "SELECT req.id AS \"id!\", + (CASE + WHEN usr.email IS NULL THEN 'deleted' + WHEN workspace.deleted THEN 'archived' + ELSE 'active' + END) AS \"status!\" + FROM unnest($1::text[]) AS req(id) + LEFT JOIN workspace ON workspace.id = req.id + LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2", + &req.workspace_ids[..], + email, + ) + .fetch_all(&db) + .await?; + let statuses = rows.into_iter().map(|r| (r.id, r.status)).collect(); + Ok(Json(statuses)) +} + pub async fn check_w_id_conflict<'c>(tx: &mut Transaction<'c, Postgres>, w_id: &str) -> Result<()> { if w_id == "global" { return Err(windmill_common::error::Error::BadRequest( diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index 152c512b46..2696fa1c42 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -997,6 +997,39 @@ paths: schema: $ref: "#/components/schemas/UserWorkspaceList" + /workspaces/session_workspace_status: + post: + summary: get the lifecycle status of workspaces referenced by client-side sessions + operationId: getSessionWorkspaceStatus + tags: + - workspace + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + workspace_ids: + type: array + items: + type: string + required: + - workspace_ids + responses: + "200": + description: map of workspace id to status (active, archived, or deleted) + content: + application/json: + schema: + type: object + additionalProperties: + type: string + enum: + - active + - archived + - deleted + /w/{workspace}/workspaces/get_as_superadmin: get: summary: get workspace as super admin (require to be super admin) diff --git a/frontend/src/lib/components/sessions/SessionFilterMenu.svelte b/frontend/src/lib/components/sessions/SessionFilterMenu.svelte index d026eb7cf6..e5e67d1567 100644 --- a/frontend/src/lib/components/sessions/SessionFilterMenu.svelte +++ b/frontend/src/lib/components/sessions/SessionFilterMenu.svelte @@ -11,20 +11,25 @@ // called against this specific menu instance (same pattern as DropdownSubmenuItem). builders: MenubarMenuBuilders showArchived: boolean + showAllWorkspaces: boolean archivedCount: number } - let { builders, showArchived = $bindable(), archivedCount }: Props = $props() + let { + builders, + showArchived = $bindable(), + showAllWorkspaces = $bindable(), + archivedCount + }: Props = $props() const { elements: { subTrigger, subMenu }, states: { subOpen } } = untrack(() => builders).createSubmenu() - // Count of active filters, surfaced as a subtle badge on the trigger so an - // applied filter is visible without opening the submenu. Grows as more - // filters are added here. - let activeCount = $derived(showArchived ? 1 : 0) + // Count of active (non-default) filters, surfaced as a subtle badge on the + // trigger so an applied filter is visible without opening the submenu. + let activeCount = $derived((showArchived ? 1 : 0) + (showAllWorkspaces ? 1 : 0)) -
    - startRename(session) - }, - session.archived - ? { - displayName: 'Unarchive', - icon: ArchiveRestore, - action: () => setSessionArchived(session.id, false) - } - : { - displayName: 'Archive', - icon: Archive, - action: () => setSessionArchived(session.id, true) - }, - { - displayName: 'Delete', - icon: Trash2, - type: 'delete', - action: () => (pendingDelete = session) - } - ]} + {#each sessionGroups as group (group.rootId)} + {#if showGroupHeaders} +
    + {group.name} +
    + {/if} + {#each group.sessions as session (session.id)} + {@const runtime = getRuntime(session.id)} + {@const status = runtime ? getSessionChatStatus(runtime) : 'idle'} + {@const isSelected = onSessionsPage && session.id === sessionState.currentSessionId} + {@const isEditing = editingId === session.id} + {@const unread = unreadFor(session)} + {@const draft = hasDraft(session)} +
    + {#if isEditing} + + + + { + if (e.key === 'Enter') commitRename() + else if (e.key === 'Escape') cancelRename() + }} + onblur={commitRename} + placeholder="Untitled session" + autofocus + spellcheck="false" + class="flex-1 min-w-0 bg-transparent border-0 outline-none text-xs font-normal text-primary" + /> + + {:else} +
    - {/if} -
    + {/if} + +
    + startRename(session) + }, + ...(session.archived + ? // No Unarchive when the workspace is gone — it can't persist + // (putSession guard) and reconcile would re-archive it. + isUnavailableFork(session) + ? [] + : [ + { + displayName: 'Unarchive', + icon: ArchiveRestore, + action: () => setSessionArchived(session.id, false) + } + ] + : [ + { + displayName: 'Archive', + icon: Archive, + action: () => setSessionArchived(session.id, true) + } + ]), + { + displayName: 'Delete', + icon: Trash2, + type: 'delete', + action: () => (pendingDelete = session) + } + ]} + > + {#snippet buttonReplacement()} + + + + {/snippet} + +
    + {/if} +
    + {/each} {/each}
    {/if} diff --git a/frontend/src/lib/components/sessions/SessionWrapper.svelte b/frontend/src/lib/components/sessions/SessionWrapper.svelte index c74bf294e6..027089d4ae 100644 --- a/frontend/src/lib/components/sessions/SessionWrapper.svelte +++ b/frontend/src/lib/components/sessions/SessionWrapper.svelte @@ -7,7 +7,7 @@ import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' import { AIChatManager } from '$lib/components/copilot/chat/AIChatManager.svelte' - import { userWorkspaces, usersWorkspaceStore, workspaceStore } from '$lib/stores' + import { userWorkspaces, workspaceStore } from '$lib/stores' import { WorkspaceService } from '$lib/gen' import { sendUserToast } from '$lib/toast' import Toggle from '$lib/components/Toggle.svelte' @@ -32,9 +32,11 @@ import SessionDraftBar from './SessionDraftBar.svelte' import { createSession, + deleteSessionsForWorkspace, getEffectiveWorkspaceId, moveSessionToNewFork, moveSessionToWorkspace, + reconcileAfterWorkspaceChange, renameSession, selectSession, sessionState, @@ -101,16 +103,6 @@ let archiveConfirmOpen = $state(false) let archiveAlsoFork = $state(false) - async function refreshWorkspaceList() { - // Match the SidebarContent.deleteFork pattern: replace the in-memory - // list rather than nulling it. See B1 fix. - try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) - } catch (e) { - console.error('Failed to refresh workspaces', e) - } - } - async function handleConfirmedDelete() { deleteConfirmOpen = false if (!session) return @@ -125,8 +117,9 @@ if (forkToDelete) { try { await WorkspaceService.deleteWorkspace({ workspace: forkToDelete }) + await deleteSessionsForWorkspace(forkToDelete) sendUserToast(`Deleted forked workspace ${forkToDelete}`) - await refreshWorkspaceList() + await reconcileAfterWorkspaceChange() } catch (e: any) { sendUserToast(`Failed to delete fork ${forkToDelete}: ${e?.body ?? e}`, true) } @@ -150,7 +143,7 @@ try { await WorkspaceService.archiveWorkspace({ workspace: forkToArchive }) sendUserToast(`Archived forked workspace ${forkToArchive}`) - await refreshWorkspaceList() + await reconcileAfterWorkspaceChange() } catch (e: any) { sendUserToast(`Failed to archive fork ${forkToArchive}: ${e?.body ?? e}`, true) } @@ -280,6 +273,29 @@ is position:fixed, so it doesn't count as a flex item — no stray gap when only one bar shows. -->
    + {#if session.archived && !isUnavailable} + +
    +
    + + This session is archived +
    + +
    + {/if} moveAndActivate(workspaceId)} @@ -320,17 +336,25 @@ icon: Pencil, action: () => summaryInput?.edit() }, - session.archived - ? { - displayName: 'Unarchive', - icon: ArchiveRestore, - action: () => setSessionArchived(session.id, false) - } - : { - displayName: 'Archive', - icon: Archive, - action: () => archiveAndReset() - }, + ...(session.archived + ? // No Unarchive when the workspace is gone — it can't persist + // (putSession guard) and reconcile would re-archive it. + isUnavailable + ? [] + : [ + { + displayName: 'Unarchive', + icon: ArchiveRestore, + action: () => setSessionArchived(session.id, false) + } + ] + : [ + { + displayName: 'Archive', + icon: Archive, + action: () => archiveAndReset() + } + ]), { displayName: 'Delete', icon: Trash2, @@ -403,10 +427,12 @@ hideHeader hideModeSelector wideLayout - forceDisabled={isUnavailable} + forceDisabled={isUnavailable || !!session.archived} forceDisabledMessage={isUnavailable ? 'This session is linked to a workspace that no longer exists. Move it or discard it from the banner above to keep working.' - : ''} + : session.archived + ? 'This session is archived. Unarchive it from the banner above to keep working.' + : ''} emptyHint={sessionEmptyHint} {inputPreface} /> diff --git a/frontend/src/lib/components/sessions/sessionScope.svelte.ts b/frontend/src/lib/components/sessions/sessionScope.svelte.ts index a69074fdf8..9bcf1bfd8a 100644 --- a/frontend/src/lib/components/sessions/sessionScope.svelte.ts +++ b/frontend/src/lib/components/sessions/sessionScope.svelte.ts @@ -1,32 +1,20 @@ import { derived, type Readable } from 'svelte/store' import { workspaceStore, userWorkspaces, type UserWorkspace } from '$lib/stores' -import { findWorkspaceDescendants } from '$lib/utils/workspaceHierarchy' -// Walk up parent_workspace_id chain to find the root of the fork family -// containing `id`. Falls back to the workspace itself if it has no parent -// (or its parent isn't in the user's list). -function findFamilyRoot(id: string, all: UserWorkspace[]): UserWorkspace | undefined { - let current = all.find((w) => w.id === id) - while (current?.parent_workspace_id) { - const parent = all.find((w) => w.id === current!.parent_workspace_id) - if (!parent) break - current = parent +export function workspaceRootId(id: string | undefined, all: UserWorkspace[]): string | undefined { + if (!id) return undefined + let cur = all.find((w) => w.id === id) + if (!cur) return id + while (cur?.parent_workspace_id) { + const parentId = cur.parent_workspace_id + const parent = all.find((w) => w.id === parentId) + if (!parent) return parentId + cur = parent } - return current + return cur?.id ?? id } -// Set of workspace ids a session must belong to for the user to see it in -// the sidebar list. The whole fork family is visible from any node: when -// the user is inside fork A whose root is R, sessions belonging to R or -// any sibling fork of A are listed too. Recomputes when the user switches -// workspace or when the workspace list refreshes. -export const visibleWorkspaceIds: Readable> = derived( +export const currentWorkspaceRootId: Readable = derived( [workspaceStore, userWorkspaces], - ([ws, all]) => { - if (!ws) return new Set() - const root = findFamilyRoot(ws, all) ?? ({ id: ws } as UserWorkspace) - const ids = new Set([root.id]) - for (const d of findWorkspaceDescendants(root.id, all)) ids.add(d.id) - return ids - } + ([ws, all]) => workspaceRootId(ws ?? undefined, all) ) diff --git a/frontend/src/lib/components/sessions/sessionState.svelte.ts b/frontend/src/lib/components/sessions/sessionState.svelte.ts index 64f35fca19..e3d9e620f6 100644 --- a/frontend/src/lib/components/sessions/sessionState.svelte.ts +++ b/frontend/src/lib/components/sessions/sessionState.svelte.ts @@ -11,8 +11,9 @@ import { } from '$lib/stores' import { switchWorkspace } from '$lib/storeUtils' import { getLocalSetting, storeLocalSetting } from '$lib/utils' +import { workspaceRootId } from './sessionScope.svelte' +import { type DBSchema, type IDBPDatabase } from 'idb' import { userScopedDb } from '$lib/userScopedDb' -import type { DBSchema, IDBPDatabase } from 'idb' import { deleteItemsForSession } from '../copilot/chat/files/attachedFilesDB' // Switch the global workspace iff the target differs from the active one @@ -118,6 +119,9 @@ export type Session = { // deferred to first send (via commitSessionWorkspace) so cancelling // the draft doesn't leave an orphan fork behind. pending_fork?: PendingFork + // Stable root workspace id used only for sidebar grouping. Lifecycle follows + // workspace_id, not this field. Root sessions store the same id in both fields. + workspace_root_id?: string chatId?: string target?: SessionTarget summary?: string @@ -127,6 +131,10 @@ export type Session = { // (toggleable via the picker filter). Archive is reversible — distinct // from delete, which removes the session entirely. archived?: boolean + // Set when `archived` was applied because the session's workspace was + // archived (not by the user). Lets reconciliation auto-unarchive the session + // when the workspace is unarchived, while leaving user-archived sessions be. + archivedByWorkspace?: boolean // In-memory-only flag: the session exists but isn't written to // IndexedDB until the user sends their first message. Avoids // piling abandoned drafts across `+` clicks — createSession reuses @@ -139,10 +147,10 @@ export type Session = { lastSeenCount?: number } -// Sessions live in a per-user IndexedDB (windmill-sessions::email), one record -// per session in the `sessions` store keyed by `id`. IndexedDB is the sole -// store — no localStorage fallback. The bare localStorage keys below are the -// pre-namespacing source migrated once into the first connecting user's DB. +// Sessions live in one per-user IndexedDB, one record per session in the +// `sessions` store keyed by `id`. IndexedDB is the sole store — no localStorage +// fallback. The bare localStorage keys below are the oldest (pre-namespacing) +// source, claimed once during the legacy migration. const SESSIONS_DB = 'windmill-sessions' const LEGACY_SESSIONS_KEY = 'windmill_sessions' const LEGACY_LAST_SEEN_KEY = 'windmill_sessions_last_seen_counts' @@ -216,6 +224,45 @@ async function migrateSessionsFromLocalStorage(db: IDBPDatabase): storeLocalSetting(LEGACY_LAST_SEEN_KEY, undefined) } +function sessionRootId(s: Session): string | undefined { + return ( + s.workspace_root_id ?? + workspaceRootId(s.workspace_id ?? s.pending_workspace_id, get(userWorkspaces)) ?? + s.workspace_id ?? + s.pending_workspace_id + ) +} + +function ensureSessionRootId(s: Session): boolean { + if (s.workspace_root_id || s.transient) return false + const workspaceId = s.workspace_id ?? s.pending_workspace_id + if (workspaceId && get(userWorkspaces).length === 0) return false + const root = sessionRootId(s) + if (!root) return false + s.workspace_root_id = root + return true +} + +// Recompute workspace_root_id from the live parent chain, replacing a stale +// stored value. A fork family is re-rooted when an ancestor is deleted: the +// FK's ON DELETE SET NULL nulls the child's parent_workspace_id, so the topmost +// member shifts down. Without this, a sub-fork session keeps a root pointing at +// the deleted ancestor and drops out of the sidebar (grouped under a dead root) +// even though its own workspace is still alive. Only re-roots when the +// workspace resolves in the live list — otherwise workspaceRootId falls back to +// the id itself and would clobber a valid root for a merely-unavailable workspace. +function refreshSessionRootId(s: Session): boolean { + if (s.transient) return false + const workspaceId = s.workspace_id ?? s.pending_workspace_id + if (!workspaceId) return false + const all = get(userWorkspaces) + if (!all.some((w) => w.id === workspaceId)) return false + const root = workspaceRootId(workspaceId, all) + if (!root || root === s.workspace_root_id) return false + s.workspace_root_id = root + return true +} + const sessionsDb = userScopedDb(SESSIONS_DB, { version: 1, upgrade(db) { @@ -223,7 +270,9 @@ const sessionsDb = userScopedDb(SESSIONS_DB, { db.createObjectStore('sessions', { keyPath: 'id' }) } }, - migrate: migrateSessionsFromLocalStorage + async migrate(db) { + await migrateSessionsFromLocalStorage(db) + } }) // Starts empty: the list is hydrated from the user's IndexedDB by the @@ -245,6 +294,14 @@ export const sessionState = $state<{ // so callers fire-and-forget. export async function putSession(s: Session): Promise { if (!BROWSER || s.transient) return + // Never resurrect a session whose committed workspace is gone. A live runtime + // can still write through here after reconciliation deletes its record (chatId + // seed, unread watermark), so guard once the workspace list is loaded. + if (s.workspace_id) { + const all = get(userWorkspaces) + if (all.length > 0 && !all.some((w) => w.id === s.workspace_id)) return + } + ensureSessionRootId(s) const db = await sessionsDb.whenReady() if (!db) return try { @@ -270,48 +327,234 @@ export async function deleteSessionRecord(id: string): Promise { // createSession() maintains (it prepends). whenReady() reopens for the current // user automatically, so this also handles user switch; an absent DB (logged // out / open failed) yields an empty list. -async function hydrateSessions(): Promise { +async function hydrateSessions({ dropTransients = false } = {}): Promise { + // Transient (unsent) drafts live only in memory and belong to the current + // user; preserve them across an intra-user reconcile, but drop them on a user + // change so one user's draft (and its pending fork/workspace state) never + // bleeds into the next user's list. + const transients = dropTransients ? [] : sessionState.sessions.filter((s) => s.transient) const db = await sessionsDb.whenReady() if (!db) { - sessionState.sessions = [] + sessionState.sessions = transients return } try { const all = await db.getAll('sessions') + const changed = all.filter((s) => ensureSessionRootId(s)) + for (const s of changed) await db.put('sessions', s) all.sort((a, b) => b.createdAt - a.createdAt) - sessionState.sessions = all + sessionState.sessions = [...transients, ...all] } catch (e) { console.error('Failed to load sessions from IndexedDB', e) - sessionState.sessions = [] + sessionState.sessions = transients } } -// Re-hydrate whenever the logged-in email resolves or changes. On logout -// (email → undefined) or a genuine user switch (X → Y) the in-memory list and -// active-session pointer reset so one user's sessions never bleed into another. +export type WorkspaceLifecycleStatus = 'active' | 'archived' | 'deleted' + +// The never-orphaned rule as a pure function of (session, its workspace's +// status) — no IO, so the whole truth table is unit-testable. An `undefined` +// status (workspace absent from the queried set) is a no-op, never a delete. +// deleted → delete the session +// archived → archive it, tagged archivedByWorkspace (idempotent) +// active → auto-unarchive iff WE archived it (has archivedByWorkspace) +export function decideSessionLifecycle( + session: Session, + status: WorkspaceLifecycleStatus | undefined +): { action: 'delete' | 'archive' | 'unarchive' | 'noop'; patch?: Partial } { + if (status === 'deleted') return { action: 'delete' } + if (status === 'archived') { + return session.archived + ? { action: 'noop' } + : { action: 'archive', patch: { archived: true, archivedByWorkspace: true } } + } + if (status === 'active' && session.archivedByWorkspace) { + return { action: 'unarchive', patch: { archived: undefined, archivedByWorkspace: undefined } } + } + return { action: 'noop' } +} + +// Apply a decision patch in place: `undefined` removes the key (the unarchive +// path needs the flags gone, not set to undefined), any other value assigns. +function applyLifecyclePatch(session: Session, patch: Partial): void { + for (const [k, v] of Object.entries(patch)) { + if (v === undefined) delete (session as Record)[k] + else (session as Record)[k] = v + } +} + +// Workspace switches reconcile too (to catch a workspace deleted/archived on +// another device), but throttled so rapid switching doesn't spam the status +// endpoint. Mutation-driven reconciles are unthrottled. +let lastReconcileAt = 0 +const RECONCILE_THROTTLE_MS = 30_000 + +// Reconcile every stored session against its workspace's lifecycle. Sessions are +// client-only, so the backend can't delete/archive them directly — instead the +// client asks the backend for the status of every workspace its sessions +// reference and applies the rule (see decideSessionLifecycle) the user can't see +// happen otherwise. Reached via reconcileAfterWorkspaceChange (mutations), +// load, and a throttled workspace switch. +export async function reconcileSessionsLifecycle(): Promise { + if (!BROWSER) return + lastReconcileAt = Date.now() + const db = await sessionsDb.whenReady() + if (!db) return + const wsIds = new Set() + const sessions = await db.getAll('sessions') + for (const s of sessions) if (s.workspace_id) wsIds.add(s.workspace_id) + if (wsIds.size === 0) return + + let status: Record + try { + status = await WorkspaceService.getSessionWorkspaceStatus({ + requestBody: { workspace_ids: [...wsIds] } + }) + } catch (e) { + console.error('Failed to reconcile session lifecycle', e) + return + } + + const deletedIds = new Set() + for (const s of sessions) { + if (!s.workspace_id) continue + const { action, patch } = decideSessionLifecycle(s, status[s.workspace_id]) + if (action === 'delete') { + await db.delete('sessions', s.id) + // GC linked files too, matching deleteSession — a record-only delete + // here would orphan the session's attached-file blobs/handles. + void deleteItemsForSession(s.id) + deletedIds.add(s.id) + continue + } + let changed = action === 'archive' || action === 'unarchive' + if (changed && patch) applyLifecyclePatch(s, patch) + // Re-root surviving sessions whose family topmost member shifted (an + // ancestor was deleted); fall back to backfilling a missing root. + if (refreshSessionRootId(s) || ensureSessionRootId(s)) changed = true + if (changed) await db.put('sessions', s) + } + await hydrateSessions() + // If the session the user was on lived in a now-deleted workspace, it was just + // removed — drop the dangling pointer so the page falls back to "no session + // selected" instead of a ghost. + if (sessionState.currentSessionId && deletedIds.has(sessionState.currentSessionId)) { + sessionState.currentSessionId = undefined + } +} + +// The single seam for "a workspace just changed — bring sessions back in sync." +// Refresh the workspace list FIRST — both reconcile and the putSession guard +// read it, so it must reflect the change before reconcile runs — then reconcile. +// Call this from every workspace create/delete/archive/unarchive site. +export async function reconcileAfterWorkspaceChange(): Promise { + if (!BROWSER) return + try { + usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) + } catch (e) { + console.error('Failed to refresh workspace list before reconcile', e) + } + await reconcileSessionsLifecycle() +} + +// Count non-transient sessions committed to a given workspace — used to warn the +// user, before archiving/deleting a workspace, how many AI sessions go with it. +export async function countSessionsForWorkspace(workspaceId: string): Promise { + if (!BROWSER) return 0 + const db = await sessionsDb.whenReady() + if (!db) return 0 + try { + const all = await db.getAll('sessions') + return all.filter((s) => s.workspace_id === workspaceId && !s.transient).length + } catch { + return 0 + } +} + +export async function archiveSessionsForWorkspace(workspaceId: string): Promise { + if (!BROWSER || !workspaceId) return + const db = await sessionsDb.whenReady() + if (!db) return + const all = await db.getAll('sessions') + for (const s of all) { + // Skip sessions the user already archived: tagging them archivedByWorkspace + // would make a later workspace-unarchive auto-restore them, discarding the + // user's manual archive. Mirrors decideSessionLifecycle's already-archived noop. + if (s.transient || s.workspace_id !== workspaceId || s.archived) continue + s.archived = true + s.archivedByWorkspace = true + ensureSessionRootId(s) + await db.put('sessions', s) + } + for (const s of sessionState.sessions) { + if (s.transient || s.workspace_id !== workspaceId || s.archived) continue + s.archived = true + s.archivedByWorkspace = true + } +} + +export async function deleteSessionsForWorkspace(workspaceId: string): Promise { + if (!BROWSER || !workspaceId) return + const db = await sessionsDb.whenReady() + if (!db) return + const all = await db.getAll('sessions') + const ids = new Set( + all.filter((s) => s.workspace_id === workspaceId && !s.transient).map((s) => s.id) + ) + for (const id of ids) { + await db.delete('sessions', id) + // GC linked files too (matches deleteSession) so a workspace teardown + // doesn't leave the sessions' attached-file blobs/handles orphaned. + void deleteItemsForSession(id) + } + sessionState.sessions = sessionState.sessions.filter((s) => !ids.has(s.id)) + if (sessionState.currentSessionId && ids.has(sessionState.currentSessionId)) { + sessionState.currentSessionId = undefined + } +} + +// Re-hydrate on user (email) change. The new user's persisted sessions are +// re-read from their own scoped DB (different email → different DB name); the +// in-memory list is rebuilt from scratch — including dropping the previous +// user's transient drafts — and the active-session pointer is cleared, so one +// user's sessions never bleed into another. onUserChange(async (email, prevEmail) => { if (!BROWSER) return - await hydrateSessions() + await hydrateSessions({ dropTransients: prevEmail !== email }) if (prevEmail !== undefined && prevEmail !== email) { sessionState.currentSessionId = undefined } + // Load-time reconcile: catch workspaces archived/deleted while away. + void reconcileSessionsLifecycle() }) +// Workspace switches do not reload or clear the active session. The sidebar +// filters the single loaded session list by workspace_root_id, while the current +// chat survives workspace switches. +if (BROWSER) { + let lastWorkspace: string | undefined + let initialized = false + workspaceStore.subscribe((ws) => { + const isSwitch = initialized && ws !== lastWorkspace + initialized = true + lastWorkspace = ws + if (isSwitch && Date.now() - lastReconcileAt > RECONCILE_THROTTLE_MS) { + void reconcileSessionsLifecycle() + } + }) +} + export function findSessionByName(name: string): Session | undefined { return sessionState.sessions.find((s) => s.name === name) } -// Walk up parent_workspace_id to the family root, given a starting -// workspace id. Returns the input id if no parent chain is visible. -function familyRootId(id: string | undefined, all: UserWorkspace[]): string | undefined { - if (!id) return undefined - let cur = all.find((w) => w.id === id) - while (cur?.parent_workspace_id) { - const parent = all.find((w) => w.id === cur!.parent_workspace_id) - if (!parent) break - cur = parent - } - return cur?.id ?? id +function defaultSessionWorkspaceId( + id: string | undefined, + all: UserWorkspace[] +): string | undefined { + const root = workspaceRootId(id, all) + if (root && all.some((w) => w.id === root)) return root + return id } export function createSession(): Session { @@ -327,11 +570,11 @@ export function createSession(): Session { .map((s) => /^session-(\d+)$/.exec(s.name)?.[1]) .map((n) => (n ? parseInt(n, 10) : 0)) const next = (existingNumbers.length ? Math.max(...existingNumbers) : 0) + 1 - // Default to the family root rather than wherever the user happens + // Default to the root workspace rather than wherever the user happens // to be — sessions usually start from "the canonical workspace" and // the picker lets them switch to a fork later. const currentWs = get(workspaceStore) - const root = familyRootId(currentWs ?? undefined, get(userWorkspaces)) + const root = defaultSessionWorkspaceId(currentWs ?? undefined, get(userWorkspaces)) const pending = root ?? currentWs // Friendly default summary so the header reads like "Zippy session" // rather than "Untitled session" — assigned at create time, the user @@ -425,11 +668,12 @@ export async function commitSessionWorkspace( void putSession(s) return undefined } - if (get(workspaceStore) !== newId) switchWorkspace(newId) s.workspace_id = newId s.pending_fork = undefined s.pending_workspace_id = undefined - void putSession(s) + s.workspace_root_id = workspaceRootId(newId, get(userWorkspaces)) ?? newId + await putSession(s) + if (get(workspaceStore) !== newId) switchWorkspace(newId) return newId } @@ -437,14 +681,15 @@ export async function commitSessionWorkspace( if (!ws) return undefined s.workspace_id = ws s.pending_workspace_id = undefined - // `pending_workspace_id` defaults to the family root when created from + s.workspace_root_id = workspaceRootId(ws, get(userWorkspaces)) ?? ws + await putSession(s) + // `pending_workspace_id` defaults to the root workspace when created from // inside a fork, so the committed workspace can differ from the active // workspaceStore. Without this sync, the very first AI request's // `logAiChat` and tool calls read the stale fork from workspaceStore // while the session metadata says root. Mirrors the `switchWorkspace` // in the pending_fork branch above. if (get(workspaceStore) !== ws) syncWorkspaceTo(ws) - void putSession(s) return ws } @@ -545,14 +790,15 @@ export async function materializeFork(fork: PendingFork): Promise { const s = sessionState.sessions.find((x) => x.id === id) if (!s) return if (s.workspace_id === newWorkspaceId) return s.workspace_id = newWorkspaceId delete s.pending_workspace_id delete s.pending_fork - void putSession(s) + s.workspace_root_id = workspaceRootId(newWorkspaceId, get(userWorkspaces)) ?? newWorkspaceId + await putSession(s) } // Create a brand-new fork and re-assign a committed session to it. Used @@ -567,8 +813,10 @@ export async function moveSessionToNewFork( if (!s) return undefined const newId = await materializeFork(fork) if (!newId) return undefined + // Persist the session before switching so the target workspace can show it + // immediately in the root-filtered sidebar. + await moveSessionToWorkspace(id, newId) if (get(workspaceStore) !== newId) switchWorkspace(newId) - moveSessionToWorkspace(id, newId) return newId } @@ -576,16 +824,19 @@ export function setSessionArchived(id: string, archived: boolean) { const s = sessionState.sessions.find((x) => x.id === id) if (!s) return const next = archived ? true : undefined - if (s.archived === next) return + if (s.archived === next && (archived || !s.archivedByWorkspace)) return if (archived) s.archived = true - else delete s.archived + else { + delete s.archived + delete s.archivedByWorkspace + } void putSession(s) } export function deleteSession(id: string) { - const idx = sessionState.sessions.findIndex((s) => s.id === id) - if (idx < 0) return - sessionState.sessions = sessionState.sessions.filter((s) => s.id !== id) + const s = sessionState.sessions.find((x) => x.id === id) + if (!s) return + sessionState.sessions = sessionState.sessions.filter((x) => x.id !== id) if (sessionState.currentSessionId === id) { sessionState.currentSessionId = sessionState.sessions[0]?.id } diff --git a/frontend/src/lib/components/sessions/sessionState.test.ts b/frontend/src/lib/components/sessions/sessionState.test.ts index 1af2055319..851ab27153 100644 --- a/frontend/src/lib/components/sessions/sessionState.test.ts +++ b/frontend/src/lib/components/sessions/sessionState.test.ts @@ -2,6 +2,7 @@ import { describe, it, expect, vi } from 'vitest' import { get } from 'svelte/store' import { commitSessionWorkspace, + decideSessionLifecycle, deriveForkStatus, isForkSession, renameSession, @@ -228,7 +229,7 @@ describe('commitSessionWorkspace — CE workspace-cap fork guard', () => { describe('commitSessionWorkspace — workspaceStore sync (non-fork branch)', () => { it('syncs workspaceStore to the committed workspace when they differ', async () => { // Repro: user is sitting in a fork workspace (wm-fork-x) and creates a - // new session whose pending_workspace_id defaults to the family root. + // new session whose pending_workspace_id defaults to the root workspace. // Without the syncWorkspaceTo call in commitSessionWorkspace's non-fork // branch, the session metadata says root while the active workspace // stays on the fork — so AIChatManager.chatRequest's logAiChat and tool @@ -247,6 +248,7 @@ describe('commitSessionWorkspace — workspaceStore sync (non-fork branch)', () expect(committed).toBe('root_ws') const s = sessionState.sessions.find((x) => x.id === id) expect(s?.workspace_id).toBe('root_ws') + expect(s?.workspace_root_id).toBe('root_ws') expect(s?.pending_workspace_id).toBeUndefined() expect(get(workspaceStore)).toBe('root_ws') } finally { @@ -269,6 +271,8 @@ describe('commitSessionWorkspace — workspaceStore sync (non-fork branch)', () try { const committed = await commitSessionWorkspace(id, undefined) expect(committed).toBe('root_ws') + const s = sessionState.sessions.find((x) => x.id === id) + expect(s?.workspace_root_id).toBe('root_ws') expect(get(workspaceStore)).toBe('root_ws') } finally { const i = sessionState.sessions.findIndex((x) => x.id === id) @@ -323,3 +327,39 @@ describe('session summary generation guards', () => { } }) }) + +describe('decideSessionLifecycle — the never-orphaned rule (pure)', () => { + const mk = (over: Partial = {}): Session => ({ + id: 'x', + name: 'session-1', + workspace_id: 'ws', + createdAt: 0, + ...over + }) + + it('deleted workspace → delete, regardless of archive state', () => { + expect(decideSessionLifecycle(mk(), 'deleted')).toEqual({ action: 'delete' }) + expect(decideSessionLifecycle(mk({ archived: true }), 'deleted')).toEqual({ action: 'delete' }) + }) + + it('archived workspace → archive (tagged) when not already archived; no-op otherwise', () => { + expect(decideSessionLifecycle(mk(), 'archived')).toEqual({ + action: 'archive', + patch: { archived: true, archivedByWorkspace: true } + }) + expect(decideSessionLifecycle(mk({ archived: true }), 'archived')).toEqual({ action: 'noop' }) + }) + + it('active workspace → unarchive only the ones WE archived (archivedByWorkspace)', () => { + expect(decideSessionLifecycle(mk({ archived: true, archivedByWorkspace: true }), 'active')).toEqual( + { action: 'unarchive', patch: { archived: undefined, archivedByWorkspace: undefined } } + ) + // user-archived (no archivedByWorkspace flag) is left alone + expect(decideSessionLifecycle(mk({ archived: true }), 'active')).toEqual({ action: 'noop' }) + expect(decideSessionLifecycle(mk(), 'active')).toEqual({ action: 'noop' }) + }) + + it('unknown status (workspace absent from the queried set) → no-op, never a delete', () => { + expect(decideSessionLifecycle(mk(), undefined)).toEqual({ action: 'noop' }) + }) +}) diff --git a/frontend/src/lib/components/sessions/sessionStateIndexedDb.test.ts b/frontend/src/lib/components/sessions/sessionStateIndexedDb.test.ts index 36a27095be..13802fb999 100644 --- a/frontend/src/lib/components/sessions/sessionStateIndexedDb.test.ts +++ b/frontend/src/lib/components/sessions/sessionStateIndexedDb.test.ts @@ -8,6 +8,13 @@ vi.mock('esm-env', async (importOriginal) => ({ BROWSER: true })) +// Spy on the attached-file GC so we can assert lifecycle deletes clean it up. +const { deleteItemsForSessionMock } = vi.hoisted(() => ({ deleteItemsForSessionMock: vi.fn() })) +vi.mock('../copilot/chat/files/attachedFilesDB', async (orig) => ({ + ...(await orig()), + deleteItemsForSession: deleteItemsForSessionMock +})) + // sessionState imports WorkspaceService; these tests don't touch the network. vi.mock('$lib/gen', async (orig) => { const actual = await orig() @@ -15,13 +22,24 @@ vi.mock('$lib/gen', async (orig) => { ...actual, WorkspaceService: { ...actual.WorkspaceService, - listUserWorkspaces: vi.fn().mockResolvedValue([]) + listUserWorkspaces: vi.fn().mockResolvedValue([]), + getSessionWorkspaceStatus: vi.fn().mockResolvedValue({}) } } }) -import { userStore, type UserExt } from '$lib/stores' -import { sessionState, putSession, deleteSessionRecord, type Session } from './sessionState.svelte' +import { userStore, usersWorkspaceStore, type UserExt } from '$lib/stores' +import { WorkspaceService } from '$lib/gen' +import { + sessionState, + putSession, + deleteSessionRecord, + archiveSessionsForWorkspace, + deleteSessionsForWorkspace, + reconcileSessionsLifecycle, + setSessionArchived, + type Session +} from './sessionState.svelte' function asUser(email: string): UserExt { return { email, username: email.split('@')[0] } as unknown as UserExt @@ -52,6 +70,7 @@ beforeEach(async () => { ;(globalThis as any).indexedDB = new IDBFactory() localStorage.clear() userStore.set(undefined) + usersWorkspaceStore.set(undefined) await flush() sessionState.sessions = [] sessionState.currentSessionId = undefined @@ -112,6 +131,23 @@ describe('sessionState IndexedDB persistence', () => { await vi.waitFor(() => expect(sessionState.sessions.map((s) => s.id)).toEqual(['a1'])) }) + it("drops the previous user's transient draft on a user change", async () => { + const a = freshUser() + const b = freshUser() + + userStore.set(a) + await flush() + // A starts an unsent draft — transient, in-memory only, never persisted. + sessionState.sessions = [session({ id: 'a-draft', transient: true }), ...sessionState.sessions] + + // Switch to B: A's transient must not bleed into B's list (it would + // otherwise be reused by createSession and inherit A's pending state). + userStore.set(b) + await vi.waitFor(() => { + expect(sessionState.sessions.some((s) => s.id === 'a-draft')).toBe(false) + }) + }) + it('resets the active session pointer on user switch', async () => { const a = freshUser() const b = freshUser() @@ -123,6 +159,13 @@ describe('sessionState IndexedDB persistence', () => { }) it('claims legacy localStorage sessions for the first connector, folding watermarks, then deletes the bare keys', async () => { + usersWorkspaceStore.set({ + email: 'u@x.com', + workspaces: [ + { id: 'root', name: 'root', disabled: false }, + { id: 'fork', name: 'fork', parent_workspace_id: 'root', disabled: false } + ] as never + }) localStorage.setItem( 'windmill_sessions', JSON.stringify([ @@ -130,18 +173,22 @@ describe('sessionState IndexedDB persistence', () => { // transient legacy entries are not persisted { id: 'leg2', name: 'L2', createdAt: 200, transient: true }, // legacy '' workspace marker is normalised away - { id: 'leg3', name: 'L3', createdAt: 50, workspace_id: '' } + { id: 'leg3', name: 'L3', createdAt: 50, workspace_id: '' }, + { id: 'leg4', name: 'L4', createdAt: 25, workspace_id: 'fork' } ]) ) localStorage.setItem('windmill_sessions_last_seen_counts', JSON.stringify({ leg1: 5 })) const user = freshUser() userStore.set(user) - await vi.waitFor(() => expect(sessionState.sessions.map((s) => s.id)).toEqual(['leg1', 'leg3'])) + await vi.waitFor(() => + expect(sessionState.sessions.map((s) => s.id)).toEqual(['leg1', 'leg3', 'leg4']) + ) const leg1 = sessionState.sessions.find((s) => s.id === 'leg1')! expect(leg1.lastSeenCount).toBe(5) expect(sessionState.sessions.find((s) => s.id === 'leg3')!.workspace_id).toBeUndefined() + expect(sessionState.sessions.find((s) => s.id === 'leg4')!.workspace_root_id).toBe('root') // Bare keys are gone so a later different user does not re-inherit them. expect(localStorage.getItem('windmill_sessions')).toBeNull() @@ -174,6 +221,135 @@ describe('sessionState IndexedDB persistence', () => { expect(localStorage.getItem('windmill_sessions_last_seen_counts')).toBeNull() }) + it('archiveSessionsForWorkspace tags only the sessions it archives, preserving user-archived ones', async () => { + const user = freshUser() + userStore.set(user) + await flush() + + // One clean session and one the user already archived by hand, same workspace. + await putSession(session({ id: 'clean', createdAt: 100, workspace_id: 'wsA' })) + await putSession( + session({ id: 'userarch', createdAt: 50, workspace_id: 'wsA', archived: true }) + ) + await rehydrate(user) + await vi.waitFor(() => expect(sessionState.sessions.length).toBe(2)) + + await archiveSessionsForWorkspace('wsA') + await rehydrate(user) + + // waitFor: rehydrate's re-population settles asynchronously, so re-find + // inside the retry rather than reading the list once immediately after. + await vi.waitFor(() => { + const clean = sessionState.sessions.find((s) => s.id === 'clean')! + const userarch = sessionState.sessions.find((s) => s.id === 'userarch')! + // Workspace-archived → tagged, so a later unarchive auto-restores it. + expect(clean.archived).toBe(true) + expect(clean.archivedByWorkspace).toBe(true) + // User-archived → left untouched (no tag), so unarchive won't resurrect it. + expect(userarch.archived).toBe(true) + expect(userarch.archivedByWorkspace).toBeUndefined() + }) + }) + + it("GCs each session's attached files when its workspace is torn down", async () => { + const user = freshUser() + userStore.set(user) + await flush() + await putSession(session({ id: 'f1', createdAt: 1, workspace_id: 'wsX' })) + await putSession(session({ id: 'f2', createdAt: 2, workspace_id: 'wsX' })) + await rehydrate(user) + await vi.waitFor(() => expect(sessionState.sessions.length).toBe(2)) + + deleteItemsForSessionMock.mockClear() + await deleteSessionsForWorkspace('wsX') + + // Both deleted sessions' linked files must be GC'd, not just their records. + const cleaned = deleteItemsForSessionMock.mock.calls.map((c) => c[0]).sort() + expect(cleaned).toEqual(['f1', 'f2']) + }) + + it('does not persist a per-session unarchive when the workspace is gone (resurrection guard)', async () => { + const user = freshUser() + // Seed the archived session while its workspace is still in the list so the + // write lands. + usersWorkspaceStore.set({ + email: user.email, + workspaces: [{ id: 'gone-ws', name: 'gone', disabled: false }] as never + }) + userStore.set(user) + await flush() + await putSession(session({ id: 'arch', createdAt: 1, workspace_id: 'gone-ws', archived: true })) + await rehydrate(user) + await vi.waitFor(() => expect(sessionState.sessions.map((s) => s.id)).toEqual(['arch'])) + + // The workspace disappears from the user's list (archived/deleted elsewhere). + // A non-empty list that omits it triggers the putSession resurrection guard. + usersWorkspaceStore.set({ + email: user.email, + workspaces: [{ id: 'live-ws', name: 'live', disabled: false }] as never + }) + await flush() + + // Attempting to unarchive in place must NOT persist — this is exactly why the + // per-session Unarchive control is hidden when the workspace is unavailable. + setSessionArchived('arch', false) + await flush() + + // Read straight from IndexedDB: the in-memory list is reassigned by the + // async reconcile that rehydrate kicks off, so asserting against it races; + // the DB is the source of truth for whether the unarchive persisted. + const db = await openDB(`windmill-sessions::${user.email}`, 1) + const rec = (await db.get('sessions' as never, 'arch')) as Session + db.close() + expect(rec.archived).toBe(true) + }) + + it('re-roots a sub-fork session on reconcile when an ancestor was deleted', async () => { + const user = freshUser() + // Family after a grandparent deletion: the FK's ON DELETE SET NULL nulled + // fork-1's parent, so fork-1 is now the topmost member; fork_of_fork still + // points at fork-1. + usersWorkspaceStore.set({ + email: user.email, + workspaces: [ + { id: 'wm-fork-fork-1', name: 'fork-1', disabled: false }, + { + id: 'wm-fork-fork_of_fork', + name: 'fork_of_fork', + parent_workspace_id: 'wm-fork-fork-1', + disabled: false + } + ] as never + }) + vi.mocked(WorkspaceService.getSessionWorkspaceStatus).mockResolvedValueOnce({ + 'wm-fork-fork_of_fork': 'active' + } as never) + + userStore.set(user) + await flush() + // Seed with a STALE root pointing at the now-deleted grandparent. + await putSession( + session({ + id: 'sub', + createdAt: 1, + workspace_id: 'wm-fork-fork_of_fork', + workspace_root_id: 'wm-grandparent-deleted' + }) + ) + await rehydrate(user) + await vi.waitFor(() => expect(sessionState.sessions.map((s) => s.id)).toEqual(['sub'])) + + await reconcileSessionsLifecycle() + await rehydrate(user) + + // Re-rooted to the new family topmost member, not left on the dead ancestor. + // waitFor: rehydrate's re-population settles asynchronously. + await vi.waitFor(() => { + const sub = sessionState.sessions.find((s) => s.id === 'sub')! + expect(sub.workspace_root_id).toBe('wm-fork-fork-1') + }) + }) + it('clears the in-memory list on logout', async () => { const user = freshUser() userStore.set(user) diff --git a/frontend/src/lib/components/sidebar/SidebarContent.svelte b/frontend/src/lib/components/sidebar/SidebarContent.svelte index 2e6c8d2448..1882ed33dc 100644 --- a/frontend/src/lib/components/sidebar/SidebarContent.svelte +++ b/frontend/src/lib/components/sidebar/SidebarContent.svelte @@ -4,7 +4,6 @@ superadmin, usedTriggerKinds, userStore, - usersWorkspaceStore, userWorkspaces, workspaceStore, isCriticalAlertsUIOpen, @@ -82,6 +81,10 @@ import MenuButton from './MenuButton.svelte' import GoogleCloudIcon from '../icons/GoogleCloudIcon.svelte' import AzureIcon from '../icons/AzureIcon.svelte' + import { + deleteSessionsForWorkspace, + reconcileAfterWorkspaceChange + } from '$lib/components/sessions/sessionState.svelte' async function leaveWorkspace() { await WorkspaceService.leaveWorkspace({ workspace: $workspaceStore ?? '' }) @@ -143,25 +146,34 @@ sendUserToast(`Failed to delete forked child ${child.id}: ${err}`, true) return } + // Backend delete is authoritative; session cleanup is best-effort so a + // local failure can't abort the remaining (parent) deletes. + await deleteSessionsForWorkspace(child.id).catch((e) => + console.error(`Session cleanup for ${child.id} failed`, e) + ) } } await WorkspaceService.deleteWorkspace({ workspace }) + await deleteSessionsForWorkspace(workspace).catch((e) => + console.error('Session cleanup after workspace delete failed', e) + ) sendUserToast('You deleted the workspace') if (parentStillAccessible && parentId) { - // Refresh the workspace list before landing on the parent. - // `clearStores()` would null `usersWorkspaceStore`, which the - // sidebar's `visibleSessions` filter reads via `$userWorkspaces` - // — with an empty list, every committed session falls into the - // "workspace_id set but not in user's list" branch and renders - // as "Fork — no longer available" until a hard reload. + // Refresh the workspace list AND reconcile session lifecycle before + // landing on the parent. The refresh keeps the sidebar's + // `visibleSessions` filter from rendering every committed session as + // "Fork — no longer available" (it reads `$userWorkspaces`, which + // `clearStores()` would null). The reconcile re-roots surviving child + // forks: deleting this fork without "delete children" re-parents them + // via the backend's ON DELETE SET NULL, so their sessions' stored + // `workspace_root_id` must be recomputed off the now-deleted ancestor. + // A reconcile failure must NOT strand the user on the just-deleted + // workspace — always fall through to the parent switch + navigation. try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) + await reconcileAfterWorkspaceChange() } catch (e) { - // A transient list-refresh failure must not strand the user on the - // just-deleted workspace — still switch + navigate (the list reloads - // on the next page load). - console.error('Failed to refresh workspaces after delete', e) + console.error('Failed to reconcile sessions after workspace delete', e) } switchWorkspace(parentId) await goto('/') diff --git a/frontend/src/lib/components/workspace/WorkspaceCard.svelte b/frontend/src/lib/components/workspace/WorkspaceCard.svelte index c4ac967b50..851743b56b 100644 --- a/frontend/src/lib/components/workspace/WorkspaceCard.svelte +++ b/frontend/src/lib/components/workspace/WorkspaceCard.svelte @@ -6,6 +6,7 @@ import type { UserWorkspace } from '$lib/stores' import { superadmin } from '$lib/stores' import { WorkspaceService } from '$lib/gen' + import { reconcileAfterWorkspaceChange } from '$lib/components/sessions/sessionState.svelte' import { pluralize } from '$lib/utils' import WorkspaceIcon from './WorkspaceIcon.svelte' import WorkspaceCard from './WorkspaceCard.svelte' @@ -64,6 +65,8 @@ if (onUnarchive) { await WorkspaceService.unarchiveWorkspace({ workspace: workspace.id }) await onUnarchive(workspace.id) + // Restore sessions auto-archived when this workspace was archived. + await reconcileAfterWorkspaceChange() } } diff --git a/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte b/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte index 024d94e8e6..30b050a9f6 100644 --- a/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte +++ b/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte @@ -22,6 +22,7 @@ import { sendUserToast } from '$lib/toast' import TestAIKey from '$lib/components/copilot/TestAIKey.svelte' import { switchWorkspace } from '$lib/storeUtils' + import { deleteSessionsForWorkspace } from '$lib/components/sessions/sessionState.svelte' import { isCloudHosted } from '$lib/cloud' import ToggleButtonGroup from '$lib/components/common/toggleButton-v2/ToggleButtonGroup.svelte' import ToggleButton from '$lib/components/common/toggleButton-v2/ToggleButton.svelte' @@ -100,6 +101,14 @@ deletingExistingFork = true try { await WorkspaceService.deleteWorkspace({ workspace: prefixedId }) + // Drop local sessions bound to this id so they don't resurface (or + // auto-unarchive) against a new fork recreated under the same id. + // Fire-and-forget: neither a slow nor a failing IndexedDB op should + // block the delete/reuse flow (cleanup completes long before the UI + // could create a session in a recreated fork). + void deleteSessionsForWorkspace(prefixedId).catch((e) => + console.error(`Session cleanup for reused fork id ${prefixedId} failed`, e) + ) sendUserToast(`Permanently deleted workspace ${prefixedId}`) deleteExistingForkOpen = false await validateName(id) diff --git a/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte b/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte index 2710167325..624db4061e 100644 --- a/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte @@ -2,9 +2,14 @@ import CompareWorkspaces from '$lib/components/CompareWorkspaces.svelte' import CompareDrafts from '$lib/components/CompareDrafts.svelte' import { WorkspaceService, type WorkspaceComparison } from '$lib/gen' + import { + archiveSessionsForWorkspace, + deleteSessionsForWorkspace, + reconcileAfterWorkspaceChange + } from '$lib/components/sessions/sessionState.svelte' import { useWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' import { page } from '$app/state' - import { userWorkspaces, usersWorkspaceStore, workspaceStore } from '$lib/stores' + import { userWorkspaces, workspaceStore } from '$lib/stores' import { onDestroy, untrack } from 'svelte' import CenteredPage from '$lib/components/CenteredPage.svelte' import PageHeader from '$lib/components/PageHeader.svelte' @@ -143,14 +148,9 @@ let acting = $state(false) async function afterForkGone() { - // Mirror SidebarContent.deleteFork (B1): refresh the workspace list - // rather than letting `clearStores()` null it, then land the user on - // the parent if still accessible. - try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) - } catch (e) { - console.error('Failed to refresh workspaces', e) - } + // The workspace list was already refreshed by reconcileAfterWorkspaceChange + // (so the just-removed fork is gone from it); land the user on the parent if + // it's still accessible. if (parentWorkspaceId && $userWorkspaces.find((w) => w.id === parentWorkspaceId)) { switchWorkspace(parentWorkspaceId) await goto('/') @@ -166,6 +166,15 @@ try { await WorkspaceService.archiveWorkspace({ workspace: currentWorkspaceId }) sendUserToast(`Archived fork ${currentWorkspaceId}`) + // Client session cleanup is best-effort: a local IndexedDB failure must + // not falsely report the (already successful) archive as failed, nor + // block navigation away from the now-archived fork. + try { + await archiveSessionsForWorkspace(currentWorkspaceId) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after fork archive failed', e) + } await afterForkGone() } catch (e: any) { sendUserToast(`Failed to archive fork: ${e?.body ?? e}`, true) @@ -181,6 +190,15 @@ try { await WorkspaceService.deleteWorkspace({ workspace: currentWorkspaceId }) sendUserToast(`Deleted fork ${currentWorkspaceId}`) + // Client session cleanup is best-effort: a local IndexedDB failure must + // not abort the redirect after a successful delete, leaving the user on + // the now-deleted workspace path. + try { + await deleteSessionsForWorkspace(currentWorkspaceId) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after fork delete failed', e) + } await afterForkGone() } catch (e: any) { sendUserToast(`Failed to delete fork: ${e?.body ?? e}`, true) diff --git a/frontend/src/routes/(root)/(logged)/sessions/+page.svelte b/frontend/src/routes/(root)/(logged)/sessions/+page.svelte index a733c325d6..1d6f3f45ec 100644 --- a/frontend/src/routes/(root)/(logged)/sessions/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/sessions/+page.svelte @@ -7,7 +7,6 @@ import SessionWrapper from '$lib/components/sessions/SessionWrapper.svelte' import { createSession, - getEffectiveWorkspaceId, selectSession, sessionState, syncWorkspaceTo @@ -19,7 +18,6 @@ promoteEditorWarm } from '$lib/components/sessions/sessionRuntime.svelte' import { markSessionSeen } from '$lib/components/sessions/sessionUnread.svelte' - import { visibleWorkspaceIds } from '$lib/components/sessions/sessionScope.svelte' import { isGlobalAiEnabled } from '$lib/components/copilot/chat/global/gate' import { userWorkspaces } from '$lib/stores' @@ -46,21 +44,10 @@ untrack(() => syncWorkspaceTo(ws)) }) - // Resolve the active session if its effective workspace is in scope - // (active workspace + its forks). Unavailable sessions — committed to - // a workspace that no longer exists — also resolve so the user can - // land on the move/discard banner instead of hitting "Session not - // found". - const activeSession = $derived( - sessionState.sessions.find((s) => { - if (s.name !== sessionName) return false - const ws = getEffectiveWorkspaceId(s) - if (!ws) return false - if ($visibleWorkspaceIds.has(ws)) return true - if (s.workspace_id && !$userWorkspaces.find((w) => w.id === s.workspace_id)) return true - return false - }) - ) + // sessionState.sessions holds every local session for the user. Resolve by + // name without applying the sidebar root filter so an open chat survives + // workspace switches. + const activeSession = $derived(sessionState.sessions.find((s) => s.name === sessionName)) // Touch the runtime for the active session so it gets created on first visit // and the pane shows up. Subsequent renders find it via listRuntimes(). @@ -94,20 +81,15 @@ }) }) - // Warm = has a live runtime (module-scoped) AND its workspace is in - // scope (or its workspace is unavailable — those sessions still need - // to render the move/discard banner instead of vanishing on us). + // Warm = sessions that currently have a live (module-scoped) runtime. The + // picker eagerly creates runtimes for its visible sessions, so this tracks + // whatever the picker shows — the current family, or every family when + // "Show all workspaces" is on. Runtimes whose session record isn't loaded + // resolve to undefined here and drop out. const warmSessions = $derived( listRuntimes() .map((r) => sessionState.sessions.find((s) => s.id === r.sessionId)) .filter((s): s is NonNullable => s != null) - .filter((s) => { - const ws = getEffectiveWorkspaceId(s) - if (!ws) return false - if ($visibleWorkspaceIds.has(ws)) return true - if (s.workspace_id && !$userWorkspaces.find((w) => w.id === s.workspace_id)) return true - return false - }) ) // Promote the active session in the LRU. Mutations untracked so the effect diff --git a/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte b/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte index e71f0bb71e..f4f36eeb4d 100644 --- a/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte @@ -68,6 +68,13 @@ type DucklakeSettingsType } from '$lib/components/workspaceSettings/DucklakeSettings.svelte' import UnsavedConfirmationModal from '$lib/components/common/confirmationModal/UnsavedConfirmationModal.svelte' + import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte' + import { + archiveSessionsForWorkspace, + countSessionsForWorkspace, + deleteSessionsForWorkspace, + reconcileAfterWorkspaceChange + } from '$lib/components/sessions/sessionState.svelte' import TextInput from '$lib/components/text_input/TextInput.svelte' import CollapseLink from '$lib/components/CollapseLink.svelte' import { validateWebhookUrl, validateEncryptionKey } from '$lib/validators/workspaceSettings' @@ -87,6 +94,65 @@ let slack_team_name: string | undefined = $state() let teams_team_id: string | undefined = $state() let teams_team_name: string | undefined = $state() + + // Workspace archive/delete cascade to the workspace's client-side AI sessions + // (archive → archive, delete → delete) via reconcileSessionsLifecycle. The + // confirmation modals warn how many sessions are affected first. + let archiveConfirmOpen = $state(false) + let deleteConfirmOpen = $state(false) + let affectedSessionCount = $state(0) + + async function openArchiveConfirm() { + affectedSessionCount = await countSessionsForWorkspace($workspaceStore ?? '') + archiveConfirmOpen = true + } + async function openDeleteConfirm() { + affectedSessionCount = await countSessionsForWorkspace($workspaceStore ?? '') + deleteConfirmOpen = true + } + async function doArchiveWorkspace() { + const ws = $workspaceStore ?? '' + // Land on the parent workspace if this is a fork and the parent is still + // accessible — otherwise fall back to the workspace picker. + const parentId = $userWorkspaces.find((w) => w.id === ws)?.parent_workspace_id + const parentStillAccessible = !!(parentId && $userWorkspaces.find((w) => w.id === parentId)) + await WorkspaceService.archiveWorkspace({ workspace: ws }) + sendUserToast(`Archived workspace ${ws}`) + // Best-effort client cleanup: a local IndexedDB failure must not strand the + // user on the just-archived workspace. The reconcile also refreshes the + // workspace list (dropping the archived one) so the parent-accessible check + // below — captured before the archive — still routes correctly. + try { + await archiveSessionsForWorkspace(ws) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after workspace archive failed', e) + } + if (parentStillAccessible && parentId) { + switchWorkspace(parentId) + await goto('/') + } else { + workspaceStore.set(undefined) + usersWorkspaceStore.set(undefined) + await goto('/user/workspaces') + } + } + async function doDeleteWorkspace() { + const ws = $workspaceStore ?? '' + await WorkspaceService.deleteWorkspace({ workspace: ws }) + sendUserToast(`Deleted workspace ${ws}`) + // Best-effort client cleanup — must not block navigation off the deleted + // workspace if a local IndexedDB op throws. + try { + await deleteSessionsForWorkspace(ws) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after workspace delete failed', e) + } + workspaceStore.set(undefined) + usersWorkspaceStore.set(undefined) + await goto('/user/workspaces') + } let useCustomSlackApp: boolean = $state(false) let slackAppType: 'instance' | 'workspace' = $state('instance') @@ -1543,34 +1609,7 @@ disabled={$workspaceStore === 'admins' || $workspaceStore === 'starter'} unifiedSize="md" btnClasses="mt-2" - on:click={async () => { - const ws = $workspaceStore ?? '' - // Land on the parent workspace if this is a fork and the - // parent is still accessible — otherwise fall back to the - // workspace picker. - const parentId = $userWorkspaces.find((w) => w.id === ws)?.parent_workspace_id - const parentStillAccessible = !!( - parentId && $userWorkspaces.find((w) => w.id === parentId) - ) - await WorkspaceService.archiveWorkspace({ workspace: ws }) - sendUserToast(`Archived workspace ${ws}`) - if (parentStillAccessible && parentId) { - // Refresh the list so the just-archived workspace drops out before - // we land on the parent. Guarded: a refresh failure must not block - // the switch (the list reloads on next page load). - try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) - } catch (e) { - console.error('Failed to refresh workspaces after archive', e) - } - switchWorkspace(parentId) - await goto('/') - } else { - workspaceStore.set(undefined) - usersWorkspaceStore.set(undefined) - await goto('/user/workspaces') - } - }} + on:click={openArchiveConfirm} > Archive workspace @@ -1581,18 +1620,51 @@ disabled={$workspaceStore === 'admins' || $workspaceStore === 'starter'} size="sm" btnClasses="mt-2" - on:click={async () => { - await WorkspaceService.deleteWorkspace({ workspace: $workspaceStore ?? '' }) - sendUserToast(`Deleted workspace ${$workspaceStore}`) - workspaceStore.set(undefined) - usersWorkspaceStore.set(undefined) - goto('/user/workspaces') - }} + on:click={openDeleteConfirm} > Delete workspace (superadmin) {/if}
    + + { + archiveConfirmOpen = false + await doArchiveWorkspace() + }} + onCanceled={() => (archiveConfirmOpen = false)} + > +
    + + Archiving this workspace also archives its AI sessions{affectedSessionCount > 0 + ? ` (${affectedSessionCount})` + : ''}. Unarchiving the workspace restores them. + +
    +     + + { + deleteConfirmOpen = false + await doDeleteWorkspace() + }} + onCanceled={() => (deleteConfirmOpen = false)} + > +
    + + Permanently deleting this workspace also permanently deletes its AI sessions{affectedSessionCount > + 0 + ? ` (${affectedSessionCount})` + : ''}. This cannot be undone. + +
    +     {:else if tab == 'webhook'} Date: Wed, 24 Jun 2026 15:33:55 +0200 Subject: [PATCH 057/117] feat: add /compact session chat command (#9764) * feat: add session chat slash commands * feat: add /compact session chat command Co-Authored-By: Claude Opus 4.8 (1M context) * fix: dedupe built-in commands against same-named workspace skills Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .../copilot/chat/AIChatManager.svelte.ts | 213 +++++++++++++++--- .../copilot/chat/AIChatManager.test.ts | 172 ++++++++++++++ .../copilot/chat/ContextTextarea.svelte | 2 +- 3 files changed, 351 insertions(+), 36 deletions(-) diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts index 45f928d71c..4d3528ac66 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts @@ -113,6 +113,11 @@ const MAX_CONSECUTIVE_COMPACTION_FAILURES = 3 // (panel teardown, save-and-clear) pass their own reason, so the queued-message // flush can tell "the user wants to move on" from "the turn was torn down". const USER_CANCEL_REASON = 'user_cancelled' +// Built-in `/compact` session command — summarizes the conversation locally +// instead of sending a turn to the model. Matched on the whole input so a +// regular message that merely mentions "/compact" mid-sentence is unaffected. +const COMPACT_COMMAND_NAME = 'compact' +const COMPACT_COMMAND_RE = /^\/compact\s*$/ const AI_AUTONOMY_MODE_STORAGE_KEY = 'ai-chat-autonomy-mode' const LEGACY_AUTO_ACCEPT_TOOL_CONFIRMATIONS_STORAGE_KEY = 'ai-chat-yolo-mode' const WEB_SEARCH_ERROR_HINT = @@ -331,6 +336,23 @@ export class AIChatManager { globalSkills = $state([]) private globalSkillsRefreshId = 0 + // Built-in session-chat slash commands, listed in the command picker + // alongside workspace skills. Unlike a skill, `/compact` runs locally + // (compactManually) and never reaches the model; the submit path intercepts + // it first, so it shadows any workspace skill of the same name. + readonly sessionBuiltinCommands: AiSkillListItem[] = [ + { name: COMPACT_COMMAND_NAME, description: 'Summarize the conversation to free up context' } + ] + + // Built-ins followed by workspace skills, with any skill whose name collides + // with a built-in dropped: the picker keys leaves by name, so a duplicate + // would break its keyed list and ambiguous-resolve nav. Built-ins win — they + // already shadow same-named skills at execution (the submit interception). + sessionCommands: AiSkillListItem[] = $derived([ + ...this.sessionBuiltinCommands, + ...this.globalSkills.filter((s) => !this.sessionBuiltinCommands.some((b) => b.name === s.name)) + ]) + allowedModes: Record = $derived({ script: this.flowAiChatHelpers === undefined && @@ -432,6 +454,64 @@ export class AIChatManager { return freed } + /** + * Core summarize + rewrite, shared by automatic and manual compaction. Sends + * the prefix to the summarizer, then replaces the summarized prefix with a + * single summary message in `messages` (as a user message) and + * `displayMessages` (as a `summary` boundary). Surviving tail user messages + * have their restart `index` re-based onto the new history: the summary + * occupies slot 0, so a tail user message that was at `keepFrom` lands at slot + * 1. `displayKeepFrom` is where the kept tail begins in `displayMessages`. + * + * Owns only the `compacting` flag and the history rewrite; callers own trigger + * policy (circuit breaker, gates) and persistence. Returns the outcome — + * 'aborted' is a user Stop (history left untouched), distinct from 'error'. + */ + private runSummarization = async ( + prefix: ChatCompletionMessageParam[], + tail: ChatCompletionMessageParam[], + keepFrom: number, + displayKeepFrom: number, + abortController: AbortController + ): Promise<'ok' | 'empty' | 'aborted' | 'error'> => { + this.compacting = true + try { + const raw = await getNonStreamingCompletion( + [...prefix, { role: 'user', content: getCompactionSummaryPrompt() }], + abortController + ) + const formatted = formatCompactSummary(raw ?? '') + if (!formatted) { + return 'empty' + } + + this.messages = [{ role: 'user', content: buildSummaryMessageContent(formatted) }, ...tail] + + // Replace the summarized display prefix with the boundary marker and + // re-base the surviving tail's restart indices (the summary occupies + // slot 0, so the tail now starts at slot 1). + this.displayMessages = [ + { role: 'summary', content: formatted }, + ...this.displayMessages + .slice(displayKeepFrom) + .map((m) => (m.role === 'user' ? { ...m, index: m.index - keepFrom + 1 } : m)) + ] + + // The provider report described the pre-compaction history; the new + // history is much smaller, so clear it and let readers re-estimate. + this.contextUsage = undefined + return 'ok' + } catch (err) { + if (abortController.signal.aborted) { + return 'aborted' + } + console.error('Conversation summarization failed', err) + return 'error' + } finally { + this.compacting = false + } + } + /** * Summary-based partial compaction. Summarizes the older PREFIX of the stored * history into a single user message and keeps the recent tail verbatim, @@ -506,45 +586,94 @@ export class AIChatManager { return false } - this.compacting = true - try { - const raw = await getNonStreamingCompletion( - [...prefix, { role: 'user', content: getCompactionSummaryPrompt() }], - abortController - ) - const formatted = formatCompactSummary(raw ?? '') - if (!formatted) { - this.consecutiveCompactionFailures++ - return false - } - - this.messages = [{ role: 'user', content: buildSummaryMessageContent(formatted) }, ...tail] - - // Replace the summarized display prefix with the boundary marker and - // re-base the surviving tail's restart indices (the summary occupies - // slot 0, so the tail now starts at slot 1). - this.displayMessages = [ - { role: 'summary', content: formatted }, - ...this.displayMessages - .slice(displayKeepFrom) - .map((m) => (m.role === 'user' ? { ...m, index: m.index - keepFrom + 1 } : m)) - ] - - // The provider report described the pre-compaction history; the new - // history is much smaller, so clear it and let readers re-estimate. - this.contextUsage = undefined + const result = await this.runSummarization( + prefix, + tail, + keepFrom, + displayKeepFrom, + abortController + ) + if (result === 'ok') { this.consecutiveCompactionFailures = 0 return true - } catch (err) { - // A user Stop aborts the in-flight summary — that's a turn cancel, not a - // compaction failure, so it doesn't count toward the circuit breaker. - if (!abortController.signal.aborted) { - console.error('Conversation summarization failed', err) - this.consecutiveCompactionFailures++ + } + // 'aborted' is a user Stop during the in-flight summary — a turn cancel, not + // a compaction failure, so it doesn't count toward the circuit breaker. + if (result === 'empty' || result === 'error') { + this.consecutiveCompactionFailures++ + } + return false + } + + /** + * Manual compaction (the `/compact` session command): summarize the ENTIRE + * stored history into a single summary message and keep nothing verbatim, so + * the next message continues from the summary alone. Unlike the automatic + * trigger it ignores the context-window budget, the circuit breaker, and the + * prefix-size gate — the user asked for it explicitly — and runs on its own + * abort controller so the Stop button (`cancel`) can interrupt the in-flight + * summary, leaving history untouched. + */ + compactManually = async (): Promise => { + if (this.loading) { + return + } + // A summary round-trip only pays off once there's a prior exchange to fold + // in; a single message (or none) has nothing to compact. + if (this.messages.length < 2) { + sendUserToast('Nothing to compact yet.') + return + } + + const abortController = new AbortController() + this.abortController = abortController + this.loading = true + let result: 'ok' | 'empty' | 'aborted' | 'error' = 'error' + try { + // Everything is the prefix, nothing is kept verbatim: keepFrom and + // displayKeepFrom point past the end so the kept tail is empty. + result = await this.runSummarization( + [...this.messages], + [], + this.messages.length, + this.displayMessages.length, + abortController + ) + switch (result) { + case 'ok': + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage + ) + sendUserToast('Conversation compacted.') + break + case 'empty': + sendUserToast( + 'Compaction produced an empty summary — conversation left unchanged.', + true + ) + break + case 'error': + sendUserToast('Failed to compact the conversation.', true) + break + // 'aborted' (user Stop): history untouched, no toast. } - return false } finally { - this.compacting = false + this.loading = false + } + + // Flush a message typed while compaction ran. Mirrors the send-turn + // epilogue (loading gated its capture): auto-send after a successful + // compaction or a deliberate user cancel — the user is ready to move on — + // while a failed/empty compaction or a programmatic cancel leaves it queued. + if ((result === 'ok' || this.wasCancelledByUser()) && this.queuedMessage) { + const next = this.queuedMessage + this.queuedMessage = '' + const accepted = await this.sendRequest({ instructions: next }) + if (accepted === false) { + this.queuedMessage = next + } } } @@ -1263,6 +1392,20 @@ export class AIChatManager { if (!this.instructions.trim()) { return false } + // Built-in `/compact` session command: summarize the conversation locally + // instead of sending a turn to the model. Intercepted here — before the + // beforeSend workspace commit, file regrants, and skill expansion — and not + // turned into a chat turn. Scoped to session chat GLOBAL mode, where the + // slash-command UI lives. + if ( + this.isSessionChat && + this.mode === AIMode.GLOBAL && + COMPACT_COMMAND_RE.test(this.instructions.trim()) + ) { + this.instructions = '' + await this.compactManually() + return false + } // Re-grant any locked File System Access handles within this send gesture, so the // file tools can read the live files. requestPermission() needs a user gesture, and // this runs before the first await/network call while the Send click is still active. diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts index 62b5999bc1..2815e451eb 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts @@ -1029,6 +1029,178 @@ describe('AIChatManager context compaction', () => { }) }) +describe('AIChatManager manual compaction', () => { + const model = { provider: 'openai', model: 'gpt-4o' } + + beforeEach(() => { + localStorage.clear() + vi.clearAllMocks() + mocks.getCurrentModel.mockReturnValue(model) + mocks.tryGetCurrentModel.mockReturnValue(model) + // changeMode(GLOBAL) refreshes workspace skills; keep it a no-op here. + mocks.listAiSkills.mockResolvedValue([]) + }) + + function seedExchange(manager: AIChatManager) { + manager.messages = [ + { role: 'user', content: 'q1' }, + { role: 'assistant', content: 'a1' }, + { role: 'user', content: 'q2' }, + { role: 'assistant', content: 'a2' } + ] + manager.displayMessages = [ + { role: 'user', content: 'q1', index: 0 }, + { role: 'assistant', content: 'a1' }, + { role: 'user', content: 'q2', index: 2 }, + { role: 'assistant', content: 'a2' } + ] + } + + it('folds the whole history into a single summary boundary, keeping nothing verbatim', async () => { + mocks.getNonStreamingCompletion.mockResolvedValue('MANUAL SUMMARY') + const manager = new AIChatManager() + seedExchange(manager) + manager.contextUsage = 123 + const saveChat = vi.spyOn(manager.historyManager, 'saveChat') + + await manager.compactManually() + + // The summarizer saw the entire history, then the summary instruction. + expect(mocks.getNonStreamingCompletion).toHaveBeenCalledTimes(1) + const summaryReq = mocks.getNonStreamingCompletion.mock.calls[0][0] + expect(summaryReq).toHaveLength(5) + expect(summaryReq[0].content).toBe('q1') + expect(summaryReq[3].content).toBe('a2') + expect(summaryReq[4].content).toContain('detailed summary') + + // Nothing kept verbatim: messages collapse to just the summary user message. + expect(manager.messages).toHaveLength(1) + expect(manager.messages[0].role).toBe('user') + expect(manager.messages[0].content).toContain('MANUAL SUMMARY') + expect(manager.messages[0].content).toContain('continued from a previous conversation') + expect(manager.messages[0].content).not.toContain('') + + // The transcript shows one summary boundary in place of the old bubbles. + expect(manager.displayMessages).toHaveLength(1) + expect(manager.displayMessages[0]).toMatchObject({ role: 'summary', content: 'MANUAL SUMMARY' }) + + expect(manager.contextUsage).toBeUndefined() + expect(saveChat).toHaveBeenCalled() + expect(mocks.sendUserToast).toHaveBeenCalledWith('Conversation compacted.') + expect(manager.loading).toBe(false) + expect(manager.compacting).toBe(false) + }) + + it('no-ops with a toast when there is nothing worth compacting', async () => { + const manager = new AIChatManager() + manager.messages = [{ role: 'user', content: 'only one' }] + + await manager.compactManually() + + expect(mocks.getNonStreamingCompletion).not.toHaveBeenCalled() + expect(mocks.sendUserToast).toHaveBeenCalledWith('Nothing to compact yet.') + expect(manager.messages).toHaveLength(1) + }) + + it('leaves history untouched when the user stops mid-summary', async () => { + mocks.getNonStreamingCompletion.mockImplementation(async (_msgs: any, ac: AbortController) => { + ac.abort('user_cancelled') + throw new Error('aborted') + }) + const manager = new AIChatManager() + seedExchange(manager) + + await manager.compactManually() + + expect(manager.messages).toHaveLength(4) + expect(manager.displayMessages.some((m) => m.role === 'summary')).toBe(false) + // An abort is a user cancel, not a failure — no toast, no destructive change. + expect(mocks.sendUserToast).not.toHaveBeenCalled() + expect(manager.loading).toBe(false) + }) + + it('routes the /compact session command to manual compaction instead of the model', async () => { + mocks.getNonStreamingCompletion.mockResolvedValue('VIA COMMAND') + const manager = new AIChatManager() + manager.isSessionChat = true + seedExchange(manager) + + const sent = await manager.sendRequest({ instructions: '/compact', mode: AIMode.GLOBAL }) + + // The command never became a chat turn... + expect(sent).toBe(false) + expect(mocks.runChatLoop).not.toHaveBeenCalled() + // ...it ran the summarizer and compacted in place, clearing the composer. + expect(mocks.getNonStreamingCompletion).toHaveBeenCalledTimes(1) + expect(manager.displayMessages[0]).toMatchObject({ role: 'summary', content: 'VIA COMMAND' }) + expect(manager.instructions).toBe('') + }) + + it('auto-sends a message queued while compaction was running', async () => { + mocks.getNonStreamingCompletion.mockResolvedValue('S') + mocks.runChatLoop.mockImplementation(async (config: any) => { + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + const manager = new AIChatManager() + manager.isSessionChat = true + manager.changeMode(AIMode.GLOBAL) + seedExchange(manager) + // A message typed while loading was true gets queued, not sent. + manager.queuedMessage = 'follow-up question' + + await manager.compactManually() + + // Compaction ran once, then the queued message went out as a real turn. + expect(mocks.getNonStreamingCompletion).toHaveBeenCalledTimes(1) + expect(mocks.runChatLoop).toHaveBeenCalledTimes(1) + const sent = mocks.runChatLoop.mock.calls[0][0].messages + expect(sent[sent.length - 1].content).toContain('follow-up question') + expect(manager.queuedMessage).toBe('') + }) + + it('does not intercept /compact outside session chat', async () => { + mocks.runChatLoop.mockImplementation(async (config: any) => { + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + const manager = new AIChatManager() + manager.isSessionChat = false + + await manager.sendRequest({ instructions: '/compact', mode: AIMode.GLOBAL }) + + // Without the session-chat command surface, /compact is a normal message. + expect(mocks.runChatLoop).toHaveBeenCalledTimes(1) + expect(mocks.getNonStreamingCompletion).not.toHaveBeenCalled() + }) + + it('shadows a workspace skill that collides with a built-in command', () => { + const manager = new AIChatManager() + manager.globalSkills = [ + { name: 'compact', description: 'a workspace skill that happens to be named compact' }, + { name: 'review-code', description: 'review code for bugs' } + ] + + // Built-in `compact` comes first and the colliding skill is dropped, so the + // picker never renders two leaves with the same `skill:compact` key. + const names = manager.sessionCommands.map((c) => c.name) + expect(names).toEqual(['compact', 'review-code']) + expect(manager.sessionCommands[0].description).toBe( + 'Summarize the conversation to free up context' + ) + }) +}) + const assistantToolCall = (id: string): ChatCompletionMessageParam => ({ role: 'assistant', content: '', diff --git a/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte b/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte index bc920be013..c6f3268e50 100644 --- a/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte +++ b/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte @@ -80,7 +80,7 @@ const commandSkills = $derived( aiChatManager.mode === AIMode.GLOBAL && aiChatManager.isSessionChat - ? aiChatManager.globalSkills + ? aiChatManager.sessionCommands : [] ) const activeTooltipWord = $derived(showContextTooltip ? contextTooltipWord : commandTooltipWord) From 2a70ccc38675c7c2353807a4f85764a8a35224e2 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Wed, 24 Jun 2026 15:37:33 +0200 Subject: [PATCH 058/117] feat(frontend): show approval wait as a distinct segment in flow timeline (#9756) Co-authored-by: Claude Opus 4.8 (1M context) --- .../components/FlowStatusViewerInner.svelte | 19 +++- .../src/lib/components/FlowTimeline.svelte | 86 ++++++++++++++++++- .../src/lib/components/TimelineBar.svelte | 30 +++++-- 3 files changed, 119 insertions(+), 16 deletions(-) diff --git a/frontend/src/lib/components/FlowStatusViewerInner.svelte b/frontend/src/lib/components/FlowStatusViewerInner.svelte index ed1c536ff5..0bc47e4437 100644 --- a/frontend/src/lib/components/FlowStatusViewerInner.svelte +++ b/frontend/src/lib/components/FlowStatusViewerInner.svelte @@ -1180,15 +1180,25 @@ export type FlowModuleForTimeline = { id: string type: FlowModuleValue['type'] + suspend?: boolean } function allModulesForTimeline( modules: FlowModule[], expandedSubflows: Record ): FlowModuleForTimeline[] { - const ids = dfs(modules, (x) => ({ id: x.id, type: x.value.type }) as FlowModuleForTimeline, { - skipToolNodes: true - }) + const ids = dfs( + modules, + (x) => + ({ + id: x.id, + type: x.value.type, + suspend: x.suspend != undefined + }) as FlowModuleForTimeline, + { + skipToolNodes: true + } + ) function rec( ids: FlowModuleForTimeline[], @@ -1208,7 +1218,8 @@ fms, (x) => ({ id: x.id.startsWith('subflow:') ? x.id : buildSubflowKey(x.id, nprefix), - type: x.value.type + type: x.value.type, + suspend: x.suspend != undefined }), { skipToolNodes: true } ), diff --git a/frontend/src/lib/components/FlowTimeline.svelte b/frontend/src/lib/components/FlowTimeline.svelte index f1610c874a..4ed13c1df1 100644 --- a/frontend/src/lib/components/FlowTimeline.svelte +++ b/frontend/src/lib/components/FlowTimeline.svelte @@ -72,6 +72,66 @@ } const barHeight = 32 + + // Whole approval-wait machinery below is inert unless a step actually has a suspend config, + // so large suspend-free flows pay nothing for the flatten/sort and the per-tick recompute. + const hasSuspendModule = $derived(flowModules.some((m) => m.suspend)) + + // Push times of every job on the timeline, ascending. Used to locate when the step + // that follows an approval step started — i.e. the moment the approval was granted. + const allCreatedAts = $derived( + hasSuspendModule + ? Object.values(items ?? {}) + .flat() + .map((j) => j.created_at) + .filter((t): t is number => t != undefined) + .sort((a, b) => a - b) + : [] + ) + + // Heuristic: the grant moment is approximated by the next job pushed anywhere on the + // timeline. Exact for sequential flows; for an approval step inside one branch of a + // parallel branchall a concurrent sibling job can land first and understate the wait. + function nextCreatedAtAfter(t: number): number | undefined { + return allCreatedAts.find((c) => c > t) + } + + // For a completed suspend/approval step, the time spent waiting for the approval is the + // gap between the step finishing and the next step being pushed (or now, if still waiting). + function approvalWait(b: { + started_at?: number + duration_ms?: number + }): { start: number; len: number; running: boolean } | undefined { + if (b.started_at == undefined || b.duration_ms == undefined) { + return undefined + } + const end = b.started_at + b.duration_ms + const next = nextCreatedAtAfter(end) + const waitEnd = next ?? (flowDone ? undefined : now) + if (waitEnd == undefined) { + return undefined + } + const len = waitEnd - end + if (len < 100) { + return undefined + } + return { start: end, len, running: next == undefined } + } + + // Approval wait per module id, computed once and consumed by both the rows and the legend. + const approvalWaitByModule = $derived.by(() => { + const result: Record = {} + for (const m of flowModules) { + if (!m.suspend) continue + const sub = (items?.[m.id] ?? []).filter((x) => x.created_at && x.started_at) + if (sub.length !== 1) continue + const aw = approvalWait(sub[0]) + if (aw) result[m.id] = aw + } + return result + }) + + const hasApprovalWait = $derived(Object.keys(approvalWaitByModule).length > 0)
    Execution
    + {#if hasApprovalWait} +
    +
    + Approval wait +
    + {/if} {#if max && min} {msToSec(max - min, 1)}s {/if} @@ -113,7 +179,7 @@ /> {/if} - {#each flowModules as { id: k, type: typ } (k)} + {#each flowModules as { id: k, type: typ, suspend: isSuspend } (k)} {@const subItems = items?.[k]?.filter((x) => x.created_at && x.started_at)}
    @@ -161,6 +227,7 @@ ? 0 : now - b?.created_at : 0} + {@const aw = isSuspend ? approvalWaitByModule[k] : undefined}
    {#if b.started_at} {/if} + {#if aw} + + {/if}
    {:else}
    @@ -196,7 +277,6 @@
    {/each}
    - {:else} {/if} diff --git a/frontend/src/lib/components/TimelineBar.svelte b/frontend/src/lib/components/TimelineBar.svelte index 5d1f53e961..1d8e05008d 100644 --- a/frontend/src/lib/components/TimelineBar.svelte +++ b/frontend/src/lib/components/TimelineBar.svelte @@ -15,6 +15,10 @@ concat?: boolean gray?: boolean spacerClass?: string + /** Overrides the default gray/blue bar color (e.g. to mark an approval wait). */ + colorClass?: string + /** Tooltip label shown on hover instead of the default job link. */ + tooltip?: string } let { @@ -27,7 +31,9 @@ running, concat = false, gray = false, - spacerClass = '' + spacerClass = '', + colorClass = undefined, + tooltip = undefined }: Props = $props() @@ -37,20 +43,26 @@ {/if} {#snippet text()} - {id} + {#if tooltip} + {tooltip} + {:else} + {id} + {/if} {/snippet} {#if len > 0} {@const narrow = len / total < 0.09} From de6192bec1695883a07452f7db2fb51c94dbfd43 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Wed, 24 Jun 2026 15:37:56 +0200 Subject: [PATCH 059/117] fix(frontend): highlight the runtime-chosen branch in flow graph viewer (#9755) Co-authored-by: Claude Opus 4.8 (1M context) --- .../components/flows/map/VirtualItem.svelte | 22 ++++++++++++++----- .../renderers/nodes/BranchAllStart.svelte | 6 +++++ .../renderers/nodes/BranchOneStart.svelte | 8 +++++++ .../lib/components/graph/renderers/utils.ts | 3 ++- 4 files changed, 32 insertions(+), 7 deletions(-) diff --git a/frontend/src/lib/components/flows/map/VirtualItem.svelte b/frontend/src/lib/components/flows/map/VirtualItem.svelte index f7a08f1933..652cf4041d 100644 --- a/frontend/src/lib/components/flows/map/VirtualItem.svelte +++ b/frontend/src/lib/components/flows/map/VirtualItem.svelte @@ -9,7 +9,11 @@ import { Database, Square } from 'lucide-svelte' import FlowGraphPreviewButton from './FlowGraphPreviewButton.svelte' import type { Job } from '$lib/gen' - import { getNodeColorClasses, aiActionToNodeState } from '$lib/components/graph' + import { + getNodeColorClasses, + aiActionToNodeState, + type FlowNodeState + } from '$lib/components/graph' import { getGraphContext } from '$lib/components/graph/graphContext' interface Props { @@ -39,6 +43,9 @@ job?: Job showJobStatus?: boolean flowHasChanged?: boolean + /** When set, overrides the node outline with this run-state's colored outline. + * Used to mark the branch taken at runtime on branchone/branchall nodes. */ + borderState?: FlowNodeState } let { @@ -67,14 +74,13 @@ individualStepTests = false, job, showJobStatus = false, - flowHasChanged = false + flowHasChanged = false, + borderState = undefined }: Props = $props() const flowGraphContext = getGraphContext() - let isMultiSelected = $derived( - (flowGraphContext?.selectionManager?.selectedIds?.length ?? 0) > 1 - ) + let isMultiSelected = $derived((flowGraphContext?.selectionManager?.selectedIds?.length ?? 0) > 1) const outputPickerVisible = $derived( (nodeKind || (inputJson && Object.keys(inputJson).length > 0)) && editMode @@ -96,6 +102,10 @@ // AI action colors take priority over execution state, fallback to _VirtualItem const effectiveState = $derived(aiActionToNodeState(action) ?? outputType ?? '_VirtualItem') let colorClasses = $derived(getNodeColorClasses(effectiveState, selected)) + // The branch taken at runtime keeps its outline regardless of selection so it stays visible. + let outlineClasses = $derived( + borderState ? getNodeColorClasses(borderState, true).outline : colorClasses.outline + )
    diff --git a/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte b/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte index 1f2623e552..9682d941b6 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte @@ -6,6 +6,7 @@ import { X } from 'lucide-svelte' import type { BranchAllStartN } from '../../graphBuilder.svelte' import { getGraphContext } from '../../graphContext' + import { computeBorderStatus } from '../utils' interface Props { data: BranchAllStartN['data'] id: string @@ -14,6 +15,10 @@ let { data, id }: Props = $props() const { selectionManager } = getGraphContext() + + let borderStatus = $derived( + computeBorderStatus(data.branchIndex, 'branchall', data.flowModuleState) + ) @@ -22,6 +27,7 @@ label={data.label} selectable selected={selectionManager && selectionManager.isNodeSelected(id)} + borderState={borderStatus} on:select={() => { setTimeout(() => data.eventHandlers.select(data.id)) }} diff --git a/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte b/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte index 6333d32845..1cc634962f 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte @@ -6,6 +6,7 @@ import { X } from 'lucide-svelte' import type { BranchOneStartN } from '../../graphBuilder.svelte' import { getGraphContext } from '../../graphContext' + import { computeBorderStatus } from '../utils' interface Props { data: BranchOneStartN['data'] id: string @@ -13,6 +14,12 @@ const { selectionManager } = getGraphContext() let { data, id }: Props = $props() + + // branchIndex is -1 for the default branch and 0-based for explicit branches; + // branchChosen is 0 for default and 1-based, hence the +1. + let borderStatus = $derived( + computeBorderStatus(data.branchIndex + 1, 'branchone', data.flowModuleState) + ) @@ -22,6 +29,7 @@ preLabel={data.preLabel} selectable selected={selectionManager && selectionManager.isNodeSelected(id)} + borderState={borderStatus} on:select={() => { setTimeout(() => data?.eventHandlers?.select(data.id)) }} diff --git a/frontend/src/lib/components/graph/renderers/utils.ts b/frontend/src/lib/components/graph/renderers/utils.ts index 8b393f1de9..c8c6ca5109 100644 --- a/frontend/src/lib/components/graph/renderers/utils.ts +++ b/frontend/src/lib/components/graph/renderers/utils.ts @@ -19,7 +19,8 @@ export function computeBorderStatus( } else { let flow_jobs_success = graphModuleState?.flow_jobs_success if (!flow_jobs_success) { - return 'WaitingForPriorSteps' + // No run yet: leave the branch border neutral instead of forcing a highlight. + return undefined } else { let status = flow_jobs_success?.[branchIndex] if (status == undefined) { From b5bd8245d81b84fc14d3ea955bf1e66ac576bf37 Mon Sep 17 00:00:00 2001 From: hugocasa Date: Wed, 24 Jun 2026 16:08:45 +0200 Subject: [PATCH 060/117] fix: reject symlink traversal in job-dir path validation (#9713) * fix: reject symlink traversal in job-dir path validation Co-Authored-By: Claude Opus 4.8 (1M context) * test: cover dangling symlink in job-dir path validation Co-Authored-By: Claude Opus 4.8 (1M context) * fix: close symlink-traversal bypass via in-bounds `..` in path check Walk the normalized relative path instead of raw user components, so an in-bounds `..` (e.g. `foo/../evil/payload`) can no longer drift the walk past a planted symlink. Adds regression coverage. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- backend/windmill-common/src/worker.rs | 99 ++++++++++++++++++++++++++- 1 file changed, 97 insertions(+), 2 deletions(-) diff --git a/backend/windmill-common/src/worker.rs b/backend/windmill-common/src/worker.rs index e3bd7ef7ef..3a1469d5e3 100644 --- a/backend/windmill-common/src/worker.rs +++ b/backend/windmill-common/src/worker.rs @@ -693,8 +693,6 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error let full_path = job_dir.join(&user_path); - // let normalized_job_dir = std::fs::canonicalize(job_dir)?; - // let normalized_full_path = std::fs::canonicalize(&full_path)?; let normalized_job_dir = normalize_path(job_dir); let normalized_full_path = normalize_path(&full_path); @@ -706,6 +704,36 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error .into()); } + // The lexical check above cannot see symlinks: a symlink planted inside the + // job dir - e.g. by an earlier Ansible `git_repos` clone whose tracked + // content includes one - would let a later `git clone` or file write follow + // it out of the job dir while still passing the textual `starts_with` check. + // Walk the *normalized* relative path (`..`/`.` already collapsed) so each + // step matches the real on-disk resolution, and reject any existing component + // that is a symlink. Walking the raw user path would drift on an in-bounds + // `..` (e.g. `foo/../link`, which normalizes back inside the job dir) and miss + // the real symlinked component. Not-yet-existing components are safe: a path + // that does not exist cannot itself be a symlink. + let relative = normalized_full_path + .strip_prefix(&normalized_job_dir) + .unwrap_or(&normalized_full_path); + let mut current = normalized_job_dir.clone(); + for component in relative.components() { + if let Component::Normal(c) = component { + current.push(c); + if std::fs::symlink_metadata(¤t) + .map(|m| m.file_type().is_symlink()) + .unwrap_or(false) + { + return Err(std::io::Error::new( + std::io::ErrorKind::PermissionDenied, + "Path traverses a symlink, which is not allowed.", + ) + .into()); + } + } + } + Ok(normalized_full_path) } @@ -2828,4 +2856,71 @@ mod tests { let _ = std::fs::remove_dir_all(&base); } + + #[test] + fn test_is_allowed_file_location_allows_plain_relative() { + let base = std::env::temp_dir().join(format!("wm_allowed_loc_ok_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + let out = is_allowed_file_location(job_dir_str, "repo/sub/playbook.yml").unwrap(); + assert_eq!(out, normalize_path(&job_dir.join("repo/sub/playbook.yml"))); + + let _ = std::fs::remove_dir_all(&base); + } + + #[test] + fn test_is_allowed_file_location_rejects_parent_and_absolute() { + let base = + std::env::temp_dir().join(format!("wm_allowed_loc_esc_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + assert!(is_allowed_file_location(job_dir_str, "../escape").is_err()); + assert!(is_allowed_file_location(job_dir_str, "a/../../escape").is_err()); + assert!(is_allowed_file_location(job_dir_str, "/etc/passwd").is_err()); + + let _ = std::fs::remove_dir_all(&base); + } + + // Regression for GHSA-v934-cvpf-6fjw: a symlink planted inside the job dir + // (e.g. by an earlier `git_repos` clone) must not let a later target traverse + // it out of the job dir, even though the lexical path stays "inside". + #[cfg(unix)] + #[test] + fn test_is_allowed_file_location_rejects_symlink_traversal() { + let base = + std::env::temp_dir().join(format!("wm_allowed_loc_symlink_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + // Stand-in for the shared cache dir living outside the job dir. + let outside = base.join("outside"); + std::fs::create_dir_all(&outside).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + // Plant `job/repo` -> `../outside`, as a malicious first clone would. + let planted = job_dir.join("repo"); + std::os::unix::fs::symlink(&outside, &planted).unwrap(); + + // Both the symlink itself and any path traversing it are rejected. + assert!(is_allowed_file_location(job_dir_str, "repo").is_err()); + assert!(is_allowed_file_location(job_dir_str, "repo/payload").is_err()); + assert!(is_allowed_file_location(job_dir_str, "repo/sub/payload").is_err()); + + // An in-bounds `..` must not bypass the check: `foo/../repo/payload` + // normalizes back to `repo/payload` and still traverses the symlink. + assert!(is_allowed_file_location(job_dir_str, "foo/../repo/payload").is_err()); + std::fs::create_dir(job_dir.join("real")).unwrap(); + assert!(is_allowed_file_location(job_dir_str, "real/../repo/payload").is_err()); + + // A dangling symlink (target does not exist yet) is still caught: + // `symlink_metadata` does not follow the link. + let dangling = job_dir.join("dangling"); + std::os::unix::fs::symlink(base.join("nonexistent"), &dangling).unwrap(); + assert!(is_allowed_file_location(job_dir_str, "dangling/payload").is_err()); + + let _ = std::fs::remove_dir_all(&base); + } } From 4dbf8737238ccc4dc2c67365e6d43f04f46c75b5 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Wed, 24 Jun 2026 16:40:07 +0200 Subject: [PATCH 061/117] fix(frontend): stop flow step id generation from being poisoned by non-canonical keys (#9766) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(frontend): stop flow step id generation from being poisoned by non-canonical keys nextId computed the next step id from the max of charsToNumber over every module id and flowState key. Only canonical auto-ids (a, b, ... aa, ab) have a meaningful charsToNumber value, but flowState also holds copy ids ("z2"), subflow result keys ("subflow:..."), reserved keys ("failure"/"preprocessor") and user-renamed ids. The old `length >= 4` guard filtered long junk but let short junk through, so e.g. duplicating step "z" (key "z2", charsToNumber 629) made the next new step jump to "xg" and escalate from there. nextId now only counts a key if it round-trips through numberToChars and is not reserved, and the broken length cap is removed so large flows still get correct ids. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(frontend): keep length cap in nextId to avoid regressing long renames Address CI review: removing the length cap made all-lowercase renamed step ids (e.g. "process", which round-trips through numberToChars) feed into the max and poison id generation again — a regression versus the prior behavior, since step ids can be renamed to ^[a-zA-Z][a-zA-Z0-9_]*$. Restore the length>=4 skip and pair it with the round-trip canonical check, so short non-canonical keys (copy ids "z2"/"c10", reserved/renamed short ids) no longer poison the max while long renames stay out of the sequence. Update the tests to reflect the actual coverage. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .../components/flows/flowModuleNextId.test.ts | 53 +++++++++++++++++++ .../lib/components/flows/flowModuleNextId.ts | 30 ++++++++--- 2 files changed, 76 insertions(+), 7 deletions(-) create mode 100644 frontend/src/lib/components/flows/flowModuleNextId.test.ts diff --git a/frontend/src/lib/components/flows/flowModuleNextId.test.ts b/frontend/src/lib/components/flows/flowModuleNextId.test.ts new file mode 100644 index 0000000000..dcf007e925 --- /dev/null +++ b/frontend/src/lib/components/flows/flowModuleNextId.test.ts @@ -0,0 +1,53 @@ +import { describe, expect, it } from 'vitest' + +import type { OpenFlow } from '$lib/gen' +import type { FlowState } from './flowState' +import { nextId } from './flowModuleNextId' + +function flowWith(ids: string[]): OpenFlow { + return { + summary: '', + value: { + modules: ids.map((id) => ({ id, value: { type: 'identity' } as any })) + } + } as OpenFlow +} + +function stateWith(keys: string[]): FlowState { + return Object.fromEntries(keys.map((k) => [k, {}])) as FlowState +} + +describe('nextId', () => { + it('produces a, b, c, ... for a fresh flow', () => { + expect(nextId(stateWith(['failure']), flowWith([]))).toBe('a') + expect(nextId(stateWith(['a', 'failure']), flowWith(['a']))).toBe('b') + expect(nextId(stateWith(['a', 'b', 'c', 'failure']), flowWith(['a', 'b', 'c']))).toBe('d') + }) + + it('ignores the reserved failure/preprocessor keys always present in flowState', () => { + expect(nextId(stateWith(['failure', 'preprocessor']), flowWith([]))).toBe('a') + }) + + // Regression: copy ids ("z2"), subflow result keys and other non-canonical keys land in + // flowState; charsToNumber on them used to leak into the max and made new steps jump to + // garbage ids like "bzw". + it('is not poisoned by copy ids', () => { + const ids = ['a', 'b', 'c'] + const state = stateWith([...ids, 'c2', 'a2', 'z2', 'c10', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('d') + }) + + it('is not poisoned by subflow result keys', () => { + const ids = ['a', 'b'] + const state = stateWith([...ids, 'subflow:abcd', 'Result', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('c') + }) + + // A step renamed to a long lowercase word ("process") is a valid base-26 string and would + // otherwise inflate the max; the length cutoff keeps such renames out of the sequence. + it('is not poisoned by renames to long lowercase words or underscored ids', () => { + const ids = ['a', 'b'] + const state = stateWith([...ids, 'process', 'my_step', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('c') + }) +}) diff --git a/frontend/src/lib/components/flows/flowModuleNextId.ts b/frontend/src/lib/components/flows/flowModuleNextId.ts index e10c900982..48b2eb5ac2 100644 --- a/frontend/src/lib/components/flows/flowModuleNextId.ts +++ b/frontend/src/lib/components/flows/flowModuleNextId.ts @@ -1,19 +1,35 @@ import type { OpenFlow } from '$lib/gen' import { dfs } from './dfs' import type { FlowState } from './flowState' -import { charsToNumber, numberToChars } from './idUtils' +import { charsToNumber, forbiddenIds, numberToChars } from './idUtils' + +const reservedIds = new Set(forbiddenIds) + +// Returns the base-26 value of a key only if it is a short, auto-generated step id +// (a, b, ..., z, aa, ...). flowState/module-id keys also include copy ids ("a2"), subflow +// result keys ("subflow:..."), reserved keys and user-renamed ids; feeding those through +// charsToNumber yields meaningless (often huge) numbers that would poison id generation and +// make new steps jump to ids like "bzw". Short non-canonical keys are rejected via a +// round-trip check; longer keys are skipped entirely, which also leaves user renames to long +// lowercase words (e.g. "process") out of the sequence. +function autoIdNumber(key: string): number | undefined { + if (key.length >= 4 || reservedIds.has(key)) { + return undefined + } + const num = charsToNumber(key) + if (num < 0 || numberToChars(num) !== key) { + return undefined + } + return num +} // Computes the next available id export function nextId(flowState: FlowState, fullFlow: OpenFlow): string { const allIds = dfs(fullFlow.value.modules, (fm) => fm.id) const max = allIds.concat(Object.keys(flowState)).reduce((acc, key) => { - if (key.length >= 4) { - return acc - } else { - const num = charsToNumber(key) - return Math.max(acc, num + 1) - } + const num = autoIdNumber(key) + return num === undefined ? acc : Math.max(acc, num + 1) }, 0) return numberToChars(max) } From 2e020b2ccc7a649a5923bff72a98f07d4fc85381 Mon Sep 17 00:00:00 2001 From: Guilhem Date: Wed, 24 Jun 2026 17:07:20 +0200 Subject: [PATCH 062/117] feat(ai-chat): context usage gauge + unified model settings menu (#9763) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(ai-chat): show context usage as a gauge with hover tooltip Co-Authored-By: Claude Opus 4.8 (1M context) * feat(ai-chat): consolidate model, thinking & params into one dropdown Merge the model picker, reasoning-effort selector and prompt settings into a single dropdown with a model list, a thinking-effort slider and a hover-revealed Parameters submenu. The trigger shows the model and effort. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(ai-chat): polish model/thinking dropdown interactions Register the model rows and thinking slider as melt menu items (roving highlight + arrow-key navigation), keep the menu open on selection via a new DropdownV2 closeOnItemClick prop, use melt's createSubmenu for the Parameters flyout so it flips on screen edges, and use the brand accent for the context-usage gauge and slider. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(ai-chat): stop popover drift and keep Thinking section when unsupported Freeze the trigger width while the dropdown is open so the bottom-end popover doesn't shift as the effort label resizes (released on close, so no reserved padding). When a model has no reasoning support, show the Thinking section disabled with a message instead of removing it. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(ai-chat): restore reasoning slider drag inside the menu The slider lives in a melt menu item, whose roving focus blurs the focused element on pointermove and aborted the native thumb drag. Stop the slider's pointer events from bubbling to the item so melt leaves it alone; focus-based highlighting still works. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(ai-chat): move Parameters to the top of the model settings menu Co-Authored-By: Claude Opus 4.8 (1M context) * feat(ai-chat): hide the @ context picker in global mode Co-Authored-By: Claude Opus 4.8 (1M context) * fix(ai-chat): only mark context gauge as a meter when the window is known A meter is a 0–100% reading; with an unknown context window there is no max to measure against, so role/aria-value* are dropped (previously valuenow fell back to the raw token count against an implicit valuemax of 100). Co-Authored-By: Claude Opus 4.8 (1M context) * docs(frontend): note closeOnItemClick is read at mount-time Addresses a non-blocking review note on DropdownV2. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(ai-chat): fix showContextPicker comment to match GLOBAL removal Addresses Pi review P2: GLOBAL no longer offers the @ context picker. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(ai-chat): clarify showContextPicker hides only the manual @ button In GLOBAL, @-context is still invoked inline by typing @ in the input; only the redundant picker button is hidden. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- frontend/src/lib/components/DropdownV2.svelte | 24 +- .../copilot/chat/AIChatDisplay.svelte | 19 +- .../copilot/chat/AIChatModelSettings.svelte | 456 ++++++++++++++++++ .../copilot/chat/AIChatSettingsMenu.svelte | 186 ------- .../copilot/chat/ContextUsageIndicator.svelte | 52 +- .../meltComponents/MenuItemWrapper.svelte | 28 ++ 6 files changed, 557 insertions(+), 208 deletions(-) create mode 100644 frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte delete mode 100644 frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte create mode 100644 frontend/src/lib/components/meltComponents/MenuItemWrapper.svelte diff --git a/frontend/src/lib/components/DropdownV2.svelte b/frontend/src/lib/components/DropdownV2.svelte index 200f66f538..bf83d59886 100644 --- a/frontend/src/lib/components/DropdownV2.svelte +++ b/frontend/src/lib/components/DropdownV2.svelte @@ -30,6 +30,10 @@ placement?: Placement usePointerDownOutside?: boolean closeOnOtherDropdownOpen?: boolean + // When false the menu stays open after an item is selected (melt's closeOnItemClick). + // Consumers that keep the menu open must close it themselves where appropriate. + // Read once at menu creation (like `placement`); changing it after mount has no effect. + closeOnItemClick?: boolean fixedHeight?: boolean hidePopup?: boolean open?: boolean @@ -41,10 +45,18 @@ size?: ButtonType.UnifiedSize btnText?: string buttonReplacement?: import('svelte').Snippet - // In customMenu mode the snippet receives the melt-ui `item` action - // store so consumers can wrap their own rows in (or - // `use:melt={$item}`) and get arrow-key navigation + aria wiring. - menu?: import('svelte').Snippet<[{ item: MenubarMenuElements['item']; close: () => void }]> + // In customMenu mode the snippet receives the melt-ui `item` action store + // (so consumers can wrap rows in for arrow-key navigation + aria) + // and `builders` (so they can compose melt submenus, e.g. via DropdownSubmenuItem). + menu?: import('svelte').Snippet< + [ + { + item: MenubarMenuElements['item'] + close: () => void + builders: ReturnType['builders'] + } + ] + > maxHeight?: string | undefined } @@ -56,6 +68,7 @@ placement = 'bottom-end', usePointerDownOutside = false, closeOnOtherDropdownOpen = true, + closeOnItemClick = true, fixedHeight = true, hidePopup = false, open = $bindable(false), @@ -82,6 +95,7 @@ positioning: { placement: untrack(() => placement) }, + closeOnItemClick: untrack(() => closeOnItemClick), loop: true, onOpenChange: ({ next }) => { if (closeOnOtherDropdownOpen) { @@ -176,7 +190,7 @@ transition:fly={{ duration: enableFlyTransition ? 100 : 0, y: -16 }} > {#if customMenu} - {@render menu?.({ item, close })} + {@render menu?.({ item, close, builders })} {:else} {/if}
    -
    - -
    {#if (aiChatManager.mode === AIMode.NAVIGATOR || aiChatManager.mode === AIMode.ASK) && suggestions.length > 0 && messages.filter((m) => m.role === 'user').length === 0 && !disabled}
    diff --git a/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte b/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte new file mode 100644 index 0000000000..0a911134b6 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte @@ -0,0 +1,456 @@ + + +{#snippet externalLinkIcon()} + +{/snippet} + + + {#snippet buttonReplacement()} +
    + +
    + {/snippet} + {#snippet menu({ item, builders, close })} +
    + + + +
    +
    Model
    +
    + {#each models as m (m.provider + m.model)} + selectModel(m)} + > + {m.model} + {#if m.model === providerModel.model && m.provider === providerModel.provider} + + {/if} + + {/each} +
    + +
    + {#if capability.supported} + + +
    + Thinking + {currentStop} +
    + {#if stops.length > 1} + +
    + selectReasoning(stops[+e.currentTarget.value])} + use:isolatePointer + class="lean-range no-default-style w-full" + aria-label="Reasoning effort" + /> +
    + {/if} +     + {:else} + +
    +
    Thinking
    +
    Not supported by this model
    +
    + {/if} +
    + {/snippet} +     + + + + diff --git a/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte b/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte deleted file mode 100644 index 683bf3c586..0000000000 --- a/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte +++ /dev/null @@ -1,186 +0,0 @@ - - -{#snippet externalLinkIcon()} - -{/snippet} - - - {#snippet buttonReplacement()} - +
    + +
    + {#if schemas.loading} +
    + Loading schema… +
    + {:else if schemas.error} +

    Failed to load: {schemas.error.message}

    + {:else if !schemas.current?.length} +

    + No schema captured yet. The output schema is recorded automatically after a // materialize run. +

    + {:else if !canEvolve} + +
    +
    + + This asset's schema is fixed — an append / merge / partitioned materialize INSERTs into + a fixed-schema table, so the columns can't change run-to-run. +
    + {#if selected} + {@render columnsTable(selected.columns)} + {/if} +
    + {:else} +
    + + +
    + {#each schemas.current as s, i (s.version)} + + {/each} +
    +     + +
    + {#if selected} + {@render columnsTable(selected.columns)} + {/if} +
    +     +     +
    + {/if} +
    + diff --git a/frontend/src/lib/components/assets/AssetGraph/types.ts b/frontend/src/lib/components/assets/AssetGraph/types.ts index 4a5d16d693..c942d28934 100644 --- a/frontend/src/lib/components/assets/AssetGraph/types.ts +++ b/frontend/src/lib/components/assets/AssetGraph/types.ts @@ -33,6 +33,11 @@ export interface AssetGraphRunnableNode { // asset. Surfaced as a count badge (with a per-test breakdown in the title) // so test coverage is visible on the node without opening the pane. data_tests?: DataTest[] + // Managed `// materialize` write strategy. Absent for non-materializing or + // `manual` scripts. Used (with `partition_kind`) to decide whether a + // produced asset's schema can evolve: only whole-table `replace` can, since + // `append`/`merge`/partitioned writes INSERT into a fixed-schema table. + materialize_strategy?: 'replace' | 'append' | 'merge' // Synthesized by the page from a local draft; the script doesn't exist // in the DB yet. Drives a dashed/lower-opacity rendering to mirror how // unsaved triggers are styled — visually distinct from persisted nodes. diff --git a/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte index e7f52ddbd4..3548ba6017 100644 --- a/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte @@ -1958,6 +1958,27 @@ .map((e) => ({ kind: e.runnable_kind, path: e.runnable_path, unsaved: e.unsaved })) }) + // Whether the selected ducklake asset's captured schema can *evolve* (drives + // the asset panel's Schema tab: version history vs. a single fixed schema). + // Only a whole-table `replace` producer (CREATE OR REPLACE) can change + // columns run-to-run; `append`/`merge`/partitioned writes INSERT into a + // fixed-schema table, so their schema is pinned at first materialize. + // + // Fail open: show the fixed view only when we're *sure* — every producer is a + // known insert-style write. A producer with no `materialize_strategy` + // metadata (e.g. a draft-overlay runnable, which the graph synthesizes + // without it) is treated as unknown → evolvable, so captured history is never + // hidden behind a stale "fixed" verdict. + let schemaCanEvolve = $derived.by(() => { + const sel = selection + if (!sel || sel.kind !== 'asset' || sel.asset_kind !== 'ducklake') return true + const producerPaths = new Set(selectionProducers.map((p) => p.path)) + const producers = graphWithDraft.runnables.filter((r) => producerPaths.has(r.path)) + const knownFixed = (r: (typeof producers)[number]) => + !!r.materialize_strategy && !(r.materialize_strategy === 'replace' && !r.partition_kind) + return producers.length === 0 || !producers.every(knownFixed) + }) + // Downstream subscriber count for the currently-edited script. Drives // the Test button's cascade UX: when > 0, ScriptEditor renders a split // button exposing "just this step" (default, with `_wmill_skip_asset_dispatch`) @@ -2531,6 +2552,7 @@ onRunByPath={runByPathLegit} selection={activeDraft ? undefined : selection} selectionProducers={activeDraft ? [] : selectionProducers} + {schemaCanEvolve} {runsRefreshKey} {runsPendingJobId} {activeRunnable} From 577ceeee8679f054c6898d1a7889df30ab830f8f Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Fri, 26 Jun 2026 21:37:33 +0200 Subject: [PATCH 110/117] perf(audit): re-anchor S3 audit export on enable + opt-in backfill (#9818) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * [ee] perf(audit): re-anchor S3 audit export on enable + opt-in backfill The S3/GCS audit-log export's steady-state query filters by `age(xmin)` (unindexable), so the only scan bound is the timestamp floor. On a fresh enable the floor was epoch, and on a re-enable the cursor resumed from its pre-disable position — either way the first run scanned the whole `audit_partitioned` table. Under a `statement_timeout` (e.g. Aiven) that scan never completes: the cursor never advances, nothing is exported, and the repeated full scans saturate the database. Re-anchor on enable (EE companion, windmill-ee-private#634): - New trigger migration records a recent timestamp floor instead of the epoch sentinel and `DO UPDATE`s the cursor to the current snapshot xmin on re-enable, so the export always resumes from ~now and never rescans history. Includes a one-time fixup for legacy epoch-sentinel checkpoints on upgrade. Opt-in historical backfill (new `audit_logs_s3_backfill` module + endpoints): - Exports a chosen `[from, to)` window on demand, scanning strictly by `timestamp` (the partition key) in bounded keyset pages — each query is an index scan capped at one page (verified via EXPLAIN: later partitions `never executed`, ~11ms/page), so it stays well under any statement timeout regardless of window size. Writes alongside the steady-state objects under logs/audit/, without touching the xmin cursor. - POST /settings/audit_logs_s3_backfill {from,to} (super-admin + Enterprise), GET /settings/audit_logs_s3_backfill_status. Also repurposes the status endpoint's `bootstrapping` flag to mean "draining a backlog" (the cursor is capped and catching up), and updates the setting description to point operators at the backfill. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(audit): heartbeat backfill lease per object; bump EE ref Address review (cubic): persist progress (refreshing the lease heartbeat) after every object PUT in the backfill page loop, not only once per page, so the gap between heartbeats stays well under STALE_HEARTBEAT_SECS even on slow uploads and another replica can't re-claim mid-page and run a concurrent backfill. Bumps ee-repo-ref.txt to pull in the EE test-race fix (folding the backlog-drain regression into the single audit e2e test). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(audit): reject unstable backfill windows; bump EE ref Address review (P1): the backfill keyset-pages over rows visible at scan time and declares completion when the scan runs dry, but a row's `timestamp` is its inserting transaction's `xact_start`. A window whose upper bound is recent or in the future could silently omit a transaction that started inside `[from, to)` but commits after the scan passed that timestamp. `try_start` now rejects any `to` newer than the oldest in-flight `xact_start` (everything strictly older than the oldest running transaction is committed and stable), using the same trustworthy stats gating as the exporter's floor (restricted role / 2PC → a 7-day-old cutoff). Bumps ee-repo-ref.txt for the EE monotonic-checkpoint fix. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(audit): re-anchor legacy epoch checkpoints instead of synthetic floor Address review (P1): the legacy-checkpoint fixup stamped last_oldest_inflight_ts to now()-7d while leaving the old last_xmin in place. On an instance that enabled export on the old code >7 days ago and got stuck before the first successful batch, the next run would filter post-enable rows older than 7 days out via `timestamp >= ts_floor` while still advancing last_xmin over the interval — silently dropping them (the same floor-vs-cursor loss class fixed elsewhere in this PR), and contradicting the "nothing committed after enabling is skipped" guarantee. A stuck epoch-sentinel checkpoint cannot be safely resumed (its backlog can be arbitrarily old, so any recent floor prunes rows the cursor then skips, and an epoch floor reintroduces the full scan). Re-anchor it to the migration's current snapshot xmin instead — exactly like a fresh enable — so the export resumes cleanly from ~now and the never-exported pre-upgrade window is recovered via the opt-in backfill rather than silently dropped. Reword the setting description so it no longer implies the disabled/legacy window is covered by the cursor. Co-Authored-By: Claude Opus 4.8 (1M context) * test(audit): end-to-end integration tests for the object-store backfill The backfill previously had only SQL-level/EXPLAIN validation. Add real integration tests (in-memory object store, sqlx::test) exercising the public path: - backfill_exports_window_in_pages: with the page size forced to 2 rows, a settled 3-day window is exported across multiple keyset pages; asserts every in-window row lands exactly once, rows outside [from,to) are excluded, a day that straddles a page boundary yields more than one object, progress counts match, and a re-run is idempotent (deterministic keys overwritten, no dupes). - backfill_rejects_unstable_window: a future/live `to` is rejected as unstable, a window safely in the past is accepted. Adds a test-only PAGE_ROWS override so multi-page behaviour is exercised with a handful of rows. Co-Authored-By: Claude Opus 4.8 (1M context) * docs(audit): note backfill scope is audit_partitioned only Make explicit that, like the steady-state export, the backfill reads only audit_partitioned; the pre-partitioning `audit` table is intentionally out of scope (not a missed case). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(audit): reject backfill windows before the partitioned boundary Address review (Codex P1): the backfill reads only audit_partitioned, but pre-partitioning history lives in the legacy `audit` table (still read by audit list/get via UNION ALL, and retained for the configured period — 365 days by default on EE). Since the setting text points operators at this API for "pre-existing history", a window overlapping legacy rows would report completion while silently omitting them. Per the decision to not export the legacy table, reject instead of silently omit: try_start now rejects a `from` earlier than the oldest audit_partitioned timestamp (every legacy row predates the partition cutover, so a `from` at/after that boundary can never overlap them). Reworded the setting text to scope the backfill to the partitioned era. Added a regression test, plus an RAII guard (cubic P2) so the test-only globals are restored even if an assertion panics. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(audit): backfill object keys per-window; require trustworthy settled cutoff Address review (two P1s): - Object-key overwrite loss: keys were `dt=/audit_backfill_.ndjson`. A narrower, overlapping backfill can start a day's page at the same first row (same min_id) but hold fewer rows, and `put` would overwrite a broader run's object — silently dropping the rows only that object held. Include the requested window in the key so different ranges write disjoint objects (same window re-runs stay idempotent; consumers dedupe overlapping rows by id). New regression test (verified red→green). - Untrustworthy settled cutoff: when min(xact_start) isn't trustworthy (role lacks pg_read_all_stats/superuser, or a prepared 2PC txn exists), the old now()-7d fallback could still let an old transaction commit rows inside an accepted window after the scan, so a "complete" backfill silently missed them. Since a backfill asserts completeness, reject in those cases instead of falling back. (The continuous exporter keeps its 7-day fallback — it only claims bounded lag.) Also makes the tests robust under the parallel runner: run_backfill takes the store as a param, so tests pass a local in-memory store (no global OBJECT_STORE_SETTINGS race) and serialize on the PAGE_ROWS override. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(audit): reject backfill overlapping legacy table; regen deref openapi; trim migration comment Address review (1 P1 + 2 P2): - Empty-partition backfill (P1): the min(audit_partitioned) guard no-ops when audit_partitioned is empty, so an upgraded instance with legacy `audit` rows but no partitioned rows yet would accept a window and complete with zero rows, silently omitting the legacy rows. Check the legacy `audit` table directly: reject any window that overlaps a legacy row (subsumes the boundary check and covers the empty-partitioned case). Test updated accordingly. - openapi-deref (P2): regenerate openapi-deref.yaml/json (served via include_str!) so /openapi.{yaml,json} expose the new backfill endpoints. - Migration comment (P2): trim the PR-history narration to the durable constraints (why a recent floor and a monotonic cursor are required), per AGENTS.md. Co-Authored-By: Claude Opus 4.8 (1M context) * chore: update ee-repo-ref to b821fecccbcba2efed544890576bf2b84321d70d This commit updates the EE repository reference after PR #634 was merged in windmill-ee-private. Previous ee-repo-ref: 6b191b77aabcf77658ad4f9031576e0d7b66bf89 New ee-repo-ref: b821fecccbcba2efed544890576bf2b84321d70d Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: windmill-internal-app[bot] --- ...100e926bb678171947e71d6d87fdd0c3f9299.json | 15 + ...b64b33c722afde7457912e8abd5743de8829.json} | 7 +- ...f172c89da5b216d3530c2743319ab1f4ca853.json | 20 + ...790770c32716ef953a2ea4af17d122d0a3b3c.json | 44 ++ backend/ee-repo-ref.txt | 2 +- ..._audit_logs_s3_reanchor_on_enable.down.sql | 19 + ...51_audit_logs_s3_reanchor_on_enable.up.sql | 70 ++ .../src/audit_logs_s3.rs | 9 +- .../src/audit_logs_s3_backfill.rs | 709 ++++++++++++++++++ backend/windmill-api-settings/src/lib.rs | 35 +- backend/windmill-api/openapi-deref.json | 211 +++++- backend/windmill-api/openapi-deref.yaml | 158 +++- backend/windmill-api/openapi.yaml | 86 +++ .../src/lib/components/instanceSettings.ts | 2 +- 14 files changed, 1375 insertions(+), 12 deletions(-) create mode 100644 backend/.sqlx/query-3accb7e0eab75fcd34bf5b6d75e100e926bb678171947e71d6d87fdd0c3f9299.json rename backend/.sqlx/{query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json => query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json} (68%) create mode 100644 backend/.sqlx/query-a54f686b1bfb16e4e1da2bc143ef172c89da5b216d3530c2743319ab1f4ca853.json create mode 100644 backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json create mode 100644 backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql create mode 100644 backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql create mode 100644 backend/windmill-api-settings/src/audit_logs_s3_backfill.rs diff --git a/backend/.sqlx/query-3accb7e0eab75fcd34bf5b6d75e100e926bb678171947e71d6d87fdd0c3f9299.json b/backend/.sqlx/query-3accb7e0eab75fcd34bf5b6d75e100e926bb678171947e71d6d87fdd0c3f9299.json new file mode 100644 index 0000000000..54db7ff310 --- /dev/null +++ b/backend/.sqlx/query-3accb7e0eab75fcd34bf5b6d75e100e926bb678171947e71d6d87fdd0c3f9299.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO background_task_state (name, value)\n SELECT $1, jsonb_build_object(\n 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint,\n 'last_ts', now(),\n 'last_oldest_inflight_ts', COALESCE(\n CASE WHEN (current_setting('is_superuser') = 'on'\n OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'))\n AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts)\n THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL)\n ELSE NULL END,\n now() - interval '7 days'))\n WHERE NOT EXISTS (SELECT 1 FROM global_settings WHERE name = $2)\n ON CONFLICT (name) DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "3accb7e0eab75fcd34bf5b6d75e100e926bb678171947e71d6d87fdd0c3f9299" +} diff --git a/backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json b/backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json similarity index 68% rename from backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json rename to backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json index 56b887e8b9..f8f8511709 100644 --- a/backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json +++ b/backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json @@ -1,16 +1,17 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, false, $3, now(), now(), now())\n ON CONFLICT (name) DO UPDATE SET\n value = $2, running = false, owner = $3,\n finished_at = now(), updated_at = now()", + "query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, false, $3, now(), now(), now())\n ON CONFLICT (name) DO UPDATE SET\n value = $2, running = false, owner = $3,\n finished_at = now(), updated_at = now()\n WHERE (background_task_state.value->>'last_xmin')::bigint <= $4", "describe": { "columns": [], "parameters": { "Left": [ "Text", "Jsonb", - "Text" + "Text", + "Int8" ] }, "nullable": [] }, - "hash": "8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d" + "hash": "881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829" } diff --git a/backend/.sqlx/query-a54f686b1bfb16e4e1da2bc143ef172c89da5b216d3530c2743319ab1f4ca853.json b/backend/.sqlx/query-a54f686b1bfb16e4e1da2bc143ef172c89da5b216d3530c2743319ab1f4ca853.json new file mode 100644 index 0000000000..4c0d0ae14f --- /dev/null +++ b/backend/.sqlx/query-a54f686b1bfb16e4e1da2bc143ef172c89da5b216d3530c2743319ab1f4ca853.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT CASE WHEN (current_setting('is_superuser') = 'on'\n OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'))\n AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts)\n THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL)\n ELSE NULL END AS \"cutoff?\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "cutoff?", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "a54f686b1bfb16e4e1da2bc143ef172c89da5b216d3530c2743319ab1f4ca853" +} diff --git a/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json b/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json new file mode 100644 index 0000000000..5ac11c6a5e --- /dev/null +++ b/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json @@ -0,0 +1,44 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT to_char(timestamp AT TIME ZONE 'UTC', 'YYYY-MM-DD') AS \"day!\",\n id AS \"id!\",\n timestamp AS \"ts!\",\n row_to_json(r)::text AS \"line!\"\n FROM (\n SELECT workspace_id, id, timestamp, username, operation,\n action_kind::text AS action_kind, resource, parameters, email, span\n FROM audit_partitioned\n WHERE timestamp >= $1 AND timestamp < $2\n AND (timestamp, id) > ($3, $4)\n ORDER BY timestamp, id\n LIMIT $5\n ) r\n ORDER BY timestamp, id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "day!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "id!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "ts!", + "type_info": "Timestamptz" + }, + { + "ordinal": 3, + "name": "line!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Timestamptz", + "Timestamptz", + "Timestamptz", + "Int8", + "Int8" + ] + }, + "nullable": [ + null, + false, + false, + null + ] + }, + "hash": "fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c" +} diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index 85d87fe2ca..294707d466 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -7b92c8e0de4cfc6d986499d60a5f79cd1c6b9d0b +b821fecccbcba2efed544890576bf2b84321d70d diff --git a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql new file mode 100644 index 0000000000..2c9e78659e --- /dev/null +++ b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql @@ -0,0 +1,19 @@ +-- Restore the previous anchor: epoch sentinel + preserve-cursor (DO NOTHING). +CREATE OR REPLACE FUNCTION audit_logs_s3_anchor_on_enable() +RETURNS TRIGGER AS $$ +BEGIN + IF NEW.value = to_jsonb(true) + AND (TG_OP = 'INSERT' OR OLD.value IS DISTINCT FROM NEW.value) THEN + INSERT INTO background_task_state (name, value) + VALUES ( + 'audit_logs_s3_export', + jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', '1970-01-01T00:00:00+00:00' + ) + ) + ON CONFLICT (name) DO NOTHING; + END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; diff --git a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql new file mode 100644 index 0000000000..3fd759da9e --- /dev/null +++ b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql @@ -0,0 +1,70 @@ +-- Anchors the audit→object-store export cursor when the setting is enabled. +-- +-- `last_ts`/`last_oldest_inflight_ts` must be a *recent* floor, not epoch: the +-- export's `timestamp >= floor` predicate is the only partition-pruning bound (the +-- `age(xmin)` cursor is unindexable), so an epoch floor would scan the whole +-- `audit_partitioned` table on the first run and never finish under a +-- `statement_timeout`. The floor must be at or below the timestamp of any row whose +-- xid >= this snapshot xmin; the oldest in-flight `xact_start` is that bound when +-- stats are visible (no restricted role / prepared 2PC txn), else a bounded 7-day +-- window. +-- +-- `ON CONFLICT DO UPDATE ... WHERE last_xmin <` keeps the cursor monotonic: a +-- re-enable re-anchors it forward (so the export resumes from ~now rather than +-- rescanning the disabled gap — that gap is the backfill's job), but it never moves +-- backwards, so it is HA-safe and can't be regressed by a slower concurrent writer. +-- +-- The task name literal must match +-- `windmill_common::global_settings::AUDIT_LOGS_S3_EXPORT_TASK`. + +CREATE OR REPLACE FUNCTION audit_logs_s3_anchor_on_enable() +RETURNS TRIGGER AS $$ +DECLARE + v_floor timestamptz; +BEGIN + IF NEW.value = to_jsonb(true) + AND (TG_OP = 'INSERT' OR OLD.value IS DISTINCT FROM NEW.value) THEN + v_floor := COALESCE( + CASE WHEN (current_setting('is_superuser') = 'on' + OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE')) + AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts) + THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL) + ELSE NULL END, + now() - interval '7 days'); + INSERT INTO background_task_state (name, value) + VALUES ( + 'audit_logs_s3_export', + jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', now(), + 'last_oldest_inflight_ts', v_floor + ) + ) + ON CONFLICT (name) DO UPDATE + SET value = EXCLUDED.value + WHERE (background_task_state.value->>'last_xmin')::bigint + < (EXCLUDED.value->>'last_xmin')::bigint; + END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +-- Recovery for a legacy epoch-sentinel checkpoint (`last_ts = epoch`, never +-- drained). It cannot be safely resumed: its un-drained backlog can be arbitrarily +-- old, so stamping a recent floor over the old xmin would prune the older rows while +-- the cursor advanced past them (silent loss), and keeping the epoch floor would +-- reintroduce the full scan. Re-anchor it to now like a fresh enable; the pre-anchor +-- window is recoverable via the opt-in backfill, not silently dropped. +UPDATE background_task_state +SET value = jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', to_jsonb(now()), + 'last_oldest_inflight_ts', to_jsonb(COALESCE( + CASE WHEN (current_setting('is_superuser') = 'on' + OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE')) + AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts) + THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL) + ELSE NULL END, + now() - interval '7 days'))) +WHERE name = 'audit_logs_s3_export' + AND (value->>'last_ts')::timestamptz <= 'epoch'::timestamptz; diff --git a/backend/windmill-api-settings/src/audit_logs_s3.rs b/backend/windmill-api-settings/src/audit_logs_s3.rs index 9fa4574a1e..d7d749cb58 100644 --- a/backend/windmill-api-settings/src/audit_logs_s3.rs +++ b/backend/windmill-api-settings/src/audit_logs_s3.rs @@ -20,9 +20,11 @@ use windmill_common::DB; pub struct AuditLogsS3ExportStatus { /// xid cursor: rows of transactions below this have been exported. pub last_xmin: i64, - /// Partition-pruning floor (the epoch sentinel while still bootstrapping). + /// Partition-pruning floor: the latest audit-row timestamp the cursor has + /// reached (also the read side's 7-day-fallback anchor). pub last_ts: Option>, - /// True until the initial post-enable backlog has been fully drained. + /// True while the exporter is draining a backlog — the last run was capped + /// at `MAX_XID_INTERVAL` xids and has not yet caught up to the live snapshot. pub bootstrapping: bool, /// The latest audit-row timestamp actually written to object storage so /// far (monotonic) — the "how current is the mirror" figure. @@ -50,8 +52,7 @@ pub async fn get_status(db: &DB) -> error::Result::from_timestamp(0, 0).unwrap(); - let bootstrapping = last_ts.map(|t| t <= epoch).unwrap_or(true); + let bootstrapping = v.get("draining").and_then(|x| x.as_bool()).unwrap_or(false); Ok(Some(AuditLogsS3ExportStatus { last_xmin: v.get("last_xmin").and_then(|x| x.as_i64()).unwrap_or(0), last_ts, diff --git a/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs b/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs new file mode 100644 index 0000000000..e86ceb6448 --- /dev/null +++ b/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs @@ -0,0 +1,709 @@ +#![cfg(feature = "parquet")] +//! Opt-in historical backfill of audit logs to the instance object store. +//! +//! The steady-state exporter (the EE `export_audit_logs_to_object_store`) cursors +//! on transaction xmin and, by design, only exports rows committed *after* the +//! feature was enabled — it never rescans history (an `age(xmin)` predicate is +//! unindexable, so scanning the whole partitioned table can't survive a +//! `statement_timeout`). This module covers the complementary need: exporting a +//! chosen historical `[from, to)` window (e.g. the gap left while the export was +//! disabled) on demand. +//! +//! It is safe to run on a large table because it scans strictly by `timestamp` +//! (the partition key — pruned and indexed) in bounded keyset pages, so every +//! query touches at most one page worth of rows and survives a statement timeout. +//! It does not touch the xmin cursor / checkpoint at all. +//! +//! Objects are written next to the steady-state ones under `logs/audit/dt=/` +//! as `audit_backfill__.ndjson`, with the exact same row shape, so +//! a consumer reads them uniformly. The key includes the requested window so two +//! different backfill ranges never overwrite each other (a per-page `min_id` alone +//! is not unique across windows). Re-running the *same* window is deterministic +//! (audit history is append-only), so it overwrites the same objects rather than +//! duplicating. A window that overlaps already-exported steady-state rows simply +//! re-emits them under a different key; consumers dedupe by `id`. +//! +//! Scope: like the steady-state export, this reads only `audit_partitioned`. The +//! pre-partitioning `audit` table is intentionally not exported; a window that +//! overlaps any legacy `audit` row is rejected (see [`try_start`]) so a backfill +//! never silently reports success while omitting them. +//! +//! Progress is persisted in `background_task_state` (name [`TASK_NAME`]) so any +//! API replica can serve the status endpoint, mirroring `log_cleanup`. + +use std::sync::Arc; + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use tokio::sync::RwLock; + +use crate::background_task; +use windmill_common::error::{self}; +use windmill_common::tracing_init::LOGS_AUDIT; +use windmill_common::{DB, INSTANCE_NAME}; + +use windmill_object_store::object_store_reexports::{ObjectStore, Path as ObjectPath}; + +pub const TASK_NAME: &str = "audit_logs_s3_backfill"; + +/// Rows fetched per keyset page. Bounds each query so it stays well under any +/// `statement_timeout` even on a busy partition, and bounds peak memory (one +/// page of ndjson is buffered before the day-grouped PUTs). +const PAGE_ROWS: i64 = 10_000; + +/// Test-only override for [`PAGE_ROWS`] (0 = use the default), so a test can force +/// multi-page / page-spanning-day keyset behaviour with only a handful of rows. +#[cfg(test)] +static PAGE_ROWS_OVERRIDE: std::sync::atomic::AtomicI64 = std::sync::atomic::AtomicI64::new(0); + +fn page_rows() -> i64 { + #[cfg(test)] + { + match PAGE_ROWS_OVERRIDE.load(std::sync::atomic::Ordering::Relaxed) { + 0 => PAGE_ROWS, + n => n, + } + } + #[cfg(not(test))] + { + PAGE_ROWS + } +} + +#[derive(Clone, Serialize, Deserialize)] +pub struct AuditBackfillProgress { + pub running: bool, + pub started_at: DateTime, + pub finished_at: Option>, + /// Human-readable description of the current phase. + pub phase: String, + /// Inclusive lower / exclusive upper bound of the window being exported. + pub from: DateTime, + pub to: DateTime, + /// Audit rows written to object storage so far. + pub rows_written: u64, + /// Object PUTs issued so far (one per day per page). + pub objects_written: u64, + /// Keyset cursor: the timestamp of the last row exported (how far the + /// backfill has progressed through the window). + pub last_ts: Option>, + pub errors: u64, + pub last_error: Option, +} + +impl AuditBackfillProgress { + fn new_running(from: DateTime, to: DateTime) -> Self { + Self { + running: true, + started_at: Utc::now(), + finished_at: None, + phase: "starting".to_string(), + from, + to, + rows_written: 0, + objects_written: 0, + last_ts: None, + errors: 0, + last_error: None, + } + } +} + +struct Session { + db: DB, + owner: String, + progress: RwLock, +} + +impl Session { + async fn update(&self, f: F) { + let snapshot = { + let mut p = self.progress.write().await; + f(&mut p); + p.clone() + }; + if let Err(e) = + background_task::update_state(&self.db, TASK_NAME, &self.owner, &snapshot).await + { + tracing::warn!("audit backfill: failed to persist progress: {e:#}"); + } + } + + async fn record_error(&self, msg: String) { + tracing::error!("audit backfill: {msg}"); + self.update(|p| { + p.errors = p.errors.saturating_add(1); + p.last_error = Some(msg); + }) + .await; + } + + async fn release(&self) { + let snapshot = { + let mut p = self.progress.write().await; + p.running = false; + p.finished_at = Some(Utc::now()); + p.phase = "done".to_string(); + p.clone() + }; + tracing::info!( + "audit backfill finished: {} row(s) in {} object(s) for [{}, {}), {} error(s)", + snapshot.rows_written, + snapshot.objects_written, + snapshot.from, + snapshot.to, + snapshot.errors + ); + if let Err(e) = background_task::release(&self.db, TASK_NAME, &self.owner, &snapshot).await + { + tracing::warn!("audit backfill: failed to release lease: {e:#}"); + } + } +} + +#[derive(Deserialize)] +pub struct BackfillRequest { + pub from: DateTime, + pub to: DateTime, +} + +/// Atomically claim the backfill lease, or error if one is already running. +pub async fn try_start(db: &DB, from: DateTime, to: DateTime) -> error::Result<()> { + if from >= to { + return Err(error::Error::BadRequest( + "audit backfill: `from` must be strictly before `to`".to_string(), + )); + } + // The backfill keyset-pages by `(timestamp, id)` over rows visible at scan time and + // declares the window fully exported once the scan runs dry. But a row's `timestamp` + // is its inserting transaction's `xact_start`, so a transaction that started inside + // `[from, to)` yet commits after the scan has passed that timestamp — or any row + // committed when `to` is in the future — would be silently omitted. Require `to` to + // be at or before the oldest in-flight `xact_start`: everything strictly older than + // the oldest running transaction is already committed and stable. + // + // That bound is only sound when we can see every xmin-holding transaction. A role + // without pg_read_all_stats/superuser sees only its own sessions, and a prepared + // (2PC) transaction is invisible to pg_stat_activity — in either case an old + // transaction could still commit rows inside an accepted window after our scan ends. + // Since a backfill asserts completeness, we REJECT in those cases rather than fall + // back to a best-effort margin (NULL below). (The continuous exporter, which only + // claims bounded lag, keeps the 7-day fallback instead.) + let settled_cutoff: Option> = sqlx::query_scalar!( + r#"SELECT CASE WHEN (current_setting('is_superuser') = 'on' + OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE')) + AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts) + THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL) + ELSE NULL END AS "cutoff?""# + ) + .fetch_one(db) + .await?; + let Some(settled_cutoff) = settled_cutoff else { + return Err(error::Error::BadRequest( + "audit backfill: cannot determine a trustworthy settled-time boundary, so completeness \ + can't be guaranteed. The windmill DB role needs pg_read_all_stats (or superuser) and \ + there must be no prepared (2PC) transactions in progress — otherwise an old or \ + invisible transaction could later commit audit rows inside the requested window and \ + the backfill would miss them. Grant the privilege / resolve prepared transactions and \ + retry." + .to_string(), + )); + }; + if to > settled_cutoff { + return Err(error::Error::BadRequest(format!( + "audit backfill: `to` ({to}) must be at or before {settled_cutoff}, the newest point \ + guaranteed settled (the oldest in-flight transaction's start); choose an earlier \ + upper bound." + ))); + } + // The backfill (like the steady-state export) reads only `audit_partitioned`. Audit + // history from before partitioning was introduced lives in the legacy `audit` table + // and is intentionally not exported. If the requested window overlaps any legacy row, + // reject — otherwise a "completed" backfill would silently omit them. Checking the + // legacy table directly (rather than min(audit_partitioned)) also covers an upgraded + // instance whose `audit_partitioned` is still empty, where a min() guard would no-op. + // Non-macro query: no compile-time-checked entry needed. + let overlaps_legacy: bool = sqlx::query_scalar::<_, bool>( + "SELECT EXISTS (SELECT 1 FROM audit WHERE timestamp >= $1 AND timestamp < $2)", + ) + .bind(from) + .bind(to) + .fetch_one(db) + .await?; + if overlaps_legacy { + return Err(error::Error::BadRequest( + "audit backfill: the requested window overlaps rows in the legacy (pre-partitioning) \ + `audit` table, which is not exported to object storage. Restrict the window to the \ + partitioned era (after audit-log partitioning was introduced)." + .to_string(), + )); + } + let claimed = background_task::try_claim( + db, + TASK_NAME, + &*INSTANCE_NAME, + &AuditBackfillProgress::new_running(from, to), + ) + .await?; + if !claimed { + return Err(error::Error::BadRequest( + "An audit log backfill is already running".to_string(), + )); + } + Ok(()) +} + +/// Fetch the current backfill status. Any API server can call this. +pub async fn get_status(db: &DB) -> error::Result> { + let Some(r) = background_task::get(db, TASK_NAME).await? else { + return Ok(None); + }; + match serde_json::from_value::(r.value) { + Ok(mut p) => { + // get() collapses `running` to false when the heartbeat is stale. + p.running = r.running; + Ok(Some(p)) + } + Err(e) => Err(error::Error::internal_err(format!( + "deserialize audit backfill progress: {e:#}" + ))), + } +} + +pub fn spawn_backfill(db: DB, from: DateTime, to: DateTime) { + use futures::FutureExt; + use std::panic::AssertUnwindSafe; + + tokio::spawn(async move { + let session = Arc::new(Session { + db: db.clone(), + owner: INSTANCE_NAME.clone(), + progress: RwLock::new(AuditBackfillProgress::new_running(from, to)), + }); + + let s = session.clone(); + let task = async move { + let store = match windmill_object_store::get_object_store().await { + Some(st) => st, + None => { + s.record_error("Object storage is not configured".to_string()) + .await; + return; + } + }; + if let Err(e) = run_backfill(&s, &db, &store, from, to).await { + s.record_error(format!("backfill failed: {e:#}")).await; + } + }; + + // catch_unwind so a panic can't leave the lease held forever. + if let Err(panic) = AssertUnwindSafe(task).catch_unwind().await { + let msg = panic + .downcast_ref::<&str>() + .map(|s| s.to_string()) + .or_else(|| panic.downcast_ref::().cloned()) + .unwrap_or_else(|| "unknown panic".to_string()); + session + .record_error(format!("backfill task panicked: {msg}")) + .await; + } + + session.release().await; + }); +} + +/// Export `[from, to)` in keyset pages ordered by `(timestamp, id)`. Each page is +/// a bounded, partition-pruned scan; rows are grouped by UTC day and written one +/// object per day per page. +async fn run_backfill( + session: &Session, + db: &DB, + store: &Arc, + from: DateTime, + to: DateTime, +) -> error::Result<()> { + session.update(|p| p.phase = "exporting".to_string()).await; + + // Keyset cursor over (timestamp, id). `id` starts below any real value so the + // first page includes rows at exactly `from`. + let mut cursor_ts = from; + let mut cursor_id: i64 = -1; + let page_rows = page_rows(); + // Namespace object keys by the requested window. The per-page `min_id` alone is not + // unique across runs: a narrower, overlapping backfill can start a day's page at the + // same first row (same `min_id`) but contain fewer rows, and `put` would overwrite a + // broader run's object — silently dropping the rows only that object held. Including + // the window makes different ranges write disjoint keys (same window re-runs stay + // idempotent); consumers already dedupe overlapping rows by `id`. + let window_key = format!("{}_{}", from.timestamp_millis(), to.timestamp_millis()); + + loop { + let rows = sqlx::query!( + r#"SELECT to_char(timestamp AT TIME ZONE 'UTC', 'YYYY-MM-DD') AS "day!", + id AS "id!", + timestamp AS "ts!", + row_to_json(r)::text AS "line!" + FROM ( + SELECT workspace_id, id, timestamp, username, operation, + action_kind::text AS action_kind, resource, parameters, email, span + FROM audit_partitioned + WHERE timestamp >= $1 AND timestamp < $2 + AND (timestamp, id) > ($3, $4) + ORDER BY timestamp, id + LIMIT $5 + ) r + ORDER BY timestamp, id"#, + from, + to, + cursor_ts, + cursor_id, + page_rows + ) + .fetch_all(db) + .await?; + + if rows.is_empty() { + break; + } + + // Group this page's ndjson lines by day, preserving (timestamp, id) order, + // and track the min id per day for a deterministic, collision-free key. + let mut by_day: Vec<(String, i64, String)> = Vec::new(); // (day, min_id, ndjson) + for row in &rows { + match by_day.last_mut() { + Some((day, _min_id, acc)) if *day == row.day => { + acc.push('\n'); + acc.push_str(&row.line); + } + _ => by_day.push((row.day.clone(), row.id, row.line.clone())), + } + } + + for (day, min_id, ndjson) in &by_day { + let object_path = ObjectPath::from(format!( + "{LOGS_AUDIT}dt={day}/audit_backfill_{window_key}_{min_id}.ndjson" + )); + store + .put(&object_path, ndjson.clone().into_bytes().into()) + .await + .map_err(|e| error::Error::internal_err(format!("upload {object_path}: {e:#}")))?; + let n = ndjson.lines().count() as u64; + // Persist progress (and refresh the lease heartbeat) after every object, + // not just once the page completes: a stale heartbeat lets another replica + // re-claim the lease and run a concurrent backfill, so the gap between + // heartbeats must stay well under STALE_HEARTBEAT_SECS even if a page's + // uploads are slow. + session + .update(|p| { + p.rows_written = p.rows_written.saturating_add(n); + p.objects_written = p.objects_written.saturating_add(1); + }) + .await; + } + + // Advance the keyset cursor past the last row of this page. + let last = rows.last().expect("page is non-empty"); + cursor_ts = last.ts; + cursor_id = last.id; + + let new_last_ts = last.ts; + session.update(|p| p.last_ts = Some(new_last_ts)).await; + + // A short page means the window is exhausted. + if (rows.len() as i64) < page_rows { + break; + } + } + + Ok(()) +} + +#[cfg(all(test, feature = "parquet"))] +mod tests { + use super::*; + use futures::stream::StreamExt; + use std::sync::atomic::Ordering; + use std::sync::Arc; + use windmill_object_store::object_store_reexports::{InMemory, ObjectStore, Path as OsPath}; + + /// A private, per-test object store. `run_backfill` takes the store as a parameter, + /// so tests use a local one and never touch the process-global `OBJECT_STORE_SETTINGS` + /// (which would otherwise race across the parallel test runner). + fn local_store() -> (Arc, Arc) { + let store = Arc::new(InMemory::new()); + let dynstore: Arc = store.clone(); + (store, dynstore) + } + + /// Serializes the tests that touch the `PAGE_ROWS_OVERRIDE` process global (read + /// inside `run_backfill`) so they can't observe each other's value under the parallel + /// runner. + static PAGE_OVERRIDE_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); + + /// Resets `PAGE_ROWS_OVERRIDE` on drop so a failing assertion can't leak a non-default + /// page size into another test. + struct ResetPageOverride; + impl Drop for ResetPageOverride { + fn drop(&mut self) { + PAGE_ROWS_OVERRIDE.store(0, Ordering::Relaxed); + } + } + + /// Insert an audit row `days` days in the past (creating the daily partition if + /// needed). The row's `timestamp` defaults to that point, landing it in the + /// matching partition. + async fn insert_audit_days_ago(db: &DB, operation: &str, days: i64) -> i64 { + sqlx::query(&format!( + "DO $$ DECLARE d date := current_date - {days}; BEGIN \ + EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_partitioned \ + FOR VALUES FROM (%L) TO (%L)', 'audit_'||to_char(d,'YYYYMMDD'), d, d + 1); END $$;" + )) + .execute(db) + .await + .ok(); + sqlx::query_scalar::<_, i64>(&format!( + "INSERT INTO audit_partitioned + (workspace_id, username, operation, action_kind, parameters, timestamp) + VALUES ('test-ws','tester',$1,'create'::action_kind,'{{}}'::jsonb, + now() - interval '{days} days') + RETURNING id" + )) + .bind(operation) + .fetch_one(db) + .await + .expect("insert audit row") + } + + /// Insert an audit row at an exact timestamp (creating the daily partition if + /// needed), for tests that need distinct in-day timestamps. + async fn insert_audit_at(db: &DB, operation: &str, ts: DateTime) -> i64 { + sqlx::query(&format!( + "DO $$ DECLARE d date := '{}'; BEGIN \ + EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_partitioned \ + FOR VALUES FROM (%L) TO (%L)', 'audit_'||to_char(d,'YYYYMMDD'), d, d + 1); END $$;", + ts.format("%Y-%m-%d") + )) + .execute(db) + .await + .ok(); + sqlx::query_scalar::<_, i64>( + "INSERT INTO audit_partitioned + (workspace_id, username, operation, action_kind, parameters, timestamp) + VALUES ('test-ws','tester',$1,'create'::action_kind,'{}'::jsonb,$2) + RETURNING id", + ) + .bind(operation) + .bind(ts) + .fetch_one(db) + .await + .expect("insert audit row") + } + + /// All ids across every `audit_backfill_*.ndjson` object, and the set of object + /// paths (to assert pagination/day keying). + async fn backfilled(store: &InMemory) -> (Vec, Vec) { + let prefix = OsPath::from("logs/audit"); + let metas = store + .list(Some(&prefix)) + .collect::>() + .await + .into_iter() + .map(|m| m.expect("list object")) + .collect::>(); + let mut ids = Vec::new(); + let mut paths = Vec::new(); + for meta in metas { + paths.push(meta.location.to_string()); + let bytes = store + .get(&meta.location) + .await + .expect("get object") + .bytes() + .await + .expect("read bytes"); + for line in std::str::from_utf8(&bytes).unwrap().lines() { + if line.is_empty() { + continue; + } + let v: serde_json::Value = serde_json::from_str(line).expect("valid ndjson"); + ids.push(v.get("id").and_then(|x| x.as_i64()).expect("row has id")); + } + } + ids.sort(); + paths.sort(); + (ids, paths) + } + + fn session(db: &DB, from: DateTime, to: DateTime) -> Session { + Session { + db: db.clone(), + owner: INSTANCE_NAME.clone(), + progress: RwLock::new(AuditBackfillProgress::new_running(from, to)), + } + } + + // End-to-end backfill: a settled multi-day window is exported in bounded keyset + // pages (forced to 2 rows/page) — every in-window row lands exactly once, rows + // outside [from,to) are excluded, a day that spans a page boundary produces more + // than one object, and re-running is idempotent (same keys overwritten, no dupes). + #[sqlx::test(migrations = "../migrations")] + async fn backfill_exports_window_in_pages(db: DB) -> anyhow::Result<()> { + let _serial = PAGE_OVERRIDE_LOCK.lock().await; + let _reset = ResetPageOverride; // restores the page override even on panic + let (store, dyn_store) = local_store(); + // Force multi-page / page-spanning-day keyset behaviour with a handful of rows. + PAGE_ROWS_OVERRIDE.store(2, Ordering::Relaxed); + + // In window [now-6d, now-2d): days 5, 4, 3 ago. + let mut want = Vec::new(); + for i in 0..3 { + want.push(insert_audit_days_ago(&db, &format!("bf.d5.{i}"), 5).await); + } + for i in 0..2 { + want.push(insert_audit_days_ago(&db, &format!("bf.d4.{i}"), 4).await); + } + for i in 0..2 { + want.push(insert_audit_days_ago(&db, &format!("bf.d3.{i}"), 3).await); + } + want.sort(); + // Out of window: before `from` and at/after `to`. + let before = insert_audit_days_ago(&db, "bf.before", 7).await; + let after = insert_audit_days_ago(&db, "bf.after", 1).await; + + let from = Utc::now() - chrono::Duration::days(6); + let to = Utc::now() - chrono::Duration::days(2); + + let s = session(&db, from, to); + run_backfill(&s, &db, &dyn_store, from, to).await?; + + let (ids, paths) = backfilled(&store).await; + assert_eq!(ids, want, "exactly the in-window rows, each once: {ids:?}"); + assert!( + !ids.contains(&before) && !ids.contains(&after), + "rows outside [from,to) must not be exported" + ); + // 3 rows on the day-5 partition at a 2-row page size => that day spans pages, + // so it yields >1 object — proving keyset paging across a day boundary. + let day5_objects = paths + .iter() + .filter(|p| p.contains("audit_backfill_")) + .count(); + assert!( + day5_objects >= 4, + "expected multiple paged objects (incl. a split day), got {paths:?}" + ); + { + let p = s.progress.read().await; + assert_eq!(p.rows_written, want.len() as u64, "progress row count"); + } + + // Idempotent re-run: deterministic keys are overwritten, never duplicated. + let s2 = session(&db, from, to); + run_backfill(&s2, &db, &dyn_store, from, to).await?; + let (ids2, _) = backfilled(&store).await; + assert_eq!(ids2, want, "re-run stays exactly once per row: {ids2:?}"); + + Ok(()) + } + + // A narrower backfill overlapping a broader one must not overwrite (and drop rows + // from) the broader run's object: the object key includes the window. The two + // windows share a day and the same first row (so the same `min_id`), but the + // narrower one holds fewer rows. + #[sqlx::test(migrations = "../migrations")] + async fn backfill_window_in_key_prevents_overwrite(db: DB) -> anyhow::Result<()> { + // Hold the lock so no concurrent test's PAGE_ROWS_OVERRIDE is observed; this test + // wants the default (large) page size so each day is one object per window. + let _serial = PAGE_OVERRIDE_LOCK.lock().await; + let (store, dyn_store) = local_store(); + + // Four rows on the same day at distinct times. + let base = Utc::now() - chrono::Duration::days(5); + let r0 = insert_audit_at(&db, "ov.0", base).await; + let r1 = insert_audit_at(&db, "ov.1", base + chrono::Duration::seconds(10)).await; + let r2 = insert_audit_at(&db, "ov.2", base + chrono::Duration::seconds(20)).await; + let r3 = insert_audit_at(&db, "ov.3", base + chrono::Duration::seconds(30)).await; + + // Broad run covers all four (one object for the day, keyed by r0). + let a_from = base - chrono::Duration::seconds(1); + let a_to = base + chrono::Duration::seconds(31); + run_backfill(&session(&db, a_from, a_to), &db, &dyn_store, a_from, a_to).await?; + + // Narrow run starts at the same first row (same min_id) but holds only r0, r1. + let b_from = base - chrono::Duration::seconds(1); + let b_to = base + chrono::Duration::seconds(15); + run_backfill(&session(&db, b_from, b_to), &db, &dyn_store, b_from, b_to).await?; + + let (ids, _) = backfilled(&store).await; + for id in [r0, r1, r2, r3] { + assert!( + ids.contains(&id), + "row {id} lost — a narrower overlapping window overwrote the broader run's \ + object: {ids:?}" + ); + } + Ok(()) + } + + // The endpoint rejects a window whose upper bound is not yet settled (a row's + // timestamp is its txn's xact_start, so a future/live `to` could miss late + // commits), but accepts a window safely in the past. + #[sqlx::test(migrations = "../migrations")] + async fn backfill_rejects_unstable_window(db: DB) -> anyhow::Result<()> { + let future = Utc::now() + chrono::Duration::days(1); + let past_from = Utc::now() - chrono::Duration::days(2); + let err = try_start(&db, past_from, future).await.unwrap_err(); + assert!( + matches!(err, error::Error::BadRequest(_)), + "a future `to` must be rejected as unstable, got {err:?}" + ); + + // A window fully in the settled past is accepted. + let from = Utc::now() - chrono::Duration::days(3); + let to = Utc::now() - chrono::Duration::days(2); + try_start(&db, from, to) + .await + .expect("a settled past window is accepted"); + Ok(()) + } + + /// Insert a row into the legacy (non-partitioned) `audit` table at an exact time. + async fn insert_legacy_audit_at(db: &DB, operation: &str, ts: DateTime) { + sqlx::query( + "INSERT INTO audit (workspace_id, username, operation, action_kind, parameters, timestamp) + VALUES ('test-ws','tester',$1,'create'::action_kind,'{}'::jsonb,$2)", + ) + .bind(operation) + .bind(ts) + .execute(db) + .await + .expect("insert legacy audit row"); + } + + // A window overlapping rows in the legacy (non-partitioned) `audit` table is rejected: + // those rows are not exported, so the backfill must not report success while silently + // omitting them. Covers the empty-`audit_partitioned` case (a min(partitioned) guard + // would no-op there). + #[sqlx::test(migrations = "../migrations")] + async fn backfill_rejects_window_overlapping_legacy(db: DB) -> anyhow::Result<()> { + // A legacy row ~5 days ago, and no partitioned rows at all. + insert_legacy_audit_at(&db, "legacy.row", Utc::now() - chrono::Duration::days(5)).await; + + // A window covering it is rejected. + let from = Utc::now() - chrono::Duration::days(6); + let to = Utc::now() - chrono::Duration::days(2); + let err = try_start(&db, from, to).await.unwrap_err(); + assert!( + matches!(err, error::Error::BadRequest(_)), + "a window overlapping legacy audit rows must be rejected, got {err:?}" + ); + + // A window clear of any legacy row is accepted. + let from_ok = Utc::now() - chrono::Duration::days(2); + let to_ok = Utc::now() - chrono::Duration::days(1); + try_start(&db, from_ok, to_ok) + .await + .expect("a window with no legacy overlap is accepted"); + Ok(()) + } +} diff --git a/backend/windmill-api-settings/src/lib.rs b/backend/windmill-api-settings/src/lib.rs index 3c179168f5..a7e1322ad9 100644 --- a/backend/windmill-api-settings/src/lib.rs +++ b/backend/windmill-api-settings/src/lib.rs @@ -14,6 +14,8 @@ use std::{ #[cfg(feature = "parquet")] mod audit_logs_s3; #[cfg(feature = "parquet")] +mod audit_logs_s3_backfill; +#[cfg(feature = "parquet")] mod background_task; #[cfg(feature = "private")] mod ee; @@ -204,7 +206,12 @@ pub fn global_service() -> Router { ) .route("/run_log_cleanup", post(run_log_cleanup)) .route("/log_cleanup_status", get(log_cleanup_status)) - .route("/audit_logs_s3_status", get(audit_logs_s3_status)); + .route("/audit_logs_s3_status", get(audit_logs_s3_status)) + .route("/audit_logs_s3_backfill", post(run_audit_logs_s3_backfill)) + .route( + "/audit_logs_s3_backfill_status", + get(audit_logs_s3_backfill_status), + ); } #[cfg(not(feature = "parquet"))] @@ -601,6 +608,32 @@ async fn audit_logs_s3_status( Ok(Json(audit_logs_s3::get_status(&db).await?)) } +#[cfg(feature = "parquet")] +async fn run_audit_logs_s3_backfill( + Extension(db): Extension, + authed: ApiAuthed, + Json(req): Json, +) -> error::Result { + require_super_admin(&db, &authed.email).await?; + if !matches!(get_license_plan().await, LicensePlan::Enterprise) { + return Err(error::Error::BadRequest( + "Audit log export to object storage is an Enterprise feature".to_string(), + )); + } + audit_logs_s3_backfill::try_start(&db, req.from, req.to).await?; + audit_logs_s3_backfill::spawn_backfill(db.clone(), req.from, req.to); + Ok(axum::http::StatusCode::ACCEPTED) +} + +#[cfg(feature = "parquet")] +async fn audit_logs_s3_backfill_status( + Extension(db): Extension, + authed: ApiAuthed, +) -> error::JsonResult> { + require_super_admin(&db, &authed.email).await?; + Ok(Json(audit_logs_s3_backfill::get_status(&db).await?)) +} + #[derive(Deserialize)] pub struct TestKey { pub license_key: String, diff --git a/backend/windmill-api/openapi-deref.json b/backend/windmill-api/openapi-deref.json index 540d6bea52..e511578821 100644 --- a/backend/windmill-api/openapi-deref.json +++ b/backend/windmill-api/openapi-deref.json @@ -1,7 +1,7 @@ { "openapi": "3.0.3", "info": { - "version": "1.739.0", + "version": "1.740.0", "title": "Windmill API", "contact": { "name": "Windmill Team", @@ -2719,6 +2719,124 @@ } } }, + "/settings/audit_logs_s3_backfill": { + "post": { + "summary": "start an opt-in historical backfill of audit logs to object storage", + "operationId": "runAuditLogsS3Backfill", + "tags": [ + "setting" + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "from": { + "type": "string", + "format": "date-time", + "description": "inclusive lower bound of the window to export" + }, + "to": { + "type": "string", + "format": "date-time", + "description": "exclusive upper bound of the window to export" + } + }, + "required": [ + "from", + "to" + ] + } + } + } + }, + "responses": { + "202": { + "description": "backfill started" + } + } + } + }, + "/settings/audit_logs_s3_backfill_status": { + "get": { + "summary": "get status of the audit-log object-store historical backfill", + "operationId": "getAuditLogsS3BackfillStatus", + "tags": [ + "setting" + ], + "responses": { + "200": { + "description": "current backfill status (null if never run)", + "content": { + "application/json": { + "schema": { + "nullable": true, + "type": "object", + "properties": { + "running": { + "type": "boolean" + }, + "started_at": { + "type": "string", + "format": "date-time" + }, + "finished_at": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "phase": { + "type": "string" + }, + "from": { + "type": "string", + "format": "date-time" + }, + "to": { + "type": "string", + "format": "date-time" + }, + "rows_written": { + "type": "integer", + "format": "int64" + }, + "objects_written": { + "type": "integer", + "format": "int64" + }, + "last_ts": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "errors": { + "type": "integer", + "format": "int64" + }, + "last_error": { + "type": "string", + "nullable": true + } + }, + "required": [ + "running", + "started_at", + "phase", + "from", + "to", + "rows_written", + "objects_written", + "errors" + ] + } + } + } + } + } + } + }, "/settings/send_stats": { "post": { "summary": "send stats", @@ -18440,6 +18558,14 @@ "type": "boolean" } }, + { + "name": "timeout", + "description": "custom timeout in seconds for this preview run", + "in": "query", + "schema": { + "type": "integer" + } + }, { "$ref": "#/components/parameters/NewJobId" } @@ -20465,6 +20591,89 @@ } } }, + "/w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}": { + "get": { + "summary": "get all logs for a flow job in a structured format", + "operationId": "getFlowAllLogsStructured", + "tags": [ + "job" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "$ref": "#/components/parameters/JobId" + } + ], + "responses": { + "200": { + "description": "structured logs of all flow steps, one entry per job", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "properties": { + "job_id": { + "type": "string" + }, + "label": { + "type": "string", + "description": "human-readable label describing the job's position in the flow tree" + }, + "kind": { + "type": "string", + "description": "job kind (script, flow, forloopflow, ...)" + }, + "flow_step_id": { + "type": "string", + "nullable": true + }, + "step_path": { + "type": "string", + "nullable": true, + "description": "materialized step path (e.g. \"a/b\")" + }, + "depth": { + "type": "integer", + "description": "depth in the flow tree (0 for the root flow job)" + }, + "parent_module_type": { + "type": "string", + "nullable": true, + "description": "parent module type (forloopflow, branchall, ...)" + }, + "sibling_index": { + "type": "integer", + "description": "1-based index of this job among siblings sharing the same step" + }, + "sibling_count": { + "type": "integer", + "description": "total number of siblings sharing the same step" + }, + "logs": { + "type": "string" + } + }, + "required": [ + "job_id", + "label", + "kind", + "depth", + "sibling_index", + "sibling_count", + "logs" + ] + } + } + } + } + } + } + } + }, "/w/{workspace}/jobs_u/get_completed_logs_tail/{id}": { "get": { "summary": "get completed job logs tail", diff --git a/backend/windmill-api/openapi-deref.yaml b/backend/windmill-api/openapi-deref.yaml index 127fda6bed..e17c38efaf 100644 --- a/backend/windmill-api/openapi-deref.yaml +++ b/backend/windmill-api/openapi-deref.yaml @@ -1,6 +1,6 @@ openapi: 3.0.3 info: - version: 1.739.0 + version: 1.740.0 title: Windmill API contact: name: Windmill Team @@ -2727,6 +2727,90 @@ paths: - bootstrapping - last_run_exported - updated_at + /settings/audit_logs_s3_backfill: + post: + summary: start an opt-in historical backfill of audit logs to object storage + operationId: runAuditLogsS3Backfill + tags: + - setting + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + from: + type: string + format: date-time + description: inclusive lower bound of the window to export + to: + type: string + format: date-time + description: exclusive upper bound of the window to export + required: + - from + - to + responses: + '202': + description: backfill started + /settings/audit_logs_s3_backfill_status: + get: + summary: get status of the audit-log object-store historical backfill + operationId: getAuditLogsS3BackfillStatus + tags: + - setting + responses: + '200': + description: current backfill status (null if never run) + content: + application/json: + schema: + nullable: true + type: object + properties: + running: + type: boolean + started_at: + type: string + format: date-time + finished_at: + type: string + format: date-time + nullable: true + phase: + type: string + from: + type: string + format: date-time + to: + type: string + format: date-time + rows_written: + type: integer + format: int64 + objects_written: + type: integer + format: int64 + last_ts: + type: string + format: date-time + nullable: true + errors: + type: integer + format: int64 + last_error: + type: string + nullable: true + required: + - running + - started_at + - phase + - from + - to + - rows_written + - objects_written + - errors /settings/send_stats: post: summary: send stats @@ -18747,6 +18831,11 @@ paths: in: query schema: type: boolean + - name: timeout + description: custom timeout in seconds for this preview run + in: query + schema: + type: integer - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -21710,6 +21799,73 @@ paths: text/plain: schema: type: string + /w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}: + get: + summary: get all logs for a flow job in a structured format + operationId: getFlowAllLogsStructured + tags: + - job + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: id + in: path + required: true + schema: *ref_178 + responses: + '200': + description: structured logs of all flow steps, one entry per job + content: + application/json: + schema: + type: array + items: + type: object + properties: + job_id: + type: string + label: + type: string + description: >- + human-readable label describing the job's position in + the flow tree + kind: + type: string + description: job kind (script, flow, forloopflow, ...) + flow_step_id: + type: string + nullable: true + step_path: + type: string + nullable: true + description: materialized step path (e.g. "a/b") + depth: + type: integer + description: depth in the flow tree (0 for the root flow job) + parent_module_type: + type: string + nullable: true + description: parent module type (forloopflow, branchall, ...) + sibling_index: + type: integer + description: >- + 1-based index of this job among siblings sharing the + same step + sibling_count: + type: integer + description: total number of siblings sharing the same step + logs: + type: string + required: + - job_id + - label + - kind + - depth + - sibling_index + - sibling_count + - logs /w/{workspace}/jobs_u/get_completed_logs_tail/{id}: get: summary: get completed job logs tail diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index 0bbce245d3..7b7f4c4388 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1812,6 +1812,92 @@ paths: - last_run_exported - updated_at + /settings/audit_logs_s3_backfill: + post: + summary: start an opt-in historical backfill of audit logs to object storage + operationId: runAuditLogsS3Backfill + tags: + - setting + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + from: + type: string + format: date-time + description: inclusive lower bound of the window to export + to: + type: string + format: date-time + description: exclusive upper bound of the window to export + required: + - from + - to + responses: + "202": + description: backfill started + + /settings/audit_logs_s3_backfill_status: + get: + summary: get status of the audit-log object-store historical backfill + operationId: getAuditLogsS3BackfillStatus + tags: + - setting + responses: + "200": + description: current backfill status (null if never run) + content: + application/json: + schema: + nullable: true + type: object + properties: + running: + type: boolean + started_at: + type: string + format: date-time + finished_at: + type: string + format: date-time + nullable: true + phase: + type: string + from: + type: string + format: date-time + to: + type: string + format: date-time + rows_written: + type: integer + format: int64 + objects_written: + type: integer + format: int64 + last_ts: + type: string + format: date-time + nullable: true + errors: + type: integer + format: int64 + last_error: + type: string + nullable: true + required: + - running + - started_at + - phase + - from + - to + - rows_written + - objects_written + - errors + /settings/send_stats: post: summary: send stats diff --git a/frontend/src/lib/components/instanceSettings.ts b/frontend/src/lib/components/instanceSettings.ts index 72ccbeca2b..81a7b841b7 100644 --- a/frontend/src/lib/components/instanceSettings.ts +++ b/frontend/src/lib/components/instanceSettings.ts @@ -443,7 +443,7 @@ export const settings: Record = { { label: 'Store audit logs in object storage', description: - 'When enabled and instance object storage is configured, audit logs are also exported as newline-delimited JSON to the dedicated logs/audit/ folder (partitioned by day). Export is incremental and runs off the hot path. Pre-existing history is not backfilled: export starts from when the setting is enabled (transactions in flight at that moment may include a bounded set of just-prior rows). No audit log committed after enabling is ever skipped.', + 'When enabled and instance object storage is configured, audit logs are also exported as newline-delimited JSON to the dedicated logs/audit/ folder (partitioned by day). Export is incremental and runs off the hot path. Enabling (or re-enabling) anchors the export at ~now: while it stays enabled, every audit log committed from that point on is exported (transactions in flight at the moment of enabling may include a bounded set of just-prior rows). Pre-existing history, and any window during which export was disabled, are NOT exported by this cursor — use the opt-in backfill API to export a chosen historical range, back to when audit-log partitioning was introduced (older rows in the legacy audit table are not exported, and a window overlapping them is rejected): POST /settings/audit_logs_s3_backfill {from, to} (status at GET /settings/audit_logs_s3_backfill_status).', key: 'store_audit_logs_s3', fieldType: 'boolean', storage: 'setting', From 9172a0945bef7c82432006cc580ac3e635e0c72e Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Fri, 26 Jun 2026 21:43:02 +0200 Subject: [PATCH 111/117] chore(main): release 1.741.0 (#9804) * chore(main): release 1.741.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> --- CHANGELOG.md | 27 +++ backend/Cargo.lock | 160 +++++++++--------- backend/Cargo.toml | 4 +- .../parsers/windmill-parser-wasm/Cargo.lock | 48 +++--- .../parsers/windmill-parser-wasm/Cargo.toml | 2 +- backend/windmill-api/openapi.yaml | 2 +- benchmarks/lib.ts | 2 +- cli/src/core/constants.ts | 2 +- frontend/package-lock.json | 4 +- frontend/package.json | 2 +- lsp/Pipfile | 2 +- openflow.openapi.yaml | 2 +- .../WindmillClient/WindmillClient.psd1 | 2 +- python-client/wmill/pyproject.toml | 2 +- typescript-client/jsr.json | 2 +- typescript-client/package.json | 2 +- version.txt | 2 +- 17 files changed, 147 insertions(+), 120 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c3b1808a0f..5f07f85012 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,32 @@ # Changelog +## [1.741.0](https://github.com/windmill-labs/windmill/compare/v1.740.0...v1.741.0) (2026-06-26) + + +### Features + +* **ai-chat:** add create_folder tool to global chat ([#9819](https://github.com/windmill-labs/windmill/issues/9819)) ([44c25de](https://github.com/windmill-labs/windmill/commit/44c25de418612ab98341adb15d5671222b54367e)) +* **ai-chat:** hint /compact in context usage tooltip ([#9777](https://github.com/windmill-labs/windmill/issues/9777)) ([aadfb62](https://github.com/windmill-labs/windmill/commit/aadfb620c0b7dcd7e94367b761875e14ef9abe69)) +* **ai-chat:** let global chat edit the user's personal instructions ([#9771](https://github.com/windmill-labs/windmill/issues/9771)) ([3be2752](https://github.com/windmill-labs/windmill/commit/3be27521b05de33e48582e80c6651071f889f048)) +* **ai-chat:** surface raw apps in the @-mention context picker ([#9800](https://github.com/windmill-labs/windmill/issues/9800)) ([1602244](https://github.com/windmill-labs/windmill/commit/16022447c7b445be753b9545b10b4c67da0893d5)) +* capture managed-materialize output schema as asset metadata ([#2](https://github.com/windmill-labs/windmill/issues/2)a) ([#9812](https://github.com/windmill-labs/windmill/issues/9812)) ([ade74b2](https://github.com/windmill-labs/windmill/commit/ade74b297f6a03441e700a20ffc2d7291c8a85fd)) +* **sdk:** allow overriding worker tag when running jobs (WIN-2105) ([#9807](https://github.com/windmill-labs/windmill/issues/9807)) ([52fc7bf](https://github.com/windmill-labs/windmill/commit/52fc7bf94cf3f87f68d9dba9884944d87e7d5d57)) + + +### Bug Fixes + +* apply step timeout to 'Test this step' preview ([#9810](https://github.com/windmill-labs/windmill/issues/9810)) ([d04062b](https://github.com/windmill-labs/windmill/commit/d04062bff58c9c4c79ce542a4321e71bcbcf0e98)) +* **flows:** reject corrupt step paths at deploy + atomic cache writes ([#9751](https://github.com/windmill-labs/windmill/issues/9751)) ([#9813](https://github.com/windmill-labs/windmill/issues/9813)) ([3cda447](https://github.com/windmill-labs/windmill/commit/3cda44762148bcd2ee5c0ea821db884950376ead)) +* **frontend:** clarify instance data table unavailable on cloud ([#9806](https://github.com/windmill-labs/windmill/issues/9806)) ([c3e8c78](https://github.com/windmill-labs/windmill/commit/c3e8c789ac05c9c28991d9ab6f2358f61fa87971)) +* hide GCS service account key behind a reveal in object storage settings ([#9815](https://github.com/windmill-labs/windmill/issues/9815)) ([0ec5061](https://github.com/windmill-labs/windmill/commit/0ec5061270749ed078e01f5a4bc7397a1755ca32)) +* ping job during volume setup to prevent false zombie restarts ([#9803](https://github.com/windmill-labs/windmill/issues/9803)) ([43bb676](https://github.com/windmill-labs/windmill/commit/43bb676dc5652cb06fe1414b8d3aacf295bae36b)) +* skipped suspend step no longer parks the flow forever ([#9821](https://github.com/windmill-labs/windmill/issues/9821)) ([40110bc](https://github.com/windmill-labs/windmill/commit/40110bc7158bc42c3d84bd4637a12b82fcd72a9a)) + + +### Performance Improvements + +* **audit:** re-anchor S3 audit export on enable + opt-in backfill ([#9818](https://github.com/windmill-labs/windmill/issues/9818)) ([577ceee](https://github.com/windmill-labs/windmill/commit/577ceeee8679f054c6898d1a7889df30ab830f8f)) + ## [1.740.0](https://github.com/windmill-labs/windmill/compare/v1.739.0...v1.740.0) (2026-06-25) diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 9830bed379..8f11a0cb98 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -3412,9 +3412,9 @@ checksum = "c286de4e81ea2590afc24d754e0f83810c566f50a1388fa75ebd57928c0d9745" [[package]] name = "debug-helper" -version = "0.3.13" +version = "0.3.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f578e8e2c440e7297e008bb5486a3a8a194775224bbc23729b0dbdfaeebf162e" +checksum = "80a4af69c60438a1a82af89d362f4729fd38db7b73f305a237636fad31ceb2bf" [[package]] name = "debugid" @@ -13734,7 +13734,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-nats", @@ -13816,7 +13816,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.740.0" +version = "1.741.0" dependencies = [ "async-stream", "async-trait", @@ -13849,7 +13849,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -13862,7 +13862,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "argon2", @@ -14000,7 +14000,7 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14023,7 +14023,7 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14037,7 +14037,7 @@ dependencies = [ [[package]] name = "windmill-api-auth" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14063,7 +14063,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.740.0" +version = "1.741.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14073,7 +14073,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14090,7 +14090,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14112,7 +14112,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14135,7 +14135,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14151,7 +14151,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14172,7 +14172,7 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14193,7 +14193,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14207,7 +14207,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-nats", @@ -14242,7 +14242,7 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14267,7 +14267,7 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "flate2", @@ -14285,7 +14285,7 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14307,7 +14307,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14327,7 +14327,7 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14364,7 +14364,7 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14392,7 +14392,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.740.0" +version = "1.741.0" dependencies = [ "lazy_static", "serde", @@ -14404,7 +14404,7 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.740.0" +version = "1.741.0" dependencies = [ "argon2", "axum 0.8.9", @@ -14429,7 +14429,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14443,7 +14443,7 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.740.0" +version = "1.741.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14476,7 +14476,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.740.0" +version = "1.741.0" dependencies = [ "chrono", "lazy_static", @@ -14490,7 +14490,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14509,7 +14509,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.740.0" +version = "1.741.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -14611,7 +14611,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.740.0" +version = "1.741.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -14630,7 +14630,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.740.0" +version = "1.741.0" dependencies = [ "regex", "serde", @@ -14645,7 +14645,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -14669,7 +14669,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "futures", @@ -14686,7 +14686,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.740.0" +version = "1.741.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -14702,7 +14702,7 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -14723,7 +14723,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -14754,7 +14754,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "arc-swap", @@ -14779,7 +14779,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-stream", @@ -14813,7 +14813,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "futures", @@ -14831,7 +14831,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.740.0" +version = "1.741.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -14840,7 +14840,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "lazy_static", @@ -14852,7 +14852,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde_json", @@ -14864,7 +14864,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "gosyn", @@ -14876,7 +14876,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "lazy_static", @@ -14888,7 +14888,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde_json", @@ -14900,7 +14900,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "nu-parser", @@ -14911,7 +14911,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14922,7 +14922,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14934,7 +14934,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "rustpython-ast", @@ -14945,7 +14945,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-recursion", @@ -14967,7 +14967,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde_json", @@ -14979,7 +14979,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "lazy_static", @@ -14993,7 +14993,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15010,7 +15010,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "lazy_static", @@ -15023,7 +15023,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde", @@ -15035,7 +15035,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "lazy_static", @@ -15053,7 +15053,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15069,7 +15069,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15085,7 +15085,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde", @@ -15096,7 +15096,7 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-recursion", @@ -15135,7 +15135,7 @@ dependencies = [ [[package]] name = "windmill-runtime-nativets" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "const_format", @@ -15174,7 +15174,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.740.0" +version = "1.741.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15185,7 +15185,7 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-recursion", @@ -15219,7 +15219,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15243,7 +15243,7 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15276,7 +15276,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15309,7 +15309,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15329,7 +15329,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15363,7 +15363,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15399,7 +15399,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15422,7 +15422,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15446,7 +15446,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-nats", @@ -15470,7 +15470,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15505,7 +15505,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15533,7 +15533,7 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-trait", @@ -15558,7 +15558,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "bitflags 2.13.0", @@ -15577,7 +15577,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-once-cell", @@ -15687,7 +15687,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.740.0" +version = "1.741.0" dependencies = [ "bytes", "futures", diff --git a/backend/Cargo.toml b/backend/Cargo.toml index f6453298cd..d721dcc272 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.740.0" +version = "1.741.0" authors.workspace = true edition.workspace = true @@ -87,7 +87,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.740.0" +version = "1.741.0" authors = ["Ruben Fiszel "] edition = "2021" diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index 7616f1b324..78db9dd2b0 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.740.0" +version = "1.741.0" dependencies = [ "aho-corasick", "anyhow", @@ -6272,7 +6272,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.740.0" +version = "1.741.0" dependencies = [ "proc-macro2", "quote", @@ -6284,7 +6284,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.740.0" +version = "1.741.0" dependencies = [ "convert_case", "serde", @@ -6293,7 +6293,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "lazy_static", @@ -6305,7 +6305,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde_json", @@ -6317,7 +6317,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "gosyn", @@ -6329,7 +6329,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "lazy_static", @@ -6341,7 +6341,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde_json", @@ -6353,7 +6353,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "nu-parser", @@ -6364,7 +6364,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6375,7 +6375,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6387,7 +6387,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6398,7 +6398,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "async-recursion", @@ -6420,7 +6420,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde_json", @@ -6432,7 +6432,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "lazy_static", @@ -6446,7 +6446,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "convert_case", @@ -6463,7 +6463,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "lazy_static", @@ -6476,7 +6476,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde", @@ -6488,7 +6488,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "lazy_static", @@ -6506,7 +6506,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6522,7 +6522,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6538,7 +6538,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6570,7 +6570,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "serde", @@ -6581,7 +6581,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.740.0" +version = "1.741.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index 4538d1be39..6faadc1cec 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.740.0" +version = "1.741.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index 7b7f4c4388..e9c57c3fb0 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: - version: 1.740.0 + version: 1.741.0 title: Windmill API contact: diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index 596737fb93..2b3b23746a 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.740.0"; +export const VERSION = "v1.741.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ diff --git a/cli/src/core/constants.ts b/cli/src/core/constants.ts index b84b5a5939..1e40e04ac7 100644 --- a/cli/src/core/constants.ts +++ b/cli/src/core/constants.ts @@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork"; // (e.g. utils.ts) can read it without importing main.ts and creating a circular // dependency (main → workspace → utils → main) that triggers a TDZ. // Re-exported from main.ts for backwards compatibility. -export const VERSION = "1.740.0"; +export const VERSION = "1.741.0"; diff --git a/frontend/package-lock.json b/frontend/package-lock.json index beb9b9fe12..ed9dd1d7ea 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "@windmill-labs/components", - "version": "1.740.0", + "version": "1.741.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@windmill-labs/components", - "version": "1.740.0", + "version": "1.741.0", "hasInstallScript": true, "license": "AGPL-3.0", "dependencies": { diff --git a/frontend/package.json b/frontend/package.json index db92ba42f2..154497ef76 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "@windmill-labs/components", - "version": "1.740.0", + "version": "1.741.0", "scripts": { "dev": "vite dev", "dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev", diff --git a/lsp/Pipfile b/lsp/Pipfile index 5b1de97770..cda87ce8d7 100644 --- a/lsp/Pipfile +++ b/lsp/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -wmill = ">=1.740.0" +wmill = ">=1.741.0" sendgrid = "*" mysql-connector-python = "*" pymongo = "*" diff --git a/openflow.openapi.yaml b/openflow.openapi.yaml index a14b6e3d1d..9ccfa3ceab 100644 --- a/openflow.openapi.yaml +++ b/openflow.openapi.yaml @@ -1,7 +1,7 @@ openapi: '3.0.3' info: - version: 1.740.0 + version: 1.741.0 title: OpenFlow Spec contact: name: Ruben Fiszel diff --git a/powershell-client/WindmillClient/WindmillClient.psd1 b/powershell-client/WindmillClient/WindmillClient.psd1 index 8048bc9d50..ebdfaa8215 100644 --- a/powershell-client/WindmillClient/WindmillClient.psd1 +++ b/powershell-client/WindmillClient/WindmillClient.psd1 @@ -12,7 +12,7 @@ RootModule = 'WindmillClient.psm1' # Version number of this module. - ModuleVersion = '1.740.0' + ModuleVersion = '1.741.0' # Supported PSEditions # CompatiblePSEditions = @() diff --git a/python-client/wmill/pyproject.toml b/python-client/wmill/pyproject.toml index 4244d43338..d0942b025c 100644 --- a/python-client/wmill/pyproject.toml +++ b/python-client/wmill/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "wmill" -version = "1.740.0" +version = "1.741.0" description = "A client library for accessing Windmill server wrapping the Windmill client API" license = "Apache-2.0" homepage = "https://windmill.dev" diff --git a/typescript-client/jsr.json b/typescript-client/jsr.json index 2ba27adde2..13d6388e30 100644 --- a/typescript-client/jsr.json +++ b/typescript-client/jsr.json @@ -1,6 +1,6 @@ { "name": "@windmill/windmill", - "version": "1.740.0", + "version": "1.741.0", "exports": "./src/index.ts", "publish": { "exclude": ["!src", "./s3Types.ts", "./sqlUtils.ts", "./client.ts"] diff --git a/typescript-client/package.json b/typescript-client/package.json index f710adb116..42ebc2096d 100644 --- a/typescript-client/package.json +++ b/typescript-client/package.json @@ -1,7 +1,7 @@ { "name": "windmill-client", "description": "Windmill SDK client for browsers and Node.js", - "version": "1.740.0", + "version": "1.741.0", "author": "Ruben Fiszel", "license": "Apache 2.0", "homepage": "https://github.com/windmill-labs/windmill/tree/main/typescript-client#readme", diff --git a/version.txt b/version.txt index d964f54b8d..b27f19abcc 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.740.0 +1.741.0 From 003a262a4e9d6c2a63ada01aa8429aea1fbb6031 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Sat, 27 Jun 2026 21:08:00 +0200 Subject: [PATCH 112/117] feat: column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) (#9814) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: column-level lineage for ducklake pipelines via // column annotation Co-Authored-By: Claude Opus 4.8 * feat: auto-derive column lineage from DuckDB SQL AST (annotation as override) Co-Authored-By: Claude Opus 4.8 * docs: clarify column-lineage inference is server-side; drafts use annotations Co-Authored-By: Claude Opus 4.8 * feat(frontend): surface inferred column lineage in live pipeline drafts Threads the DuckDB SQL-AST column lineage (from the WASM asset parser) through ScriptEditor -> details pane -> page -> resolveGraph, merged with // column annotations (annotation wins) so the live preview matches the deployed graph. Takes effect once windmill-parser-wasm-asset is republished with the inference. Co-Authored-By: Claude Opus 4.8 * chore(frontend): bump windmill-parser-wasm-asset to 1.740.0 for SQL column-lineage inference Co-Authored-By: Claude Opus 4.8 * docs: column-lineage inference now runs live (WASM) too, merged with annotations Co-Authored-By: Claude Opus 4.8 * feat(frontend): transitive column-lineage trace (impact analysis) Stitches every producer's column_lineage into a pipeline-wide column graph (columnLineageGraph.ts) and replaces the single-hop diagram with an interactive ColumnLineageTrace: select an asset to see its columns' full upstream/downstream lineage across scripts; click any column to highlight its complete transitive impact set (forward + backward) and dim the rest. Co-Authored-By: Claude Opus 4.8 * fix: address CI review on column lineage (parse-fallback, node-id, perf, leak) - backend: DuckDB SQL parse failure now falls back to `// column` annotation lineage instead of dropping it (Codex P1) - columnLineageGraph: collision-proof JSON node ids; deterministic first-write output anchoring when a producer has multiple ducklake writes (cubic P2 ×2) - pipeline page: gate buildColumnGraph to a ducklake-asset selection so it doesn't rebuild on every editor keystroke (cubic P2) - ScriptEditor: clear inferredColumnLineage on parse error so it can't leak across a script switch (cubic P2) - AssetGraphEdge: widen badge stacking offset 12px->18px to fully clear (cubic P3) Co-Authored-By: Claude Opus 4.8 * fix: resolve JOIN inputs + anchor column lineage to // materialize target Addresses the second Codex review pass (two P1s): - SQL inference now walks JOINed tables: build_from_maps maps every FROM entry AND its joins into the alias map, and single-table attribution requires no joins. `SELECT o.x, c.y FROM a o JOIN b c` now resolves c.y (was dropped). - The column graph anchors a producer's lineage to its declared // materialize target (surfaced on the runnable node) instead of guessing a ducklake write-edge, which is unordered for deployed graphs and ambiguous for multi-output scripts. Falls back to a write-edge when no materialize target. Co-Authored-By: Claude Opus 4.8 * fix: gate column-lineage badge to the // materialize target write-edge The canvas badge keyed on `e.asset_kind === 'ducklake'`, so a multi-output producer showed the same column mapping on every ducklake write-edge. Use the same materialize-target anchor as buildColumnGraph: the badge lands only on the declared output's edge, falling back to the ducklake write-edge when there's no materialize annotation. (Codex P1) Co-Authored-By: Claude Opus 4.8 * fix: build column trace from displayGraph so View hides draft lineage The transitive column trace was built from graphWithDraft regardless of mode, so in View with drafts hidden it could surface draft `// column` lineage the deployed canvas doesn't show. Build it from `displayGraph` (the graph the canvas actually renders) so the trace matches: draft overlays in edit / show-drafts, deployed-only in plain View. (Codex P2) Co-Authored-By: Claude Opus 4.8 * fix: don't infer column lineage for local/temp staging CTAS A CTAS into a local/temp staging table isn't the materialized output, but its projection was inferred and (flat) column_lineage anchored to the script's // materialize target — so staging columns showed up as the final asset's. Gate inference to the actual output: a top-level managed-materialize SELECT, or a CTAS/CREATE VIEW whose target resolves to a real asset. (Codex P1) Co-Authored-By: Claude Opus 4.8 * fix: scope inferred column lineage to one output asset Inference accumulated columns from every output-producing query into one flat list, all anchored (frontend) to the script's // materialize target — so an auxiliary CTAS into a different asset showed its columns on the materialized one. Tag each inferred entry with its output asset and, in parse_assets, scope the list to the // materialize target (keeping untagged top-level-SELECT entries); with no declared target, drop inference when entries span multiple output assets rather than attribute them to an arbitrary one. Parser-internal — no wire change. (Codex P1) Co-Authored-By: Claude Opus 4.8 * fix: treat CREATE TEMP TABLE/VIEW as local even under an active USE A one-part temp name under `USE dl` resolved to an asset (ducklake://…/tmp) before being registered local, so a final SELECT reading it invented `final.total <- warehouse/tmp.amt` (a phantom DuckLake column) and recorded a phantom asset. track_table_definition now registers any temporary table/view as local up front, bypassing active-asset resolution; CreateTable/CreateView pass their `temporary` flag. (Codex P1) Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: Claude Opus 4.8 --- ...ecdac6806e1a306dca880943d97bb6d6a889d.json | 67 ++ backend/Cargo.lock | 1 + .../src/asset_parser.rs | 669 ++++++++++++++++-- .../windmill-parser/src/asset_parser.rs | 233 ++++++ .../tests/fixtures/pipeline_annotations.json | 91 +++ .../tests/pipeline_annotations_parity.rs | 13 + backend/windmill-api-assets/Cargo.toml | 1 + backend/windmill-api-assets/src/lib.rs | 63 +- backend/windmill-common/src/assets.rs | 4 +- docs/pipelines-vs-dbt.md | 38 +- frontend/package-lock.json | 58 +- frontend/package.json | 2 +- .../src/lib/components/ScriptEditor.svelte | 20 +- .../assets/AssetGraph/AssetGraphCanvas.svelte | 42 +- .../AssetGraph/AssetGraphDetailsPane.svelte | 45 +- .../assets/AssetGraph/AssetGraphEdge.svelte | 50 +- .../AssetGraph/ColumnLineageTrace.svelte | 173 +++++ .../AssetGraph/columnLineageGraph.test.ts | 162 +++++ .../assets/AssetGraph/columnLineageGraph.ts | 169 +++++ .../parsePipelineAnnotations.parity.test.ts | 8 +- .../parsePipelineAnnotations.test.ts | 32 +- .../AssetGraph/parsePipelineAnnotations.ts | 83 ++- .../assets/AssetGraph/resolveGraph.ts | 36 +- .../lib/components/assets/AssetGraph/types.ts | 10 +- frontend/src/lib/infer.ts | 5 + .../(logged)/pipeline/[folder]/+page.svelte | 47 +- 26 files changed, 1973 insertions(+), 149 deletions(-) create mode 100644 backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json create mode 100644 frontend/src/lib/components/assets/AssetGraph/ColumnLineageTrace.svelte create mode 100644 frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.test.ts create mode 100644 frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.ts diff --git a/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json b/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json new file mode 100644 index 0000000000..ffce4491e3 --- /dev/null +++ b/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json @@ -0,0 +1,67 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\",\n language AS \"language!: windmill_common::scripts::ScriptLang\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ORDER BY path, created_at DESC\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "content!", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "language!: windmill_common::scripts::ScriptLang", + "type_info": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby", + "rlang" + ] + } + } + } + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d" +} diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 8f11a0cb98..c970648000 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -14033,6 +14033,7 @@ dependencies = [ "tracing", "windmill-api-auth", "windmill-common", + "windmill-parser-sql-asset", ] [[package]] diff --git a/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs b/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs index 812bfa785d..f7c578a274 100644 --- a/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs +++ b/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs @@ -9,8 +9,9 @@ use sqlparser::{ parser::Parser, }; use windmill_parser::asset_parser::{ - asset_was_used, merge_assets, parse_asset_syntax, parse_pipeline_annotations, AssetKind, - AssetUsageAccessType, ParseAssetsOutput, ParseAssetsResult, + asset_was_used, merge_assets, merge_column_lineage, parse_asset_syntax, + parse_pipeline_annotations, AssetKind, AssetUsageAccessType, ColumnLineage, ColumnRef, + ParseAssetsOutput, ParseAssetsResult, }; use AssetUsageAccessType::*; @@ -33,7 +34,55 @@ pub fn parse_assets(input: &str) -> anyhow::Result { } } - let pipeline = parse_pipeline_annotations(input); + let mut pipeline = parse_pipeline_annotations(input); + // Scope inferred lineage to a single output asset so columns from an + // auxiliary CTAS aren't attributed to the materialized one (the flat list + // has no per-entry output on the wire). The `// materialize` target, when + // declared, IS the output: keep entries tagged with it plus untagged + // top-level-SELECT entries (which describe that target). Without a declared + // target, keep inference only when every tagged entry shares one output + // asset — otherwise it's ambiguous which asset the flat list describes, so + // drop it rather than show false dependencies. + let target = pipeline + .materialize + .as_ref() + .map(|m| (m.target_kind, m.target_path.clone())); + let inferred: Vec = match &target { + Some(t) => collector + .column_lineage + .into_iter() + .filter(|(out, _)| out.as_ref().map_or(true, |o| o == t)) + .map(|(_, cl)| cl) + .collect(), + None => { + let mut first: Option<&(AssetKind, String)> = None; + let mut ambiguous = false; + for (out, _) in &collector.column_lineage { + if let Some(o) = out { + match first { + None => first = Some(o), + Some(f) if f != o => { + ambiguous = true; + break; + } + _ => {} + } + } + } + if ambiguous { + Vec::new() + } else { + collector + .column_lineage + .into_iter() + .map(|(_, cl)| cl) + .collect() + } + } + }; + // Body-inferred column lineage, with `// column` annotations taking + // precedence per output column (explicit declaration overrides inference). + pipeline.column_lineage = merge_column_lineage(inferred, pipeline.column_lineage); Ok(ParseAssetsOutput::new( merge_assets(collector.assets), Vec::new(), @@ -54,6 +103,16 @@ struct AssetCollector { cte_name_stack: Vec>, // Locally created tables (not attached to an asset) local_table_names: HashSet, + // Inferred column-level lineage: one entry per output column of an + // output-producing query, mapping it to the upstream source columns its + // expression reads. Each is tagged with the *output asset* it belongs to — + // `Some((kind, path))` for a CTAS / CREATE VIEW into a real asset, `None` + // for a top-level managed-materialize SELECT (its output is the `// + // materialize` target, known only in `parse_assets`). `parse_assets` uses + // the tag to scope the flat list to a single output asset so columns from an + // auxiliary output don't get attributed to the materialized one. Best-effort: + // dynamic/raw SQL, INSERT…SELECT, and wildcards are left to annotations. + column_lineage: Vec<(Option<(AssetKind, String)>, ColumnLineage)>, } impl AssetCollector { @@ -65,15 +124,24 @@ impl AssetCollector { currently_used_asset: None, cte_name_stack: Vec::new(), local_table_names: HashSet::new(), + column_lineage: Vec::new(), } } - /// If the name resolves to an attached asset, record it. Otherwise, register it as a local - /// table/view so that subsequent references are not mistakenly attributed to the active asset. - fn track_table_definition(&mut self, name: &ObjectName) { - if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { - self.assets.push(asset); - } else if let Some(simple_name) = get_trivial_obj_name(name) { + /// Record a `CREATE TABLE`/`VIEW` target. A *temporary* table/view is always + /// local — even a one-part name under an active `USE dl`, which would + /// otherwise resolve to an asset (`ducklake://…/tmp`) and then leak as a + /// column source for later references. A non-temp name that resolves to an + /// attached asset is recorded as that asset; anything else is registered + /// local so subsequent references aren't attributed to the active asset. + fn track_table_definition(&mut self, name: &ObjectName, is_temporary: bool) { + if !is_temporary { + if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { + self.assets.push(asset); + return; + } + } + if let Some(simple_name) = get_trivial_obj_name(name) { self.local_table_names.insert(simple_name.to_lowercase()); } } @@ -280,6 +348,22 @@ impl AssetCollector { } } + // Infer the output-column lineage of a query that produces an asset, tagging + // each entry with its `output` asset. Called only for an output-producing + // query — a top-level managed-materialize SELECT (`output: None`, resolved + // to the `// materialize` target later) or a CTAS / CREATE VIEW into a real + // asset (`output: Some`). A CTAS into a local/temp staging table is never + // an output, so it's simply not passed here. + fn infer_query_output( + &mut self, + query: &sqlparser::ast::Query, + output: Option<(AssetKind, String)>, + ) { + if let Some(select) = query.body.as_select() { + self.infer_column_lineage(&select.projection, &select.from, output); + } + } + fn handle_table_with_joins( &mut self, table_with_joins: &sqlparser::ast::TableWithJoins, @@ -302,17 +386,57 @@ impl AssetCollector { } } - // Extract columns from SELECT items and create individual asset results for each column - // Only processes columns that reference known assets to avoid false positives - fn extract_column_assets( - &mut self, - projection: &[SelectItem], + // The alias-map entry (key → asset) for one FROM/JOIN table factor, or + // `None` if it isn't an asset-backed table. The key is its alias, else the + // bare table name; S3 table-functions and string-literal tables are only + // keyed when aliased (an unaliased one is ambiguous). Returns the asset with + // a single matched relation so the caller can attribute qualified columns. + fn table_alias_entry(&self, relation: &TableFactor) -> Option<(String, ParseAssetsResult)> { + let TableFactor::Table { name, alias, args, .. } = relation else { + return None; + }; + let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); + if has_args { + let alias = alias.as_ref()?; + let asset = self.get_s3_asset_from_table_function(relation)?; + return Some((alias.name.value.clone(), asset)); + } + let asset = self + .get_associated_asset_from_obj_name(name, Some(R)) + .or_else(|| self.get_s3_asset_from_str_literal_table(relation))?; + if get_str_lit_from_obj_name(name).is_some() { + // String-literal S3 table: only unambiguous when aliased. + let alias = alias.as_ref()?; + return Some((alias.name.value.clone(), asset)); + } + let key = match alias { + Some(a) => a.name.value.clone(), + None => name + .0 + .last() + .and_then(|id| id.as_ident()) + .map(|id| id.value.clone()) + .unwrap_or_default(), + }; + Some((key, asset)) + } + + // Resolve a query's FROM clause into (single-table asset, alias→asset map). + // `single_table` is `Some` only for an unambiguous one-table FROM with no + // joins (so bare column refs can be attributed); `table_to_asset` keys by + // alias/table name for qualified refs and includes every JOINed table. + // Shared by `extract_column_assets` (read columns) and `infer_column_lineage` + // (output→input edges) so both resolve identically. + fn build_from_maps( + &self, from_tables: &[sqlparser::ast::TableWithJoins], + ) -> ( + Option, + BTreeMap, ) { - // Check if this is a single-table SELECT (to avoid ambiguity). - // For S3 table functions (read_parquet/read_csv/read_json), detect the asset even - // though args are present, since we know the file path from the string literal arg. - let single_table = if from_tables.len() == 1 { + // Single unambiguous table only when there's exactly one FROM entry AND + // it has no joins — otherwise a bare column could belong to any side. + let single_table = if from_tables.len() == 1 && from_tables[0].joins.is_empty() { let relation = &from_tables[0].relation; if let TableFactor::Table { name, args, .. } = relation { let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); @@ -329,51 +453,30 @@ impl AssetCollector { None }; - // Build a map of table aliases/names to assets for multi-table queries. - // For S3 table functions, only aliased references are unambiguous - // (e.g. SELECT t.col1 FROM read_parquet('s3://...') AS t). + // Alias → asset for qualified column refs, across every FROM entry AND + // its JOINed tables (so `c.col` in `FROM a JOIN c` resolves). let mut table_to_asset: BTreeMap = BTreeMap::new(); for table_with_joins in from_tables { - if let TableFactor::Table { name, alias, args, .. } = &table_with_joins.relation { - let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); - if has_args { - // For table functions, only add to the alias map when an alias is present - if let Some(alias) = alias { - if let Some(asset) = - self.get_s3_asset_from_table_function(&table_with_joins.relation) - { - table_to_asset.insert(alias.name.value.clone(), asset); - } - } - } else if let Some(asset) = self - .get_associated_asset_from_obj_name(name, Some(R)) - .or_else(|| { - self.get_s3_asset_from_str_literal_table(&table_with_joins.relation) - }) - { - // For string literal S3 tables (e.g. FROM 's3:///file.parquet'), only add to - // the alias map when an alias is present (to avoid false positives). - // For regular named tables, use alias or table name as key. - let is_str_literal = get_str_lit_from_obj_name(name).is_some(); - if is_str_literal { - if let Some(alias) = alias { - table_to_asset.insert(alias.name.value.clone(), asset); - } - } else { - let table_key = if let Some(alias) = alias { - alias.name.value.clone() - } else { - name.0 - .last() - .and_then(|id| id.as_ident()) - .map(|id| id.value.clone()) - .unwrap_or_default() - }; - table_to_asset.insert(table_key, asset); - } + if let Some((k, a)) = self.table_alias_entry(&table_with_joins.relation) { + table_to_asset.insert(k, a); + } + for join in &table_with_joins.joins { + if let Some((k, a)) = self.table_alias_entry(&join.relation) { + table_to_asset.insert(k, a); } } } + (single_table, table_to_asset) + } + + // Extract columns from SELECT items and create individual asset results for each column + // Only processes columns that reference known assets to avoid false positives + fn extract_column_assets( + &mut self, + projection: &[SelectItem], + from_tables: &[sqlparser::ast::TableWithJoins], + ) { + let (single_table, table_to_asset) = self.build_from_maps(from_tables); // Process each SELECT item for item in projection { @@ -446,6 +549,136 @@ impl AssetCollector { } } } + + // Infer column-level lineage for an output-producing query's projection: + // each *named* output column → the upstream source columns its expression + // reads. Covers passthroughs (`amount`) and computed columns + // (`amount + tax AS total`) alike. Skipped: wildcards and unaliased + // expressions (no stable output name), and inputs that don't resolve to a + // known asset. A column with no resolved inputs is dropped. + // + // Best-effort and intentionally flat: results from every output query in the + // script accumulate into one list (the graph hangs them off the materialize + // write-edge), with no per-output-table association. This is exact for the + // common single-output member (a managed-materialize SELECT, or one CTAS), + // but a multi-statement script that stages through a TEMP table reports the + // *intermediate* column names (the final SELECT reads the temp table, whose + // columns don't resolve to an asset, so they drop out). A `// column` + // annotation overrides any output column where inference is wrong or coarse. + fn infer_column_lineage( + &mut self, + projection: &[SelectItem], + from_tables: &[sqlparser::ast::TableWithJoins], + output: Option<(AssetKind, String)>, + ) { + let (single_table, table_to_asset) = self.build_from_maps(from_tables); + for item in projection { + let (out_col, expr) = match item { + SelectItem::ExprWithAlias { expr, alias } => (alias.value.clone(), expr), + SelectItem::UnnamedExpr(expr @ Expr::Identifier(id)) => (id.value.clone(), expr), + SelectItem::UnnamedExpr(expr @ Expr::CompoundIdentifier(parts)) => { + match parts.last() { + Some(last) => (last.value.clone(), expr), + None => continue, + } + } + _ => continue, + }; + let mut collector = ColumnIdentCollector { refs: Vec::new(), query_depth: 0 }; + let _ = expr.visit(&mut collector); + let mut inputs: Vec = Vec::new(); + for parts in &collector.refs { + if let Some(cr) = self.resolve_column_ref(parts, &single_table, &table_to_asset) { + if !inputs.contains(&cr) { + inputs.push(cr); + } + } + } + if !inputs.is_empty() { + self.column_lineage + .push((output.clone(), ColumnLineage { column: out_col, inputs })); + } + } + } + + // Resolve identifier `parts` (e.g. `["t","amount"]` or `["amount"]`) to the + // source asset column it reads, mirroring `extract_column_assets`' + // resolution: a bare ident needs an unambiguous single-table FROM; a + // qualified ident resolves its prefix via the alias map, or (≥3 parts) as a + // db/schema-qualified object name. + fn resolve_column_ref( + &self, + parts: &[String], + single_table: &Option, + table_to_asset: &BTreeMap, + ) -> Option { + let asset_to_ref = |asset: &ParseAssetsResult, col: &str| ColumnRef { + from_kind: asset.kind, + from_path: asset.path.clone(), + from_column: col.to_string(), + }; + match parts { + [col] => single_table.as_ref().map(|a| asset_to_ref(a, col)), + [.., col] => { + let prefix = parts.first()?; + if let Some(asset) = table_to_asset.get(prefix) { + Some(asset_to_ref(asset, col)) + } else if parts.len() >= 3 { + let obj_parts: Vec = parts[..parts.len() - 1] + .iter() + .map(|p| ObjectNamePart::Identifier(sqlparser::ast::Ident::new(p.clone()))) + .collect(); + let asset = + self.get_associated_asset_from_obj_name(&ObjectName(obj_parts), Some(R))?; + Some(asset_to_ref(&asset, col)) + } else { + None + } + } + [] => None, + } + } +} + +// Collects the identifier paths an expression reads, for column-lineage +// inference: `Expr::Identifier(a)` → `["a"]`, `Expr::CompoundIdentifier(t.a)` → +// `["t","a"]`. The derived `Visit` walk recurses through operators, functions, +// casts and CASE, so every leaf identifier of the outer expression is captured. +struct ColumnIdentCollector { + refs: Vec>, + // Depth of nested (sub)queries inside the expression. Identifiers are only + // captured at depth 0: a scalar/correlated subquery's columns belong to ITS + // own FROM, not the outer projection's, so descending would misattribute + // (e.g. `(SELECT x FROM other) AS c FROM orders` must NOT bind `c` to + // `orders.x`). Subquery-derived columns are simply left to annotations. + query_depth: usize, +} + +impl Visitor for ColumnIdentCollector { + type Break = (); + + fn pre_visit_query(&mut self, _query: &sqlparser::ast::Query) -> std::ops::ControlFlow<()> { + self.query_depth += 1; + std::ops::ControlFlow::Continue(()) + } + + fn post_visit_query(&mut self, _query: &sqlparser::ast::Query) -> std::ops::ControlFlow<()> { + self.query_depth = self.query_depth.saturating_sub(1); + std::ops::ControlFlow::Continue(()) + } + + fn pre_visit_expr(&mut self, expr: &Expr) -> std::ops::ControlFlow { + if self.query_depth == 0 { + match expr { + Expr::Identifier(id) => self.refs.push(vec![id.value.clone()]), + Expr::CompoundIdentifier(parts) => self + .refs + .push(parts.iter().map(|id| id.value.clone()).collect()), + _ => {} + } + } + std::ops::ControlFlow::Continue(()) + } } impl Visitor for AssetCollector { @@ -505,7 +738,11 @@ impl Visitor for AssetCollector { ) -> std::ops::ControlFlow { match statement { sqlparser::ast::Statement::Query(q) => { + // A top-level SELECT is the managed-materialize output, so its + // columns ARE the materialized asset's columns (output resolved + // to the `// materialize` target in `parse_assets`). self.handle_query_reads(q); + self.infer_query_output(q, None); } sqlparser::ast::Statement::Insert(insert) => { @@ -658,18 +895,32 @@ impl Visitor for AssetCollector { } sqlparser::ast::Statement::CreateTable(create_table) => { - self.track_table_definition(&create_table.name); + self.track_table_definition(&create_table.name, create_table.temporary); // `CREATE TABLE x AS SELECT … FROM y` reads y. The AS-query // isn't a `Statement::Query`, so its FROM tables are only - // caught here. + // caught here. Only infer output lineage when `x` is a real + // asset — a CTAS into a local/temp staging table is not the + // materialized output (its columns aren't the asset's). if let Some(query) = &create_table.query { self.handle_query_reads(query); + // Infer only when `x` is a real asset (its output columns + // ARE that asset's), tagged with it so `parse_assets` can + // scope lineage per output. A local/temp staging table is + // not an asset → not inferred. + if let Some(asset) = + self.get_associated_asset_from_obj_name(&create_table.name, Some(W)) + { + self.infer_query_output(query, Some((asset.kind, asset.path))); + } } } - sqlparser::ast::Statement::CreateView { name, query, .. } => { - self.track_table_definition(name); + sqlparser::ast::Statement::CreateView { name, query, temporary, .. } => { + self.track_table_definition(name, *temporary); self.handle_query_reads(query); + if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { + self.infer_query_output(query, Some((asset.kind, asset.path))); + } } // DROP TABLE/VIEW is a write to the dropped object — the @@ -687,7 +938,9 @@ impl Visitor for AssetCollector { | sqlparser::ast::ObjectType::MaterializedView ) { for name in names { - self.track_table_definition(name); + // DROP is a write to the named object; resolve it as an + // asset if it is one (not a temp-creation context). + self.track_table_definition(name, false); } } } @@ -1920,8 +2173,9 @@ mod ctas_read_tests { assets ); assert!( - assets.iter().any(|a| a.path == "main/exciting_809" - && a.access_type == Some(W)), + assets + .iter() + .any(|a| a.path == "main/exciting_809" && a.access_type == Some(W)), "expected write of main/exciting_809, got {:?}", assets ); @@ -1935,9 +2189,292 @@ mod ctas_read_tests { "#; let assets = parse_assets(input).unwrap().assets; assert!( - assets.iter().any(|a| a.path == "main/fx_rates" && a.access_type == Some(R)), + assets + .iter() + .any(|a| a.path == "main/fx_rates" && a.access_type == Some(R)), "expected read of main/fx_rates, got {:?}", assets ); } + + fn lineage(input: &str) -> Vec { + parse_assets(input).unwrap().column_lineage + } + + #[test] + fn test_infer_lineage_computed_and_passthrough() { + // CTAS with a computed column (amount + tax) and a passthrough (id). + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, dl.orders.amount + dl.orders.tax AS order_total + FROM dl.orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ + ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }, + ColumnLineage { + column: "order_total".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "tax".to_string(), + }, + ], + }, + ] + ); + } + + #[test] + fn test_infer_lineage_bare_column_single_table() { + // Managed-materialize form: a plain top-level SELECT, bare columns + // attributed to the single FROM table. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + USE dl; + SELECT amount AS revenue FROM orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ColumnLineage { + column: "revenue".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn test_infer_lineage_annotation_overrides() { + // The `// column` annotation for `order_total` wins; `id` stays inferred. + let input = r#" + -- column order_total <- datatable://prod/manual.grand_total + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, dl.orders.amount + dl.orders.tax AS order_total + FROM dl.orders; + "#; + let got = lineage(input); + // Annotation entry is authoritative and listed first. + assert_eq!(got[0].column, "order_total"); + assert_eq!(got[0].inputs[0].from_path, "prod/manual"); + assert_eq!(got[0].inputs[0].from_column, "grand_total"); + // Inferred `id` survives; inferred `order_total` dropped (no dup). + assert!(got.iter().any(|c| c.column == "id")); + assert_eq!(got.iter().filter(|c| c.column == "order_total").count(), 1); + } + + #[test] + fn test_infer_lineage_skips_local_staging_ctas() { + // A CTAS into a TEMP/local table is NOT the materialized output, so its + // columns must not be reported (they'd be anchored to the script's + // `// materialize` target as if they were the final asset's columns). + // The final SELECT reads the local staging table → unresolved → empty. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TEMP TABLE tmp AS SELECT dl.orders.amount AS amt FROM dl.orders; + SELECT amt AS total FROM tmp; + "#; + assert!( + lineage(input).is_empty(), + "staging columns must not be reported as final output; got {:?}", + lineage(input) + ); + } + + #[test] + fn test_infer_lineage_ctas_into_asset_still_inferred() { + // A CTAS whose target IS an asset is the output, so it's still inferred. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS SELECT dl.orders.amount AS amt FROM dl.orders; + "#; + assert_eq!( + lineage(input), + vec![ColumnLineage { + column: "amt".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn test_infer_lineage_temp_table_under_use_is_local() { + // A one-part TEMP table name under an active `USE dl` must NOT resolve to + // an asset (`warehouse/tmp`); it's local, so the final SELECT reading it + // can't invent `warehouse/tmp.amt` as a column source for the output. + let input = r#" + -- materialize ducklake://warehouse/final + ATTACH 'ducklake://warehouse' AS dl; + USE dl; + CREATE TEMP TABLE tmp AS SELECT amount AS amt FROM orders; + SELECT amt AS total FROM tmp; + "#; + let got = lineage(input); + assert!( + got.is_empty(), + "temp staging under USE must not leak warehouse/tmp as a source; got {:?}", + got + ); + // And no phantom `warehouse/tmp` asset is recorded. + let assets = parse_assets(input).unwrap().assets; + assert!( + !assets.iter().any(|a| a.path == "warehouse/tmp"), + "temp table must not be recorded as an asset; got {:?}", + assets + ); + } + + #[test] + fn test_infer_lineage_scopes_to_materialize_target() { + // A script with a `// materialize` target plus an AUXILIARY CTAS into a + // different asset: only the materialized target's columns are reported; + // the auxiliary output's columns must not be attributed to it. + let input = r#" + -- materialize ducklake://warehouse/final + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.audit AS SELECT dl.orders.id AS aid FROM dl.orders; + SELECT dl.orders.amount AS total FROM dl.orders; + "#; + assert_eq!( + lineage(input), + vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }], + "auxiliary `audit` columns must not appear on the materialized target" + ); + } + + #[test] + fn test_infer_lineage_drops_ambiguous_multi_output() { + // No `// materialize` target and two real CTAS outputs: which asset the + // flat lineage describes is ambiguous, so inference is dropped rather + // than attributed to an arbitrary one. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.a AS SELECT dl.orders.id AS x FROM dl.orders; + CREATE TABLE dl.b AS SELECT dl.orders.amount AS y FROM dl.orders; + "#; + assert!( + lineage(input).is_empty(), + "ambiguous multi-output must drop inference" + ); + } + + #[test] + fn test_infer_lineage_wildcard_yields_nothing() { + // `SELECT *` has no enumerable output columns → no inferred lineage. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS SELECT * FROM dl.orders; + "#; + assert!(lineage(input).is_empty()); + } + + #[test] + fn test_infer_lineage_resolves_joined_inputs() { + // Columns from BOTH sides of an explicit JOIN must resolve, incl. a + // computed column mixing the two. A bare column is dropped (ambiguous + // across the join) rather than misattributed to the first table. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT o.id, c.region AS cust_region, o.amount + c.discount AS net + FROM dl.orders o + JOIN dl.customers c ON c.id = o.customer_id; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ + ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }, + ColumnLineage { + column: "cust_region".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/customers".to_string(), + from_column: "region".to_string(), + }], + }, + ColumnLineage { + column: "net".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/customers".to_string(), + from_column: "discount".to_string(), + }, + ], + }, + ] + ); + } + + #[test] + fn test_infer_lineage_does_not_descend_into_subqueries() { + // A scalar subquery's bare column (`amount`) belongs to the subquery's + // own FROM, NOT the outer `dl.orders` — it must not be attributed to the + // outer table. The subquery column is left to annotations; the + // passthrough `id` still resolves. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, (SELECT amount FROM dl.other LIMIT 1) AS c + FROM dl.orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }], + "subquery column `c` must be dropped, not misattributed to orders" + ); + } } diff --git a/backend/parsers/windmill-parser/src/asset_parser.rs b/backend/parsers/windmill-parser/src/asset_parser.rs index 2103e9937b..2b7f6ecdbf 100644 --- a/backend/parsers/windmill-parser/src/asset_parser.rs +++ b/backend/parsers/windmill-parser/src/asset_parser.rs @@ -117,6 +117,11 @@ pub struct ParseAssetsOutput { // lines allowed). Drives the worker's post-materialize verifier probes. #[serde(skip_serializing_if = "Vec::is_empty", default)] pub data_tests: Vec, + // `// column <- .[, …]` — declared column-level lineage, + // one entry per output column. Accumulating. Pure metadata: drives the + // column-lineage graph view, executes nothing. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub column_lineage: Vec, } #[derive(Serialize, Debug, PartialEq, Clone)] @@ -275,6 +280,39 @@ pub enum DataTest { Custom { path: String }, } +// `// column <- .[, …]` — declared column-level +// lineage: one output column of this script's produced asset and the upstream +// source columns it derives from. A sibling of `DataTest` in the extensible +// annotation family (`docs/pipelines-vs-dbt.md` §3): same parse shape — a head +// token (the output column) then a per-variant tail — but accumulating, one +// line per output column. Unlike `data_test` these are pure metadata: they +// drive the column-lineage graph view, never a runtime probe. +// +// dbt derives column lineage from SQL-AST parsing; Windmill is polyglot +// (Python/TS/Bash/SQL in one DAG), so a uniform AST is not available. The +// annotation is the explicit, language-agnostic declaration — the same +// "annotations are real comments parsed strictly" stance as the rest of the +// pipeline grammar. Body-inferred per-asset column *sets* (`columns` on +// `ParseAssetsResult`) complement it but cannot express column→column edges. +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct ColumnLineage { + // The produced asset's output column this line describes. + pub column: String, + // Upstream source columns it derives from (≥1; malformed refs dropped). + pub inputs: Vec, +} + +// One `.` upstream reference inside a `// column` line. The +// asset URI accepts the default-syntax shorthands (like `// materialize` / +// `// data_test relationships`); the column is the segment after the final +// `.` (so a schema-qualified `warehouse/main.orders.amount` keeps `amount`). +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct ColumnRef { + pub from_kind: AssetKind, + pub from_path: String, + pub from_column: String, +} + // `// trigger any` (default) vs `// trigger all`. `Any` = OR: any trigger // firing runs the script (current behaviour). `All` = AND: the script // runs only once every partition-bearing input has materialized at the @@ -307,6 +345,7 @@ pub struct PipelineAnnotations { pub retry: Option, pub materialize: Option, pub data_tests: Vec, + pub column_lineage: Vec, } impl ParseAssetsOutput { @@ -332,10 +371,33 @@ impl ParseAssetsOutput { retry: pipeline.retry, materialize: pipeline.materialize, data_tests: pipeline.data_tests, + column_lineage: pipeline.column_lineage, } } } +// Combine column lineage inferred from the body (SQL AST) with lineage declared +// via `// column` annotations. The annotation is the *override*: where both +// describe the same output column, the explicit declaration wins and the +// inferred entry is dropped. Inferred entries are also deduped by output column +// among themselves (first wins). Used by the language asset-parsers so a +// `// column` line can correct a mis-inferred edge without disabling inference +// for the rest of the columns. +pub fn merge_column_lineage( + inferred: Vec, + annotated: Vec, +) -> Vec { + let mut seen: std::collections::HashSet = + annotated.iter().map(|c| c.column.clone()).collect(); + let mut out = annotated; + for c in inferred { + if seen.insert(c.column.clone()) { + out.push(c); + } + } + out +} + #[derive(Debug, Clone, Serialize)] pub struct DelegateToGitRepoDetails { pub resource: String, @@ -680,6 +742,17 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { continue; } + // `// column <- .[, …]` — accumulating column lineage. + // A complete word, so it never swallows a body comment that happens to + // start with `column` followed by non-lineage prose (that has no `<-` + // and is dropped fail-safe). Checked before `on`/asset shorthands. + if let Some(after_kw) = consume_keyword(rest, "column") { + if let Some(spec) = parse_column_lineage_spec(after_kw.trim()) { + out.column_lineage.push(spec); + } + continue; + } + if let Some(after_kw) = consume_keyword(rest, "on") { let spec_text = after_kw.trim(); if spec_text.is_empty() { @@ -841,6 +914,38 @@ fn parse_relationships(s: &str) -> Option { Some(DataTest::Relationships { column, to_kind, to_path: to_path.to_string(), to_column }) } +// Parse a `// column <- [, …]` right-hand side. The head +// (before `<-`) is the output column; the tail is a comma-separated list of +// `.` upstream references. Mirrors `parse_accepted_values`' +// "drop empties, require ≥1" stance: individually malformed refs are dropped +// and the line is kept iff at least one ref parses; a missing `<-`, a non-ident +// output column, or zero valid refs drops the whole line (fail-safe). +fn parse_column_lineage_spec(s: &str) -> Option { + let (out_col, refs) = s.split_once("<-")?; + let column = single_ident(out_col)?; + let inputs: Vec = refs + .split(',') + .filter_map(|r| parse_column_ref(r.trim())) + .collect(); + if inputs.is_empty() { + return None; + } + Some(ColumnLineage { column, inputs }) +} + +// `.` — the referenced column is the segment after the final +// `.`; everything before it is the asset URI (default-syntax shorthands +// enabled, like `// materialize`). Same shape as `parse_relationships`' target. +fn parse_column_ref(s: &str) -> Option { + let (asset_uri, ref_col) = s.rsplit_once('.')?; + let from_column = single_ident(ref_col)?; + let (from_kind, from_path) = parse_asset_syntax(asset_uri.trim(), true)?; + if from_path.is_empty() { + return None; + } + Some(ColumnRef { from_kind, from_path: from_path.to_string(), from_column }) +} + // Parse a `// partitioned [opts]` right-hand side. Recognized kinds: // `daily`, `hourly`, `weekly`, `monthly` (with optional tz/format/start), // and `dynamic key=""` (plus optional format). @@ -1545,4 +1650,132 @@ mod pipeline_annotation_tests { vec![DataTest::Unique { column: "id".to_string() }] ); } + + #[test] + fn column_lineage_basic() { + let code = concat!( + "// column order_total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.tax\n", + "// column user_name <- datatable://prod/users.name\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!( + out.column_lineage, + vec![ + ColumnLineage { + column: "order_total".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "tax".to_string(), + }, + ], + }, + ColumnLineage { + column: "user_name".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/users".to_string(), + from_column: "name".to_string(), + }], + }, + ] + ); + } + + #[test] + fn column_lineage_schema_qualified_keeps_last_dot_as_column() { + // The column is the segment after the FINAL dot, so a schema-qualified + // ducklake table (`main.dim_products`) survives intact. + let out = parse_pipeline_annotations( + "// column sku <- ducklake://warehouse/main.dim_products.sku", + ); + assert_eq!( + out.column_lineage, + vec![ColumnLineage { + column: "sku".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/main.dim_products".to_string(), + from_column: "sku".to_string(), + }], + }] + ); + } + + #[test] + fn column_lineage_drops_malformed_refs_keeps_valid() { + // `bad_no_dot` has no `.col` and is dropped; the line survives on its + // one valid ref. Mirrors accepted_values' drop-empties-keep-≥1 stance. + let out = parse_pipeline_annotations( + "// column total <- bad_no_dot, datatable://prod/orders.amount", + ); + assert_eq!( + out.column_lineage, + vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn merge_column_lineage_annotation_overrides_inferred() { + let inferred = vec![ + ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "w/o".to_string(), + from_column: "amount".to_string(), + }], + }, + ColumnLineage { + column: "qty".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "w/o".to_string(), + from_column: "qty".to_string(), + }], + }, + ]; + // Annotation redefines `total` (wins) and leaves `qty` to inference. + let annotated = vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/x".to_string(), + from_column: "grand_total".to_string(), + }], + }]; + let merged = merge_column_lineage(inferred, annotated); + assert_eq!(merged.len(), 2); + // Annotation entry kept first and authoritative. + assert_eq!(merged[0].column, "total"); + assert_eq!(merged[0].inputs[0].from_column, "grand_total"); + // Inferred `qty` survives (no annotation for it); inferred `total` dropped. + assert_eq!(merged[1].column, "qty"); + } + + #[test] + fn column_lineage_malformed_lines_dropped_fail_safe() { + // No arrow, a multi-token output column, and a line whose every ref is + // malformed are all dropped entirely. + let out = parse_pipeline_annotations(concat!( + "// column no_arrow datatable://prod/x.y\n", // missing `<-` + "// column a b <- datatable://prod/x.y\n", // output not a single ident + "// column total <- bad_no_dot\n", // no valid ref + "// column\n", // bare keyword + )); + assert!(out.column_lineage.is_empty()); + } } diff --git a/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json b/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json index 304a8c3842..4ecbc992bc 100644 --- a/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json +++ b/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json @@ -415,5 +415,96 @@ "retry": null, "data_tests": [{ "type": "unique", "column": "id" }] } + }, + { + "name": "column lineage maps output columns to upstream sources", + "code": "// column order_total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.tax\n// column user_name <- datatable://prod/users.name\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "order_total", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + }, + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "tax" + } + ] + }, + { + "column": "user_name", + "inputs": [ + { "from_kind": "datatable", "from_path": "prod/users", "from_column": "name" } + ] + } + ] + } + }, + { + "name": "column lineage keeps duplicate input refs (dedup is a view concern)", + "code": "// column total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.amount\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "total", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + }, + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + } + ] + } + ] + } + }, + { + "name": "column lineage keeps schema-qualified table, drops malformed refs", + "code": "// column sku <- ducklake://warehouse/main.dim_products.sku, bad_no_dot\n// column no_arrow datatable://prod/x.y\n// column total <- bad_no_dot\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "sku", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/main.dim_products", + "from_column": "sku" + } + ] + } + ] + } } ] diff --git a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs index 094419300b..883ddebcbc 100644 --- a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs +++ b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs @@ -42,6 +42,11 @@ struct Expected { // compared against `serde_json::to_value(got.data_tests)`. Absent === []. #[serde(default)] data_tests: Vec, + // Snake_case `ColumnLineage` serde shape (e.g. {"column":"x","inputs": + // [{"from_kind":"datatable","from_path":"p","from_column":"c"}]}), compared + // against `to_value(got.column_lineage)`. Absent === []. + #[serde(default)] + column_lineage: Vec, } #[derive(Deserialize)] @@ -200,5 +205,13 @@ fn pipeline_annotation_fixtures_match() { serde_json::Value::Array(f.expected.data_tests.clone()), "{ctx}: data tests" ); + + let got_lineage = + serde_json::to_value(&got.column_lineage).expect("column_lineage serialize"); + assert_eq!( + got_lineage, + serde_json::Value::Array(f.expected.column_lineage.clone()), + "{ctx}: column lineage" + ); } } diff --git a/backend/windmill-api-assets/Cargo.toml b/backend/windmill-api-assets/Cargo.toml index c6ba973f80..c6e7676b6b 100644 --- a/backend/windmill-api-assets/Cargo.toml +++ b/backend/windmill-api-assets/Cargo.toml @@ -11,6 +11,7 @@ path = "src/lib.rs" [dependencies] windmill-api-auth.workspace = true windmill-common = { workspace = true, default-features = false } +windmill-parser-sql-asset.workspace = true axum.workspace = true chrono.workspace = true serde.workspace = true diff --git a/backend/windmill-api-assets/src/lib.rs b/backend/windmill-api-assets/src/lib.rs index d25bb4db5f..29122bd793 100644 --- a/backend/windmill-api-assets/src/lib.rs +++ b/backend/windmill-api-assets/src/lib.rs @@ -519,6 +519,17 @@ struct GraphRunnableNode { retry: Option, #[serde(skip_serializing_if = "Vec::is_empty", default)] data_tests: Vec, + // `// column <- .` declared column-level lineage, surfaced + // so the canvas can draw the column-lineage view on deployed nodes (not + // only live drafts). Lockstep with TS `AssetGraphRunnableNode.column_lineage`. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + column_lineage: Vec, + // `// materialize ` target — the asset this script's `column_lineage` + // describes. Lets the column-graph anchor lineage to the exact output asset + // instead of guessing a ducklake write-edge (a multi-output script writes + // several). Absent for scripts with no `// materialize` annotation. + #[serde(skip_serializing_if = "Option::is_none", default)] + materialize_target: Option, // Managed `// materialize` write strategy (`replace` | `append` | `merge`), // absent for non-materializing or `manual` scripts. Surfaced so the asset // panel can tell whether the captured schema can evolve: only whole-table @@ -528,6 +539,14 @@ struct GraphRunnableNode { materialize_strategy: Option, } +// The output asset a producer's column lineage belongs to (the `// materialize` +// target). Kept minimal — the column graph only needs (kind, path) to anchor. +#[derive(Serialize, Debug)] +struct MaterializeTargetNode { + kind: windmill_common::assets::AssetKind, + path: String, +} + // The partition's kind word for the node badge (the full PartitionSpec carries // tz/format/start, which the badge doesn't need). fn partition_kind_word(kind: &windmill_common::assets::PartitionKind) -> &'static str { @@ -738,7 +757,8 @@ async fn asset_graph( // defensive against transient overlaps). let pipeline_member_paths = sqlx::query!( r#" - SELECT DISTINCT ON (path) path AS "path!", content AS "content!" + SELECT DISTINCT ON (path) path AS "path!", content AS "content!", + language AS "language!: windmill_common::scripts::ScriptLang" FROM script WHERE workspace_id = $1 AND auto_kind = 'pipeline' @@ -790,6 +810,34 @@ async fn asset_graph( ) }) .collect(); + // Column-level lineage per member. The annotation-only lineage (already + // parsed above) is the baseline. For DuckDB scripts we additionally run the + // full SQL asset parser to infer output→input column edges from the AST; it + // merges them with the `// column` annotations (annotation wins). If the SQL + // can't be parsed (DuckDB accepts grammar `sqlparser` rejects), we fall back + // to the annotation-only baseline rather than dropping explicit annotations. + let column_lineage_by_path: std::collections::HashMap< + String, + Vec, + > = pipeline_member_paths + .iter() + .map(|r| { + let annotated = || { + annotations_by_path + .get(&r.path) + .map(|a| a.column_lineage.clone()) + .unwrap_or_default() + }; + let lineage = if r.language == windmill_common::scripts::ScriptLang::DuckDb { + windmill_parser_sql_asset::parse_assets(&r.content) + .map(|o| o.column_lineage) + .unwrap_or_else(|_| annotated()) + } else { + annotated() + }; + (r.path.clone(), lineage) + }) + .collect(); let pipeline_member_script_paths: std::collections::HashSet = pipeline_member_paths.into_iter().map(|r| r.path).collect(); let existing_script_paths: std::collections::HashSet = @@ -930,6 +978,19 @@ async fn asset_graph( tag: ann.and_then(|a| a.tag.clone()), retry: ann.and_then(|a| a.retry.clone()), data_tests: ann.map(|a| a.data_tests.clone()).unwrap_or_default(), + // Inferred (DuckDB AST) + annotation column lineage, gated to + // scripts like the badges above. + column_lineage: (usage_kind == AssetUsageKind::Script) + .then(|| column_lineage_by_path.get(&path)) + .flatten() + .cloned() + .unwrap_or_default(), + materialize_target: ann.and_then(|a| a.materialize.as_ref()).map(|m| { + MaterializeTargetNode { + kind: windmill_common::assets::asset_kind_from_parser(m.target_kind), + path: m.target_path.clone(), + } + }), materialize_strategy: ann.and_then(|a| a.materialize.as_ref()).and_then(|m| { if m.manual { None diff --git a/backend/windmill-common/src/assets.rs b/backend/windmill-common/src/assets.rs index e24b6806a2..c186f59141 100644 --- a/backend/windmill-common/src/assets.rs +++ b/backend/windmill-common/src/assets.rs @@ -5,8 +5,8 @@ use sqlx::{PgExecutor, Postgres, Transaction}; use crate::{error, scripts::ScriptHash}; pub use windmill_parser::asset_parser::{ - parse_pipeline_annotations, DataTest, PartitionKind, PipelineAnnotations, RetrySpec, - TriggerSpec, PARTITION_TOKEN, + merge_column_lineage, parse_pipeline_annotations, ColumnLineage, ColumnRef, DataTest, + PartitionKind, PipelineAnnotations, RetrySpec, TriggerSpec, PARTITION_TOKEN, }; pub use windmill_types::assets::*; diff --git a/docs/pipelines-vs-dbt.md b/docs/pipelines-vs-dbt.md index 5dd93b5a37..4344ef2296 100644 --- a/docs/pipelines-vs-dbt.md +++ b/docs/pipelines-vs-dbt.md @@ -48,7 +48,7 @@ Asset-centric, polyglot, annotation-driven, event-aware: |---|---|---| | Data tests | No | **Shipped** (`// data_test`) | | Incremental materializations | No, but pick a philosophy | TODO with design decision | -| Column lineage + docs site | No | Pure TODO | +| Column lineage | No | **Shipped** (`// column`); docs site still TODO | | Snapshots / SCD2 | No | New output kind | | Selective execution grammar | No | UI/CLI surface | | Schema contracts | No, but design metadata model | TODO with design work | @@ -84,10 +84,38 @@ See [Incremental deep-dive](#incremental-deep-dive) below. dbt: SQL-AST parsing for column-level deps; `dbt docs serve` produces a static lineage site with descriptions. -Today: graph is asset-level. `SqlQueryDetails` in the parser -(`backend/parsers/windmill-parser/src/asset_parser.rs:44`) already has a -column map — the scaffolding exists. No `// column` annotation, no docs -surface. Pure TODO; no abstraction stands in the way. +**Shipped**, inferred-first. For DuckDB scripts the column lineage is **derived +automatically from the SQL AST** — `windmill-parser-sql-asset` walks each +output-producing query's projection and maps every output column to the source +columns its expression reads (passthroughs *and* computed columns like +`amount + tax AS total`), resolving each input to its asset via the same +`ATTACH`/alias machinery the asset parser already uses. This is the dbt-style +AST lineage, and it needs no annotation. + +The `// column <- .[, …]` annotation is the +**override / escape hatch**, for the cases inference can't reach: polyglot +transforms (Python/TS/Bash — no SQL AST), dynamic SQL (`${sql.raw(...)}`, flagged +by `SqlQueryDetails.has_raw_interpolation`), or correcting a mis-inferred edge. +Inferred and annotated lineage are merged per output column with the annotation +winning (`merge_column_lineage`). The annotation is the second *extensible* +annotation family after `// data_test` — same head-then-tail parse shape (see +`ColumnLineage`/`ColumnRef` in `asset_parser.rs`) — and is pure metadata: it +drives the graph surface, never a runtime probe. + +Surfaced two ways in the asset graph: a count badge on the +producer→materialized-asset write-edge, and a **transitive column-lineage +trace** (`ColumnLineageTrace.svelte`, over the cross-script graph built by +`columnLineageGraph.ts`) in the asset details pane — select an asset to see its +columns and their full upstream/downstream lineage, click any column to +highlight its complete impact set across the pipeline (forward + backward). + +SQL-AST inference runs both **server-side** (the graph endpoint + deploy via +`parse_assets`, for *deployed* members) and **in the live editor** — the same +parser compiled to WASM (`windmill-parser-wasm-asset`) runs on the open draft's +buffer, and `resolveGraph` merges its `column_lineage` with the buffer's +`// column` annotations under the same annotation-wins precedence, so the draft +preview matches what deploys. The `dbt docs serve`-style static lineage *site* +is still TODO. ### 4. Snapshots / SCD2 diff --git a/frontend/package-lock.json b/frontend/package-lock.json index ed9dd1d7ea..bd2619a54e 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -79,7 +79,7 @@ "vscode-languageclient": "~9.0.1", "vscode-uri": "~3.1.0", "vscode-ws-jsonrpc": "~3.5.0", - "windmill-parser-wasm-asset": "1.728.1", + "windmill-parser-wasm-asset": "1.740.0", "windmill-parser-wasm-csharp": "1.510.1", "windmill-parser-wasm-go": "1.510.1", "windmill-parser-wasm-java": "1.510.1", @@ -878,7 +878,6 @@ "version": "1.10.0", "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.10.0.tgz", "integrity": "sha512-yq6OkJ4p82CAfPl0u9mQebQHKPJkY7WrIuk205cTYnYe+k2Z8YBh11FrbRG/H6ihirqcacOgl2BIO8oyMQLeXw==", - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -890,7 +889,6 @@ "version": "1.10.0", "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz", "integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==", - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -901,7 +899,6 @@ "version": "1.2.1", "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.1.tgz", "integrity": "sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w==", - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1417,7 +1414,6 @@ "version": "1.1.4", "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.4.tgz", "integrity": "sha512-3NQNNgA1YSlJb/kMH1ildASP9HW7/7kYnRI2szWJaofaS1hWmbGI4H+d3+22aGzXXN9IJ+n+GiFVcGipJP18ow==", - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1566,7 +1562,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1583,7 +1578,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1600,7 +1594,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1617,7 +1610,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1634,7 +1626,6 @@ "cpu": [ "arm" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1651,7 +1642,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1668,7 +1658,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1685,7 +1674,6 @@ "cpu": [ "ppc64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1702,7 +1690,6 @@ "cpu": [ "s390x" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1719,7 +1706,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1736,7 +1722,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1753,7 +1738,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1770,7 +1754,6 @@ "cpu": [ "wasm32" ], - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1789,7 +1772,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1806,7 +1788,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MIT", "optional": true, "os": [ @@ -2112,7 +2093,6 @@ "version": "0.10.2", "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.2.tgz", "integrity": "sha512-RoBvJ2X0wuKlWFIjrwffGw1IqZHKQqzIchKaadZZfnNpsAYp2mM0h36JtPCjNDAHGgYez/15uMBpfGwchhiMgg==", - "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -7348,7 +7328,7 @@ "version": "1.21.7", "resolved": "https://registry.npmjs.org/jiti/-/jiti-1.21.7.tgz", "integrity": "sha512-/imKNG4EbWNrVjoNC/1H5/9GFy+tqjGBHCaSsN+P2RnPqjsLmv6UD3Ej+Kj8nBWaRAwyk7kK5ZUc+OEatnTR3A==", - "dev": true, + "devOptional": true, "license": "MIT", "bin": { "jiti": "bin/jiti.js" @@ -7883,7 +7863,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7904,7 +7883,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7925,7 +7903,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7946,7 +7923,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7967,7 +7943,6 @@ "cpu": [ "arm" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7988,7 +7963,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8009,7 +7983,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8030,7 +8003,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8051,7 +8023,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8072,7 +8043,6 @@ "cpu": [ "arm64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8093,7 +8063,6 @@ "cpu": [ "x64" ], - "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -12781,21 +12750,6 @@ } } }, - "node_modules/svelte-check/node_modules/picomatch": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", - "integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", - "dev": true, - "license": "MIT", - "optional": true, - "peer": true, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/jonschlinkert" - } - }, "node_modules/svelte-eslint-parser": { "version": "0.43.0", "resolved": "https://registry.npmjs.org/svelte-eslint-parser/-/svelte-eslint-parser-0.43.0.tgz", @@ -13535,7 +13489,7 @@ "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", - "dev": true, + "devOptional": true, "license": "Apache-2.0", "bin": { "tsc": "bin/tsc", @@ -14321,9 +14275,9 @@ } }, "node_modules/windmill-parser-wasm-asset": { - "version": "1.728.1", - "resolved": "https://registry.npmjs.org/windmill-parser-wasm-asset/-/windmill-parser-wasm-asset-1.728.1.tgz", - "integrity": "sha512-73cyU6XM3gYEjFBx3qOKnv+VV1t70eAr6OiT+x0QobjFVNmqZFEdA7ayMYYVCnnix8OZxcsTNEF1hT61w1XiKw==" + "version": "1.740.0", + "resolved": "https://registry.npmjs.org/windmill-parser-wasm-asset/-/windmill-parser-wasm-asset-1.740.0.tgz", + "integrity": "sha512-Dgn5sQ93vJpqTQkv9iAy25+bqH2bUnIMdCfERb2Tia0Ql9T6iHbQhi4yzH9FbGW4Drv9GP46Qyetphqvp8vFOw==" }, "node_modules/windmill-parser-wasm-csharp": { "version": "1.510.1", diff --git a/frontend/package.json b/frontend/package.json index 154497ef76..8bdf440a00 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -154,7 +154,7 @@ "vscode-languageclient": "~9.0.1", "vscode-uri": "~3.1.0", "vscode-ws-jsonrpc": "~3.5.0", - "windmill-parser-wasm-asset": "1.728.1", + "windmill-parser-wasm-asset": "1.740.0", "windmill-parser-wasm-csharp": "1.510.1", "windmill-parser-wasm-go": "1.510.1", "windmill-parser-wasm-java": "1.510.1", diff --git a/frontend/src/lib/components/ScriptEditor.svelte b/frontend/src/lib/components/ScriptEditor.svelte index 54be27f787..f1944c73cc 100644 --- a/frontend/src/lib/components/ScriptEditor.svelte +++ b/frontend/src/lib/components/ScriptEditor.svelte @@ -104,6 +104,7 @@ import AssetsDropdownButton from './assets/AssetsDropdownButton.svelte' import { canHavePreprocessor } from '$lib/script_helpers' import { assetEq, type AssetWithAltAccessType } from './assets/lib' + import type { ColumnLineage } from './assets/AssetGraph/parsePipelineAnnotations' import { editor as meditor } from 'monaco-editor' import type { ReviewChangesOpts } from './copilot/chat/monaco-adapter' import GitRepoViewer from './GitRepoViewer.svelte' @@ -171,6 +172,11 @@ lastDeployedCode?: string | undefined disableAi?: boolean assets?: AssetWithAltAccessType[] + // Body-inferred column lineage (DuckDB SQL AST), surfaced alongside + // `assets` so the pipeline editor can render inferred column lineage on + // the live graph. Empty/undefined for non-DuckDB or when the parser + // build predates the inference. + inferredColumnLineage?: ColumnLineage[] modules?: { [key: string]: ScriptModule } | null editorBarRight?: import('svelte').Snippet enablePreprocessorSnippet?: boolean @@ -229,6 +235,7 @@ lastDeployedCode = undefined, disableAi = false, assets = $bindable(), + inferredColumnLineage = $bindable(), modules = $bindable(undefined), editorBarRight, enablePreprocessorSnippet = false, @@ -585,7 +592,13 @@ watch( () => inferAssetsRes.current, () => { - if (!inferAssetsRes.current || inferAssetsRes.current?.status === 'error') return + if (!inferAssetsRes.current || inferAssetsRes.current?.status === 'error') { + // Clear stale lineage on parse error / unset, so a script switch + // whose new body fails to parse can't leave the previous script's + // inferred column lineage bound to the new path. + if (inferredColumnLineage !== undefined) inferredColumnLineage = undefined + return + } let newAssets = inferAssetsRes.current.assets as AssetWithAltAccessType[] for (const asset of newAssets) { const old = assets?.find((a) => assetEq(a, asset)) @@ -593,6 +606,11 @@ } const normalizedAssets = newAssets.length > 0 ? newAssets : undefined if (!deepEqual(assets, normalizedAssets)) assets = normalizedAssets + + const newLineage = inferAssetsRes.current.column_lineage + const normalizedLineage = newLineage && newLineage.length > 0 ? newLineage : undefined + if (!deepEqual(inferredColumnLineage, normalizedLineage)) + inferredColumnLineage = normalizedLineage } ) diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte index 81d5c58cd5..c13e9abc10 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte @@ -219,6 +219,9 @@ // Producer's `// data_test` checks, on the write-edge to the // materialized asset — rendered as a flask badge on the link. data_tests?: NonNullable + // Producer's `// column` declared lineage, on the same write-edge — + // rendered as a columns badge on the link. + column_lineage?: NonNullable } // Graph-id of the script the user just launched (zero-latency hint), @@ -340,6 +343,28 @@ producerTests.set(`${r.usage_kind}:${r.path}`, r.data_tests) } } + // Producer → its `// column` declared lineage, keyed by runnable id, so + // the write-edge to the materialized asset can carry the columns badge — + // the edge *is* the transformation, and the lineage describes its output. + const producerColumnLineage = new Map< + string, + NonNullable + >() + // The `// materialize` target the lineage describes, so the badge lands + // only on that write-edge (a multi-output script writes several ducklake + // tables) — mirrors the anchor logic in `buildColumnGraph`. + const producerMaterializeTarget = new Map< + string, + NonNullable + >() + for (const r of g.runnables) { + if (r.column_lineage && r.column_lineage.length > 0) { + producerColumnLineage.set(`${r.usage_kind}:${r.path}`, r.column_lineage) + } + if (r.materialize_target) { + producerMaterializeTarget.set(`${r.usage_kind}:${r.path}`, r.materialize_target) + } + } for (const r of g.runnables) { const rid = `${r.usage_kind}:${r.path}` // Optimistic badge: the moment a run is launched from this view @@ -413,13 +438,23 @@ // write-edge carries the badge / custom-test nodes — a producer's // other (e.g. S3/datatable) outputs must not show them. const edgeTests = e.asset_kind === 'ducklake' ? producerTests.get(runnableId) : undefined + // Column lineage describes one materialized output, so the badge + // lands only on that asset's write-edge: the declared `// materialize` + // target when known (a multi-output script writes several ducklake + // tables), else the ducklake write-edge as the unambiguous fallback. + const matTarget = producerMaterializeTarget.get(runnableId) + const isOutputEdge = matTarget + ? e.asset_kind === matTarget.kind && e.asset_path === matTarget.path + : e.asset_kind === 'ducklake' + const edgeColumnLineage = isOutputEdge ? producerColumnLineage.get(runnableId) : undefined edges.push({ id: `prod:${runnableId}->${assetId}`, source: runnableId, target: assetId, kind: 'lineage-write', unsaved: e.unsaved, - data_tests: edgeTests + data_tests: edgeTests, + column_lineage: edgeColumnLineage }) // Each custom (`// data_test `) test → its own node // below the asset it validates, with a dashed "tests" edge. @@ -898,7 +933,10 @@ // producer's last-run status (the script fails if any test // fails) tints it green/red; neutral until it has run. data_tests: e.data_tests, - testsRunStatus: e.data_tests?.length ? runStates?.get(e.source)?.status : undefined + testsRunStatus: e.data_tests?.length ? runStates?.get(e.source)?.status : undefined, + // Column-lineage badge on the same write-edge (the link is the + // transformation whose output columns the lineage describes). + column_lineage: e.column_lineage }, animated, label, diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte index b69db2e04f..46ebaaf624 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte @@ -25,7 +25,13 @@ import type { Schema } from '$lib/common' import type { AssetGraphSelection, PipelineMode } from './types' import PipelineScriptView from './PipelineScriptView.svelte' - import { parsePipelineAnnotations, type PipelineAnnotations } from './parsePipelineAnnotations' + import { + parsePipelineAnnotations, + type ColumnLineage, + type PipelineAnnotations + } from './parsePipelineAnnotations' + import ColumnLineageTrace from './ColumnLineageTrace.svelte' + import { assetColumnNodes, type ColumnLineageGraph } from './columnLineageGraph' import SummaryPathDisplay from '$lib/components/SummaryPathDisplay.svelte' import S3FilePreview from '$lib/components/S3FilePreview.svelte' import DataTablePreview from './DataTablePreview.svelte' @@ -72,7 +78,11 @@ // edges + synthesize asset nodes for drafts whose body has been // edited past the seeded template. Fires on every keystroke that // changes the inferred set. - onAssetsChange?: (scriptPath: string | undefined, assets: AssetWithAltAccessType[]) => void + onAssetsChange?: ( + scriptPath: string | undefined, + assets: AssetWithAltAccessType[], + columnLineage?: ColumnLineage[] + ) => void // Emits the live editor buffer on every keystroke so the parent can // autosave the in-flight content WITHOUT waiting for the pane teardown // (`onDraftPersist`). `onDraftPersist` stays the authoritative commit on @@ -114,6 +124,10 @@ path: string unsaved?: boolean }> + // Pipeline-wide column-lineage graph (built by the parent page from the + // resolved graph). Drives the transitive column-lineage trace shown for a + // selected materialized asset. + selectionColumnGraph?: ColumnLineageGraph // Whether the selected ducklake asset's schema can evolve (whole-table // `replace` producer). Forwarded to the Schema tab: version history when // true, a single fixed-schema view when false. Defaults to true (unknown). @@ -218,6 +232,7 @@ onScriptRenamed, onScriptRemoved, selectionProducers = [], + selectionColumnGraph, schemaCanEvolve = true, runsRefreshKey, runsPendingJobId, @@ -360,6 +375,10 @@ // edges as the user edits the body (e.g. renaming a CREATE TABLE // target updates the output asset node in real time). let liveBodyAssets = $state(undefined) + // Body-inferred column lineage (DuckDB SQL AST), bound out of ScriptEditor + // alongside `liveBodyAssets` and forwarded so the live graph can show + // inferred column lineage on the edited script before it deploys. + let liveColumnLineage = $state(undefined) // Bumped when the runs panel reports a watched job has reached a // terminal state. Drives S3FilePreview's refreshKey so the preview @@ -541,7 +560,8 @@ inPipeline: false, triggerAssets: [], nativeTriggers: [], - dataTests: [] + dataTests: [], + columnLineage: [] } ) $effect(() => { @@ -554,7 +574,7 @@ }) $effect(() => { if (readOnly) return - onAssetsChange?.(script?.path, liveBodyAssets ?? []) + onAssetsChange?.(script?.path, liveBodyAssets ?? [], liveColumnLineage) }) $effect(() => { if (readOnly) return @@ -999,7 +1019,21 @@ {#key selection.path} - +
    + {#if selectionColumnGraph && assetColumnNodes(selectionColumnGraph, selection.asset_kind, selection.path).length > 0} +
    + +
    + {/if} +
    + +
    +
    {/key} {:else}
    @@ -1109,6 +1143,7 @@ bind:code={script.content} bind:schema={script.schema} bind:assets={liveBodyAssets} + bind:inferredColumnLineage={liveColumnLineage} {onTestStateChange} {args} /> diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphEdge.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphEdge.svelte index 65607cc5a6..c4bef2dd1f 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphEdge.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphEdge.svelte @@ -1,8 +1,8 @@ + +
    +
    + + Column lineage + {#if targetLabel} + + {targetLabel} + {/if} + + {#if selected} + + {:else if component.size > 1} + click a column to trace + {/if} +
    + + {#if component.size === 0} + No column lineage for this asset. + {:else} +
    + + {#each edges as e (`${e.from}->${e.to}`)} + {@const hot = traced !== undefined && traced.has(e.from) && traced.has(e.to)} + {@const cold = traced !== undefined && !hot} + + {/each} + + + {#each [...component] as id (id)} + {@const n = graph.nodes.get(id)} + {@const p = pos.get(id)} + {#if n && p} + {@const isSeed = seedSet.has(id)} + + {/if} + {/each} +
    + {/if} +
    diff --git a/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.test.ts b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.test.ts new file mode 100644 index 0000000000..6842dcfd86 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.test.ts @@ -0,0 +1,162 @@ +import { describe, expect, it } from 'vitest' +import type { AssetGraphResponse } from './types' +import { + buildColumnGraph, + colNodeId, + traceColumn, + connectedComponent, + assetColumnNodes, + computeDepths +} from './columnLineageGraph' + +// Two scripts chained through an intermediate ducklake table: +// s1: orders.amount -> staging.amt +// s2: staging.amt -> daily.total +// s2: customers.name -> daily.cust (a second source into the sink) +function chainGraph(): AssetGraphResponse { + return { + assets: [], + triggers: [], + runnables: [ + { + path: 's1', + usage_kind: 'script', + column_lineage: [ + { + column: 'amt', + inputs: [{ from_kind: 'ducklake', from_path: 'wh/orders', from_column: 'amount' }] + } + ] + }, + { + path: 's2', + usage_kind: 'script', + column_lineage: [ + { + column: 'total', + inputs: [{ from_kind: 'ducklake', from_path: 'wh/staging', from_column: 'amt' }] + }, + { + column: 'cust', + inputs: [{ from_kind: 'ducklake', from_path: 'wh/customers', from_column: 'name' }] + } + ] + } + ], + edges: [ + { + runnable_path: 's1', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'wh/staging', + access_type: 'w' + }, + { + runnable_path: 's2', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'wh/daily', + access_type: 'w' + } + ] + } +} + +const ORDERS_AMOUNT = colNodeId('ducklake', 'wh/orders', 'amount') +const STAGING_AMT = colNodeId('ducklake', 'wh/staging', 'amt') +const DAILY_TOTAL = colNodeId('ducklake', 'wh/daily', 'total') +const DAILY_CUST = colNodeId('ducklake', 'wh/daily', 'cust') +const CUSTOMERS_NAME = colNodeId('ducklake', 'wh/customers', 'name') + +describe('buildColumnGraph', () => { + it('stitches per-script lineage into a transitive graph via shared columns', () => { + const g = buildColumnGraph(chainGraph()) + // orders.amount feeds staging.amt feeds daily.total + expect(g.up.get(STAGING_AMT)).toEqual(new Set([ORDERS_AMOUNT])) + expect(g.up.get(DAILY_TOTAL)).toEqual(new Set([STAGING_AMT])) + expect(g.down.get(ORDERS_AMOUNT)).toEqual(new Set([STAGING_AMT])) + expect(g.down.get(STAGING_AMT)).toEqual(new Set([DAILY_TOTAL])) + }) + + it('anchors to the // materialize target, not a guessed write-edge', () => { + const graph = chainGraph() + // s1 declares its materialize target and also has unordered extra + // ducklake writes; the lineage must anchor to the declared target. + const s1 = graph.runnables.find((r) => r.path === 's1')! + s1.materialize_target = { kind: 'ducklake', path: 'wh/staging' } + graph.edges.unshift({ + runnable_path: 's1', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'wh/other', + access_type: 'w' + }) + const g = buildColumnGraph(graph) + expect(g.nodes.has(STAGING_AMT)).toBe(true) // anchored to the declared target + expect(g.nodes.has(colNodeId('ducklake', 'wh/other', 'amt'))).toBe(false) + }) + + it('falls back to a ducklake write-edge when there is no materialize target', () => { + // chainGraph's runnables carry no materialize_target, so s1's lineage is + // anchored via its (single) ducklake write-edge. + const g = buildColumnGraph(chainGraph()) + expect(g.nodes.has(STAGING_AMT)).toBe(true) + }) + + it('node ids are collision-proof across `#` / `:` in paths and columns', () => { + // A delimiter-concatenated id would merge these; the JSON-encoded id must not. + expect(colNodeId('ducklake', 'a#b', 'c')).not.toBe(colNodeId('ducklake', 'a', 'b#c')) + expect(colNodeId('ducklake', 'a:b', 'c')).not.toBe(colNodeId('ducklake', 'a', 'b:c')) + }) + + it('skips producers with no ducklake output asset (columns unanchorable)', () => { + const graph = chainGraph() + graph.edges = graph.edges.filter((e) => e.runnable_path !== 's1') // s1 loses its output edge + const g = buildColumnGraph(graph) + // staging.amt is no longer produced as a node by s1... + expect(g.up.has(STAGING_AMT)).toBe(false) + // ...but s2 still anchors daily.total ← staging.amt (staging.amt as a source). + expect(g.up.get(DAILY_TOTAL)).toEqual(new Set([STAGING_AMT])) + }) +}) + +describe('traceColumn', () => { + it('returns the full upstream + downstream impact set of a source column', () => { + const g = buildColumnGraph(chainGraph()) + // from the root source, the whole chain downstream is impacted + expect(traceColumn(ORDERS_AMOUNT, g)).toEqual( + new Set([ORDERS_AMOUNT, STAGING_AMT, DAILY_TOTAL]) + ) + }) + + it('traces backward from a sink to every contributing source', () => { + const g = buildColumnGraph(chainGraph()) + expect(traceColumn(DAILY_TOTAL, g)).toEqual(new Set([DAILY_TOTAL, STAGING_AMT, ORDERS_AMOUNT])) + // the sibling output `cust` and its source are NOT in total's trace + expect(traceColumn(DAILY_TOTAL, g).has(DAILY_CUST)).toBe(false) + expect(traceColumn(DAILY_TOTAL, g).has(CUSTOMERS_NAME)).toBe(false) + }) + + it('traces an intermediate column both directions', () => { + const g = buildColumnGraph(chainGraph()) + expect(traceColumn(STAGING_AMT, g)).toEqual(new Set([STAGING_AMT, ORDERS_AMOUNT, DAILY_TOTAL])) + }) +}) + +describe('connectedComponent + depths', () => { + it('collects the neighborhood of an asset and lays it out by hop depth', () => { + const g = buildColumnGraph(chainGraph()) + const seeds = assetColumnNodes(g, 'ducklake', 'wh/daily') // the sink asset + expect(new Set(seeds)).toEqual(new Set([DAILY_TOTAL, DAILY_CUST])) + const comp = connectedComponent(seeds, g) + expect(comp).toEqual( + new Set([DAILY_TOTAL, DAILY_CUST, STAGING_AMT, ORDERS_AMOUNT, CUSTOMERS_NAME]) + ) + const depths = computeDepths(comp, g) + expect(depths.get(ORDERS_AMOUNT)).toBe(0) + expect(depths.get(STAGING_AMT)).toBe(1) + expect(depths.get(DAILY_TOTAL)).toBe(2) + expect(depths.get(CUSTOMERS_NAME)).toBe(0) + expect(depths.get(DAILY_CUST)).toBe(1) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.ts b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.ts new file mode 100644 index 0000000000..50fa1c4e7b --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.ts @@ -0,0 +1,169 @@ +import type { AssetKind } from '$lib/gen' +import type { AssetGraphResponse } from './types' + +// A node in the column-level lineage graph: one column of one asset. +export type ColumnNode = { kind: AssetKind; path: string; column: string } +export type ColumnNodeId = string + +// Collision-proof node id — JSON-encoded tuple, so a `#`/`:` inside a path or +// (quoted) column name can't merge two distinct columns into one node. +export function colNodeId(kind: AssetKind, path: string, column: string): ColumnNodeId { + return JSON.stringify([kind, path, column]) +} + +// The pipeline-wide column-lineage graph, stitched across every producer. Each +// producer's `column_lineage` contributes single-hop edges (its output column ← +// its source columns); shared (asset,column) nodes chain those hops into the +// full transitive graph (`orders.amount → staging.amt → daily.total`). +export type ColumnLineageGraph = { + nodes: Map + // outputColumn → the source columns it derives from (walk upstream). + up: Map> + // sourceColumn → the output columns derived from it (walk downstream). + down: Map> +} + +// Build the column graph from a resolved asset graph. A producer's +// `column_lineage` describes the columns of the asset it materializes; that +// output asset is the ducklake target it writes (v1 materialize target), found +// from its write-edge. Producers without a known ducklake output are skipped +// (their columns can't be anchored to an asset node). +export function buildColumnGraph(graph: AssetGraphResponse): ColumnLineageGraph { + const nodes = new Map() + const up = new Map>() + const down = new Map>() + + const addNode = (n: ColumnNode): ColumnNodeId => { + const id = colNodeId(n.kind, n.path, n.column) + if (!nodes.has(id)) nodes.set(id, n) + return id + } + const addEdge = (src: ColumnNodeId, out: ColumnNodeId) => { + if (src === out) return + ;(up.get(out) ?? up.set(out, new Set()).get(out)!).add(src) + ;(down.get(src) ?? down.set(src, new Set()).get(src)!).add(out) + } + + // The output asset a runnable's `column_lineage` describes. The declared + // `// materialize` target is authoritative (a multi-output script writes + // several ducklake tables, and the deployed write-edges are unordered, so + // picking "a" write-edge can anchor to the wrong asset). Fall back to a + // ducklake write-edge only for producers with no materialize annotation + // (e.g. a literal single-output CTAS). + const outputAsset = new Map() + for (const r of graph.runnables ?? []) { + if (r.materialize_target) { + outputAsset.set(`${r.usage_kind}:${r.path}`, r.materialize_target) + } + } + for (const e of graph.edges ?? []) { + const access = e.access_type ?? 'r' + const key = `${e.runnable_kind}:${e.runnable_path}` + if ( + (access === 'w' || access === 'rw') && + e.asset_kind === 'ducklake' && + !outputAsset.has(key) + ) { + outputAsset.set(key, { kind: e.asset_kind, path: e.asset_path }) + } + } + + for (const r of graph.runnables ?? []) { + const lineage = r.column_lineage + if (!lineage || lineage.length === 0) continue + const out = outputAsset.get(`${r.usage_kind}:${r.path}`) + if (!out) continue + for (const cl of lineage) { + const outId = addNode({ kind: out.kind, path: out.path, column: cl.column }) + for (const inp of cl.inputs) { + const srcId = addNode({ + kind: inp.from_kind, + path: inp.from_path, + column: inp.from_column + }) + addEdge(srcId, outId) + } + } + } + + return { nodes, up, down } +} + +// Every node reachable from `start` by following `adj` (transitive closure, +// excluding `start` itself). Iterative to avoid deep-recursion limits. +function reach(start: ColumnNodeId, adj: Map>): Set { + const seen = new Set() + const stack = [start] + while (stack.length) { + const n = stack.pop()! + for (const m of adj.get(n) ?? []) { + if (!seen.has(m)) { + seen.add(m) + stack.push(m) + } + } + } + return seen +} + +// The full transitive trace of a column: itself + all upstream ancestors + all +// downstream descendants. This is the impact set — "everything that feeds, or +// is fed by, this column". +export function traceColumn(id: ColumnNodeId, g: ColumnLineageGraph): Set { + const out = new Set([id]) + for (const a of reach(id, g.up)) out.add(a) + for (const d of reach(id, g.down)) out.add(d) + return out +} + +// The connected neighborhood of a set of seed columns (an asset's columns): +// the seeds plus everything upstream and downstream of any of them. This is the +// subgraph the trace view renders around a selected asset. +export function connectedComponent( + seeds: ColumnNodeId[], + g: ColumnLineageGraph +): Set { + const out = new Set() + for (const s of seeds) { + if (!g.nodes.has(s)) continue + out.add(s) + for (const a of reach(s, g.up)) out.add(a) + for (const d of reach(s, g.down)) out.add(d) + } + return out +} + +// All column-node ids belonging to one asset (its seed set for a trace). +export function assetColumnNodes( + g: ColumnLineageGraph, + kind: AssetKind, + path: string +): ColumnNodeId[] { + const ids: ColumnNodeId[] = [] + for (const [id, n] of g.nodes) if (n.kind === kind && n.path === path) ids.push(id) + return ids +} + +// Longest-path depth of each node within `ids`, sources at depth 0 and depth +// increasing downstream — so a left→right layout reads upstream→downstream. +// Cycle-guarded (lineage is a DAG, but be defensive). +export function computeDepths( + ids: Set, + g: ColumnLineageGraph +): Map { + const depth = new Map() + const visiting = new Set() + const d = (id: ColumnNodeId): number => { + const memo = depth.get(id) + if (memo !== undefined) return memo + if (visiting.has(id)) return 0 + visiting.add(id) + let m = 0 + for (const u of g.up.get(id) ?? []) if (ids.has(u)) m = Math.max(m, d(u) + 1) + visiting.delete(id) + depth.set(id, m) + return m + } + for (const id of ids) d(id) + return depth +} diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts index 008b903ad9..3eb1e4e574 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts @@ -19,7 +19,8 @@ const ASSERTED_TS_FIELDS: Record = { tag: true, retry: true, materialize: true, - dataTests: true + dataTests: true, + columnLineage: true } // Parser-parity guard: this TS parser (drives the live graph preview) and @@ -71,6 +72,9 @@ type Fixture = { // corpus drives both sides. The TS parser emits this shape verbatim // (snake_case fields), so the comparison is 1:1. Absent === []. data_tests?: Array> + // Snake_case `ColumnLineage` serde shape — TS parser emits it verbatim, + // so the comparison is 1:1. Absent === []. + column_lineage?: Array> } } @@ -161,6 +165,8 @@ describe('parsePipelineAnnotations matches the shared Rust fixture corpus', () = } expect(got.dataTests, 'data tests').toEqual(f.expected.data_tests ?? []) + + expect(got.columnLineage, 'column lineage').toEqual(f.expected.column_lineage ?? []) }) } }) diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts index 15b12b959d..5f4b063e3f 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts @@ -1,5 +1,9 @@ import { describe, expect, it } from 'vitest' -import { parsePipelineAnnotations } from './parsePipelineAnnotations' +import { + mergeColumnLineage, + parsePipelineAnnotations, + type ColumnLineage +} from './parsePipelineAnnotations' // Unit-test the TS mirror of the backend `parse_pipeline_annotations`. The // two implementations MUST stay behaviorally identical — these tests are @@ -131,3 +135,29 @@ describe('parsePipelineAnnotations: data_upload', () => { expect(out.nativeTriggers).toEqual([]) }) }) + +describe('mergeColumnLineage', () => { + const ref = (path: string, col: string): ColumnLineage['inputs'][number] => ({ + from_kind: 'ducklake', + from_path: path, + from_column: col + }) + + it('annotation wins per output column; inferred fills the rest (mirrors Rust)', () => { + const inferred: ColumnLineage[] = [ + { column: 'total', inputs: [ref('w/o', 'amount')] }, + { column: 'qty', inputs: [ref('w/o', 'qty')] } + ] + const annotated: ColumnLineage[] = [{ column: 'total', inputs: [ref('w/manual', 'grand')] }] + const merged = mergeColumnLineage(inferred, annotated) + expect(merged).toEqual([ + { column: 'total', inputs: [ref('w/manual', 'grand')] }, // annotation, first + authoritative + { column: 'qty', inputs: [ref('w/o', 'qty')] } // inferred, not overridden + ]) + }) + + it('returns annotations unchanged when there is no inferred lineage', () => { + const annotated: ColumnLineage[] = [{ column: 'a', inputs: [ref('w/o', 'a')] }] + expect(mergeColumnLineage([], annotated)).toEqual(annotated) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts index a788acdbe0..ccbc746b04 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts @@ -113,6 +113,44 @@ export type DataTest = } | { type: 'custom'; path: string } +// `// column <- .[, …]` — declared column-level +// lineage: one output column and the upstream source columns it derives from. +// See backend `ColumnLineage`. A sibling of `DataTest` in the extensible +// annotation family — same parse shape, accumulating (one line per output +// column) — but pure metadata: drives the column-lineage graph view, runs no +// probe. Field names are snake_case to match the Rust `ColumnLineage` serde +// output verbatim, so the live-draft parse and the backend graph endpoint +// (deployed nodes) produce wire-identical shapes. +export type ColumnRef = { + from_kind: AssetKind + from_path: string + from_column: string +} +export type ColumnLineage = { + column: string + inputs: ColumnRef[] +} + +// Combine body-inferred column lineage with `// column` annotations, the +// annotation winning per output column. Mirrors the Rust `merge_column_lineage` +// (`asset_parser.rs`) so the live-draft preview matches what deploys: the +// backend already merges inferred + annotated server-side, and the live graph +// must apply the same precedence to the WASM-inferred lineage. +export function mergeColumnLineage( + inferred: ColumnLineage[], + annotated: ColumnLineage[] +): ColumnLineage[] { + const seen = new Set(annotated.map((c) => c.column)) + const out = [...annotated] + for (const c of inferred) { + if (!seen.has(c.column)) { + seen.add(c.column) + out.push(c) + } + } + return out +} + export type PipelineAnnotations = { inPipeline: boolean triggerAssets: PipelineTriggerAsset[] @@ -125,6 +163,8 @@ export type PipelineAnnotations = { materialize?: MaterializeSpec // `// data_test …` — accumulating data-quality checks (multiple lines). dataTests: DataTest[] + // `// column <- .[, …]` — accumulating column lineage. + columnLineage: ColumnLineage[] } // Tokenize a `key=value [key="quoted value"] ...` option string. Bare @@ -261,6 +301,37 @@ function parseRelationships(s: string): DataTest | undefined { } } +// `.` upstream reference. The column is the segment after the +// final `.`; the rest is the asset URI (default-syntax shorthands enabled). +// Mirrors Rust `parse_column_ref`. +function parseColumnRef(s: string): ColumnRef | undefined { + const dot = s.lastIndexOf('.') + if (dot < 0) return undefined + const fromColumn = singleIdent(s.slice(dot + 1)) + if (!fromColumn) return undefined + const asset = parseAssetSyntaxDefault(s.slice(0, dot).trim()) + if (!asset || asset.path === '') return undefined + return { from_kind: asset.kind, from_path: asset.path, from_column: fromColumn } +} + +// ` <- [, …]`. Individually malformed refs are dropped; the +// line is kept iff ≥1 ref parses (mirrors `parseAcceptedValues`). A missing +// `<-`, a non-ident output column, or zero valid refs drops the line. +// Mirrors Rust `parse_column_lineage_spec`. +function parseColumnLineageSpec(s: string): ColumnLineage | undefined { + const arrow = s.indexOf('<-') + if (arrow < 0) return undefined + const column = singleIdent(s.slice(0, arrow)) + if (!column) return undefined + const inputs = s + .slice(arrow + 2) + .split(',') + .map((r) => parseColumnRef(r.trim())) + .filter((r): r is ColumnRef => r !== undefined) + if (inputs.length === 0) return undefined + return { column, inputs } +} + // Parse a `// data_test …` right-hand side into one `DataTest`. The // leading token selects the variant; anything not a built-in keyword is the // `custom` escape hatch (a single script-path token). Returns `undefined` for @@ -398,7 +469,8 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { inPipeline: false, triggerAssets: [], nativeTriggers: [], - dataTests: [] + dataTests: [], + columnLineage: [] } for (const rawLine of code.split('\n')) { @@ -475,6 +547,15 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { continue } + // `column` is a complete word; a body comment that merely starts with + // `column` has no `<-` and is dropped fail-safe. Accumulates. + const afterColumn = consumeKeyword(inner, 'column') + if (afterColumn !== undefined) { + const spec = parseColumnLineageSpec(afterColumn.trim()) + if (spec) out.columnLineage.push(spec) + continue + } + const afterOn = consumeKeyword(inner, 'on') if (afterOn !== undefined) { const specText = afterOn.trim() diff --git a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts index 0521051375..b7c82822fa 100644 --- a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts +++ b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts @@ -1,5 +1,10 @@ import type { AssetGraphResponse, NativeTriggerKind } from './types' -import { parsePipelineAnnotations, type PipelineAnnotations } from './parsePipelineAnnotations' +import { + mergeColumnLineage, + parsePipelineAnnotations, + type ColumnLineage, + type PipelineAnnotations +} from './parsePipelineAnnotations' import { extractWrites, extractReads, @@ -20,7 +25,12 @@ export type ResolveGraphInput = { /** In-flight drafts keyed by script path. */ drafts: Map /** Body assets inferred for the currently-open script (live keystrokes). */ - liveBodyAssets: { scriptPath: string | undefined; assets: AssetWithAltAccessType[] } + liveBodyAssets: { + scriptPath: string | undefined + assets: AssetWithAltAccessType[] + /** Body-inferred column lineage (DuckDB SQL AST) for the open script. */ + columnLineage?: ColumnLineage[] + } /** Pipeline annotations parsed from the currently-open buffer. */ liveAnnotations: { scriptPath: string | undefined; annotations: PipelineAnnotations } /** Sticky session caches of inferred body writes/reads per script path. */ @@ -222,6 +232,19 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { for (const [path, d] of drafts) { const parsed = parsePipelineAnnotations(d.script.content) + // For the open script, fold in the WASM-inferred column lineage (DuckDB + // SQL AST) under the same annotation-wins precedence the backend applies + // on deploy, so the live preview matches what deploys. Only the open + // script carries live inference (`liveBodyAssets`); other drafts stay + // annotation-only until they deploy (the backend infers then). + const inferredCL = + path === liveBodyAssets.scriptPath ? (liveBodyAssets.columnLineage ?? []) : [] + const mergedCL = mergeColumnLineage(inferredCL, parsed.columnLineage) + // The `// materialize` target this draft's column lineage describes, so + // the column graph anchors to it rather than guessing a write-edge. + const materializeTarget = parsed.materialize + ? { kind: parsed.materialize.targetKind, path: parsed.materialize.targetPath } + : undefined // A draft can coexist with a base entry — during save the refetch // lands before drafts cleanup, and a user re-editing a deployed // script also produces both. In that case we mutate the existing @@ -240,15 +263,20 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { tag: parsed.tag, retry: parsed.retry, data_tests: parsed.dataTests.length > 0 ? parsed.dataTests : undefined, + column_lineage: mergedCL.length > 0 ? mergedCL : undefined, + materialize_target: materializeTarget, unsaved: true }) } else { // Refresh annotation-derived badges from the live parse too, so - // adding/removing `// data_test` lines on an already-deployed script - // updates the badge immediately (not only after redeploy/refetch). + // adding/removing `// data_test` / `// column` lines on an + // already-deployed script updates the badge immediately (not only + // after redeploy/refetch). runnables[baseIdx] = { ...runnables[baseIdx], data_tests: parsed.dataTests.length > 0 ? parsed.dataTests : undefined, + column_lineage: mergedCL.length > 0 ? mergedCL : undefined, + materialize_target: materializeTarget, unsaved: true } } diff --git a/frontend/src/lib/components/assets/AssetGraph/types.ts b/frontend/src/lib/components/assets/AssetGraph/types.ts index c942d28934..9276f500af 100644 --- a/frontend/src/lib/components/assets/AssetGraph/types.ts +++ b/frontend/src/lib/components/assets/AssetGraph/types.ts @@ -1,5 +1,5 @@ import type { AssetKind } from '$lib/gen' -import type { DataTest } from './parsePipelineAnnotations' +import type { ColumnLineage, DataTest } from './parsePipelineAnnotations' export type GraphUsageKind = 'script' | 'flow' @@ -33,6 +33,14 @@ export interface AssetGraphRunnableNode { // asset. Surfaced as a count badge (with a per-test breakdown in the title) // so test coverage is visible on the node without opening the pane. data_tests?: DataTest[] + // `// column <- .` declared column-level lineage for this + // script's materialized output. Surfaced as a count badge on the write-edge + // and as a column-to-column diagram in the asset details pane. + column_lineage?: ColumnLineage[] + // `// materialize ` target — the asset `column_lineage` describes. + // Lets the column graph anchor lineage to the exact output instead of + // guessing a ducklake write-edge (a multi-output script writes several). + materialize_target?: { kind: AssetKind; path: string } // Managed `// materialize` write strategy. Absent for non-materializing or // `manual` scripts. Used (with `partition_kind`) to decide whether a // produced asset's schema can evolve: only whole-table `replace` can, since diff --git a/frontend/src/lib/infer.ts b/frontend/src/lib/infer.ts index 7c8b83a64f..a70dd7c745 100644 --- a/frontend/src/lib/infer.ts +++ b/frontend/src/lib/infer.ts @@ -68,6 +68,7 @@ import wasmUrlWac from 'windmill-parser-wasm-wac/windmill_parser_wasm_bg.wasm?ur import { workspaceStore } from './stores.js' import { argSigToJsonSchemaType } from 'windmill-utils-internal' import { type AssetWithAccessType } from './components/assets/lib.js' +import { type ColumnLineage } from './components/assets/AssetGraph/parsePipelineAnnotations' const loadSchemaLastRun = writable< | [ @@ -169,6 +170,10 @@ type InferAssetsResult = assets: AssetWithAccessType[] sql_queries?: InferAssetsSqlQueryDetails[] columns?: Record + // Body-inferred column lineage (DuckDB SQL AST). Present once the + // `windmill-parser-wasm-asset` package is rebuilt with the inference; + // the spread below already forwards it from the parser output. + column_lineage?: ColumnLineage[] } | { status: 'error' diff --git a/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte index 3548ba6017..3056245e51 100644 --- a/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte @@ -33,8 +33,13 @@ import PipelineModeToggle from '$lib/components/assets/AssetGraph/PipelineModeToggle.svelte' import { parsePipelineAnnotations, + type ColumnLineage, type PipelineAnnotations } from '$lib/components/assets/AssetGraph/parsePipelineAnnotations' + import { + buildColumnGraph, + type ColumnLineageGraph + } from '$lib/components/assets/AssetGraph/columnLineageGraph' import { resolveGraph } from '$lib/components/assets/AssetGraph/resolveGraph' import { computeDownstreamClosure, @@ -472,7 +477,8 @@ inPipeline: false, triggerAssets: [], nativeTriggers: [], - dataTests: [] + dataTests: [], + columnLineage: [] } }) @@ -485,6 +491,7 @@ let liveBodyAssets = $state<{ scriptPath: string | undefined assets: AssetWithAltAccessType[] + columnLineage?: ColumnLineage[] }>({ scriptPath: undefined, assets: [] }) // The open draft's live editor buffer, emitted by the pane on every @@ -502,7 +509,13 @@ const EMPTY_LIVE_ASSETS = { scriptPath: undefined, assets: [] } const EMPTY_LIVE_ANNOTATIONS = { scriptPath: undefined, - annotations: { inPipeline: false, triggerAssets: [], nativeTriggers: [], dataTests: [] } + annotations: { + inPipeline: false, + triggerAssets: [], + nativeTriggers: [], + dataTests: [], + columnLineage: [] + } } // Reset every live editor overlay (annotations / body assets / content) @@ -1190,14 +1203,18 @@ ) { liveAnnotations = { scriptPath, annotations } } - function handleAssetsChange(scriptPath: string | undefined, assets: AssetWithAltAccessType[]) { + function handleAssetsChange( + scriptPath: string | undefined, + assets: AssetWithAltAccessType[], + columnLineage?: ColumnLineage[] + ) { // Single update site for the live overlay. `inferredWritesByPath` // / `inferredReadsByPath` are now derived from `liveBodyAssets` // (for the open script) + `inferredAssetsByPath` (prefetched // snapshot for every other script), so we don't have to write // into those caches here — the derive picks up our update on the // next reactive tick. - liveBodyAssets = { scriptPath, assets } + liveBodyAssets = { scriptPath, assets, columnLineage } } function handleContentChange(scriptPath: string | undefined, content: string) { liveContent = { scriptPath, content } @@ -1958,6 +1975,27 @@ .map((e) => ({ kind: e.runnable_kind, path: e.runnable_path, unsaved: e.unsaved })) }) + // Empty graph reused when the trace isn't shown (no ducklake-asset selection, + // or a draft is actively edited) so the pane blanks out like the other + // selection overlays and `buildColumnGraph` doesn't run. + const EMPTY_COLUMN_GRAPH: ColumnLineageGraph = { + nodes: new Map(), + up: new Map(), + down: new Map() + } + // Pipeline-wide column-lineage graph, stitched across every producer's + // (inferred + annotated) `column_lineage` and the asset write-edges. Drives + // the transitive column trace in the details pane. Built from `displayGraph` + // — the exact graph the canvas renders — so the trace matches it: draft + // overlays in edit / show-drafts, deployed-only in plain View. Gated to a + // ducklake-asset selection so it isn't rebuilt on every editor keystroke when + // the trace UI isn't even shown. + let columnGraph = $derived( + selection?.kind === 'asset' && selection.asset_kind === 'ducklake' + ? buildColumnGraph(displayGraph) + : EMPTY_COLUMN_GRAPH + ) + // Whether the selected ducklake asset's captured schema can *evolve* (drives // the asset panel's Schema tab: version history vs. a single fixed schema). // Only a whole-table `replace` producer (CREATE OR REPLACE) can change @@ -2552,6 +2590,7 @@ onRunByPath={runByPathLegit} selection={activeDraft ? undefined : selection} selectionProducers={activeDraft ? [] : selectionProducers} + selectionColumnGraph={activeDraft ? EMPTY_COLUMN_GRAPH : columnGraph} {schemaCanEvolve} {runsRefreshKey} {runsPendingJobId} From c479afab8ebceccbee050e923dc5c27a6712ea62 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Sun, 28 Jun 2026 11:25:59 +0200 Subject: [PATCH 113/117] fix: redeploy older app version from deployment history (#9826) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix: redeploy older app version from deployment history Co-Authored-By: Claude Opus 4.8 (1M context) * fix: apply restored app version to low-code editor on redeploy Redeploying an older app version from Deployment History fired the restore callback (toast shown) but the canvas kept displaying the current version, and Deploy then shipped that current value. AppEditor seeds its working state from `appDraftHandle.draft ?? app`, preferring the per-path autosave over the freshly restored `app` prop. The remount triggered by the restore therefore re-read the stale pre-restore draft. `reloadDeployed` already clears the draft before remounting for the reset-to-deployed flow; `onRestore` was missing the same step. Drop the autosave in `onRestore` so the remounted editor seeds from the restored value. Raw apps are unaffected: RawAppEditor binds `files` directly (no draft precedence), and `extractRawApp` mutates that bound state in place. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(raw-apps): convert savedNewAppPath event forwarding to a callback prop `svelte-check` (CI `npm check`) failed with one error: forwarding the `savedNewAppPath` createEventDispatcher event through the runes-mode RawAppEditor → RawAppEditorHeader chain types as "not assignable to never". This is the same legacy-forwarding-through-runes pattern already removed for `restore` in this PR — `on:savedNewAppPath` would likewise be dropped at runtime, breaking navigation to the new path after a deploy that renames the app. Replace the `on:savedNewAppPath` forwarding with an `onSavedNewAppPath` callback prop threaded page → RawAppEditor → RawAppEditorHeader, matching `onRestore`. The header now invokes the callback instead of dispatching, and its now-unused createEventDispatcher is removed. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .../components/apps/editor/AppEditor.svelte | 5 +++-- .../apps/editor/AppEditorHeader.svelte | 9 ++++++-- frontend/src/lib/components/apps/types.ts | 4 ++++ .../components/raw_apps/RawAppEditor.svelte | 16 +++++++++++--- .../raw_apps/RawAppEditorHeader.svelte | 22 +++++++++++++------ .../(logged)/apps/edit/[...path]/+page.svelte | 11 +++++++--- .../apps_raw/edit/[...path]/+page.svelte | 12 +++++----- 7 files changed, 56 insertions(+), 23 deletions(-) diff --git a/frontend/src/lib/components/apps/editor/AppEditor.svelte b/frontend/src/lib/components/apps/editor/AppEditor.svelte index 0cffdc2400..f6e3245c2c 100644 --- a/frontend/src/lib/components/apps/editor/AppEditor.svelte +++ b/frontend/src/lib/components/apps/editor/AppEditor.svelte @@ -83,7 +83,8 @@ onResetToDeployed, loadedFromDraft = false, othersDraftsCount = 0, - onOpenOthersDrafts + onOpenOthersDrafts, + onRestore }: AppEditorProps = $props() migrateApp(untrack(() => app)) @@ -890,7 +891,7 @@ {loadedFromDraft} {othersDraftsCount} {onOpenOthersDrafts} - on:restore + {onRestore} {policy} {fromHub} bind:this={appEditorHeader} diff --git a/frontend/src/lib/components/apps/editor/AppEditorHeader.svelte b/frontend/src/lib/components/apps/editor/AppEditorHeader.svelte index 1de186c301..0ed389f52c 100644 --- a/frontend/src/lib/components/apps/editor/AppEditorHeader.svelte +++ b/frontend/src/lib/components/apps/editor/AppEditorHeader.svelte @@ -112,6 +112,10 @@ loadedFromDraft?: boolean othersDraftsCount?: number onOpenOthersDrafts?: () => void + // Restoring an older deployment from the history drawer. A callback prop + // (not `on:restore` forwarding): forwarding a `createEventDispatcher` + // event up through these runes-mode components silently drops it. + onRestore?: (restoredApp: any) => void } let { @@ -137,7 +141,8 @@ onResetToDeployed, loadedFromDraft = false, othersDraftsCount = 0, - onOpenOthersDrafts + onOpenOthersDrafts, + onRestore }: Props = $props() /** Mirror of the path the user is editing in the pen popover. Initialized @@ -862,7 +867,7 @@ (historyBrowserDrawerOpen = false)}> - + onRestore?.(e.detail)} appPath={$appPath} /> diff --git a/frontend/src/lib/components/apps/types.ts b/frontend/src/lib/components/apps/types.ts index de34f4676d..d8ac000ba5 100644 --- a/frontend/src/lib/components/apps/types.ts +++ b/frontend/src/lib/components/apps/types.ts @@ -176,6 +176,10 @@ export interface AppEditorProps { loadedFromDraft?: boolean othersDraftsCount?: number onOpenOthersDrafts?: () => void + // Restoring an older deployment from the history drawer. Threaded through + // AppEditorHeader as a callback prop rather than `on:restore` forwarding, + // which does not propagate through these runes-mode components. + onRestore?: (restoredApp: any) => void } export type App = { diff --git a/frontend/src/lib/components/raw_apps/RawAppEditor.svelte b/frontend/src/lib/components/raw_apps/RawAppEditor.svelte index c8d15e6e7e..18f9c1af0c 100644 --- a/frontend/src/lib/components/raw_apps/RawAppEditor.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppEditor.svelte @@ -121,6 +121,14 @@ onOpenOthersDrafts?: () => void onRuntimeLogRequester?: (requester: RawAppRuntimeLogRequester | undefined) => void onRunsProvider?: (provider: RawAppRunsProvider | undefined) => void + // Restoring an older deployment from the history drawer. A callback prop + // (not `on:restore` forwarding): forwarding a `createEventDispatcher` + // event up through these runes-mode components silently drops it. + onRestore?: (restoredApp: any) => void + // Deploy created the app at a new path; the page navigates to it. Callback + // prop for the same reason as `onRestore` — `on:savedNewAppPath` forwarding + // through these runes-mode components is dropped. + onSavedNewAppPath?: (path: string) => void } let { @@ -148,7 +156,9 @@ othersDraftsCount = 0, onOpenOthersDrafts, onRuntimeLogRequester = undefined, - onRunsProvider = undefined + onRunsProvider = undefined, + onRestore, + onSavedNewAppPath }: Props = $props() export const version: number | undefined = undefined @@ -1596,8 +1606,8 @@ bind:savedApp bind:summary bind:pendingDraftPath - on:restore - on:savedNewAppPath + {onRestore} + {onSavedNewAppPath} {policy} {diffDrawer} {newApp} diff --git a/frontend/src/lib/components/raw_apps/RawAppEditorHeader.svelte b/frontend/src/lib/components/raw_apps/RawAppEditorHeader.svelte index 3963e1b16d..9853fb69bd 100644 --- a/frontend/src/lib/components/raw_apps/RawAppEditorHeader.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppEditorHeader.svelte @@ -26,7 +26,7 @@ Undo, WandSparkles } from 'lucide-svelte' - import { createEventDispatcher, untrack } from 'svelte' + import { untrack } from 'svelte' import { orderedJsonStringify, type Value, replaceFalseWithUndefined } from '../../utils' import { random_adj } from '$lib/components/random_positive_adjetive' @@ -151,6 +151,14 @@ loadedFromDraft?: boolean othersDraftsCount?: number onOpenOthersDrafts?: () => void + // Restoring an older deployment from the history drawer. A callback prop + // (not `on:restore` forwarding): forwarding a `createEventDispatcher` + // event up through these runes-mode components silently drops it. + onRestore?: (restoredApp: any) => void + // Deploy created the app at a new path; the page navigates to it. Callback + // prop for the same reason as `onRestore` — `on:savedNewAppPath` forwarding + // through these runes-mode components is dropped. + onSavedNewAppPath?: (path: string) => void } let { @@ -184,7 +192,9 @@ onResetToDeployed, loadedFromDraft = false, othersDraftsCount = 0, - onOpenOthersDrafts + onOpenOthersDrafts, + onRestore, + onSavedNewAppPath }: Props = $props() // Set by the on-behalf-of selector when the publisher picks a user other than @@ -352,7 +362,7 @@ path: appPath }) } - dispatch('savedNewAppPath', path) + onSavedNewAppPath?.(path) onDeploy?.({ path }) } catch (e) { sendUserToast(`Error creating app: ${e.body ?? e.message}`, true) @@ -475,7 +485,7 @@ }) } if (appPath !== npath) { - dispatch('savedNewAppPath', npath) + onSavedNewAppPath?.(npath) } onDeploy?.({ path: npath }) } @@ -575,8 +585,6 @@ } ]) - const dispatch = createEventDispatcher() - let customPath = $state(savedApp?.custom_path) let customPathError = $state('') @@ -688,7 +696,7 @@ (historyBrowserDrawerOpen = false)}> - + onRestore?.(e.detail)} {appPath} /> diff --git a/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte index 8c0efa0af4..ea87409171 100644 --- a/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte @@ -388,9 +388,14 @@ let diffDrawer: DiffDrawer | undefined = $state() - function onRestore(ev: any) { + function onRestore(restoredApp: any) { sendUserToast('App restored from previous deployment') - app = ev.detail + // Drop the stale pre-restore autosave. The remounted AppEditor seeds its + // state from `appDraftHandle.draft ?? app`, so without this it keeps showing + // the old draft instead of the restored version. Same reason `reloadDeployed` + // removes the draft before remounting. + UserDraft.remove('app', path) + app = restoredApp // Re-pin the stale-draft fork base to the current head. A restored value // carries the `parent_version` baked in when that older version was deployed, // which would make the deploy guard (`compareVersions`) falsely report "not on @@ -470,7 +475,7 @@ app.path = url } }} - on:restore={onRestore} + {onRestore} summary={app.summary} app={app.value} {deployedBaseline} diff --git a/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte index 94c6d72e71..0cf374f0b1 100644 --- a/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte @@ -449,9 +449,9 @@ let diffDrawer: DiffDrawer | undefined = $state(undefined) - function onRestore(ev: any) { + function onRestore(restoredApp: any) { sendUserToast('App restored from previous deployment') - let prev = ev.detail + let prev = restoredApp extractRawApp(prev) savedApp = { summary: prev.summary, @@ -536,12 +536,12 @@ {#key redraw}
    { + onSavedNewAppPath={(savedPath) => { draftSync.remove() - goto(`/apps_raw/edit/${event.detail}`) - newPath = event.detail + goto(`/apps_raw/edit/${savedPath}`) + newPath = savedPath }} - on:restore={onRestore} + {onRestore} bind:files bind:runnables bind:data From da45e699c8aefeede172c90769ef4f4b182fec0c Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Sun, 28 Jun 2026 11:43:09 +0200 Subject: [PATCH 114/117] feat(apps): add labels input to app editor deploy drawer (#9828) * feat(apps): add labels input to app editor deploy drawer The labels feature (c4c9ef5fd) wired LabelsInput into the script, flow, schedule, resource and variable editors but left the app editor out: it had no labels state and createApp/updateApp never sent labels, so apps could not be labeled from the UI despite full backend support. Thread the deployed app's labels from the edit page through AppEditor into AppEditorHeader, render LabelsInput in AppEditorHeaderDeploy after the summary field (matching ScriptBuilder/FlowSettings), and include labels in the create/update request bodies, the savedApp snapshot, and the diff/deploy comparison values. The raw-app editor shares the deploy drawer, so it is wired symmetrically (createAppRaw/updateAppRaw + the raw page loader) to avoid leaking a non-functional input there. Fixes WIN-2107 Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(apps): drop redundant labels cast in app edit restore path Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): include labels in deploy-drawer Diff current value The Diff button inside the deploy drawer built its current value without labels, so the approval preview could hide label changes that would be deployed. (Identified by cubic.) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(apps): reset raw-app labels on new-draft seed The raw-app edit route keeps labels as route-level state and the ?new_draft=true seed-template branch never cleared it. Since the route is reused across raw-app navigations, opening a labeled raw app then creating a fresh one could remount RawAppEditor with the previous app's labels and deploy them via createAppRaw. Reset labels with the other bleed-prevention resets at the top of the new-draft branch. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .../components/apps/editor/AppEditor.svelte | 2 ++ .../apps/editor/AppEditorHeader.svelte | 30 ++++++++++++++----- .../apps/editor/AppEditorHeaderDeploy.svelte | 5 ++++ frontend/src/lib/components/apps/types.ts | 3 ++ .../components/raw_apps/RawAppEditor.svelte | 5 ++++ .../raw_apps/RawAppEditorHeader.svelte | 21 +++++++++---- .../(logged)/apps/edit/[...path]/+page.svelte | 8 +++-- .../apps_raw/edit/[...path]/+page.svelte | 12 +++++++- 8 files changed, 70 insertions(+), 16 deletions(-) diff --git a/frontend/src/lib/components/apps/editor/AppEditor.svelte b/frontend/src/lib/components/apps/editor/AppEditor.svelte index f6e3245c2c..64cc94f551 100644 --- a/frontend/src/lib/components/apps/editor/AppEditor.svelte +++ b/frontend/src/lib/components/apps/editor/AppEditor.svelte @@ -69,6 +69,7 @@ path, policy, summary, + labels, deployedBaseline = undefined, fromHub = false, diffDrawer = undefined, @@ -886,6 +887,7 @@ void @@ -129,6 +132,7 @@ bottomPanelHidden = false, newApp, newPath = '', + labels: initialLabels = undefined, userDraftPath = '', onSavedNewAppPath, onShowLeftPanel, @@ -261,7 +265,8 @@ policy, deployment_message: deploymentMsg, custom_path: customPath, - preserve_on_behalf_of: preserveOnBehalfOf || undefined + preserve_on_behalf_of: preserveOnBehalfOf || undefined, + labels } }) // New path now exists server-side — drop the autocomplete cache so @@ -272,7 +277,8 @@ value: structuredClone($state.snapshot($app)), path: path, policy: policy, - custom_path: customPath + custom_path: customPath, + labels: $state.snapshot(labels) } closeSaveDrawer() sendUserToast('App deployed successfully') @@ -315,7 +321,8 @@ value: $app, path: newEditedPath || savedApp.path, policy, - custom_path: customPath + custom_path: customPath, + labels }) ) ) { @@ -366,7 +373,8 @@ // it also means that customPath needs to be set to '' instead of undefined to unset it (when admin) custom_path: $userStore?.is_admin || $userStore?.is_super_admin ? (customPath ?? '') : undefined, - preserve_on_behalf_of: preserveOnBehalfOf || undefined + preserve_on_behalf_of: preserveOnBehalfOf || undefined, + labels } }) invalidateWorkspacePaths($workspaceStore!) @@ -375,7 +383,8 @@ value: structuredClone($state.snapshot($app)), path: npath, policy, - custom_path: customPath + custom_path: customPath, + labels: $state.snapshot(labels) } const appHistory = await AppService.getAppHistoryByPath({ workspace: $workspaceStore!, @@ -629,7 +638,8 @@ value: $app, path: newEditedPath || savedApp.path, policy, - custom_path: customPath + custom_path: customPath, + labels } }) }, @@ -728,6 +738,7 @@ }) let customPath = $state(savedApp?.custom_path) + let labels = $state(untrack(() => initialLabels)) $effect(() => { if ($openDebugRun == undefined) { @@ -757,7 +768,8 @@ value: $app, path: newEditedPath || savedApp?.path, policy, - custom_path: customPath + custom_path: customPath, + labels }} /> @@ -801,7 +813,8 @@ value: $app, path: newEditedPath || savedApp.path, policy, - custom_path: customPath + custom_path: customPath, + labels }, button: { text: 'Looks good, deploy', @@ -854,6 +867,7 @@ bind:pathError bind:newEditedPath bind:preserveOnBehalfOf + bind:labels hideSecretUrl={false} /> diff --git a/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte b/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte index 0d5fc0c44a..01c0cb3028 100644 --- a/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte +++ b/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte @@ -16,6 +16,7 @@ import { isCloudHosted } from '$lib/cloud' import EEOnly from '$lib/components/EEOnly.svelte' import TextInput from '$lib/components/text_input/TextInput.svelte' + import LabelsInput from '$lib/components/LabelsInput.svelte' import OnBehalfOfSelector, { type OnBehalfOfChoice } from '$lib/components/OnBehalfOfSelector.svelte' @@ -38,6 +39,7 @@ newPath, hideSecretUrl = false, preserveOnBehalfOf = $bindable(false), + labels = $bindable(), rawApp = false, newApp = false }: { @@ -55,6 +57,7 @@ newPath: string hideSecretUrl?: boolean preserveOnBehalfOf?: boolean + labels?: string[] | undefined // Raw apps need cross-origin isolation (wm_coep) to be embeddable. Classic // (low-code) apps must NOT get the flag — it would force COEP on the // document and break no-CORP cross-origin subresources (external images, @@ -201,6 +204,8 @@ bind:value={summary} />
    +
    +
    diff --git a/frontend/src/lib/components/apps/types.ts b/frontend/src/lib/components/apps/types.ts index d8ac000ba5..960e273ac3 100644 --- a/frontend/src/lib/components/apps/types.ts +++ b/frontend/src/lib/components/apps/types.ts @@ -144,6 +144,8 @@ export interface AppEditorProps { path: string policy: Policy summary: string + /** Initial labels for the app, threaded from the loaded app data. */ + labels?: string[] /** Deployed app value the autosave `discardIf` compares against, so an * edit reverting to deployed clears the draft instead of leaving a no-op. * `undefined` for draft-only paths (no deployed baseline). */ @@ -157,6 +159,7 @@ export interface AppEditorProps { summary: string policy: any custom_path?: string + labels?: string[] } | undefined version?: number | undefined diff --git a/frontend/src/lib/components/raw_apps/RawAppEditor.svelte b/frontend/src/lib/components/raw_apps/RawAppEditor.svelte index 18f9c1af0c..d93da9a46c 100644 --- a/frontend/src/lib/components/raw_apps/RawAppEditor.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppEditor.svelte @@ -71,6 +71,8 @@ summary?: string path: string newPath?: string | undefined + /** Initial labels for the app, threaded from the loaded app data. */ + labels?: string[] savedApp?: | { value: any @@ -82,6 +84,7 @@ /** No deployed counterpart exists (draft-only); disables Diff. */ no_deployed?: boolean custom_path?: string + labels?: string[] } | undefined diffDrawer?: DiffDrawer | undefined @@ -140,6 +143,7 @@ summary = $bindable(''), path, newPath = undefined, + labels = undefined, savedApp = $bindable(undefined), diffDrawer = undefined, onNavigate, @@ -1612,6 +1616,7 @@ {diffDrawer} {newApp} {newPath} + {labels} appPath={path} {liveEditorDraftStoragePath} {autosaveWorkspace} diff --git a/frontend/src/lib/components/raw_apps/RawAppEditorHeader.svelte b/frontend/src/lib/components/raw_apps/RawAppEditorHeader.svelte index 9853fb69bd..2d7bed8333 100644 --- a/frontend/src/lib/components/raw_apps/RawAppEditorHeader.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppEditorHeader.svelte @@ -102,6 +102,7 @@ summary: string policy: any custom_path?: string + labels?: string[] /** No deployed counterpart exists (draft-only); disables Diff. */ no_deployed?: boolean } @@ -109,6 +110,8 @@ version?: number | undefined newApp: boolean newPath?: string + /** Initial labels for the app, threaded from the loaded app data. */ + labels?: string[] appPath: string runnables: Record files: Record | undefined @@ -169,6 +172,7 @@ version = $bindable(undefined), newApp, newPath = '', + labels: initialLabels = undefined, appPath, runnables, data, @@ -331,7 +335,8 @@ policy, deployment_message: deploymentMsg, custom_path: customPath, - preserve_on_behalf_of: preserveOnBehalfOf || undefined + preserve_on_behalf_of: preserveOnBehalfOf || undefined, + labels }, js, css @@ -345,7 +350,8 @@ value: structuredClone(stateSnapshot(app)), path: path, policy: policy, - custom_path: customPath + custom_path: customPath, + labels: $state.snapshot(labels) } closeSaveDrawer() sendUserToast('App deployed successfully') @@ -454,7 +460,8 @@ // custom_path requires admin so to accept update without it, we need to send as undefined when non-admin (when undefined, it will be ignored) // it also means that customPath needs to be set to '' instead of undefined to unset it (when admin) custom_path: - $userStore?.is_admin || $userStore?.is_super_admin ? (customPath ?? '') : undefined + $userStore?.is_admin || $userStore?.is_super_admin ? (customPath ?? '') : undefined, + labels }, js, css @@ -466,7 +473,8 @@ value: structuredClone(stateSnapshot(app)), path: npath, policy, - custom_path: customPath + custom_path: customPath, + labels: $state.snapshot(labels) } const appHistory = await AppService.getAppHistoryByPath({ workspace: $workspaceStore!, @@ -587,6 +595,7 @@ let customPath = $state(savedApp?.custom_path) let customPathError = $state('') + let labels = $state(untrack(() => initialLabels)) let jobsDrawerOpen = $state(false) @@ -600,7 +609,8 @@ value: app, path: newEditedPath || savedApp?.path, policy, - custom_path: customPath + custom_path: customPath, + labels }) ) @@ -690,6 +700,7 @@ bind:pathError bind:newEditedPath bind:preserveOnBehalfOf + bind:labels /> diff --git a/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte index ea87409171..7eb5d30106 100644 --- a/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte @@ -33,6 +33,7 @@ summary: string policy: any custom_path?: string + labels?: string[] } | undefined = $state(undefined) let redraw = $state(0) @@ -297,7 +298,8 @@ value: backendApp_.value as App, path: backendApp_.path, policy: backendApp_.policy, - custom_path: backendApp_.custom_path + custom_path: backendApp_.custom_path, + labels: backendApp_.labels } // "Load another user's draft" handoff: render their value. Overlay mode (we // have our own draft) hard-locks saves until the user confirms overwriting @@ -409,7 +411,8 @@ value: app_.value as App, path: app_.path, policy: app_.policy, - custom_path: app_.custom_path + custom_path: app_.custom_path, + labels: app_.labels } redraw++ } @@ -478,6 +481,7 @@ {onRestore} summary={app.summary} app={app.value} + labels={app.labels} {deployedBaseline} newPath={app.value?.draft_path ?? app.path} path={page.params.path ?? ''} diff --git a/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte index 0cf374f0b1..6227776ec4 100644 --- a/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte @@ -57,6 +57,7 @@ // let lastVersion = 0 let policy: any = $state({}) let summary = $state('') + let labels = $state(undefined) /** User-typed path from `RawAppEditorHeader` when it differs from * `savedApp.path`; mirrored into the draft below as `draft_path` for the * home list's friendly name. */ @@ -72,6 +73,7 @@ summary: string policy: any custom_path?: string + labels?: string[] no_deployed?: boolean } | undefined = $state(undefined) @@ -140,6 +142,7 @@ if (extractedData) data = extractedData files = app.value.files summary = app.summary + labels = app.labels // lastVersion = app.version policy = app.policy // Prefer the saved `draft_path` so the topbar shows the pending name, not @@ -179,6 +182,10 @@ loadedFromDraft = false draftSavedAt = undefined deployedAt = undefined + // `labels` is route-level state; reset it too so a fresh draft doesn't + // inherit (and then deploy) the previously-opened app's labels. The + // import branch re-seeds it via extractRawApp below. + labels = undefined // Brand-new raw app: no deployed baseline, so never discard-on-equal. deployedBaseline = undefined // Suspend autosave across the bootstrap: the seed template and the @@ -355,6 +362,7 @@ path: backendApp_.path, policy: backendApp_.policy, custom_path: backendApp_.custom_path, + labels: backendApp_.labels, no_deployed: backendApp_.no_deployed } // Extract the effective raw app into the editor's local pieces. The bundle @@ -458,7 +466,8 @@ value: structuredClone(stateSnapshot(prev.value)), path: prev.path, policy: structuredClone(stateSnapshot(policy)), - custom_path: prev.custom_path + custom_path: prev.custom_path, + labels: prev.labels } redraw++ } @@ -548,6 +557,7 @@ bind:summary bind:pendingDraftPath {newPath} + {labels} path={page.params.path ?? ''} liveEditorDraftStoragePath={path} {policy} From c0768de0acdf63eaba5fb97d04bfc64f2f03b93d Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Sun, 28 Jun 2026 11:49:27 +0200 Subject: [PATCH 115/117] fix: close unauthenticated DAP debugger program-mode launch bypass (#9829) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The /ws_debug debugger WebSocket gated JWT signature verification on inline `code` being present (`if (code && REQUIRE_SIGNED_REQUESTS)`), so a `program`-mode launch (naming an arbitrary server-side file path that is read and executed) skipped verification entirely — even with REQUIRE_SIGNED_DEBUG_REQUESTS=true. The WS handshake also performed no Origin check, allowing cross-origin (CSWSH) drive-by from a malicious page. - Enforce signing on every launch in both handlers (Python + Bun/TS): reject program-mode outright and require+verify a token for inline code. - Add opt-in DEBUG_ALLOWED_ORIGINS allowlist enforced at the WS handshake. - Default docker-compose REQUIRE_SIGNED_DEBUG_REQUESTS to true. - Update THREAT_MODEL T8/EP15 to reflect the root cause and mitigation. Co-authored-by: Claude Opus 4.8 (1M context) --- backend/THREAT_MODEL.md | 4 +-- debugger/dap_debug_service.ts | 38 +++++++++++++++++++++++++--- debugger/dap_websocket_server_bun.ts | 36 +++++++++++++++++++++++--- docker-compose.yml | 3 ++- 4 files changed, 70 insertions(+), 11 deletions(-) diff --git a/backend/THREAT_MODEL.md b/backend/THREAT_MODEL.md index dc98a5a625..9501b5bf0d 100644 --- a/backend/THREAT_MODEL.md +++ b/backend/THREAT_MODEL.md @@ -88,7 +88,7 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical). | EP12 Stored-content rendering | App builder HTML component, markdown, S3 download response headers | stored user content → admin browser (same origin) | Admin session, account takeover | | EP13 Log/file reading & export endpoints | `service_logs`, `jobs_u/getupdate` log file read (symlinks), workspace/tarball export | authed/unauth request → arbitrary file or admin-only config | Arbitrary files, global settings | | EP14 Secret-value & resource-value caches | In-memory caches in `windmill-store` keyed (historically un-keyed) by path | cache lookup crossing identity/folder boundary | Secret variables, resource creds | -| EP15 Deployment & runtime config | docker-compose defaults: dind, debugger (`REQUIRE_SIGNED_DEBUG_REQUESTS=false`), CORS `Any`, default admin/`changeme`, exposed Postgres, `SUPERADMIN_SECRET`, `ENABLE_NSJAIL=false`, privileged containers | operator/infra default → full instance | All assets | +| EP15 Deployment & runtime config | docker-compose defaults: dind, debugger (`REQUIRE_SIGNED_DEBUG_REQUESTS` now defaults to `true`; can still be overridden to `false`), CORS `Any`, default admin/`changeme`, exposed Postgres, `SUPERADMIN_SECRET`, `ENABLE_NSJAIL=false`, privileged containers | operator/infra default → full instance | All assets | | EP16 Supply chain | Cached hub scripts, GitHub workflow actions, vendored deps, Docker base image | build/update-time input → host & build integrity | Worker host, build integrity | | EP17 Token lifecycle | Token create/rescope/refresh, script-issued JWTs | scoped caller → broader privilege | Tokens, accounts, isolation | @@ -103,7 +103,7 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical). | T5 | Worker compromise & cross-tenant access via weak-by-default isolation (nsjail off by default → user code runs with only PID-ns `unshare`); sandbox escape where nsjail/dind/podman is enabled | remote_auth | EP9, EP15 | Worker host, isolation, downstream | critical | likely | unmitigated | nsjail off by default everywhere (`DISABLE_NSJAIL=true`); shipped compose gives PID-ns `unshare` only (`FAVOR_UNSHARE_PID=true`), bare installs get no isolation. Where nsjail enabled: read-only remounts, jail-tmp refusal, podman socket gating | GHSA-6qr8-xhg4-453q, GHSA-3vpp-vf62-wqp6, f8467f38c8, df5aec0f5d, f1b6746e0e | | T6 | Disclosure of secrets, resource credentials, and workspace encryption keys across the authorization boundary (AI proxy, MCP, caches, export); database read additionally yields plaintext instance-level `global_settings` secrets | remote_auth | EP6, EP14, EP13 | Secret variables, encryption keys, resource creds, global settings | critical | likely | partially_mitigated | RLS on `$var:`, cache scoping by caller, admin checks on export; per-workspace secret *variables* encrypted at rest, but `global_settings` is plaintext under the default DB secret backend | GHSA-jwg4-v3cj-rvfm, GHSA-8m2p-2crh-9h3w, GHSA-6635-6fch-v8px, GHSA-437f-725p-7w84, GHSA-f27g-j463-q85w (CVE-2026-26964), GHSA-j679-v6vj-jfxc, GHSA-6vrr-fq33-qpfp, 0ba128afe7, 7836a4e733, ff8e39c69b | | T7 | Full instance compromise from insecure deployment defaults (dind control, default admin/`changeme`, exposed Postgres, publicly readable SUPERADMIN_SECRET) | remote_unauth | EP15 | All assets | critical | likely | partially_mitigated | first-time-setup warning on default admin; docs recommend hardening | GHSA-3vpp-vf62-wqp6, GHSA-24fr-44f8-fqwg (CVE-2026-29059), GHSA-6q36-5p3h-766j | -| T8 | Unauthenticated RCE via the Debugger WebSocket in the default `windmill_extra` configuration | remote_unauth | EP15 | Worker host, all assets | critical | possible | unmitigated | `REQUIRE_SIGNED_DEBUG_REQUESTS` exists but defaults to false | GHSA-725h-99vx-9xr4 | +| T8 | Unauthenticated RCE via the Debugger WebSocket: `/ws_debug/*` exposed by the gateway/ingress with the debugger service as the auth boundary; signature gate was bypassable via `program`-mode launches (read+exec an arbitrary server-side file path, never signed) even with signing on, and the WS handshake had no Origin check (CSWSH) | remote_unauth | EP15 | Worker host, all assets | critical | possible | partially_mitigated | `program`-mode launches now rejected when `REQUIRE_SIGNED_DEBUG_REQUESTS` is on (signing covers every launch, not just inline `code`); shipped `docker-compose` now defaults `REQUIRE_SIGNED_DEBUG_REQUESTS=true`; opt-in `DEBUG_ALLOWED_ORIGINS` allowlist rejects cross-origin handshakes. Residual: code default is secure but operators can still set `=false`; origin allowlist is opt-in | GHSA-725h-99vx-9xr4 | | T9 | Supply-chain compromise via cached hub scripts, GitHub workflow command injection, or vulnerable base-image deps | supply_chain | EP16 | Worker host, build integrity | critical | possible | partially_mitigated | hub-script re-pin to patched versions; HUB_BASE_URL override | GHSA-w2m9-q5f7-3gpq, edf340c4d4, GHSA-8rq7-w7g6-8wvr, GHSA-vch9-39v5-4wg7 (CVE-2024-37371) | | T10 | Unauthenticated disclosure of job results, args, logs, and admin config via missing-authz public endpoints | remote_unauth | EP2, EP13 | Job results/args/logs, global settings, scripts | high | likely | partially_mitigated | anonymous-job checks, log-endpoint authz hardening | GHSA-qfg7-x243-5hg4, GHSA-v448-fmm4-52fp, 108a88a180, bb90f4ce83 | | T11 | Stored XSS leading to admin/account takeover via app HTML component, markdown, or S3 download content-type | remote_auth | EP12 | Admin session, accounts | high | likely | partially_mitigated | DOMPurify markdown sanitization, `X-Content-Type-Options: nosniff` + CSP sandbox on downloads | GHSA-9c5c-hh3c-r9mc, GHSA-qxj7-hpx3-r892, GHSA-cf2x-rg8c-v63v, bb78b1c06d, 625b67dff0 | diff --git a/debugger/dap_debug_service.ts b/debugger/dap_debug_service.ts index 0486814ff3..2b9ce46272 100644 --- a/debugger/dap_debug_service.ts +++ b/debugger/dap_debug_service.ts @@ -149,6 +149,22 @@ const logger = { const WINDMILL_BASE_URL = process.env.WINDMILL_BASE_URL || process.env.BASE_INTERNAL_URL // e.g., http://localhost:8000 const REQUIRE_SIGNED_REQUESTS = process.env.REQUIRE_SIGNED_DEBUG_REQUESTS !== 'false' +// Opt-in cross-origin protection (CSWSH defense-in-depth). When +// DEBUG_ALLOWED_ORIGINS is set (comma-separated list of origins), browser +// requests carrying a non-matching Origin header are rejected at the +// handshake. Non-browser clients send no Origin and are unaffected; code +// execution is independently gated by signed-token verification on launch. +const ALLOWED_ORIGINS = (process.env.DEBUG_ALLOWED_ORIGINS || '') + .split(',') + .map(o => o.trim()) + .filter(Boolean) + +function isOriginRejected(req: Request): boolean { + const origin = req.headers.get('origin') + if (!origin || ALLOWED_ORIGINS.length === 0) return false + return !ALLOWED_ORIGINS.includes(origin) +} + interface JWK { kty: string crv: string @@ -897,9 +913,18 @@ class PythonDebugSession extends BaseDebugSession { this.mainArgs = (args.args as Record) || {} this.envVars = (args.env as Record) || {} - // Verify JWT token if code is provided (signed debug request) - // The token is passed in the launch arguments - if (code && REQUIRE_SIGNED_REQUESTS) { + // Enforce signing on every launch. The token is passed in the launch + // arguments and is verified against the inline `code` (see windmill-api-debug). + if (REQUIRE_SIGNED_REQUESTS) { + // The backend only signs inline `code`; a `program`-mode launch names an + // arbitrary server-side file path that gets read and executed and is never + // signed. Refuse it so it cannot bypass token verification entirely. + if (this.scriptPath) { + logger.error('Rejected program-mode launch: only signed inline code is permitted') + this.sendResponse(request, false, {}, 'program-mode launch is not permitted; submit signed code instead') + return + } + const token = args.token as string | undefined if (!token) { logger.error('No debug token provided but signed requests are required') @@ -907,7 +932,7 @@ class PythonDebugSession extends BaseDebugSession { return } - const verificationError = await verifyDebugToken(token, code) + const verificationError = await verifyDebugToken(token, code ?? '') if (verificationError) { logger.error(`Token verification failed: ${verificationError}`) this.sendResponse(request, false, {}, `Token verification failed: ${verificationError}`) @@ -1067,6 +1092,11 @@ const server = Bun.serve({ const url = new URL(req.url) const path = url.pathname + if (isOriginRejected(req)) { + logger.warn(`Rejected request from disallowed origin: ${req.headers.get('origin')}`) + return new Response('Forbidden origin', { status: 403 }) + } + // Handle WebSocket upgrade with path-based routing if (server.upgrade(req, { data: { path } })) { logger.info(`WS upgrade: ${path}`) diff --git a/debugger/dap_websocket_server_bun.ts b/debugger/dap_websocket_server_bun.ts index 9caa0d4e61..d6ed8967ad 100644 --- a/debugger/dap_websocket_server_bun.ts +++ b/debugger/dap_websocket_server_bun.ts @@ -222,6 +222,21 @@ function generateMainCallArgs(code: string, args: Record): stri const WINDMILL_BASE_URL = process.env.WINDMILL_BASE_URL || process.env.BASE_INTERNAL_URL // e.g., http://localhost:8000 const REQUIRE_SIGNED_REQUESTS = process.env.REQUIRE_SIGNED_DEBUG_REQUESTS !== 'false' +// Opt-in cross-origin protection (CSWSH defense-in-depth); see +// dap_debug_service.ts for the rationale. Only enforced for this file's +// standalone Bun.serve entrypoint (the windmill-extra runtime imports the +// DebugSession class and runs the guarded server in dap_debug_service.ts). +const ALLOWED_ORIGINS = (process.env.DEBUG_ALLOWED_ORIGINS || '') + .split(',') + .map(o => o.trim()) + .filter(Boolean) + +function isOriginRejected(req: Request): boolean { + const origin = req.headers.get('origin') + if (!origin || ALLOWED_ORIGINS.length === 0) return false + return !ALLOWED_ORIGINS.includes(origin) +} + interface JWK { kty: string crv: string @@ -1428,9 +1443,18 @@ export class DebugSession { this.mainArgs = (args.args as Record) || {} this.envVars = (args.env as Record) || {} - // Verify JWT token if code is provided (signed debug request) - // The token is passed in the launch arguments - if (code && REQUIRE_SIGNED_REQUESTS) { + // Enforce signing on every launch. The token is passed in the launch + // arguments and is verified against the inline `code` (see windmill-api-debug). + if (REQUIRE_SIGNED_REQUESTS) { + // The backend only signs inline `code`; a `program`-mode launch names an + // arbitrary server-side file path that gets read and executed and is never + // signed. Refuse it so it cannot bypass token verification entirely. + if (this.scriptPath) { + logger.error('Rejected program-mode launch: only signed inline code is permitted') + this.sendResponse(request, false, {}, 'program-mode launch is not permitted; submit signed code instead') + return + } + const token = args.token as string | undefined if (!token) { logger.error('No debug token provided but signed requests are required') @@ -1438,7 +1462,7 @@ export class DebugSession { return } - const verificationError = await verifyDebugToken(token, code) + const verificationError = await verifyDebugToken(token, code ?? '') if (verificationError) { logger.error(`Token verification failed: ${verificationError}`) this.sendResponse(request, false, {}, `Token verification failed: ${verificationError}`) @@ -2535,6 +2559,10 @@ if (import.meta.main) { hostname: host, port, fetch(req, server) { + if (isOriginRejected(req)) { + logger.warn(`Rejected request from disallowed origin: ${req.headers.get('origin')}`) + return new Response('Forbidden origin', { status: 403 }) + } // Upgrade to WebSocket if (server.upgrade(req)) { return undefined as unknown as Response diff --git a/docker-compose.yml b/docker-compose.yml index 8b30479481..60be2185b0 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -182,8 +182,9 @@ services: - ENABLE_DEBUGGER=true # Set to true to enable debugger - DEBUGGER_PORT=3003 # Debugger service port - ENABLE_NSJAIL=false # Set to true for nsjail sandboxing (requires privileged: true) - - REQUIRE_SIGNED_DEBUG_REQUESTS=false # Set to true to require JWT tokens for debug sessions + - REQUIRE_SIGNED_DEBUG_REQUESTS=true # Require backend-signed JWT tokens for debug sessions. Do NOT set to false on any internet-reachable deployment: it exposes an unauthenticated code-execution debugger. - WINDMILL_BASE_URL=http://windmill_server:8000 + # - DEBUG_ALLOWED_ORIGINS=https://your-windmill-host # Optional CSWSH hardening: comma-separated allowlist of browser Origins permitted to open debug WebSockets volumes: - lsp_cache:/pyls/.cache logging: *default-logging From 75ba81b2d27fb0722095780312064cb93d20287e Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Sun, 28 Jun 2026 14:21:38 +0200 Subject: [PATCH 116/117] fix(audit): don't read pg_authid from an elevated context in S3 export migration (#9832) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(audit): don't read pg_authid from an elevated context in S3 export migration Migration 20260626132251 aborted instance startup on managed Postgres (e.g. Cloud SQL) with "Modifying pg_authid or pg_auth_members is not allowed in elevated context": the audit S3 export "oldest in-flight xact_start" floor probe calls pg_has_role(...), which reads pg_authid, and managed providers forbid that read from an elevated context. The migration ran the probe inline in its UPDATE, so the whole migration — and the instance boot — failed. Extract the probe into a shared SQL function audit_logs_s3_oldest_inflight_ts() that returns the oldest in-flight xact_start (when cluster-wide stats are visible) or NULL otherwise. The pg_has_role read is wrapped in a plpgsql BEGIN/EXCEPTION subtransaction, so a pg_authid failure returns NULL (callers fall back to a conservative 7-day window / reject) instead of aborting. is_superuser (a GUC, no catalog read) is checked first to short-circuit. The migration's trigger and UPDATE, the OSS backfill try_start, and the EE exporter/startup anchor (companion windmill-ee-private PR) all route through it. Because 20260626132251 already shipped, it is added to the potentially_stale list in windmill-api/src/db.rs: on startup the stale _sqlx_migrations row (checksum mismatch) is deleted and the fixed, idempotent migration re-applies, so already-migrated instances upgrade without a checksum-mismatch boot failure. Fixes WIN-2108 Co-Authored-By: Claude Opus 4.8 (1M context) * chore: update ee-repo-ref to 95352c13c4c82247d8cfd80936f9203aeb079802 This commit updates the EE repository reference after PR #635 was merged in windmill-ee-private. Previous ee-repo-ref: 136f49a52af922868acac33abf8198913a9e835c New ee-repo-ref: 95352c13c4c82247d8cfd80936f9203aeb079802 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: windmill-internal-app[bot] --- ...100e926bb678171947e71d6d87fdd0c3f9299.json | 15 ------ ...01699bf9968a02675f9f598c97a19c6df089b.json | 20 -------- ...41b344a1710715fb230ac97aca875a9ef38a5.json | 20 ++++++++ ...f172c89da5b216d3530c2743319ab1f4ca853.json | 20 -------- ...fb76daed7d908be80f664a8a30e9896ab1f7e.json | 20 ++++++++ ...95eadc68642592e3c56d4bf0b89b6440869b1.json | 15 ++++++ backend/ee-repo-ref.txt | 2 +- ..._audit_logs_s3_reanchor_on_enable.down.sql | 2 + ...51_audit_logs_s3_reanchor_on_enable.up.sql | 50 ++++++++++++------- .../src/audit_logs_s3_backfill.rs | 23 ++++----- backend/windmill-api/src/db.rs | 6 +++ 11 files changed, 108 insertions(+), 85 deletions(-) delete mode 100644 backend/.sqlx/query-3accb7e0eab75fcd34bf5b6d75e100e926bb678171947e71d6d87fdd0c3f9299.json delete mode 100644 backend/.sqlx/query-5d84f1ede2fbe09923a36d80d7c01699bf9968a02675f9f598c97a19c6df089b.json create mode 100644 backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json delete mode 100644 backend/.sqlx/query-a54f686b1bfb16e4e1da2bc143ef172c89da5b216d3530c2743319ab1f4ca853.json create mode 100644 backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json create mode 100644 backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json diff --git a/backend/.sqlx/query-3accb7e0eab75fcd34bf5b6d75e100e926bb678171947e71d6d87fdd0c3f9299.json b/backend/.sqlx/query-3accb7e0eab75fcd34bf5b6d75e100e926bb678171947e71d6d87fdd0c3f9299.json deleted file mode 100644 index 54db7ff310..0000000000 --- a/backend/.sqlx/query-3accb7e0eab75fcd34bf5b6d75e100e926bb678171947e71d6d87fdd0c3f9299.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO background_task_state (name, value)\n SELECT $1, jsonb_build_object(\n 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint,\n 'last_ts', now(),\n 'last_oldest_inflight_ts', COALESCE(\n CASE WHEN (current_setting('is_superuser') = 'on'\n OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'))\n AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts)\n THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL)\n ELSE NULL END,\n now() - interval '7 days'))\n WHERE NOT EXISTS (SELECT 1 FROM global_settings WHERE name = $2)\n ON CONFLICT (name) DO NOTHING", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "3accb7e0eab75fcd34bf5b6d75e100e926bb678171947e71d6d87fdd0c3f9299" -} diff --git a/backend/.sqlx/query-5d84f1ede2fbe09923a36d80d7c01699bf9968a02675f9f598c97a19c6df089b.json b/backend/.sqlx/query-5d84f1ede2fbe09923a36d80d7c01699bf9968a02675f9f598c97a19c6df089b.json deleted file mode 100644 index ae27ba252d..0000000000 --- a/backend/.sqlx/query-5d84f1ede2fbe09923a36d80d7c01699bf9968a02675f9f598c97a19c6df089b.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT CASE WHEN (current_setting('is_superuser') = 'on'\n OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'))\n AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts)\n THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL)\n ELSE NULL END AS \"x\"", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "x", - "type_info": "Timestamptz" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - null - ] - }, - "hash": "5d84f1ede2fbe09923a36d80d7c01699bf9968a02675f9f598c97a19c6df089b" -} diff --git a/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json b/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json new file mode 100644 index 0000000000..f909a33b4c --- /dev/null +++ b/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT audit_logs_s3_oldest_inflight_ts() AS \"x\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "x", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5" +} diff --git a/backend/.sqlx/query-a54f686b1bfb16e4e1da2bc143ef172c89da5b216d3530c2743319ab1f4ca853.json b/backend/.sqlx/query-a54f686b1bfb16e4e1da2bc143ef172c89da5b216d3530c2743319ab1f4ca853.json deleted file mode 100644 index 4c0d0ae14f..0000000000 --- a/backend/.sqlx/query-a54f686b1bfb16e4e1da2bc143ef172c89da5b216d3530c2743319ab1f4ca853.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT CASE WHEN (current_setting('is_superuser') = 'on'\n OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'))\n AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts)\n THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL)\n ELSE NULL END AS \"cutoff?\"", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "cutoff?", - "type_info": "Timestamptz" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - null - ] - }, - "hash": "a54f686b1bfb16e4e1da2bc143ef172c89da5b216d3530c2743319ab1f4ca853" -} diff --git a/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json b/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json new file mode 100644 index 0000000000..dc82db7591 --- /dev/null +++ b/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT audit_logs_s3_oldest_inflight_ts() AS \"cutoff?\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "cutoff?", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e" +} diff --git a/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json b/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json new file mode 100644 index 0000000000..2c775af0b1 --- /dev/null +++ b/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO background_task_state (name, value)\n SELECT $1, jsonb_build_object(\n 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint,\n 'last_ts', now(),\n 'last_oldest_inflight_ts',\n COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days'))\n WHERE NOT EXISTS (SELECT 1 FROM global_settings WHERE name = $2)\n ON CONFLICT (name) DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1" +} diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index 294707d466..ce3db8ebcb 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -b821fecccbcba2efed544890576bf2b84321d70d +95352c13c4c82247d8cfd80936f9203aeb079802 diff --git a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql index 2c9e78659e..1ce2c852b7 100644 --- a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql +++ b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql @@ -17,3 +17,5 @@ BEGIN RETURN NEW; END; $$ LANGUAGE plpgsql; + +DROP FUNCTION IF EXISTS audit_logs_s3_oldest_inflight_ts(); diff --git a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql index 3fd759da9e..ff2ed110c6 100644 --- a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql +++ b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql @@ -17,27 +17,48 @@ -- The task name literal must match -- `windmill_common::global_settings::AUDIT_LOGS_S3_EXPORT_TASK`. +-- Oldest in-flight `xact_start` when this role can observe *all* sessions (so the +-- min is a true cluster-wide bound), else NULL — callers substitute a conservative +-- window. The stats-visibility check goes through `is_superuser` (a preset GUC, no +-- catalog read) first, then a best-effort `pg_has_role` probe guarded by EXCEPTION: +-- `pg_has_role` reads `pg_authid`, which some managed providers (e.g. Cloud SQL) +-- forbid even to read from an elevated context, raising "Modifying pg_authid or +-- pg_auth_members is not allowed in elevated context". The EXCEPTION block runs in +-- its own subtransaction, so a failure there returns NULL (→ conservative fallback) +-- without aborting the caller — the migration-time UPDATE below, the trigger, or the +-- export task, none of which must fail just because the optimization is unavailable. +CREATE OR REPLACE FUNCTION audit_logs_s3_oldest_inflight_ts() +RETURNS timestamptz AS $$ +DECLARE + v_can_read_all_stats boolean := current_setting('is_superuser') = 'on'; +BEGIN + IF NOT v_can_read_all_stats THEN + BEGIN + v_can_read_all_stats := pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'); + EXCEPTION WHEN OTHERS THEN + v_can_read_all_stats := false; + END; + END IF; + IF v_can_read_all_stats AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts) THEN + RETURN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL); + END IF; + RETURN NULL; +END; +$$ LANGUAGE plpgsql; + CREATE OR REPLACE FUNCTION audit_logs_s3_anchor_on_enable() RETURNS TRIGGER AS $$ -DECLARE - v_floor timestamptz; BEGIN IF NEW.value = to_jsonb(true) AND (TG_OP = 'INSERT' OR OLD.value IS DISTINCT FROM NEW.value) THEN - v_floor := COALESCE( - CASE WHEN (current_setting('is_superuser') = 'on' - OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE')) - AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts) - THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL) - ELSE NULL END, - now() - interval '7 days'); INSERT INTO background_task_state (name, value) VALUES ( 'audit_logs_s3_export', jsonb_build_object( 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, 'last_ts', now(), - 'last_oldest_inflight_ts', v_floor + 'last_oldest_inflight_ts', + COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days') ) ) ON CONFLICT (name) DO UPDATE @@ -59,12 +80,7 @@ UPDATE background_task_state SET value = jsonb_build_object( 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, 'last_ts', to_jsonb(now()), - 'last_oldest_inflight_ts', to_jsonb(COALESCE( - CASE WHEN (current_setting('is_superuser') = 'on' - OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE')) - AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts) - THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL) - ELSE NULL END, - now() - interval '7 days'))) + 'last_oldest_inflight_ts', + to_jsonb(COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days'))) WHERE name = 'audit_logs_s3_export' AND (value->>'last_ts')::timestamptz <= 'epoch'::timestamptz; diff --git a/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs b/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs index e86ceb6448..0c5c55f7ed 100644 --- a/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs +++ b/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs @@ -186,18 +186,17 @@ pub async fn try_start(db: &DB, from: DateTime, to: DateTime) -> error // without pg_read_all_stats/superuser sees only its own sessions, and a prepared // (2PC) transaction is invisible to pg_stat_activity — in either case an old // transaction could still commit rows inside an accepted window after our scan ends. - // Since a backfill asserts completeness, we REJECT in those cases rather than fall - // back to a best-effort margin (NULL below). (The continuous exporter, which only - // claims bounded lag, keeps the 7-day fallback instead.) - let settled_cutoff: Option> = sqlx::query_scalar!( - r#"SELECT CASE WHEN (current_setting('is_superuser') = 'on' - OR pg_has_role(current_user, 'pg_read_all_stats', 'USAGE')) - AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts) - THEN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL) - ELSE NULL END AS "cutoff?""# - ) - .fetch_one(db) - .await?; + // Since a backfill asserts completeness, we REJECT in those cases (the function + // returns NULL). (The continuous exporter, which only claims bounded lag, keeps the + // 7-day fallback instead.) The probe lives in the `audit_logs_s3_oldest_inflight_ts()` + // SQL function (migration 20260626132251) so its `pg_has_role`/`pg_authid` read is + // wrapped in a subtransaction EXCEPTION: managed providers (e.g. Cloud SQL) forbid + // reading pg_authid from an elevated context, which would otherwise surface here as + // an opaque error instead of NULL → the actionable rejection below. + let settled_cutoff: Option> = + sqlx::query_scalar!(r#"SELECT audit_logs_s3_oldest_inflight_ts() AS "cutoff?""#) + .fetch_one(db) + .await?; let Some(settled_cutoff) = settled_cutoff else { return Err(error::Error::BadRequest( "audit backfill: cannot determine a trustworthy settled-time boundary, so completeness \ diff --git a/backend/windmill-api/src/db.rs b/backend/windmill-api/src/db.rs index df6c8cc6b5..b81a9dbb84 100644 --- a/backend/windmill-api/src/db.rs +++ b/backend/windmill-api/src/db.rs @@ -294,6 +294,12 @@ pub async fn migrate( // idempotent, so re-applying on an already-migrated DB is a no-op. 20260423050000, 20260523055641, + // Reworked to stop reading pg_authid (via pg_has_role) from an elevated + // context, which managed providers (e.g. Cloud SQL) forbid — the original + // aborted startup. The new file is idempotent (CREATE OR REPLACE + an + // epoch-guarded UPDATE that no-ops once anchored), so re-applying on an + // already-migrated DB is safe. + 20260626132251, ]; for m in migrator.migrations.iter() { if m.migration_type.is_down_migration() { From 96c0ff65bddd90c3f306b4803a4322d2e1e064fc Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Sun, 28 Jun 2026 14:33:27 +0200 Subject: [PATCH 117/117] chore(main): release 1.742.0 (#9830) * chore(main): release 1.742.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> --- CHANGELOG.md | 15 ++ backend/Cargo.lock | 176 +++++++++--------- backend/Cargo.toml | 4 +- .../parsers/windmill-parser-wasm/Cargo.lock | 48 ++--- .../parsers/windmill-parser-wasm/Cargo.toml | 2 +- backend/windmill-api/openapi.yaml | 2 +- benchmarks/lib.ts | 2 +- cli/src/core/constants.ts | 2 +- frontend/package-lock.json | 54 +++++- frontend/package.json | 2 +- lsp/Pipfile | 2 +- openflow.openapi.yaml | 2 +- .../WindmillClient/WindmillClient.psd1 | 2 +- python-client/wmill/pyproject.toml | 2 +- typescript-client/jsr.json | 2 +- typescript-client/package.json | 2 +- version.txt | 2 +- 17 files changed, 191 insertions(+), 130 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f07f85012..361b6fbd91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## [1.742.0](https://github.com/windmill-labs/windmill/compare/v1.741.0...v1.742.0) (2026-06-28) + + +### Features + +* **apps:** add labels input to app editor deploy drawer ([#9828](https://github.com/windmill-labs/windmill/issues/9828)) ([da45e69](https://github.com/windmill-labs/windmill/commit/da45e699c8aefeede172c90769ef4f4b182fec0c)) +* column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) ([#9814](https://github.com/windmill-labs/windmill/issues/9814)) ([003a262](https://github.com/windmill-labs/windmill/commit/003a262a4e9d6c2a63ada01aa8429aea1fbb6031)) + + +### Bug Fixes + +* **audit:** don't read pg_authid from an elevated context in S3 export migration ([#9832](https://github.com/windmill-labs/windmill/issues/9832)) ([75ba81b](https://github.com/windmill-labs/windmill/commit/75ba81b2d27fb0722095780312064cb93d20287e)) +* close unauthenticated DAP debugger program-mode launch bypass ([#9829](https://github.com/windmill-labs/windmill/issues/9829)) ([c0768de](https://github.com/windmill-labs/windmill/commit/c0768de0acdf63eaba5fb97d04bfc64f2f03b93d)) +* redeploy older app version from deployment history ([#9826](https://github.com/windmill-labs/windmill/issues/9826)) ([c479afa](https://github.com/windmill-labs/windmill/commit/c479afab8ebceccbee050e923dc5c27a6712ea62)) + ## [1.741.0](https://github.com/windmill-labs/windmill/compare/v1.740.0...v1.741.0) (2026-06-26) diff --git a/backend/Cargo.lock b/backend/Cargo.lock index c970648000..7072389c21 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -1855,9 +1855,9 @@ dependencies = [ [[package]] name = "byte-unit" -version = "5.2.3" +version = "5.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37bcaa4a0975bed4a760af3efe4368825098ce5f9d37a30c5a021d635dc63d8f" +checksum = "4a813de7f2bbedb7dce265b64f1cf5908ebe4d56281ece8d847e98113788b9b0" dependencies = [ "rust_decimal", "schemars 1.2.1", @@ -4371,18 +4371,18 @@ dependencies = [ [[package]] name = "enum-ordinalize" -version = "4.3.2" +version = "4.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a1091a7bb1f8f2c4b28f1fe2cef4980ca2d410a3d727d67ecc3178c9b0800f0" +checksum = "07f808d588c10e464ea6f7d3eaed500049eff30aaac103460f61828c2d65b3eb" dependencies = [ "enum-ordinalize-derive", ] [[package]] name = "enum-ordinalize-derive" -version = "4.3.2" +version = "4.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ca9601fb2d62598ee17836250842873a413586e5d7ed88b356e38ddbb0ec631" +checksum = "42e528e2d34ba8a67a1a650b86beae8ef69fc5fdb638016f386b973226590432" dependencies = [ "proc-macro2", "quote", @@ -8962,13 +8962,13 @@ dependencies = [ [[package]] name = "quick_cache" -version = "0.6.24" +version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9c6658afe513a3b484e3abfdaa0d03ef3c0bbf017542c178dd55f94eb3051f9" +checksum = "403c1a912fec895cafb223201e368234842acb9220aaf08ab042ae89ba5f135c" dependencies = [ - "ahash 0.8.12", "equivalent", - "hashbrown 0.16.1", + "foldhash 0.2.0", + "hashbrown 0.17.1", "parking_lot", ] @@ -13734,7 +13734,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-nats", @@ -13816,7 +13816,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.741.0" +version = "1.742.0" dependencies = [ "async-stream", "async-trait", @@ -13849,7 +13849,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -13862,7 +13862,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "argon2", @@ -14000,7 +14000,7 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14023,7 +14023,7 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14038,7 +14038,7 @@ dependencies = [ [[package]] name = "windmill-api-auth" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14064,7 +14064,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.741.0" +version = "1.742.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14074,7 +14074,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14091,7 +14091,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14113,7 +14113,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14136,7 +14136,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14152,7 +14152,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14173,7 +14173,7 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14194,7 +14194,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14208,7 +14208,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-nats", @@ -14243,7 +14243,7 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14268,7 +14268,7 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "flate2", @@ -14286,7 +14286,7 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14308,7 +14308,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14328,7 +14328,7 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14365,7 +14365,7 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14393,7 +14393,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.741.0" +version = "1.742.0" dependencies = [ "lazy_static", "serde", @@ -14405,7 +14405,7 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.741.0" +version = "1.742.0" dependencies = [ "argon2", "axum 0.8.9", @@ -14430,7 +14430,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14444,7 +14444,7 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.741.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14477,7 +14477,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.741.0" +version = "1.742.0" dependencies = [ "chrono", "lazy_static", @@ -14491,7 +14491,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14510,7 +14510,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.741.0" +version = "1.742.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -14612,7 +14612,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.741.0" +version = "1.742.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -14631,7 +14631,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.741.0" +version = "1.742.0" dependencies = [ "regex", "serde", @@ -14646,7 +14646,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -14670,7 +14670,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "futures", @@ -14687,7 +14687,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.741.0" +version = "1.742.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -14703,7 +14703,7 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -14724,7 +14724,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -14755,7 +14755,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "arc-swap", @@ -14780,7 +14780,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-stream", @@ -14814,7 +14814,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "futures", @@ -14832,7 +14832,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.741.0" +version = "1.742.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -14841,7 +14841,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -14853,7 +14853,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -14865,7 +14865,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "gosyn", @@ -14877,7 +14877,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -14889,7 +14889,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -14901,7 +14901,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "nu-parser", @@ -14912,7 +14912,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14923,7 +14923,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14935,7 +14935,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -14946,7 +14946,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", @@ -14968,7 +14968,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -14980,7 +14980,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -14994,7 +14994,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15011,7 +15011,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -15024,7 +15024,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -15036,7 +15036,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -15054,7 +15054,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15070,7 +15070,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15086,7 +15086,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -15097,7 +15097,7 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", @@ -15136,7 +15136,7 @@ dependencies = [ [[package]] name = "windmill-runtime-nativets" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "const_format", @@ -15175,7 +15175,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.741.0" +version = "1.742.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15186,7 +15186,7 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", @@ -15220,7 +15220,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15244,7 +15244,7 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15277,7 +15277,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15310,7 +15310,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15330,7 +15330,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15364,7 +15364,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15400,7 +15400,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15423,7 +15423,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15447,7 +15447,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-nats", @@ -15471,7 +15471,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15506,7 +15506,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15534,7 +15534,7 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15559,7 +15559,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "bitflags 2.13.0", @@ -15578,7 +15578,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-once-cell", @@ -15688,7 +15688,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.741.0" +version = "1.742.0" dependencies = [ "bytes", "futures", diff --git a/backend/Cargo.toml b/backend/Cargo.toml index d721dcc272..45682215cf 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.741.0" +version = "1.742.0" authors.workspace = true edition.workspace = true @@ -87,7 +87,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.741.0" +version = "1.742.0" authors = ["Ruben Fiszel "] edition = "2021" diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index 78db9dd2b0..096d93163b 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.741.0" +version = "1.742.0" dependencies = [ "aho-corasick", "anyhow", @@ -6272,7 +6272,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.741.0" +version = "1.742.0" dependencies = [ "proc-macro2", "quote", @@ -6284,7 +6284,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.741.0" +version = "1.742.0" dependencies = [ "convert_case", "serde", @@ -6293,7 +6293,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6305,7 +6305,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -6317,7 +6317,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "gosyn", @@ -6329,7 +6329,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6341,7 +6341,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -6353,7 +6353,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "nu-parser", @@ -6364,7 +6364,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6375,7 +6375,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6387,7 +6387,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6398,7 +6398,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", @@ -6420,7 +6420,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -6432,7 +6432,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6446,7 +6446,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "convert_case", @@ -6463,7 +6463,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6476,7 +6476,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -6488,7 +6488,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6506,7 +6506,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6522,7 +6522,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6538,7 +6538,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6570,7 +6570,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -6581,7 +6581,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.741.0" +version = "1.742.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index 6faadc1cec..e2afd2483b 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.741.0" +version = "1.742.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index e9c57c3fb0..f9a15ef72d 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: - version: 1.741.0 + version: 1.742.0 title: Windmill API contact: diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index 2b3b23746a..f8cc748bdb 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.741.0"; +export const VERSION = "v1.742.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ diff --git a/cli/src/core/constants.ts b/cli/src/core/constants.ts index 1e40e04ac7..42d8109c49 100644 --- a/cli/src/core/constants.ts +++ b/cli/src/core/constants.ts @@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork"; // (e.g. utils.ts) can read it without importing main.ts and creating a circular // dependency (main → workspace → utils → main) that triggers a TDZ. // Re-exported from main.ts for backwards compatibility. -export const VERSION = "1.741.0"; +export const VERSION = "1.742.0"; diff --git a/frontend/package-lock.json b/frontend/package-lock.json index bd2619a54e..4a3fcf0696 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "@windmill-labs/components", - "version": "1.741.0", + "version": "1.742.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@windmill-labs/components", - "version": "1.741.0", + "version": "1.742.0", "hasInstallScript": true, "license": "AGPL-3.0", "dependencies": { @@ -878,6 +878,7 @@ "version": "1.10.0", "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.10.0.tgz", "integrity": "sha512-yq6OkJ4p82CAfPl0u9mQebQHKPJkY7WrIuk205cTYnYe+k2Z8YBh11FrbRG/H6ihirqcacOgl2BIO8oyMQLeXw==", + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -889,6 +890,7 @@ "version": "1.10.0", "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz", "integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==", + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -899,6 +901,7 @@ "version": "1.2.1", "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.1.tgz", "integrity": "sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w==", + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1414,6 +1417,7 @@ "version": "1.1.4", "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.4.tgz", "integrity": "sha512-3NQNNgA1YSlJb/kMH1ildASP9HW7/7kYnRI2szWJaofaS1hWmbGI4H+d3+22aGzXXN9IJ+n+GiFVcGipJP18ow==", + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1562,6 +1566,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1578,6 +1583,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1594,6 +1600,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1610,6 +1617,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1626,6 +1634,7 @@ "cpu": [ "arm" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1642,6 +1651,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1658,6 +1668,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1674,6 +1685,7 @@ "cpu": [ "ppc64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1690,6 +1702,7 @@ "cpu": [ "s390x" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1706,6 +1719,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1722,6 +1736,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1738,6 +1753,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1754,6 +1770,7 @@ "cpu": [ "wasm32" ], + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -1772,6 +1789,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -1788,6 +1806,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MIT", "optional": true, "os": [ @@ -2093,6 +2112,7 @@ "version": "0.10.2", "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.2.tgz", "integrity": "sha512-RoBvJ2X0wuKlWFIjrwffGw1IqZHKQqzIchKaadZZfnNpsAYp2mM0h36JtPCjNDAHGgYez/15uMBpfGwchhiMgg==", + "dev": true, "license": "MIT", "optional": true, "dependencies": { @@ -7328,7 +7348,7 @@ "version": "1.21.7", "resolved": "https://registry.npmjs.org/jiti/-/jiti-1.21.7.tgz", "integrity": "sha512-/imKNG4EbWNrVjoNC/1H5/9GFy+tqjGBHCaSsN+P2RnPqjsLmv6UD3Ej+Kj8nBWaRAwyk7kK5ZUc+OEatnTR3A==", - "devOptional": true, + "dev": true, "license": "MIT", "bin": { "jiti": "bin/jiti.js" @@ -7863,6 +7883,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7883,6 +7904,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7903,6 +7925,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7923,6 +7946,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7943,6 +7967,7 @@ "cpu": [ "arm" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7963,6 +7988,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -7983,6 +8009,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8003,6 +8030,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8023,6 +8051,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8043,6 +8072,7 @@ "cpu": [ "arm64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -8063,6 +8093,7 @@ "cpu": [ "x64" ], + "dev": true, "license": "MPL-2.0", "optional": true, "os": [ @@ -12750,6 +12781,21 @@ } } }, + "node_modules/svelte-check/node_modules/picomatch": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", + "integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", + "dev": true, + "license": "MIT", + "optional": true, + "peer": true, + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, "node_modules/svelte-eslint-parser": { "version": "0.43.0", "resolved": "https://registry.npmjs.org/svelte-eslint-parser/-/svelte-eslint-parser-0.43.0.tgz", @@ -13489,7 +13535,7 @@ "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", - "devOptional": true, + "dev": true, "license": "Apache-2.0", "bin": { "tsc": "bin/tsc", diff --git a/frontend/package.json b/frontend/package.json index 8bdf440a00..e9ec640b02 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "@windmill-labs/components", - "version": "1.741.0", + "version": "1.742.0", "scripts": { "dev": "vite dev", "dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev", diff --git a/lsp/Pipfile b/lsp/Pipfile index cda87ce8d7..7f9d00903a 100644 --- a/lsp/Pipfile +++ b/lsp/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -wmill = ">=1.741.0" +wmill = ">=1.742.0" sendgrid = "*" mysql-connector-python = "*" pymongo = "*" diff --git a/openflow.openapi.yaml b/openflow.openapi.yaml index 9ccfa3ceab..f4342930e9 100644 --- a/openflow.openapi.yaml +++ b/openflow.openapi.yaml @@ -1,7 +1,7 @@ openapi: '3.0.3' info: - version: 1.741.0 + version: 1.742.0 title: OpenFlow Spec contact: name: Ruben Fiszel diff --git a/powershell-client/WindmillClient/WindmillClient.psd1 b/powershell-client/WindmillClient/WindmillClient.psd1 index ebdfaa8215..e5b553e6f3 100644 --- a/powershell-client/WindmillClient/WindmillClient.psd1 +++ b/powershell-client/WindmillClient/WindmillClient.psd1 @@ -12,7 +12,7 @@ RootModule = 'WindmillClient.psm1' # Version number of this module. - ModuleVersion = '1.741.0' + ModuleVersion = '1.742.0' # Supported PSEditions # CompatiblePSEditions = @() diff --git a/python-client/wmill/pyproject.toml b/python-client/wmill/pyproject.toml index d0942b025c..5310b72896 100644 --- a/python-client/wmill/pyproject.toml +++ b/python-client/wmill/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "wmill" -version = "1.741.0" +version = "1.742.0" description = "A client library for accessing Windmill server wrapping the Windmill client API" license = "Apache-2.0" homepage = "https://windmill.dev" diff --git a/typescript-client/jsr.json b/typescript-client/jsr.json index 13d6388e30..7ae2023a7c 100644 --- a/typescript-client/jsr.json +++ b/typescript-client/jsr.json @@ -1,6 +1,6 @@ { "name": "@windmill/windmill", - "version": "1.741.0", + "version": "1.742.0", "exports": "./src/index.ts", "publish": { "exclude": ["!src", "./s3Types.ts", "./sqlUtils.ts", "./client.ts"] diff --git a/typescript-client/package.json b/typescript-client/package.json index 42ebc2096d..8edc248c45 100644 --- a/typescript-client/package.json +++ b/typescript-client/package.json @@ -1,7 +1,7 @@ { "name": "windmill-client", "description": "Windmill SDK client for browsers and Node.js", - "version": "1.741.0", + "version": "1.742.0", "author": "Ruben Fiszel", "license": "Apache 2.0", "homepage": "https://github.com/windmill-labs/windmill/tree/main/typescript-client#readme", diff --git a/version.txt b/version.txt index b27f19abcc..bee795f5f0 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.741.0 +1.742.0