Files
windmill/frontend/src/lib/components/S3FilePreview.svelte
T
hugocasaandClaude Opus 4.8 7a139ab23e feat(ai-chat): image attachments and agent raw-app screenshots (#10130)
* feat(ai-chat): add image attachments and agent raw-app screenshots

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(ai-chat): generalise take_screenshot fidelity caveat

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): keep compaction boundary on a displayed user message

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(raw-apps): count line boxes by vertical overlap, not rect count

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): enforce vision gating and bound image attachments

Refuse images on known text-only models instead of warning and sending them
anyway; cap input bytes before decode; keep clipboard text when it accompanies
a bitmap; don't queue a message whose images can't ride the plain-text queue.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* perf(ai-chat): trim take_screenshot schema and shrink the card's copy

Move the fidelity caveat from the tool def onto the tool result: the def is
re-sent every global iteration (~258 tok), while the caveat only matters once a
capture exists. Keep a downscaled copy in displayMessages when it is actually
smaller — those are never compacted and are re-cloned on every saveChat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): carry attached images through the message queue

Enter during a streaming turn queued the text and silently dropped the images,
so the auto-send was not the message the user submitted. The queue now holds
both, moved together via takeQueue/clearQueue/restoreQueue so none of the three
flush sites, the dequeue-to-composer path, or the two conversation-switch drops
can leak one without the other.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): gate screenshots on vision, narrow when the tool fires

take_screenshot buffered an image unconditionally, so a text-only model got an
image_url and rejected the turn; the attach-time check never covered it, nor a
model switched after attaching. Gate before capture and again at send.

Only reach for the tool when the user raises how the app looks, rather than
after every UI edit.

A collapsed preview keeps the iframe mounted at zero width, passing the ready
checks and then failing inside the rasteriser as '[object Event]'. Name it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): hold sending while attachments decode

addImages read the free-slot count before its await and appended after it, so a
send during the ~50-800ms decode cleared images while the closure still wrote to
them, landing the picture on the following message; two drops also claimed the
same slots and could pass the cap. Reserve slots up front, block sending until
they resolve, and show a placeholder so the held send is explained.

Keep only a bounded copy in the transcript: displayMessages are never compacted
and are re-cloned on every save. Measured 6.1x smaller per attachment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): route screenshots to the visible tab, resend full-res on retry

Every mounted raw-app editor claimed the runtime's single screenshot slot, so
take_screenshot could capture a background tab's app; ownership now follows the
visible tab and only the owner releases it.

restartGeneration resent displayMessages' images, which became a 384px thumbnail
when the transcript copy was bounded — retries downgraded the model's own input.
Recover the sent parts from the API message instead.

Move modelSupportsVision to modelConfig: it was untestable behind lib.ts's monaco
import chain, and the denylist missed bundled text-only defaults (Groq/Together
Llama 3.3, Foundry Phi-4 and Mistral-Large). Llama 3.2 and Phi-4 split by variant,
so both are matched narrowly. Pinned against the shipped defaultModels.

Decode attachments one at a time and derive the preview from the bounded copy: a
12MP bitmap is ~48MB and the batch was held live at once, decoded twice each.

The attach tooltip claimed nothing is uploaded, which is untrue for images.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): keep images out of text-only turns and bound the queue

The vision gate only dropped the current turn's images, so history's image parts
still went out after a switch to a text-only model and failed the request; strip
the outbound copy instead, leaving history intact for a switch back.

queueMessage had no cap, and each queued send clears the composer for another
eight, so repeated sends stacked an unbounded batch into one message.

Editing a message resent displayMessages' bounded copy, downgrading the model's
own input; retries recovered the full-size one but then re-persisted it at full
resolution. storedImages pairs the API message with its transcript entry so both
paths resend the original and re-persist the bounded copy.

Reserve image slots before awaiting text attachments: the gap left sending
enabled with an image pending, measured ~90ms for a 40-file drop, now ~8ms
regardless of batch size.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): treat deepseek-v4 as text-only

deepseek-v4-pro ships as a bundled default and the gate let images through to
it, so an attachment would fail the turn. DeepSeek's vision line is deepseek-vl.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): drop a rejected image instead of wedging the conversation

A provider that refuses an image leaves it in history, so every later turn
resends it and fails identically: the chat is stuck until the user edits the
message or starts over, and Retry re-sends the same image. The vision gate only
knows the models we ship, so this is the net for the rest. Strip the parts on an
image-related rejection and say so; unrelated failures keep the image.

Verified at the wire that no provider rejects a base64 data URL: anthropic
(source.base64), openai/gpt-4o (input_image), googleai and aws_bedrock/claude
(image_url passthrough) all 200 and read the image.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): match text-only models exactly, from provider API docs

The gate guessed by substring over model names, which answers the wrong
question. What matters is whether a provider's API accepts image parts, not
whether the model can see: DeepSeek V4 ships vision in its chat product that its
API has no content type for, and o3-mini gained vision in ChatGPT the API never
exposed. Neither is inferable from a name.

Substrings also block working models. 'mistral-large' matches Mistral Large 3,
which takes images; 'phi-4' matches Phi-4-multimodal, which does too. A wrong
entry blocks with no override, while a missing one costs a turn and recovers via
the rejection path, so the list is now exact ids only, each backed by a provider
doc.

Verdicts verified against provider API docs rather than recall. Live-checked
where a doc was contradicted: Bedrock's compatibility matrix claims no Anthropic
model is served over chat completions, but it serves images fine.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): stop retry resurrecting a rejected image

The rejection fallback strips the image from history but leaves the bubble's
thumbnail so the user can still see what they sent. storedImages fell back to
that thumbnail when the API message had no parts, so Retry re-attached the very
image the provider had just refused and failed identically — the conversation
stayed wedged through the one control offered to escape it.

Found by retrying in the UI; unit tests, wire tests and four review passes all
missed it, since it only exists between two separate fixes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ai-chat): harden image rejection recovery and drop-path attachment

* fix(ai-chat): fix image drop race, mid-turn vision gate, retry aliasing

* fix(ai-chat): key vision denylist by provider, flatten alpha before jpeg

* feat(ai-chat): offer take_screenshot on chromium only, ask for one elsewhere

* feat(ai-chat): image-only sends and click-to-expand image previews

* fix(ai-chat): capture screenshots at 2x and expand tool images full-res

* feat(frontend): expandable image previews in composer and result views

* fix(ai-chat): image-only send edge cases from review round

* fix(ai-chat): keep image-only drafts on rollback, track failing model id

* fix(ai-chat): gate rejection recovery on the failing iteration's model

* refactor(ai-chat): record iteration model via onBeforeIteration, trim tests

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): restore composer draft when beforeSend preflight fails

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): bound cumulative outbound image bytes per request

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): make the image byte bound part-granular so over-cap turns keep a subset

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): evict newest-first within a message in the image byte bound

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): prune over-cap images from stored history, not just requests

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): bound history at every save boundary, keep thumbnail pairing across eviction

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): slot-align storedImages so the bubble expands the right image after eviction

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): match rejection keywords as whole words so provisioning errors keep images

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): match input_image rejections, restore images refused by non-GLOBAL modes

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): defer non-GLOBAL image refusal restore past the composer clear

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): persist full tool screenshots for post-reload expansion

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(ai-chat): persist chat images out-of-band via blob-store refs

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): scope image blobs per chat and stop cap-eviction rotation

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): keep blob-cap chronology across drop-oldest compaction

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(ai-chat): derive blob eviction from the saved record, not write times

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): serialize chat history DB writes per manager

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): pin queued history writes to the enqueue-time user database

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): delete stale image blobs only after the chat record commits

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): don't double-restore a queued image-only draft on vision refusal

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): label image-only chats and evicted image-only bubbles

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): keep the in-memory chat mirror hydrated for DB-less sessions

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): converge the chat mirror to refs after a successful DB commit

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): guard mirror convergence against rewinding newer saves

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): invalidate pending convergences on identity re-init, keep retry image names

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-chat): bound the screenshot raster before rasterization

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(ai-chat): drop the no-IndexedDB in-memory image fallback

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 19:36:51 +02:00

304 lines
11 KiB
Svelte

<script lang="ts">
// Standalone preview for an S3-backed object. Extracted from
// S3FilePickerInner so the asset detail pane (and any other surface that
// wants to peek at S3 contents) can render the same image/PDF/CSV/text
// renderings without dragging in the whole picker UI.
//
// The component owns its own metadata + preview loads — callers only
// need to hand it a `fileKey`. CSV separator/header are local state so
// the user can re-preview the same file with different parsing flags.
import { FileX2, Loader2 } from 'lucide-svelte'
import { workspaceStore } from '$lib/stores'
import {
HelpersService,
type CancelablePromise,
type LoadFileMetadataData,
type LoadFileMetadataResponse,
type LoadFilePreviewData,
type LoadFilePreviewResponse
} from '$lib/gen'
import { displayDate, displaySize, emptyString } from '$lib/utils'
import { twMerge } from 'tailwind-merge'
import ExpandableImage from '$lib/components/common/image/ExpandableImage.svelte'
interface Props {
fileKey: string | undefined
// Optional override of the storage backend (matches the picker's
// `storage` prop). Empty/undefined uses the workspace default.
storage?: string | undefined
// Override hooks for non-default backends (e.g. workspace settings
// preview before the storage is committed). Default to the standard
// helpers service — same defaults as S3FilePickerInner.
loadFilePreviewRequest?: (d: LoadFilePreviewData) => CancelablePromise<LoadFilePreviewResponse>
loadFileMetadataRequest?: (
d: LoadFileMetadataData
) => CancelablePromise<LoadFileMetadataResponse>
// When true, surface a small metadata strip above the preview
// (size, last-modified, mime). Off by default so callers can lay
// metadata out themselves (the picker already has its own).
showMetadata?: boolean
class?: string
// Bump this to force a re-fetch (metadata + preview). Used by the
// asset detail pane after an upstream producer run completes —
// without it, the "Asset not yet materialized" empty state stays
// pinned until the user re-selects the asset.
refreshKey?: any
}
let {
fileKey,
storage = undefined,
loadFilePreviewRequest = HelpersService.loadFilePreview,
loadFileMetadataRequest = HelpersService.loadFileMetadata,
showMetadata = false,
class: className = '',
refreshKey
}: Props = $props()
let csvSeparatorChar: string = $state(',')
let csvHasHeader: boolean = $state(true)
let fileMetadata = $state<
| {
fileKey: string
mimeType: string | undefined
size: number | undefined
sizeStr: string | undefined
lastModified: string | undefined
}
| undefined
>(undefined)
let filePreview = $state<
| {
fileKey: string
contentPreview: string | undefined
contentType: string | undefined
}
| undefined
>(undefined)
let filePreviewLoading = $state(false)
let fileInfoLoading = $state(false)
// Distinct error states. `notFound` is the common case for assets that
// have been declared but never actually materialized (e.g. a fresh
// pipeline whose producer hasn't run yet) — surfaced as a calm empty
// state rather than a scary HTTP error. `loadError` is everything else
// (auth failures, transient S3 hiccups, malformed keys) and shows the
// raw message so users can debug.
let notFound = $state(false)
let loadError = $state<string | undefined>(undefined)
function isNotFoundError(err: any): boolean {
// HelpersService surfaces backend errors as ApiError with a `status`
// field plus a serialized body. We accept either a 404 status or a
// "not found" substring (case-insensitive) to be robust against
// future error wrapping changes.
const status = err?.status ?? err?.response?.status
if (status === 404) return true
const body = String(err?.body ?? err?.message ?? err ?? '').toLowerCase()
return body.includes('not found') || body.includes('404')
}
// Reload whenever the file key, workspace, or external refreshKey
// changes. The refreshKey path is what lets the asset pane re-check
// existence after an upstream run completes — moving from the
// "not yet materialized" empty state to the actual preview without
// requiring the user to re-click the asset.
$effect(() => {
const key = fileKey
const ws = $workspaceStore
void refreshKey
if (!key || !ws) {
fileMetadata = undefined
filePreview = undefined
notFound = false
loadError = undefined
return
}
void loadAll(key)
})
async function loadAll(key: string) {
if (emptyString(key)) {
fileInfoLoading = false
return
}
fileInfoLoading = true
filePreview = undefined
fileMetadata = undefined
notFound = false
loadError = undefined
try {
const meta = await loadFileMetadataRequest({
workspace: $workspaceStore!,
fileKey: key,
storage
})
if (meta !== undefined) {
fileMetadata = {
fileKey: key,
size: meta.size_in_bytes,
sizeStr: displaySize(meta.size_in_bytes),
mimeType: meta.mime_type,
lastModified: displayDate(meta.last_modified)
}
}
await reloadPreview(key, meta?.size_in_bytes, meta?.mime_type)
} catch (err: any) {
if (isNotFoundError(err)) {
notFound = true
} else {
loadError = err?.body ?? err?.message ?? String(err)
}
} finally {
fileInfoLoading = false
}
}
async function reloadPreview(key: string, size?: number, mimeType?: string) {
filePreviewLoading = true
try {
const raw = await loadFilePreviewRequest({
workspace: $workspaceStore!,
fileKey: key,
fileSizeInBytes: size,
fileMimeType: mimeType,
csvSeparator: csvSeparatorChar,
csvHasHeader: csvHasHeader,
readBytesFrom: 0,
readBytesLength: 128 * 1024,
storage
})
let content = raw.content
if (content !== null && content !== undefined && content.length >= 128 * 1024) {
content = content.substring(0, 128 * 1024 - 35) + '\n\n ... FILE CONTENT TRUNCATED ...\n\n'
}
if (raw !== undefined) {
filePreview = { fileKey: key, contentPreview: content, contentType: raw.content_type }
if (fileMetadata) {
fileMetadata.mimeType =
((key.endsWith('.png') ||
key.endsWith('.jpg') ||
key.endsWith('.jpeg') ||
key.endsWith('.webp')) &&
'Image') ||
(key.endsWith('.pdf') && 'PDF') ||
filePreview.contentType
}
}
} finally {
filePreviewLoading = false
}
}
// `storage` is keyed by the workspace's S3 storage config name — used as
// a query-string suffix on the image/PDF preview URLs.
let storageQS = $derived(storage ? `&storage=${storage}` : '')
function onCsvControlsChanged() {
if (fileMetadata?.fileKey) {
void reloadPreview(fileMetadata.fileKey, fileMetadata.size, fileMetadata.mimeType)
}
}
</script>
<div class={twMerge('flex flex-col h-full w-full overflow-auto text-xs', className)}>
{#if showMetadata && fileMetadata}
<div class="text-2xs text-tertiary px-3 py-1.5 border-b flex flex-wrap gap-x-3 gap-y-0.5">
{#if fileMetadata.sizeStr}<span>{fileMetadata.sizeStr}</span>{/if}
{#if fileMetadata.mimeType}<span>{fileMetadata.mimeType}</span>{/if}
{#if fileMetadata.lastModified}<span>{fileMetadata.lastModified}</span>{/if}
</div>
{/if}
<div class="flex-1 min-h-0 overflow-auto p-4 bg-surface-secondary">
{#if !fileKey}
<div class="text-tertiary text-xs">No file selected.</div>
{:else if fileInfoLoading && !fileMetadata && !notFound && !loadError}
<div class="flex items-center text-primary gap-1">
<Loader2 size={12} class="animate-spin" /> Loading…
</div>
{:else if notFound}
<!-- Calm empty state for assets that have been declared (e.g. by a
`// out s3://...` annotation or parsed write) but never
actually written. Common during pipeline authoring before
any producer has run. We surface the file key so users can
verify they're looking at the right place. -->
<div class="flex flex-col items-center justify-center text-center gap-2 py-8">
<div
class="flex h-10 w-10 items-center justify-center rounded-full bg-surface-tertiary text-tertiary"
>
<FileX2 size={18} />
</div>
<div class="text-sm font-medium text-primary">Asset not yet materialized</div>
<div class="text-2xs text-tertiary max-w-sm">
This asset has been declared in the pipeline but no producer has written to it yet. Run an
upstream script to populate it.
</div>
<div class="text-3xs font-mono text-tertiary mt-1 break-all max-w-md">{fileKey}</div>
</div>
{:else if loadError}
<!-- Non-404 errors: still structured but show the message so
users can debug auth / connectivity / malformed-key issues. -->
<div class="flex flex-col items-start gap-1 py-2">
<div class="text-sm font-medium text-red-700 dark:text-red-400">Could not load asset</div>
<div class="text-2xs text-tertiary break-all">{loadError}</div>
</div>
{:else if fileMetadata?.fileKey.endsWith('.png') || fileMetadata?.fileKey.endsWith('.jpg') || fileMetadata?.fileKey.endsWith('.jpeg') || fileMetadata?.fileKey.endsWith('.webp')}
<div>
<ExpandableImage
src={`/api/w/${$workspaceStore}/job_helpers/load_image_preview?file_key=${encodeURIComponent(
fileMetadata.fileKey
)}${storageQS}`}
alt="S3 preview"
title={fileMetadata.fileKey}
/>
</div>
{:else if fileMetadata?.fileKey.endsWith('.pdf')}
<div class="w-full h-[950px] border">
{#await import('$lib/components/display/PdfViewer.svelte')}
<Loader2 class="animate-spin" />
{:then Module}
<Module.default
source={`/api/w/${$workspaceStore}/job_helpers/load_image_preview?file_key=${encodeURIComponent(
fileMetadata.fileKey
)}${storageQS}`}
/>
{/await}
</div>
{:else if filePreviewLoading}
<div class="flex h-6 items-center text-primary mb-4">
<Loader2 size={12} class="animate-spin mr-1" /> File preview loading
</div>
{:else if fileMetadata !== undefined && filePreview !== undefined}
<div class="flex items-center text-primary mb-4">
{#if filePreview.contentType === 'Unknown'}
Type of file not supported for preview.
{:else if filePreview.contentType === 'Csv'}
Previewing a {filePreview.contentType?.toLowerCase()} file. Separator character:
<div class="inline-flex w-12 ml-2 mr-2">
<select class="h-8" bind:value={csvSeparatorChar} onchange={onCsvControlsChanged}>
<option value=",">,</option>
<option value=";">;</option>
<option value="\t">\t</option>
<option value="|">|</option>
</select>
</div>
Header row:
<div class="inline-flex item-center w-4 ml-2 mr-2">
<input
type="checkbox"
class="h-5"
bind:checked={csvHasHeader}
onchange={onCsvControlsChanged}
/>
</div>
{:else}
Previewing a {filePreview.contentType?.toLowerCase()} file.
{/if}
</div>
<pre class="grow whitespace-no-wrap break-words"
>{#if !emptyString(filePreview.contentPreview)}{filePreview.contentPreview}{:else if filePreview.contentType !== undefined}Preview impossible.{/if}</pre
>
{/if}
</div>
</div>