mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-21 00:02:30 +00:00
fix(ai-chat): make an attachment's key agree with its bytes, and the test backfill find anything
Addresses the CI reviewers on f41174e, plus what the local rounds found on top.
- An image is re-encoded to PNG/JPEG on the way in but kept the picked filename,
and the worker reads an attachment's media type from the object key alone
(`mime_guess::from_path`), never from the stored content type — so a `.webp`
reached the provider as PNG bytes labelled webp, which Anthropic rejects.
`storedAttachmentName` now makes the extension match the encoding; the same
rule covers a PDF picked without one, whose key would guess as `image/png`.
- The `is_test` backfill matched no existing conversation: no message row held
the flow job before this release, so it now walks to the run's root, ordered
as `get_root_job_id` is — `flow_innermost_root_job` stops at the closest flow
scope, which would miss an agent inside a subflow.
- A stored value for the attachments input no longer rides an attachment-free
message: the paperclip is that input's only writer wherever it is offered.
- The lowest thinking stop is now clickable from the `default` state, where the
range input's value already read 0 and so fired no event.
- Dropped `footerControls` and the `settleStreamingRows` export, neither of
which ever had a caller.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QN7VboDEm9HAB1t4sMxMdE
This commit is contained in:
co-authored by
Claude Opus 5
parent
f41174e8cc
commit
2c79a2f143
@@ -3,13 +3,25 @@
|
||||
-- lists tell a trial apart from a real conversation.
|
||||
ALTER TABLE flow_conversation ADD COLUMN is_test BOOLEAN NOT NULL DEFAULT false;
|
||||
|
||||
-- Existing rows: a conversation whose messages came from a flowpreview job was a test.
|
||||
-- Existing rows: a conversation whose messages came from a flowpreview run was a test.
|
||||
-- Derived once here because the job is purged on retention, after which the origin of an
|
||||
-- old conversation is unknowable.
|
||||
--
|
||||
-- Walked to the root job rather than matched directly: an existing message row never holds
|
||||
-- the flow job itself. Only this migration's release starts storing it on the user row, and
|
||||
-- the rows written before it point at the step that produced them — the AI agent's job for
|
||||
-- an answer, the tool's own job for a tool call — whose kind is never 'flowpreview'.
|
||||
--
|
||||
-- `root_job` first, matching `get_root_job_id` (windmill-worker/src/common.rs): only it
|
||||
-- reaches the top of the run. `flow_innermost_root_job` stops at the closest flow scope by
|
||||
-- design, so an agent inside a subflow would land on that subflow's 'flow' row and the
|
||||
-- conversation would read as deployed.
|
||||
UPDATE flow_conversation c
|
||||
SET is_test = true
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM flow_conversation_message m
|
||||
JOIN v2_job j ON j.id = m.job_id
|
||||
WHERE m.conversation_id = c.id AND j.kind = 'flowpreview'
|
||||
JOIN v2_job root
|
||||
ON root.id = coalesce(j.root_job, j.flow_innermost_root_job, j.parent_job, j.id)
|
||||
WHERE m.conversation_id = c.id AND root.kind = 'flowpreview'
|
||||
);
|
||||
|
||||
@@ -33,9 +33,18 @@
|
||||
overrideLabel
|
||||
}: Props = $props()
|
||||
|
||||
// A `current` that names no stop is a real state — an agent that leaves the effort unset
|
||||
// sends nothing and the provider decides — and the thumb then rests at the start. Only
|
||||
// `overrideLabel` tells the two apart, since a range input always has a thumb somewhere.
|
||||
/**
|
||||
* A `current` naming no stop is a real state, not a missing one: an agent that leaves the
|
||||
* effort unset sends nothing and the provider decides. Three things follow, and each has
|
||||
* been got wrong on its own — keep them together.
|
||||
*
|
||||
* The thumb rests at the start, because a range input always has one somewhere, and
|
||||
* `overrideLabel` is what tells the reader this is not the lowest stop. The track is
|
||||
* unfilled there, which index 0 gives for free. And since the input's value already reads
|
||||
* 0, picking the lowest stop by pointer fires no `input` event — so a click has to be
|
||||
* committed explicitly, or that stop is reachable only by keyboard.
|
||||
*/
|
||||
const hasPosition = $derived(stops.indexOf(current) >= 0)
|
||||
const stopIndex = $derived(Math.max(0, stops.indexOf(current)))
|
||||
// Percentage filled (accent) up to the thumb; the rest of the track stays surface-secondary.
|
||||
const fillPct = $derived(
|
||||
@@ -92,6 +101,15 @@
|
||||
style="--fill: {fillPct}%"
|
||||
{disabled}
|
||||
oninput={(e) => onSelect(stops[+e.currentTarget.value])}
|
||||
onclick={(e) => {
|
||||
// `click`, not `pointerup`: it is the event that means pressed and released on
|
||||
// the track, so a press that began on the row above cannot commit an effort
|
||||
// nobody chose. Only the click that moved nothing — any other stop has already
|
||||
// committed through `oninput`, and doing it again would write it twice.
|
||||
if (!hasPosition && !disabled && +e.currentTarget.value === stopIndex) {
|
||||
onSelect(stops[stopIndex])
|
||||
}
|
||||
}}
|
||||
use:isolatePointer
|
||||
class="lean-range no-default-style w-full"
|
||||
aria-label="Reasoning effort"
|
||||
|
||||
@@ -180,7 +180,6 @@
|
||||
wideLayout = false,
|
||||
emptyHint,
|
||||
inputPreface,
|
||||
footerControls,
|
||||
footerSettings,
|
||||
initialInstructions = undefined,
|
||||
onDraftChange = undefined,
|
||||
@@ -213,8 +212,6 @@
|
||||
wideLayout?: boolean
|
||||
emptyHint?: Snippet
|
||||
inputPreface?: Snippet
|
||||
/** Extra controls at the head of the composer's footer row. */
|
||||
footerControls?: Snippet
|
||||
/** The settings control at the footer's right edge, where the copilot puts its
|
||||
* model picker. A host that configures its turn elsewhere replaces it here. */
|
||||
footerSettings?: Snippet
|
||||
@@ -648,8 +645,7 @@
|
||||
// nothing else would lose the group and the `+` with it.
|
||||
const showFooterLeftControls = $derived(
|
||||
!footerMessageShown &&
|
||||
(footerControls !== undefined ||
|
||||
canAttachFiles ||
|
||||
(canAttachFiles ||
|
||||
showContextPicker ||
|
||||
showAutonomyModeSelector ||
|
||||
(chatHost.mode === AIMode.SCRIPT && hasDiff))
|
||||
@@ -973,7 +969,6 @@ the panel, or the Escape-to-stop focus check would wrongly reject them. -->
|
||||
>
|
||||
{#if showFooterLeftControls}
|
||||
<div class="flex flex-row items-center gap-x-1.5 min-w-0 flex-wrap">
|
||||
{@render footerControls?.()}
|
||||
{#if showContextPicker && !disabled}
|
||||
<Popover placement="bottom-start">
|
||||
{#snippet trigger()}
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { dataUrlToBlob, matchesAccept } from './blobUtils'
|
||||
|
||||
function file(name: string, type: string): File {
|
||||
return new File(['x'], name, { type })
|
||||
}
|
||||
|
||||
describe('dataUrlToBlob', () => {
|
||||
// The bytes are re-uploaded verbatim, so a decode that drops or shifts one is a
|
||||
// corrupted file the reader only discovers downstream.
|
||||
it('decodes base64 back to the exact bytes', async () => {
|
||||
const bytes = new Uint8Array([0x00, 0xff, 0x10, 0x89, 0x50])
|
||||
const b64 = btoa(String.fromCharCode(...bytes))
|
||||
const blob = dataUrlToBlob(`data:application/pdf;base64,${b64}`)
|
||||
expect(blob.type).toBe('application/pdf')
|
||||
expect(new Uint8Array(await blob.arrayBuffer())).toEqual(bytes)
|
||||
})
|
||||
|
||||
it('percent-decodes a url that is not base64', async () => {
|
||||
const blob = dataUrlToBlob('data:text/plain,hello%20world')
|
||||
expect(blob.type).toBe('text/plain')
|
||||
expect(await blob.text()).toBe('hello world')
|
||||
})
|
||||
|
||||
it('falls back to a media type when the url names none', async () => {
|
||||
expect(dataUrlToBlob('data:;base64,QQ==').type).toBe('application/octet-stream')
|
||||
expect(dataUrlToBlob('data:;base64,QQ==', 'image/png').type).toBe('image/png')
|
||||
})
|
||||
})
|
||||
|
||||
describe('matchesAccept', () => {
|
||||
it('matches an extension, a type wildcard and an exact media type', () => {
|
||||
expect(matchesAccept(file('report.PDF', ''), '.pdf')).toBe(true)
|
||||
expect(matchesAccept(file('shot.png', 'image/png'), 'image/*')).toBe(true)
|
||||
expect(matchesAccept(file('shot.png', 'image/png'), 'image/png')).toBe(true)
|
||||
})
|
||||
|
||||
it('refuses a file no pattern covers, and allows everything when the list is empty', () => {
|
||||
expect(matchesAccept(file('notes.txt', 'text/plain'), '.pdf, image/*')).toBe(false)
|
||||
expect(matchesAccept(file('notes.txt', 'text/plain'), '')).toBe(true)
|
||||
})
|
||||
})
|
||||
@@ -78,8 +78,9 @@
|
||||
const STORAGE_KEY_PREFIX = 'windmill_flow_chat_inputs_'
|
||||
|
||||
let showInputsModal = $state(false)
|
||||
// Conversation settings, persisted per flow. Attachments are absent by construction:
|
||||
// they ride the composer's own draft and are cleared with it on send.
|
||||
// Conversation settings, persisted per flow. These can include a value for the
|
||||
// attachments input, saved while the modal was its editor; `sendRequest` drops that one
|
||||
// once the paperclip takes over, so a stored file never rides a later message.
|
||||
let inputValues = $state<Record<string, any>>(loadInputsFromStorage() ?? {})
|
||||
let modalDraft = $state<Record<string, any>>({})
|
||||
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import type { FlowModule, InputTransform } from '$lib/gen'
|
||||
import { parseExpressionAt } from 'acorn'
|
||||
|
||||
/**
|
||||
* The flow's own AI agent steps, including those inside loops and branches but never one
|
||||
* carried as another agent's tool.
|
||||
@@ -30,7 +32,6 @@ function agentSteps(modules: FlowModule[] | undefined): FlowModule[] {
|
||||
walk(modules ?? [])
|
||||
return found
|
||||
}
|
||||
import { parseExpressionAt } from 'acorn'
|
||||
|
||||
/**
|
||||
* AI agent inputs the chat composer can drive, in footer display order.
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { storedAttachmentName } from './attachmentNames'
|
||||
|
||||
/**
|
||||
* The worker reads an attachment's media type from the object key and nothing else, so a
|
||||
* key whose extension disagrees with the bytes reaches the provider mislabelled.
|
||||
*/
|
||||
describe('storedAttachmentName', () => {
|
||||
// The composer re-encodes images, so the picked extension is the one that lies.
|
||||
it('renames a re-encoded image to the type it was encoded as', () => {
|
||||
expect(storedAttachmentName('photo.webp', 'image/png')).toBe('photo.png')
|
||||
expect(storedAttachmentName('holiday.png', 'image/jpeg')).toBe('holiday.jpg')
|
||||
})
|
||||
|
||||
it('gives an extension to a name that has none', () => {
|
||||
expect(storedAttachmentName('attachment-1', 'image/png')).toBe('attachment-1.png')
|
||||
expect(storedAttachmentName('contract', 'application/pdf')).toBe('contract.pdf')
|
||||
})
|
||||
|
||||
it('replaces only the last extension', () => {
|
||||
expect(storedAttachmentName('report.2026.final.webp', 'image/png')).toBe(
|
||||
'report.2026.final.png'
|
||||
)
|
||||
})
|
||||
|
||||
// Blobs upload byte for byte, so a type we do not re-encode keeps the name as picked.
|
||||
it('leaves a type it does not re-encode alone', () => {
|
||||
expect(storedAttachmentName('notes.csv', 'text/csv')).toBe('notes.csv')
|
||||
expect(storedAttachmentName('archive.zip', 'application/zip')).toBe('archive.zip')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,28 @@
|
||||
/**
|
||||
* What a chat attachment is stored under in object storage.
|
||||
*
|
||||
* The worker reads an attachment's media type from the object key and nothing else —
|
||||
* `mime_guess::from_path` in `windmill-ai/src/image_handler.rs`, falling back to
|
||||
* `image/png` when it can read no extension — and never from the content type stored
|
||||
* beside it. So the key's extension is a claim about the bytes, and it has to be true.
|
||||
*/
|
||||
|
||||
/** The extension each type the composer can send must be stored under. */
|
||||
const EXTENSION_BY_MEDIA_TYPE: Record<string, string> = {
|
||||
'image/png': 'png',
|
||||
'image/jpeg': 'jpg',
|
||||
'application/pdf': 'pdf'
|
||||
}
|
||||
|
||||
/**
|
||||
* The composer re-encodes every image to PNG or JPEG, so keeping the picked `photo.webp`
|
||||
* would hand the provider PNG bytes labelled webp, which Anthropic rejects outright; and a
|
||||
* PDF picked without an extension would be read back as the `image/png` fallback. A type
|
||||
* not listed is left as picked — blobs upload byte for byte, so their name is already true.
|
||||
*/
|
||||
export function storedAttachmentName(filename: string, mediaType: string): string {
|
||||
const extension = EXTENSION_BY_MEDIA_TYPE[mediaType]
|
||||
if (!extension) return filename
|
||||
const stem = filename.replace(/\.[^./]+$/, '')
|
||||
return `${stem || filename}.${extension}`
|
||||
}
|
||||
@@ -10,6 +10,7 @@ import { ToolCallStore, type ToolCallDetails } from './toolCallContext.svelte'
|
||||
import { AttachedFilesStore } from '$lib/components/copilot/chat/files/attachedFiles.svelte'
|
||||
import { SessionArtifactsStore } from '$lib/components/copilot/chat/artifacts/artifactsState.svelte'
|
||||
import { dataUrlToBlob, type AttachedBlob } from '$lib/components/copilot/chat/blobUtils'
|
||||
import { storedAttachmentName } from './attachmentNames'
|
||||
import type { AttachedImage } from '$lib/components/copilot/chat/imageUtils'
|
||||
import type { AttachedTextFile } from '$lib/components/copilot/chat/textFileUtils'
|
||||
import { HelpersService } from '$lib/gen'
|
||||
@@ -259,6 +260,13 @@ export class FlowChatViewHost implements ChatViewHost {
|
||||
const text = options.instructions?.trim() ?? ''
|
||||
const args = { ...(this.#options.additionalInputs?.() ?? {}) }
|
||||
const target = this.#options.attachmentsTarget?.()
|
||||
// Where the paperclip is the input's editor, the stored settings have no say over it:
|
||||
// a value saved while the modal owned it — before this workspace had object storage —
|
||||
// would otherwise ride along on every later message. Attachments are set below or not
|
||||
// at all. Where the modal still owns it, what the reader typed there stands.
|
||||
if (target && this.supportsMessageAttachments) {
|
||||
delete args[target.name]
|
||||
}
|
||||
let images = options.images ?? []
|
||||
let blobs = options.blobs ?? []
|
||||
// The composer refuses an attachment-only send (requiresMessageText), so this is
|
||||
@@ -354,8 +362,11 @@ export class FlowChatViewHost implements ChatViewHost {
|
||||
const prefix = `windmill_chat_uploads/${randomUUID()}`
|
||||
return Promise.all(
|
||||
attachments.map(async (attachment, index) => {
|
||||
const filename = attachment.name ?? `attachment-${index + 1}`
|
||||
const blob = dataUrlToBlob(attachment.dataUrl, attachment.mediaType)
|
||||
const filename = storedAttachmentName(
|
||||
attachment.name ?? `attachment-${index + 1}`,
|
||||
blob.type
|
||||
)
|
||||
const { file_key } = await HelpersService.fileUpload({
|
||||
workspace,
|
||||
fileKey: `${prefix}/${filename}`,
|
||||
@@ -366,6 +377,7 @@ export class FlowChatViewHost implements ChatViewHost {
|
||||
})
|
||||
)
|
||||
}
|
||||
|
||||
cancel = () => {
|
||||
void this.#manager.cancelCurrentJob()
|
||||
}
|
||||
|
||||
@@ -52,7 +52,7 @@ function newRow(state: TurnState, id: string, patch: Partial<ChatMessage>): Chat
|
||||
}
|
||||
|
||||
/** Everything still streaming is finished: nothing further will be appended to it. */
|
||||
export function settleStreamingRows(rows: ChatMessage[]): ChatMessage[] {
|
||||
function settleStreamingRows(rows: ChatMessage[]): ChatMessage[] {
|
||||
return rows.map((row) => (row.streaming ? { ...row, streaming: false } : row))
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user