mirror of https://github.com/windmill-labs/windmill.git synced 2026-08-19 00:02:03 +00:00

Files

T

Guilhem d6cf1ef987 feat: attach text files to chat messages, read on demand via file tools (#10215 )

* feat: attach text files to chat messages, read on demand via file tools

* fix: resolve name collisions and sync message files with the transcript

* refactor: render message file chips with the shared context badge

* fix: suffix same-name attachments, sync registry on compaction, bound file bytes

* feat: carry message files across compaction, drop them from the roster

* refactor: merge context, dom and file badges into one wrapping row

* fix: dedupe identical attachments and make badge list keys collision-proof

* fix: name carried files inside the collapsed summary instead of badges

* fix: serialize registry reconciliation and correct the attachment budget

* fix: reserve pending bytes so overlapping reads honor the attachment budget

* fix: share attachment byte budget across concurrent composers

The bottom composer and the edit box are both mounted while editing an
earlier message, but each enforced MAX_CONVERSATION_FILE_BYTES against
only its own staged files plus the transcript. Two attaches near the cap
could each pass independently and overflow the persisted transcript.

Each composer now publishes its staged bytes (committed attachments +
in-flight reads) to the manager, keyed per instance, and the attach-time
budget subtracts every other live composer's stage. A message an open
composer is editing is skipped from the transcript sum since that
composer's stage stands in for it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: charge an edited message at its persisted size until the edit commits

An edit is not committed until send, so the edited message's persisted
attachments return if the edit is cancelled. Substituting only the edit
box's (possibly emptied) stage let the always-mounted bottom composer
claim headroom that vanishes on cancel: remove the near-limit files in
the editor, fill the bottom draft, cancel, and the persisted transcript
overflows MAX_CONVERSATION_FILE_BYTES.

attachmentBytesExcluding now charges a message another composer is
editing at max(persisted size, editor stage), so freed space only
becomes available once the edit actually commits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: preserve a message file's exact name when a session row clashes on rebuild

syncMessageScoped rebuilds message-scoped rows from the transcript through
collision-suffixing addFiles. Session rows load first (on restore), so one
holding a wanted name pushed the rebuilt message row to a "(2)" suffix while
the persisted prompt still referenced the bare name — get() then resolved the
reference to the session asset and the model read the wrong content.

Free the name from the conflicting session row before the rebuild so the
message row reclaims its exact reference. The rename is in-memory only: it is
deterministic and re-applied on every load, and the session roster is
regenerated live each send, so the session asset stays addressable under the
suffix without a persisted-record update.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: reserve resent files across the edit-resend gap

The edit box unmounts (dropping its staged-byte entry) the instant the user
submits an edit, but restartGeneration then awaits registry sync and beforeSend
before the optimistic bubble lands in the transcript. During that gap the
resent files were reserved nowhere, so the always-mounted bottom composer could
attach into the temporary headroom and the resend would then push the persisted
transcript past MAX_CONVERSATION_FILE_BYTES.

restartGeneration now reserves the resent files' bytes in shared manager state
before the transcript slice; sendRequest releases the reservation once it
installs the bubble (or restores the files to the composer on a pre-install
bail). The reservation bridges the gap so the budget stays honored throughout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: re-target an in-flight index when a session row is renamed on rebuild

#freeNameForMessageRow renames a restored session row so a same-named message
attachment can reclaim its exact name. But the row's #indexFile, started under
the old name during restore, stamps via #patchFile(oldName, file) — after the
rename that no longer matches, leaving the row stuck 'indexing' so read_file
rejects it and search_files excludes it.

Re-kick #indexFile under the new name when the renamed row is still indexing;
the stale completion then no-ops (its name is gone).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: release resend reservation on local-command sends; surface compaction-orphaned files

Two follow-ups to the message-attachment work:

- A resend edited to /clear or /compact runs the local-command path and returns
  before installing a bubble, so the #RESEND_KEY reservation set by
  restartGeneration was never released and its bytes stayed charged, blocking
  later attachments. Release it on every sendRequest path that exits before
  install (via #releaseResendReservation).

- Drop-oldest compaction (summary fallback) removes API messages without a
  summary, so a folded message's `## ATTACHED FILES` reference no longer reaches
  the model even though the file stays readable. The roster omits message-scoped
  files, so the model loses awareness of them. orphanedMessageFileNames() finds
  message files whose only referencing message went negative-index, and the
  roster now advertises them (summary compaction already carries its own).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: key the resend reservation per send so unrelated sends can't release it

The resend reservation used a single shared key, so a normal or concurrent
sendRequest released it at its own install/early-return even though it didn't
own it — dropping an in-flight resend's reservation and letting attachments
staged before the resend bubble lands under-count against the byte cap.

restartGeneration now mints a per-resend token, reserves under it, and threads
it through sendRequest as resendReservationKey; releases act only on that key.
A send with no token (every normal send) releases nothing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: reserve the whole wanted-name set when freeing session rows on rebuild

Freeing a session row for a message-scoped rebuild suffixed it against current
rows only. With the transcript referencing both notes (2).md and notes.md and a
session row named notes.md, freeing notes.md renamed it onto notes (2).md — also
a wanted reference — so that message row cascaded to notes (3).md while its
persisted reference stayed notes (2).md, and read_file returned the session file.

#uniqueName now accepts a reserved set; the rebuild frees each session clash
clear of the entire wanted-name set, so every message row reclaims its exact
reference regardless of collision order.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: orphan summary-carried files when drop-oldest removes the summary

orphanedMessageFileNames classified summary rows as always-live, but a summary
carries its folded files' reference on its own API message. When summary
compaction succeeds and a later summarization fails, drop-oldest can remove that
API message, yet the files stayed off the roster — so the model lost their
reference even though they remained readable.

The summary display row now tracks its API index (slot 0 at creation, re-based by
drop-oldest); a negative index reads as "counterpart gone" and its files move to
the roster, mirroring user-message orphans. The index is used only for orphan
detection, never as a restart target.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: reserve outgoing file bytes for normal and queued sends too

The resend reservation covered edit/retry, but a normal or queued send has the
same gap: the composer (or queue) clears its files the instant sendRequest is
called, dropping the staged-byte entry, while sendRequest then awaits
regrantLocked()/refreshFolders() before the bubble lands. With a locked or slow
linked folder the composer stays enabled, so a fresh drop can spend the same
headroom and overflow the 5 MB cap once the first bubble installs.

Generalize the reservation: sendRequest mints a per-send token and reserves the
outgoing files' bytes just before attachment upkeep (reusing restartGeneration's
token when present), and releases it on install or any pre-install exit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor: join attachment layers on a stable content-hash id

Sixteen review rounds kept finding the same bug family: a message file's
identity was its display name, joined by hand across the registry, the
transcript, the prompt reference, and the render keys — every same-name
collision or interleaved rebuild made two of those copies disagree.

Give each message attachment a deterministic id, attachedTextFileId(name,
content) — a synchronous pure-JS hash (works on plain-HTTP deployments) whose
exact value is pinned by test, since persisted transcripts reference it. The
prompt and roster list the id, the file tools resolve id-first (bare names
remain a fallback for legacy chats and session links), and pre-id transcripts
hydrate on load by recomputing the same hash — no migration state.

Names become display-only and may collide freely, which deletes the machinery
that defended them: the suffix-readback registration loop, session-row renaming
on rebuild (#freeNameForMessageRow, reserved-set #uniqueName), and the
reconciler's serialization guards (#syncSeq/#syncChain) — syncMessageScoped now
compares ids instead of awaiting blob text, so it is synchronous and cannot
interleave. A same-name clash within one draft gets a composer-local courtesy
rename before the id is minted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: resolve bare names to session rows and scope id searches to one row

A bare name is the roster's namespace: session links are advertised by
filename and have no other handle, so a same-named message attachment (which
is addressed by id) must not shadow them. get() now resolves session rows
first, keeping the message-row name lookup only as the fallback for
transcripts persisted before ids existed.

search_files restricted an id reference by mapping it back to the display
name and letting the worker filter on it — same-named files were then
searched together under one label. The tool now passes the resolved row
itself, so an id-scoped search can only ever hit its own file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: label search hits with resolvable ids, normalize attach batches at commit

An unscoped search_files reported hits by display name only. Names may
collide, so a hit could not be mapped back to the row that produced it —
a follow-up read_file on the bare name could return a different same-named
file. Rows carrying an id are now labeled `name (file id: …)` in hit lines,
so every hit names the reference that resolves to exactly that row.

addTextFiles normalized (deduped, courtesy-renamed) each file against a
snapshot taken during its read loop. Attach batches overlap, so a file
committed by another batch between reads escaped both checks — duplicate or
same-named unsuffixed entries in one message. Normalization now runs in the
single synchronous commit step against the live list (foldIntoDraft), where
nothing can interleave.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: dedupe renamed re-drops in foldIntoDraft, truncate queued file chips

"Same file dropped twice" means same original (name, content), but a
courtesy rename erases the original name — an identical re-drop then missed
the duplicate check and landed as a further-suffixed copy. The dedupe now
also matches entries whose suffix-stripped base name equals the read's name.

Queued file chip labels get min-w-0 so long filenames truncate inside
max-w-36 instead of overflowing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: address cubic review — line counts, chip clicks, reference robustness

Five fixes from the cubic pass:

- The prompt advertised split('\n').length lines, one more than read_file
  reports for newline-terminated files — textLineCount now matches the tool's
  numbering (0 for empty, no phantom trailing line).
- Clicking a sent message's badge opened edit mode (the wrapper's
  click-to-edit), unmounting the preview popover as it opened; the badge row
  now keeps clicks to itself.
- resolve() accepts the composite label rosters and search hits print
  (`name (file id: x)`) — models echo references verbatim, so the printed
  form must resolve.
- fileToAttachedTextFile enforces MAX_TEXT_FILE_BYTES itself (raw size +
  decoded byte length), so no ingestion path can persist an oversized
  attachment past the composer's pre-check.
- Duplicate detection after a courtesy rename now uses an explicit sourceName
  instead of inferring provenance from the display name — a user's real
  `report (2).md` is not a rename of `report.md`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: exact names win over label parsing, commit recheck uses decoded bytes

resolve() parsed any `name (file id: x)`-shaped reference as a printed label,
so a session file literally named that way became unreachable by its exact
name (the dead-id fallback resolved the base name instead). Exact id and
exact-name lookups now run before label interpretation.

Attachment admission and the pending reservation use raw File.size, but the
committed charge is the decoded UTF-8 length — malformed input decodes each
invalid byte to a 3-byte replacement character, so a file passing the 8KB
text sniff could inflate past the conversation cap. The synchronous commit
step now re-checks the live budget against decoded sizes
(admitWithinByteBudget) and drops what no longer fits, with the budget toast.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: normalize files folded into the queued message

Repeated submissions during a stream aggregate into one queued message, but
their files were concatenated raw: an identical re-attach duplicated its chip
and ate a slot (possibly displacing a distinct file at the eight-file cap),
and a same-name clash skipped the courtesy rename. The queue now folds new
files through the same commit normalization as the composer — the queued
entry is a message draft like any other.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style: compaction boundary label uses text-normal text-2xs

* fix: fold provenance survives pass-through, dequeued files fold into the draft

foldIntoDraft recorded sourceName only for renames it performed itself, so a
file already courtesy-renamed by the composer lost its provenance when folded
into the queue — a later re-attach of the original escaped dedupe. Folds now
compose: the original source name rides through every fold, and dedupe
matches on it.

dequeueMessage restored queued files into a possibly-populated composer by
raw concatenation; prependText now folds them like every other draft
aggregation (dedupe, courtesy rename) before applying the cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: one MessageDraft owns the lanes that ship with a send

Review rounds kept finding the same P2 shape: an aggregation point where
files join a draft (composer commit, queue append, dequeue restore) that
forgot one of the draft rules — fold dedupe, courtesy rename, slot caps,
byte admission, lanes moving together. The rules existed only as convention
re-implemented per site.

MessageDraft owns them once: text, pastes, images, and text files live on
one object with addFiles (fold + optional decoded-byte admission + cap),
addImages (cap), prepend (restore-merge), replaceIfEmpty (occupied-guard
restore), and take (all lanes leave together). The composer holds a draft
instead of four state vars, and the queue is a draft behind the existing
queuedMessage/queuedImages/queuedFiles accessors — an aggregation point can
no longer skip a rule, because there is no raw array to concatenate into.

Deliberately not moved: @context and DOM picks (ContextManager owns their
lifecycle), the conversation byte budget's cross-composer ledger (store-side
follow-up), and sendRequest's options shape (it decomposes immediately and
is pinned by the manager test suite).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: correct the drop-routing comment, condense the budget doc

* fix: address cubic review — name sanitization, drop hold, merge restores

Four fixes from the cubic pass:

- Attachment display names render into model-facing prompt blocks, and OS
  filenames may legally contain control characters — sanitizeAttachmentName
  strips them at attach and again at every prompt-render site (legacy names
  predate the attach-time pass), so a crafted name cannot inject prompt lines.
- Drop routing awaits handle/entry resolution before it can call
  addTextFiles; a send during that window landed the dropped files on the
  next message. The drop handler now holds sending (holdSendForIngestion,
  taken before the first await) until routing completes.
- Restoring a taken queue after a failed auto-send replaced the queued draft
  wholesale, silently losing a follow-up queued during the preflight. Both
  #restoreQueue and the unmounted-input requeue now merge via draft.prepend —
  the taken entry's text lands above the newer follow-up.
- restartGeneration validated the API restart index only after reserving the
  resend bytes and truncating the transcript, so a stale index threw with the
  reservation leaked and the display transcript half-mutated. The index is
  resolved and validated before anything is touched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: restored drafts keep chronological priority, store names stay resolvable

A failed auto-send's restore folded the taken (older) draft's attachments
AFTER a follow-up queued during preflight, so at the slot caps the older
attachments silently dropped despite the text landing first — and only one
entry's pinned context survived. prepend() now puts the restored lanes ahead
(the cap drops the newest additions) and #restoreQueue unions both pinned
contexts by identity.

Session filenames were sanitized only at prompt render, so an id-less file
whose stored name carries control characters was advertised under a name
that resolve() could not match. Names are now sanitized at every store
row-creation site (attach, folder expansion, refresh, and persisted-row
restore for pre-sanitization records), making the advertised name the stored
name everywhere — render-site sanitization remains as defense in depth.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: update ee-repo-ref to aaa6cb89b05b76139252c64f057e53b94d12ac60

This commit updates the EE repository reference after PR #680 was merged in windmill-ee-private.

Previous ee-repo-ref: 4c08634af953db5c1125b1fb03f5af211fe21db3

New ee-repo-ref: aaa6cb89b05b76139252c64f057e53b94d12ac60

Automated by sync-ee-ref workflow.

* chore: repin ee-repo-ref to main's eb3690a34b (aaa6cb89 needs the unmerged AmqpTrigger OSS companion)

* Revert "chore: repin ee-repo-ref to main's eb3690a34b (aaa6cb89 needs the unmerged AmqpTrigger OSS companion)"

This reverts commit a782e6f1c5.

* refactor: one name seam per scope — claim on the store, sanitize in the fold

Name rules (sanitize, uniquify, dedupe, resolve) lived as convention at every
creation site: four sites carried verbatim copies, two skipped uniquify
(restore, file placeholders — legacy names could collapse to one display
name), and the dedupe check compared the raw name against sanitized stored
names, so re-linking a control-char file added a spurious copy.

Store side: #claimName(raw) = sanitize + uniquify is now the only way a
session row gets its display name; addFiles derives the sanitized name once
at the top of the loop, so dedupe, uniqueness, and the stored row all see the
same string, while relPath and folder keep the raw on-disk keys they must
match. Draft side: foldIntoDraft sanitizes its reads itself instead of
assuming the reader did, making the fold self-contained. Folder names — raw
grouping keys by design — are sanitized at their model-facing render sites
(roster folder lines, not-found listing), mirroring rosterLine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: a display-name collision is not a duplicate

Sanitizing names before the dedupe compare made two DISTINCT files whose raw
names sanitize identically look like re-links — the second was silently
discarded instead of claiming a suffixed name like the restore path does.

Dedupe now means "the same file re-linked": matching stats (size + mtime)
plus a matching name, pre-suffix sourceName, or raw path. #claimName records
the pre-suffix name when uniquifying renamed a row, so re-linking a suffixed
file's original still dedupes. A same-named file with different stats links
as its own row rather than being silently swallowed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: the re-link identity is the raw name, persisted with the row

Dedupe still compared sanitized display names, so two distinct raw names
that sanitize identically collapsed whenever their stats also matched — and
the provenance recorded for suffixed rows lived only in memory, so after a
reload re-linking the original behind a suffixed row stacked another copy.

sourceName now records the RAW pre-sanitization name on every session row
(display names lose information twice — sanitize, then suffix), rides the
persisted record, and is re-derived on restore. Dedupe matches stats plus
raw identity (sourceName, or relPath for folder children) and never compares
display names. #claimName returns to a pure name function; restore claims
the display name from the persisted raw identity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>