Files
windmill/ai_evals/cases
Guilhem 9739d5a2c2 feat: unified read-only diff chat tool (drafts, fork vs parent, search) (#10211)
* feat(frontend): unified `diff` chat tool with cached snapshot, fork mode, and search

One read-only global-chat tool for every comparison: drafts vs deployed
(workspace index + per-item unified patches over stable YAML), deployed
fork vs parent workspace (against="parent_workspace", sharing the fork
banner's compareWorkspaces fetch through a single-flight store), and a
literal grep over changed diff lines. Multi-file raw apps split into
per-file text patches with folder-style index children and per-file
reads. Patches are materialized once into a per-workspace cache keyed on
draft created_at / comparison ahead-behind markers and the workspace
drafts version, so repeated queries never refetch unchanged content.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai_evals): teach the mock draft backend what the diff tool reads

The diff tool reads drafts through the get_draft overlay, the drafts
listing's draft_only flag, and per-row created_at change markers — none
of which the benchmark mock modelled (fixed timestamp, always
draft_only, overlay ignored), so in evals every draft looked absent and
the model looped to max turns. Mirror production: monotonic
deterministic created_at bumped per upsert, draft_only computed against
the deployed stores, and draft/no_deployed overlays on script/flow/app
reads (404-shaped not-found). Also drop the diff case's judge items
about conversation content the judge never sees — tool usage is already
enforced deterministically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): diff tool reads unsaved editor state instead of going stale

The pre-diff flush honors the auto-save toggle (a read-only tool must
not persist parked edits), which left a gap: with auto-save off — or
after a failed save — the persisted draft the diff reads is stale, and
a brand-new editor-only draft looks absent. Item reads now detect
unflushed parked edits (hasUnsavedDisabledChanges / failed save state)
and diff the in-memory editor value directly, bypassing the snapshot
cache (it must only hold persisted state) with an explicit unsaved-
changes note; index and search modes warn which items' unsaved edits
they exclude. Local values are canonicalized onto the persisted draft
shape so they never diff noisily against the deployed side.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(ai): invalidate diff cache the moment any draft write lands

The snapshot cache leaned on time windows (5s listing throttle, 15s
read reuse) to notice writes it didn't trigger itself — an editor
autosave landing between two diff reads could serve the pre-edit patch.
The syncer now exposes onAnySaved (fires for landed upserts AND
deletes, all keys), and the snapshot subscribes once: a landed write
marks exactly that item's patch stale and expires the listing throttle,
so the next read refetches regardless of any reuse window.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): reject the diff file arg on single-document items

Passing file for a script/flow/classic-app diff was silently ignored
and returned the whole patch — an explicit error steers the model to
call again without it. Also declares the file arg on the item handlers'
signatures it was already flowing through.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): surface empty-file additions/deletions in app diffs

An empty file appearing or disappearing produces no text patch, so the
per-file split dropped it — a draft whose only change was such a file
read "unchanged". Presence changes now keep their added/deleted entry
(patch '', 0 lines), render as "(empty file)" in summaries, and a file
read states the presence change instead of an empty window.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): include classic-app drafts in the diff index and fix ++/-- search

itemTypeForKind now maps classic `app` draft rows to the chat app type
(mirroring the read path, which already pairs app/raw_app), so their
diffs materialize in the index and search instead of reporting "not
addressable". Changed-line search is hunk-aware: `---`/`+++` file
labels only occur before the first @@ marker, so a changed source line
like `++counter` (rendered `+++counter`) now matches instead of being
mistaken for a label.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): mask every variable value in chat diffs; compare classic apps value-to-value

Variable VALUES never reach a tool result — the chat-wide invariant
read_workspace_item enforces, not just for secrets. Draft-mode diffs
mask both sides with a placeholder pair that still marks WHETHER the
value changed; fork-mode masks at fetch (and still never decrypts);
item reads carry an explicit note. The former secret-only flag is now
valueMasked.

Classic-app drafts hold the bare grid value while the deployed row
nests it beside summary/policy — diffed raw, a one-field edit read as
a whole-document rewrite. Both sides now reduce to { value } via
classicAppDraftValue (pure, unwraps legacy wrapped drafts), which also
cleans the CompareDrafts drawer for classic apps.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): honest secret-draft reporting, classic-app metadata split, glob-safe file subjects

A secret variable's sides are both masked upstream, so an empty patch
cannot prove the value is unchanged — such drafts now report
"cannot be compared; may differ" (valueUncomparable) instead of
"matches deployed", in the index and item reads.

Classic-app drafts mirror summary/draft_path into the bare grid while
the deployed row keeps summary as a column: sides now reduce to
{summary, value} via classicAppDraftParts, applied to both sides, so a
summary edit diffs as one and draft-only markers never pollute the
grid diff.

Raw-app search subjects strip the file key's leading slash so
slash-anchored globs like f/x/*.tsx match.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): classic-app local edits, comparison-relevant fork fields, conflicts as unflushed

The chat app type spans two draft kinds: item mode now flushes and
probes both raw_app and classic app keys, and the flush sweep includes
classic-app editor cells (kept out of GLOBAL_DRAFT_KINDS so
clearGlobalDrafts never clears an open classic editor).

Fork projections gain the fields the backend comparison counts that
getItemValue drops: flow schema (with a taxonomy-agnostic inline-hash
strip) and resource-type description/format_extension/is_fileset.
Folder display_name is not exposed by the API's Folder type, so it
cannot be projected.

A conflicted save leaves its payload parked with state 'none', so
index/search now count conflicts among unflushed paths and say so in
their warnings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): staged app renames diff as path; flush classic-app cells at their real keys

A staged rename (draft_path) changes where deploy lands an app, so both
app kinds now compare `path` on both sides — a rename-only draft diffs
instead of reading "matches deployed". classicAppDraftParts returns the
staged path separately from the grid.

Item mode resolves each draft kind's own storage path and additionally
asks the listing which row owns a friendly/renamed path — a renamed
classic app's cell lives at its ORIGINAL storage path, which only the
listing knows — so pending/failed/auto-save-off edits are flushed and
probed at the real keys. The appDiffSides rationale comment is
compressed to the repo's four-line limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): never claim folder parity the API cannot prove

folder.display_name exists only as a DB column — no folder endpoint
returns it — so an identical projection cannot prove a fork folder
matches its parent. Fork index and item reads for folders now say the
display name is not exposed and may be what differs, instead of
"content matches parent". Exposing the field on getFolder is a backend
follow-up.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): gap-free patch pagination; forced-fresh comparisons never join older fetches

When the char backstop cut inside a patch window, the continuation
offset still pointed past the requested window — silently skipping the
undelivered lines forever. windowPatch now cuts at the last complete
line and continues exactly there (a single over-budget line is
delivered truncated and stepped past so pagination always advances).

fetchWorkspaceComparison treats an in-flight request as being as old
as its start: maxAgeMs now gates joining it, so a freshness-forced
post-mutation read (maxAgeMs 0) always issues its own fetch instead of
adopting a tally that began before the mutation, and a superseded
request can no longer clobber a newer cached result.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): generation-ordered comparison writes; path-only fork reads for enum-less kinds

Concurrent comparison requests can share a Date.now() value, letting a
superseded request's late result overwrite a newer one and be reused
for 30s — cache writes are now ordered by a monotonic request
generation. Test pins the same-millisecond race with the newer request
resolving first.

Fork comparison kinds outside the chat type enum (folder,
resource_type, …) were listed and even advertised as readable but no
call could reach them: a fork item read without `type` is now a
path-only wildcard (ambiguous paths list their kinds and ask for
type), messages label entries by their comparison kind, and pending
index lines for enum-less kinds advertise the path-only read.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): grid-based wrapper detection, comparison invalidation on mutations, multi-kind wildcard reads, honest hidden-diff summaries

The classic-app wrapper heuristic keyed on metadata keys the editor
mirrors into every bare grid — a grid with a component named `value`
was reduced to that component. `grid` presence is the discriminator: a
bare App always has it, a legacy wrapper never does.

invalidateWorkspaceDrafts now also drops cached fork comparisons for
the workspace, so the FIRST post-deploy fork read cannot reuse a
banner-prewarmed pre-deploy tally (the snapshot-baseline check only
covered subsequent reads).

Wildcard fork reads return a section per matching kind instead of an
unactionable "pass type" for kinds the chat type enum cannot name, and
the fork index never summarizes ACL-hidden differences as parity —
hidden counts stay directional (a conflicted item counts in both).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): address cubic review batch — invalidation scope, races, edge output

Comparison cache: invalidation matches either side of the pair (a
parent deploy moves its forks' tallies), fences in-flight requests
(no new joins, late results rejected via a per-key generation floor),
and the map is LRU-capped. Eviction moves from every drafts-version
bump to deploy success only — draft saves never move the deployed
tally. Fork snapshots also baseline the PARENT's drafts version.

Draft materialization carries a stale-generation token so a save
landing mid-fetch discards that run's pre-save result instead of
repopulating the invalidated entry; a save/delete also expires the
fork cache's hasLocalDraft join. onAnySaved listeners are
error-isolated (a throwing listener must not mark a committed save
failed) and the pagehide keepalive flush notifies them on dispatch.

Output edges: folder fork lines drop the empty parenthetical, and a
patch-window offset past the end reports itself instead of an
impossible range. The eval case pins the diff call's path argument.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(ai): one fencing primitive per cache instead of per-surface races

Review rounds kept finding pairwise races between async producers and
invalidation — each patched with its own fence. Replace the class:

- diffSnapshot: a per-workspace mutation epoch, bumped by every
  invalidation. Both reconcilers run a bounded retry loop — joiners
  re-validate after awaiting, producers refuse to store results whose
  inputs predate a mutation. Covers in-flight listing adoption and
  pre-deploy fork tallies in one mechanism.
- workspaceComparison: per-WORKSPACE generation floors (either side of
  a pair). Any request started before an invalidation is fenced from
  joining and from landing in the cache — including superseded
  requests the inflight map no longer tracks.

Also: delete_workspace_item invalidates comparisons like deploy does
(deployed state moved), and empty FILTERED indexes say the filter
matched nothing instead of claiming workspace/fork parity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): invalidate comparisons on every direct deploy; keep secret caveat with metadata changes

Direct chat deploys (schedule/trigger/resource/variable/app) bypass
deployDraftToWorkspace and never evicted cached fork comparisons — the
shared deploy tail now invalidates before the fallible draft cleanup.

A secret variable whose metadata also changed produced a non-empty patch
that silently dropped the value-uncomparable caveat; item reads, the
index, and fork sections now keep the caveat alongside the patch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): scope diff caches to the authenticated identity; derive fork freshness from the comparison store

An SPA logout/login left workspace-keyed diff caches (per-user drafts,
permission-filtered fork patches) readable by the next account — both
cache modules now wipe on identity change, with a global generation
floor fencing requests started under the previous account.

The fork snapshot stamped its own fetchedAt over a comparison that
could already be near expiry, compounding the two 30s windows, and
survived comparison-store invalidation when draft cleanup failed after
a deploy. It now carries the comparison's own fetchedAt/generation and
stops reuse the moment the store fences it. Delete-item invalidation
moved before the fallible draft cleanup for the same reason.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): fence fork-reconciliation joins across account switches

ForkCache lacked the epoch stamp WorkspaceCache carries, so a joiner
arriving after an identity change (or any epoch bump landing before it)
compared its own post-bump epoch against itself and adopted the old
producer's in-flight tally. The cache now records its producer's epoch
for the joiner and reuse gates, and an identity change also discards
the in-flight reconciliation maps so no cross-identity join exists.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): surface swallowed fork-side fetch failures; include conflicted editor edits in item diffs

The shared getItemValue reads {} for any failed fetch, so a transient
API failure on a fork side rendered as a fabricated one-sided diff (or
parity when both sides failed). A fork side is only fetched when the
comparison lists it as existing, so an empty read now raises and shows
as a fetch-error entry.

Item reads promised conflicted local edits (the index says so) but the
local-override branch only covered autosave-off and failed saves — a
conflict silently fell back to the persisted draft. Conflicts now read
the in-memory editor value too, with a caveat naming which side is
shown either way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai): report failed diff materializations as unsearched instead of silently omitting them

A side-fetch failure left an index entry with status 'error' and no
patch; diff search skipped it and still presented definitive no-match
or complete-count results. Failed entries are now listed in a warning
naming what was not searched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 17:06:32 +00:00
..
2026-04-30 12:23:59 +00:00