* test: add global-mode path-selection eval cases with seeded user
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(copilot): guide global-mode path selection with injected folder list
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(copilot): tailor global-mode folder guidance for workspace admins
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(copilot): type folders_read; isolate global-eval user from store
Addresses PR review:
- Add folders_read to the User/whoami openapi schema and UserExt; the global prompt builder and eval harness now read it typed instead of via inline casts (regen the client to pick it up).
- prepareGlobalSystemMessage takes an explicit user; the eval harness passes it rather than mutating the process-global userStore, removing the concurrency race (path cases no longer need --verbose).
- Rewrite the path-selection case comment as a current invariant.
- Add buildFolderGuidance unit tests in core.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add global AI chat context-optimization plan for raw apps
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): add global raw-app debugging cases on a large fixture
Adds a ~20-file analytics_dashboard raw-app fixture (incl. a 5k-line data module
and a planted wrong-totals bug), two global cases (read-heavy debug + small-edit
baseline), app-seed support in the mock backend, directory-fixture loading, and a
decorateHelpers seam so read-dedupe is measurable. Records tokenUsage for before/
after comparison of the read-tool optimization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): cap and dedupe read_app_file to bound context in large apps
read_app_file now defaults to a head slice (1500 lines / 50k chars) with offset/
limit to page further, and skips resending a file whose earlier read is still in
context (per-conversation ledger keyed off the originating tool-call id, so it
self-heals after compaction). Bounds the file-content portion of global-chat
context when working in large raw apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): add read-heavy raw-app debug case (large data module)
global-test31 induces the model to inspect the 5k-line seedData module, exercising
the read_app_file cap/offset path. Baseline ~262k tokens vs ~200k with the cap+dedupe
change (-24%).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record A+B benchmark results and fixed-overhead finding
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): clearer read_app_file past-EOF message + unit tests for cap/dedupe
Addresses local-review nits: out-of-range offset now reports 'offset N is past the
end of the file' instead of a backwards 'lines 11-10' label; adds unit coverage for
the slicing (line cap, offset/limit window, char budget, past-EOF) and re-read dedupe
(hit + miss-when-not-retained).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): char-level paging + per-range dedupe for read_app_file
Adds char_offset/char_limit so minified/long-line files can be paged within a line
window, keys the re-read ledger by range (so reading different ranges no longer
collides), and dedupes on the full-file hash (a cached range stub is invalidated
when any byte of the file changes, not just the returned range). Tests updated for
the char-slice behavior plus single-line capping, char paging, and out-of-window
change detection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-chat): add read_app_file context micro-benchmark + re-read eval case
Adds a deterministic micro-benchmark (no LLM) that drives read_app_file through a
realistic big-project read pattern (large file, re-read, minified bundle, paging)
and asserts the cap+dedupe cut returned context >50% vs the old whole-file behavior
— isolating the feature's effect from model nondeterminism and guarding against
silent weakening. Adds global-test32, a cross-file consistency investigation that
revisits overlapping files so re-read dedupe is exercised in a real run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): clarify test32 measures the read cap, not dedupe
Verified: sonnet and haiku both read each file once per conversation and retain
it, so test32 never triggers read_app_file re-read dedupe. Dedupe is measured
deterministically by the micro-benchmark instead. Comment corrected to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): drop read_app_file re-read dedupe, ship the cap only
Benchmarking showed the per-conversation re-read dedupe never fires in practice:
across sonnet/opus/gpt-5.5/haiku, every model reads each file once per conversation
and keeps it in context (0 within-conversation re-reads). It was a correct but unused
guard, so this removes the ledger, full-file hash, retention predicate, the
AIChatManager wiring, and the eval decorateHelpers seam — keeping the read cap +
offset/limit/char paging (A), which is the lever that actually bounds context. The
micro-benchmark is now cap-only; test32 is kept as a multi-file read-load case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): add search_app grep tool for global raw-app chat (experimental)
Client-side grep over a raw app's frontend files and inline runnables (literal,
case-insensitive, optional file_glob/context_lines/max_matches, head-capped).
Completes the list -> search -> ranged-read triad. Includes the eval A/B gate
(WMILL_AI_EVAL_DISABLE_SEARCH_APP), unit tests + micro-benchmark, and a
find-all-usages eval case (global-test33).
Experimental: A/B benchmarking shows it is not an unconditional win — it helps
on find-all-usages but adds agentic iterations on navigable apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): accept search_app as a valid file-inspection tool in raw-app cases
Add requiredToolsAnyOf alternatives-group to ToolValidationSpec and switch
global-test29..32 to it so a model that locates files via search_app instead
of read_app_file no longer false-fails the tool assertion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: remove stale ai-chat context-optimization planning doc
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): drop read_app_file char paging for a hard char cap
The char_offset/char_limit params guarded minified files (a single line over
the char budget) but were effectively unused in benchmarks. Remove them and the
in-window char paging; keep the hard 50k-char budget and, when a read hits it,
tell the model to narrow the line limit (or treat the file as unreadable if a
single line exceeds the budget). Proper long-line handling is left as a TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): bake search_app context to 1 line, clarify query is literal
Drop the context_lines param (models varied it to little effect) for a fixed
SEARCH_APP_CONTEXT_LINES=1, and cap on matching lines instead of pushed rows so
max_matches stays accurate with context always on. Sharpen the query description
to state it is a literal (non-regex) substring and to suggest the call form
(e.g. formatCurrency() to hit call sites and skip formatCurrencyPrecise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): widen baked search_app context to 2 lines
Models that set the old context_lines param leaned to 2; match the lean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): count every file with a match in search_app header
Move fileHadMatch ahead of the render cap so files whose matches fall past max_matches are still counted (with a regression test). Also swap the raw NUL globstar sentinel for a printable escape (the NUL bytes made core.ts read as binary to grep) and reword two comments to describe current constraints instead of drafting history.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): drop redundant input echoes from app tool results
read_app_file and search_app no longer prefix results with the tool name or echo back the caller's own inputs (file path, query, file_glob) — the model already has them from the call args, and the unbounded query echo could push the search result past its output budget. Keeps the useful signals (line range, match/file counts, truncation) and the actionable advice. Also reword max_matches to 'matching lines' since it caps lines (each expands to context rows). Unit tests updated to the new format.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Record finalContextTokens per attempt: the input-token total of the last
model request (input + cache-creation + cache-read), i.e. how full the
context window ended up. Complements the cumulative tokenUsage.prompt,
which conflates context size with loop-iteration count.
Captured generically in the shared frontend runEval via the chat loop's
lastIterationUsage, so it covers all frontend modes (global/flow/script/
app), plus CLI mode via the last assistant turn's usage. Aggregated as
average and max over passed attempts and printed in the run summary.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add cli eval behavior checks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: harden cli eval command parsing
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add proxy transport for ai evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: strip propertyNames for gemini schemas
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: require explicit eval transport
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* chore: record app benchmark baseline
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: strengthen app benchmark persistence checks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: seed inventory tracker benchmark case
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add deterministic app diagnostics
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add app chat patch_file tool
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: add app session id micro-edit case
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: narrow app patch file content
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: stop gating app evals on lint
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: use compact json for flow patches
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: improve flow eval harness
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: record flow benchmark history
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: preserve schema in set flow json
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* style: clean set flow json schema guard
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: clean flow patch review followups
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add backend preview validation to ai evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: refresh shared preview workspace assets
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: harden shared backend preview validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
centdix