* ci: add path-gated AI agent integration tests workflow
Runs integration_tests/ai_agent_tests against real LLM providers
(Anthropic/OpenAI/Google) only when AI-agent backend code or the tests
change, since runs make paid LLM calls. Adds a conftest fixture that
skips provider-parametrized cases whose API keys are absent, so CI
exercises only the providers it has secrets for.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: add path-gated ai_evals global-mode smoke workflow
Runs the global AI chat eval (global-test1) across one cheap model per
provider (Anthropic/OpenAI/Google/DeepSeek) only when the eval harness or
copilot chat code change, since runs make paid LLM calls. Builds Windmill
CE from source as the AI proxy; global tools/drafts run in the Vitest
bridge. Gates on the deterministic draft pipeline (run succeeded +
produced a draft + used write_script), not the variable LLM judge score.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run AI smokes on PR ready-for-review instead of every push
Switch the pull_request trigger from `synchronize` (every commit) to
`ready_for_review`, with a job guard skipping draft PRs, so the paid LLM
runs only fire when a PR is marked ready to merge (plus push-to-main and
manual dispatch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai_evals): lazily load cli mode so non-cli evals skip the cli toolchain
The entrypoint eagerly imported modes/cli, which pulls the wmill CLI
guidance modules and their JSR deps (@cliffy/*). Global/flow/script/app
runs then crashed with "Cannot find module '@cliffy/ansi/colors'" when
the cli workspace deps were not installed. Import createCliModeRunner
dynamically inside runCliBenchmark instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai_agent): raise low max_completion_tokens to OpenAI's 16 minimum
OpenAI's /v1/responses rejects max_output_tokens < 16 with a 400, failing
test_low_max_tokens for openai. 16 still exercises a truncated response.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run ai_evals workflow on Node 22 for the frontend undici 8.x dep
The Vitest bridge loads frontend/node_modules/undici@8.x, which requires
Node >=22.19; Node 20 failed with "webidl.util.markAsUncloneable is not a
function" when loading vitest.config.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai_evals): run frontend evals autonomously + give global-test1 more turns
Frontend evals (flow/script/app/global) ran the production chat prompt, which
assumes an interactive human — so cheaper models burned their turn budget
asking for confirmation, waiting for approval, or presenting a plan, sometimes
hitting maxTurns without producing a draft. Append a shared autonomy note in
baseEvalRunner (the path all frontend modes share, mirroring cli mode): act
directly on clear requests; only ask on genuinely ambiguous ones (preserving
the askUserQuestion cases). Also raise global-test1's maxTurns 8 -> 10 so a
model that over-explores still converges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(ai_evals): watch draft/prompt deps outside copilot/
The global eval runs production frontend code in-process, so the smoke's
behavior depends on files outside frontend/src/lib/components/copilot/**:
the draft model (userDraft.svelte.ts, userDraftDbSyncer.svelte.ts), script
inference (infer.ts), and the chat system prompts ($system_prompts ->
system_prompts/auto-generated). Add them to both push and PR path filters so
a change there actually triggers the smoke that gates on draft production.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: skip direct provider tests without credentials
* feat: add ai evals skip judge flag
* fix: simplify ai evals ci gate
* fix: simplify ai evals smoke gate
* fix: handle ai eval workflow triggers
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add global AI chat context-optimization plan for raw apps
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): add global raw-app debugging cases on a large fixture
Adds a ~20-file analytics_dashboard raw-app fixture (incl. a 5k-line data module
and a planted wrong-totals bug), two global cases (read-heavy debug + small-edit
baseline), app-seed support in the mock backend, directory-fixture loading, and a
decorateHelpers seam so read-dedupe is measurable. Records tokenUsage for before/
after comparison of the read-tool optimization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): cap and dedupe read_app_file to bound context in large apps
read_app_file now defaults to a head slice (1500 lines / 50k chars) with offset/
limit to page further, and skips resending a file whose earlier read is still in
context (per-conversation ledger keyed off the originating tool-call id, so it
self-heals after compaction). Bounds the file-content portion of global-chat
context when working in large raw apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): add read-heavy raw-app debug case (large data module)
global-test31 induces the model to inspect the 5k-line seedData module, exercising
the read_app_file cap/offset path. Baseline ~262k tokens vs ~200k with the cap+dedupe
change (-24%).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record A+B benchmark results and fixed-overhead finding
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): clearer read_app_file past-EOF message + unit tests for cap/dedupe
Addresses local-review nits: out-of-range offset now reports 'offset N is past the
end of the file' instead of a backwards 'lines 11-10' label; adds unit coverage for
the slicing (line cap, offset/limit window, char budget, past-EOF) and re-read dedupe
(hit + miss-when-not-retained).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): char-level paging + per-range dedupe for read_app_file
Adds char_offset/char_limit so minified/long-line files can be paged within a line
window, keys the re-read ledger by range (so reading different ranges no longer
collides), and dedupes on the full-file hash (a cached range stub is invalidated
when any byte of the file changes, not just the returned range). Tests updated for
the char-slice behavior plus single-line capping, char paging, and out-of-window
change detection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-chat): add read_app_file context micro-benchmark + re-read eval case
Adds a deterministic micro-benchmark (no LLM) that drives read_app_file through a
realistic big-project read pattern (large file, re-read, minified bundle, paging)
and asserts the cap+dedupe cut returned context >50% vs the old whole-file behavior
— isolating the feature's effect from model nondeterminism and guarding against
silent weakening. Adds global-test32, a cross-file consistency investigation that
revisits overlapping files so re-read dedupe is exercised in a real run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): clarify test32 measures the read cap, not dedupe
Verified: sonnet and haiku both read each file once per conversation and retain
it, so test32 never triggers read_app_file re-read dedupe. Dedupe is measured
deterministically by the micro-benchmark instead. Comment corrected to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): drop read_app_file re-read dedupe, ship the cap only
Benchmarking showed the per-conversation re-read dedupe never fires in practice:
across sonnet/opus/gpt-5.5/haiku, every model reads each file once per conversation
and keeps it in context (0 within-conversation re-reads). It was a correct but unused
guard, so this removes the ledger, full-file hash, retention predicate, the
AIChatManager wiring, and the eval decorateHelpers seam — keeping the read cap +
offset/limit/char paging (A), which is the lever that actually bounds context. The
micro-benchmark is now cap-only; test32 is kept as a multi-file read-load case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): add search_app grep tool for global raw-app chat (experimental)
Client-side grep over a raw app's frontend files and inline runnables (literal,
case-insensitive, optional file_glob/context_lines/max_matches, head-capped).
Completes the list -> search -> ranged-read triad. Includes the eval A/B gate
(WMILL_AI_EVAL_DISABLE_SEARCH_APP), unit tests + micro-benchmark, and a
find-all-usages eval case (global-test33).
Experimental: A/B benchmarking shows it is not an unconditional win — it helps
on find-all-usages but adds agentic iterations on navigable apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): accept search_app as a valid file-inspection tool in raw-app cases
Add requiredToolsAnyOf alternatives-group to ToolValidationSpec and switch
global-test29..32 to it so a model that locates files via search_app instead
of read_app_file no longer false-fails the tool assertion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: remove stale ai-chat context-optimization planning doc
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): drop read_app_file char paging for a hard char cap
The char_offset/char_limit params guarded minified files (a single line over
the char budget) but were effectively unused in benchmarks. Remove them and the
in-window char paging; keep the hard 50k-char budget and, when a read hits it,
tell the model to narrow the line limit (or treat the file as unreadable if a
single line exceeds the budget). Proper long-line handling is left as a TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): bake search_app context to 1 line, clarify query is literal
Drop the context_lines param (models varied it to little effect) for a fixed
SEARCH_APP_CONTEXT_LINES=1, and cap on matching lines instead of pushed rows so
max_matches stays accurate with context always on. Sharpen the query description
to state it is a literal (non-regex) substring and to suggest the call form
(e.g. formatCurrency() to hit call sites and skip formatCurrencyPrecise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): widen baked search_app context to 2 lines
Models that set the old context_lines param leaned to 2; match the lean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): count every file with a match in search_app header
Move fileHadMatch ahead of the render cap so files whose matches fall past max_matches are still counted (with a regression test). Also swap the raw NUL globstar sentinel for a printable escape (the NUL bytes made core.ts read as binary to grep) and reword two comments to describe current constraints instead of drafting history.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): drop redundant input echoes from app tool results
read_app_file and search_app no longer prefix results with the tool name or echo back the caller's own inputs (file path, query, file_glob) — the model already has them from the call args, and the unbounded query echo could push the search result past its output budget. Keeps the useful signals (line range, match/file counts, truncation) and the actionable advice. Also reword max_matches to 'matching lines' since it caps lines (each expands to context rows). Unit tests updated to the new format.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Record finalContextTokens per attempt: the input-token total of the last
model request (input + cache-creation + cache-read), i.e. how full the
context window ended up. Complements the cumulative tokenUsage.prompt,
which conflates context size with loop-iteration count.
Captured generically in the shared frontend runEval via the chat loop's
lastIterationUsage, so it covers all frontend modes (global/flow/script/
app), plus CLI mode via the last assistant turn's usage. Aggregated as
average and max over passed attempts and printed in the run summary.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add datatable tool coverage to global ai_evals (stage 0+1)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add seeded datatable difficulty-ladder global ai_evals (stage 2)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: skipJudge datatable evals and make stringIncludesAnyOf existential
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: make ai_evals datatable mock reflect SQL writes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add global ai chat test tools
* fix: avoid session id in flow test preview
* test: cover global flow preview ids
* test: require script and flow test tools
* fix: harden global flow test fallback
* Revert "fix: harden global flow test fallback"
This reverts commit 97254ef33a.
* fix: fallback from inactive flow test hook
* fix: list nested flow steps in errors
* feat: add cli eval behavior checks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: harden cli eval command parsing
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add proxy transport for ai evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: strip propertyNames for gemini schemas
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: require explicit eval transport
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* chore: record app benchmark baseline
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: strengthen app benchmark persistence checks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: seed inventory tracker benchmark case
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add deterministic app diagnostics
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add app chat patch_file tool
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: add app session id micro-edit case
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: narrow app patch file content
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: stop gating app evals on lint
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: use compact json for flow patches
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: improve flow eval harness
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: record flow benchmark history
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: preserve schema in set flow json
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* style: clean set flow json schema guard
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: clean flow patch review followups
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: support special flow modules in evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: extract shared flow helper logic
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: make special flow tools openai-compatible
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: improve flow eval prompts and validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: relax flow benchmark overfits
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: record updated flow benchmark history
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: address flow review findings
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: source flow chat special module prompt
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: narrow rawscript helper return type
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: dedupe flow chat prompt guidance
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: relax flow test10 validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add backend preview validation to ai evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: refresh shared preview workspace assets
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: harden shared backend preview validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
centdix
Ruben Fiszel