* ci: add path-gated AI agent integration tests workflow
Runs integration_tests/ai_agent_tests against real LLM providers
(Anthropic/OpenAI/Google) only when AI-agent backend code or the tests
change, since runs make paid LLM calls. Adds a conftest fixture that
skips provider-parametrized cases whose API keys are absent, so CI
exercises only the providers it has secrets for.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: add path-gated ai_evals global-mode smoke workflow
Runs the global AI chat eval (global-test1) across one cheap model per
provider (Anthropic/OpenAI/Google/DeepSeek) only when the eval harness or
copilot chat code change, since runs make paid LLM calls. Builds Windmill
CE from source as the AI proxy; global tools/drafts run in the Vitest
bridge. Gates on the deterministic draft pipeline (run succeeded +
produced a draft + used write_script), not the variable LLM judge score.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run AI smokes on PR ready-for-review instead of every push
Switch the pull_request trigger from `synchronize` (every commit) to
`ready_for_review`, with a job guard skipping draft PRs, so the paid LLM
runs only fire when a PR is marked ready to merge (plus push-to-main and
manual dispatch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai_evals): lazily load cli mode so non-cli evals skip the cli toolchain
The entrypoint eagerly imported modes/cli, which pulls the wmill CLI
guidance modules and their JSR deps (@cliffy/*). Global/flow/script/app
runs then crashed with "Cannot find module '@cliffy/ansi/colors'" when
the cli workspace deps were not installed. Import createCliModeRunner
dynamically inside runCliBenchmark instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai_agent): raise low max_completion_tokens to OpenAI's 16 minimum
OpenAI's /v1/responses rejects max_output_tokens < 16 with a 400, failing
test_low_max_tokens for openai. 16 still exercises a truncated response.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run ai_evals workflow on Node 22 for the frontend undici 8.x dep
The Vitest bridge loads frontend/node_modules/undici@8.x, which requires
Node >=22.19; Node 20 failed with "webidl.util.markAsUncloneable is not a
function" when loading vitest.config.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai_evals): run frontend evals autonomously + give global-test1 more turns
Frontend evals (flow/script/app/global) ran the production chat prompt, which
assumes an interactive human — so cheaper models burned their turn budget
asking for confirmation, waiting for approval, or presenting a plan, sometimes
hitting maxTurns without producing a draft. Append a shared autonomy note in
baseEvalRunner (the path all frontend modes share, mirroring cli mode): act
directly on clear requests; only ask on genuinely ambiguous ones (preserving
the askUserQuestion cases). Also raise global-test1's maxTurns 8 -> 10 so a
model that over-explores still converges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(ai_evals): watch draft/prompt deps outside copilot/
The global eval runs production frontend code in-process, so the smoke's
behavior depends on files outside frontend/src/lib/components/copilot/**:
the draft model (userDraft.svelte.ts, userDraftDbSyncer.svelte.ts), script
inference (infer.ts), and the chat system prompts ($system_prompts ->
system_prompts/auto-generated). Add them to both push and PR path filters so
a change there actually triggers the smoke that gates on draft production.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: skip direct provider tests without credentials
* feat: add ai evals skip judge flag
* fix: simplify ai evals ci gate
* fix: simplify ai evals smoke gate
* fix: handle ai eval workflow triggers
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add global-mode path-selection eval cases with seeded user
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(copilot): guide global-mode path selection with injected folder list
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(copilot): tailor global-mode folder guidance for workspace admins
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(copilot): type folders_read; isolate global-eval user from store
Addresses PR review:
- Add folders_read to the User/whoami openapi schema and UserExt; the global prompt builder and eval harness now read it typed instead of via inline casts (regen the client to pick it up).
- prepareGlobalSystemMessage takes an explicit user; the eval harness passes it rather than mutating the process-global userStore, removing the concurrency race (path cases no longer need --verbose).
- Rewrite the path-selection case comment as a current invariant.
- Add buildFolderGuidance unit tests in core.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add global AI chat context-optimization plan for raw apps
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): add global raw-app debugging cases on a large fixture
Adds a ~20-file analytics_dashboard raw-app fixture (incl. a 5k-line data module
and a planted wrong-totals bug), two global cases (read-heavy debug + small-edit
baseline), app-seed support in the mock backend, directory-fixture loading, and a
decorateHelpers seam so read-dedupe is measurable. Records tokenUsage for before/
after comparison of the read-tool optimization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): cap and dedupe read_app_file to bound context in large apps
read_app_file now defaults to a head slice (1500 lines / 50k chars) with offset/
limit to page further, and skips resending a file whose earlier read is still in
context (per-conversation ledger keyed off the originating tool-call id, so it
self-heals after compaction). Bounds the file-content portion of global-chat
context when working in large raw apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): add read-heavy raw-app debug case (large data module)
global-test31 induces the model to inspect the 5k-line seedData module, exercising
the read_app_file cap/offset path. Baseline ~262k tokens vs ~200k with the cap+dedupe
change (-24%).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: record A+B benchmark results and fixed-overhead finding
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): clearer read_app_file past-EOF message + unit tests for cap/dedupe
Addresses local-review nits: out-of-range offset now reports 'offset N is past the
end of the file' instead of a backwards 'lines 11-10' label; adds unit coverage for
the slicing (line cap, offset/limit window, char budget, past-EOF) and re-read dedupe
(hit + miss-when-not-retained).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): char-level paging + per-range dedupe for read_app_file
Adds char_offset/char_limit so minified/long-line files can be paged within a line
window, keys the re-read ledger by range (so reading different ranges no longer
collides), and dedupes on the full-file hash (a cached range stub is invalidated
when any byte of the file changes, not just the returned range). Tests updated for
the char-slice behavior plus single-line capping, char paging, and out-of-window
change detection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-chat): add read_app_file context micro-benchmark + re-read eval case
Adds a deterministic micro-benchmark (no LLM) that drives read_app_file through a
realistic big-project read pattern (large file, re-read, minified bundle, paging)
and asserts the cap+dedupe cut returned context >50% vs the old whole-file behavior
— isolating the feature's effect from model nondeterminism and guarding against
silent weakening. Adds global-test32, a cross-file consistency investigation that
revisits overlapping files so re-read dedupe is exercised in a real run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): clarify test32 measures the read cap, not dedupe
Verified: sonnet and haiku both read each file once per conversation and retain
it, so test32 never triggers read_app_file re-read dedupe. Dedupe is measured
deterministically by the micro-benchmark instead. Comment corrected to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): drop read_app_file re-read dedupe, ship the cap only
Benchmarking showed the per-conversation re-read dedupe never fires in practice:
across sonnet/opus/gpt-5.5/haiku, every model reads each file once per conversation
and keeps it in context (0 within-conversation re-reads). It was a correct but unused
guard, so this removes the ledger, full-file hash, retention predicate, the
AIChatManager wiring, and the eval decorateHelpers seam — keeping the read cap +
offset/limit/char paging (A), which is the lever that actually bounds context. The
micro-benchmark is now cap-only; test32 is kept as a multi-file read-load case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): add search_app grep tool for global raw-app chat (experimental)
Client-side grep over a raw app's frontend files and inline runnables (literal,
case-insensitive, optional file_glob/context_lines/max_matches, head-capped).
Completes the list -> search -> ranged-read triad. Includes the eval A/B gate
(WMILL_AI_EVAL_DISABLE_SEARCH_APP), unit tests + micro-benchmark, and a
find-all-usages eval case (global-test33).
Experimental: A/B benchmarking shows it is not an unconditional win — it helps
on find-all-usages but adds agentic iterations on navigable apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): accept search_app as a valid file-inspection tool in raw-app cases
Add requiredToolsAnyOf alternatives-group to ToolValidationSpec and switch
global-test29..32 to it so a model that locates files via search_app instead
of read_app_file no longer false-fails the tool assertion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: remove stale ai-chat context-optimization planning doc
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): drop read_app_file char paging for a hard char cap
The char_offset/char_limit params guarded minified files (a single line over
the char budget) but were effectively unused in benchmarks. Remove them and the
in-window char paging; keep the hard 50k-char budget and, when a read hits it,
tell the model to narrow the line limit (or treat the file as unreadable if a
single line exceeds the budget). Proper long-line handling is left as a TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): bake search_app context to 1 line, clarify query is literal
Drop the context_lines param (models varied it to little effect) for a fixed
SEARCH_APP_CONTEXT_LINES=1, and cap on matching lines instead of pushed rows so
max_matches stays accurate with context always on. Sharpen the query description
to state it is a literal (non-regex) substring and to suggest the call form
(e.g. formatCurrency() to hit call sites and skip formatCurrencyPrecise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-chat): widen baked search_app context to 2 lines
Models that set the old context_lines param leaned to 2; match the lean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): count every file with a match in search_app header
Move fileHadMatch ahead of the render cap so files whose matches fall past max_matches are still counted (with a regression test). Also swap the raw NUL globstar sentinel for a printable escape (the NUL bytes made core.ts read as binary to grep) and reword two comments to describe current constraints instead of drafting history.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): drop redundant input echoes from app tool results
read_app_file and search_app no longer prefix results with the tool name or echo back the caller's own inputs (file path, query, file_glob) — the model already has them from the call args, and the unbounded query echo could push the search result past its output budget. Keeps the useful signals (line range, match/file counts, truncation) and the actionable advice. Also reword max_matches to 'matching lines' since it caps lines (each expands to context rows). Unit tests updated to the new format.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Record finalContextTokens per attempt: the input-token total of the last
model request (input + cache-creation + cache-read), i.e. how full the
context window ended up. Complements the cumulative tokenUsage.prompt,
which conflates context size with loop-iteration count.
Captured generically in the shared frontend runEval via the chat loop's
lastIterationUsage, so it covers all frontend modes (global/flow/script/
app), plus CLI mode via the last assistant turn's usage. Aggregated as
average and max over passed attempts and printed in the run summary.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai): add list_runs and get_job_logs tools to global chat mode
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai): always suppress ansi hint in get_job_logs, drop misnamed param
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai_evals): add global list_runs and get_job_logs eval cases
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(ai): trim get_job_logs description and format global core
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai): surface list_runs/get_job_logs output as tool result
The tools set showDetails but never set message.result, so the details panel rendered "No result yet" even on success. Set result in setToolStatus (logs go in result for get_job_logs).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add datatable tool coverage to global ai_evals (stage 0+1)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add seeded datatable difficulty-ladder global ai_evals (stage 2)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: skipJudge datatable evals and make stringIncludesAnyOf existential
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: make ai_evals datatable mock reflect SQL writes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add global ai chat test tools
* fix: avoid session id in flow test preview
* test: cover global flow preview ids
* test: require script and flow test tools
* fix: harden global flow test fallback
* Revert "fix: harden global flow test fallback"
This reverts commit 97254ef33a.
* fix: fallback from inactive flow test hook
* fix: list nested flow steps in errors
* feat: add workflow-as-code skill
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: make system prompt freshness self-contained
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Revert "fix: make system prompt freshness self-contained"
This reverts commit 7d2fde9585.
* fix: refresh wac generated guidance
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: add wac cli eval cases
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: align wac prompt imports
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* feat: add cli eval behavior checks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: harden cli eval command parsing
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add proxy transport for ai evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: strip propertyNames for gemini schemas
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: require explicit eval transport
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: make flow chat code edits explicit
* refactor: centralize flow tree lookups
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: simplify flow chat tree mutations
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: reuse flow tree lookup in schema map
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: remove flow lookup alias
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: reuse flow tree in previous results
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: reuse canonical flow module lookup
* fix: align rebased flow helpers
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs: remove flow chat cleanup plan
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: remove flow chat helper wrappers
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: preserve non-flowmodule AI agent tools in skeleton and previous_result
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: consolidate flow module ID collectors into flowTree
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: search full flow tree in test_run_step to find special modules
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: recurse into aiagent tools in collectAllFlowModuleIds
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* chore: record app benchmark baseline
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: strengthen app benchmark persistence checks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: seed inventory tracker benchmark case
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add deterministic app diagnostics
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add app chat patch_file tool
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: add app session id micro-edit case
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: narrow app patch file content
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: stop gating app evals on lint
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: use compact json for flow patches
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: improve flow eval harness
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: record flow benchmark history
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: preserve schema in set flow json
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* style: clean set flow json schema guard
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: clean flow patch review followups
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: support special flow modules in evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: extract shared flow helper logic
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: make special flow tools openai-compatible
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: improve flow eval prompts and validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: relax flow benchmark overfits
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: record updated flow benchmark history
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: address flow review findings
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: source flow chat special module prompt
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: narrow rawscript helper return type
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: dedupe flow chat prompt guidance
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: relax flow test10 validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add backend preview validation to ai evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: refresh shared preview workspace assets
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: harden shared backend preview validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
centdix
Ruben Fiszel