* ci: move CI AI models to opus 5.5, gpt-6 and deepseek v4.1 flash
* test: point the deepseek eval alias at v4.1 flash
* ci: run the codex review on gpt-6.1-sol with codex cli 0.159.3
* chore: upgrade bun to 1.4.0 in dockerfiles and CI pins
* chore: move deno last in the language picker and relabel it Deno
* chore: move deno last in the pipeline language picker too
* chore: pin debugger image to bun 1.4.0 and trim the deno picker comment
* chore: state the deno picker constraint without referencing the old order
* fix: stamp bun lockfiles back to v1 while the fleet predates bun 1.4
* fix: ask bun for a v1 lockfile instead of rewriting one, and refuse an escalated lock
* chore: warn instead of silently storing a lockfile with no readable version
* test: let global evals seed the session's preview tabs
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: advertise the pinned artifact version in get_preview_status
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: reject ambiguous preview-tab and artifact eval fixtures
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: serve getJob in the ai evals benchmark api catalog
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: scope the frontend format hook to the frontend dir
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: answer the run-by-path endpoints and mirror the real getJob entry
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: gate the format hook on a repo-root frontend, not the project dir
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Update the pinned Rust toolchain from 1.93.0 to 1.97.0 (latest stable,
released 2026-07-09) across the worker/server build Dockerfiles
(Dockerfile, docker/DockerfileFull, docker/DockerfileFullEe) and all CI
workflows that pin a toolchain.
Verified the backend compiles cleanly with 1.97.0 under `-D warnings`
(the default RUSTFLAGS used by actions-rust-lang/setup-rust-toolchain).
Fixes WIN-2155
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: add path-gated AI agent integration tests workflow
Runs integration_tests/ai_agent_tests against real LLM providers
(Anthropic/OpenAI/Google) only when AI-agent backend code or the tests
change, since runs make paid LLM calls. Adds a conftest fixture that
skips provider-parametrized cases whose API keys are absent, so CI
exercises only the providers it has secrets for.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: add path-gated ai_evals global-mode smoke workflow
Runs the global AI chat eval (global-test1) across one cheap model per
provider (Anthropic/OpenAI/Google/DeepSeek) only when the eval harness or
copilot chat code change, since runs make paid LLM calls. Builds Windmill
CE from source as the AI proxy; global tools/drafts run in the Vitest
bridge. Gates on the deterministic draft pipeline (run succeeded +
produced a draft + used write_script), not the variable LLM judge score.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run AI smokes on PR ready-for-review instead of every push
Switch the pull_request trigger from `synchronize` (every commit) to
`ready_for_review`, with a job guard skipping draft PRs, so the paid LLM
runs only fire when a PR is marked ready to merge (plus push-to-main and
manual dispatch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai_evals): lazily load cli mode so non-cli evals skip the cli toolchain
The entrypoint eagerly imported modes/cli, which pulls the wmill CLI
guidance modules and their JSR deps (@cliffy/*). Global/flow/script/app
runs then crashed with "Cannot find module '@cliffy/ansi/colors'" when
the cli workspace deps were not installed. Import createCliModeRunner
dynamically inside runCliBenchmark instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai_agent): raise low max_completion_tokens to OpenAI's 16 minimum
OpenAI's /v1/responses rejects max_output_tokens < 16 with a 400, failing
test_low_max_tokens for openai. 16 still exercises a truncated response.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run ai_evals workflow on Node 22 for the frontend undici 8.x dep
The Vitest bridge loads frontend/node_modules/undici@8.x, which requires
Node >=22.19; Node 20 failed with "webidl.util.markAsUncloneable is not a
function" when loading vitest.config.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai_evals): run frontend evals autonomously + give global-test1 more turns
Frontend evals (flow/script/app/global) ran the production chat prompt, which
assumes an interactive human — so cheaper models burned their turn budget
asking for confirmation, waiting for approval, or presenting a plan, sometimes
hitting maxTurns without producing a draft. Append a shared autonomy note in
baseEvalRunner (the path all frontend modes share, mirroring cli mode): act
directly on clear requests; only ask on genuinely ambiguous ones (preserving
the askUserQuestion cases). Also raise global-test1's maxTurns 8 -> 10 so a
model that over-explores still converges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(ai_evals): watch draft/prompt deps outside copilot/
The global eval runs production frontend code in-process, so the smoke's
behavior depends on files outside frontend/src/lib/components/copilot/**:
the draft model (userDraft.svelte.ts, userDraftDbSyncer.svelte.ts), script
inference (infer.ts), and the chat system prompts ($system_prompts ->
system_prompts/auto-generated). Add them to both push and PR path filters so
a change there actually triggers the smoke that gates on draft production.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: skip direct provider tests without credentials
* feat: add ai evals skip judge flag
* fix: simplify ai evals ci gate
* fix: simplify ai evals smoke gate
* fix: handle ai eval workflow triggers
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ruben Fiszel
dependabot[bot]
AlexRV12andClaude Opus 5
Guilhem
centdixandClaude Opus 4.8