* feat: let test_run_flow name the conversation of a chat-mode test run
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: refuse a non-UUID conversation_id and mint chat test-run ids in one place
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(ai-evals): add a chat-flow follow-up case and mock flow preview runs
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: expect the conversation id argument on the manager's flow test bridge
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(ai-evals): require the chat-flow follow-up runs to share one conversation id
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: name the test_run_flow argument memory_id after the run parameter
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: let global evals seed the session's preview tabs
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: advertise the pinned artifact version in get_preview_status
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: reject ambiguous preview-tab and artifact eval fixtures
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: stop the AI chat destroying secret variables on edit
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: clear stale staged secret values and state the draft-staging rule
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: condense the pending-secret invariant to its field
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: refuse empty and oauth-managed secret values, keep drawer-staged ones in the draft
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: resolve a variable deploy's secret from one draft snapshot
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: make the variable draft the single source of a staged secret
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: drop stale in-memory secret invariants from comments and the eval
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: stop null account/expires_at leaking into variable drafts and diffs
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: report when a variable deploy leaves the secret value unchanged
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: scope the variable-value readability claims to the chat
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: correct the secret-draft invariant in the diff masking comment
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: record why a non-secret value is resent on a partial update
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: stop "Load secret value" discarding a staged secret
The audit-logged load writes the deployed secret into the draft row the
variable drawer shares with the AI chat, so offering it while that row
already stages a value silently replaces it — and the deploy that follows
carries the old value with no sign the staged one was lost.
The gate that hid the action already existed but keyed on
`isEncryptedDraftValue`, which only holds once a draft has round-tripped
through the server. A value staged in the same tab is still plaintext, so
it slipped through. Key on "anything staged" instead; clearing stays
explicit via Reset.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: extend the variable draft's empty-value sentinel past secrets
Two gaps in the chat's variable write path, both from treating "the draft
cannot carry this value" as meaning only "the value is secret".
`variableToDraftState` drops the value of an OAuth-managed variable so a
refreshed live token is never pinned into a draft, leaving '' behind. The
deploy body resent that '' verbatim for a non-secret one, wiping the token
the refresh flow owns. The sentinel now covers every value the draft is not
allowed to hold, which also removes the divergence from
`VariableEditor.save` and the shared deployer.
Making a variable secret when it holds no value produced a secret draft
staging '', a deploy body with no `value`, and the backend's "cannot change
is_secret without updating value too" — the sibling create path already
answers that case with guidance, so answer it here too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: gate the Secret toggle's secret load on the staged value too
The toggle calls `onLoadSecret` on every change so an is_secret flip has a
value to send, but that load overwrites the shared draft row — the same
discard the button gate just closed, reached by a different control.
It now loads only when the row stages nothing, which is exactly when the
flip needs a value fetched. With a value already staged there is one to
send, and it is the one the user or the chat put there.
Blocking the load costs the side effect that used to mask a worse bug: for
a deployed variable, the load replaced an `$encrypted:` marker with real
plaintext before save. Without it, un-securing a marker would store the
marker string as the value, since the deploy endpoints only decrypt it while
is_secret stays true. So the toggle is disabled outright while a marker is
staged — Reset first. That closes the marker case for draft-only variables
as well, where no load could ever have masked it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: add path-gated AI agent integration tests workflow
Runs integration_tests/ai_agent_tests against real LLM providers
(Anthropic/OpenAI/Google) only when AI-agent backend code or the tests
change, since runs make paid LLM calls. Adds a conftest fixture that
skips provider-parametrized cases whose API keys are absent, so CI
exercises only the providers it has secrets for.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: add path-gated ai_evals global-mode smoke workflow
Runs the global AI chat eval (global-test1) across one cheap model per
provider (Anthropic/OpenAI/Google/DeepSeek) only when the eval harness or
copilot chat code change, since runs make paid LLM calls. Builds Windmill
CE from source as the AI proxy; global tools/drafts run in the Vitest
bridge. Gates on the deterministic draft pipeline (run succeeded +
produced a draft + used write_script), not the variable LLM judge score.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run AI smokes on PR ready-for-review instead of every push
Switch the pull_request trigger from `synchronize` (every commit) to
`ready_for_review`, with a job guard skipping draft PRs, so the paid LLM
runs only fire when a PR is marked ready to merge (plus push-to-main and
manual dispatch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai_evals): lazily load cli mode so non-cli evals skip the cli toolchain
The entrypoint eagerly imported modes/cli, which pulls the wmill CLI
guidance modules and their JSR deps (@cliffy/*). Global/flow/script/app
runs then crashed with "Cannot find module '@cliffy/ansi/colors'" when
the cli workspace deps were not installed. Import createCliModeRunner
dynamically inside runCliBenchmark instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai_agent): raise low max_completion_tokens to OpenAI's 16 minimum
OpenAI's /v1/responses rejects max_output_tokens < 16 with a 400, failing
test_low_max_tokens for openai. 16 still exercises a truncated response.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run ai_evals workflow on Node 22 for the frontend undici 8.x dep
The Vitest bridge loads frontend/node_modules/undici@8.x, which requires
Node >=22.19; Node 20 failed with "webidl.util.markAsUncloneable is not a
function" when loading vitest.config.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai_evals): run frontend evals autonomously + give global-test1 more turns
Frontend evals (flow/script/app/global) ran the production chat prompt, which
assumes an interactive human — so cheaper models burned their turn budget
asking for confirmation, waiting for approval, or presenting a plan, sometimes
hitting maxTurns without producing a draft. Append a shared autonomy note in
baseEvalRunner (the path all frontend modes share, mirroring cli mode): act
directly on clear requests; only ask on genuinely ambiguous ones (preserving
the askUserQuestion cases). Also raise global-test1's maxTurns 8 -> 10 so a
model that over-explores still converges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci(ai_evals): watch draft/prompt deps outside copilot/
The global eval runs production frontend code in-process, so the smoke's
behavior depends on files outside frontend/src/lib/components/copilot/**:
the draft model (userDraft.svelte.ts, userDraftDbSyncer.svelte.ts), script
inference (infer.ts), and the chat system prompts ($system_prompts ->
system_prompts/auto-generated). Add them to both push and PR path filters so
a change there actually triggers the smoke that gates on draft production.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: skip direct provider tests without credentials
* feat: add ai evals skip judge flag
* fix: simplify ai evals ci gate
* fix: simplify ai evals smoke gate
* fix: handle ai eval workflow triggers
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add datatable tool coverage to global ai_evals (stage 0+1)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add seeded datatable difficulty-ladder global ai_evals (stage 2)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: skipJudge datatable evals and make stringIncludesAnyOf existential
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: make ai_evals datatable mock reflect SQL writes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add cli eval behavior checks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: harden cli eval command parsing
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add proxy transport for ai evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: strip propertyNames for gemini schemas
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: require explicit eval transport
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* chore: record app benchmark baseline
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: strengthen app benchmark persistence checks
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: seed inventory tracker benchmark case
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add deterministic app diagnostics
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add app chat patch_file tool
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: add app session id micro-edit case
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: narrow app patch file content
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: stop gating app evals on lint
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add backend preview validation to ai evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: refresh shared preview workspace assets
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: harden shared backend preview validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
GuilhemandClaude Opus 5
AlexRV12andClaude Opus 5
centdixandClaude Opus 4.8