mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-13 08:02:47 +00:00
python-instance-baseline
29
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  hugocasaandClaude Opus 5
|
8953ca670a |
feat: make hub integrations usable as examples in global AI chat (#10599)
* feat: make hub integrations usable as examples in global chat Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep web search guidance in sync with provider capability Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: resync web search guidance before the request is built Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: carry loop web search availability into every prompt rebuild Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: drop web search guidance on the completions api fallback Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct onBeforeIteration contract for the fallback re-entry Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: give the ai chat hub script descriptions and integration metadata Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: treat authored hub metadata as evidence the scripts were curated Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: treat hub curated as three-state and speak only for a stated true Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: let the benchmark hub serve integration metadata Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: benchmark the hub tool against a real integration's scripts and metadata Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: cover two more real integrations where the scripts already answer Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: report metadata_source from whether the fixture has authored meta Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: parse the hub resource type schema instead of relaying it as a string Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: match integration suggestions on slug words instead of substrings Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: search the integration a query names outright instead of ranking past it Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: reject hub integration slugs that would re-target the proxied request Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: add a mentioned integration's hits instead of filtering the search to it Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: keep named-integration hits under the content cap and unmangle fixture placeholders Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: cut the middle of a capped hub result instead of repeating its tail Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: cap hub results by keeping the best of the ranked and named hits Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: keep a named integration that ranking placed below the content cap Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: let plan mode use the hub integration lookup Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: mark which hub integrations carry authored provider knowledge The hub flags the integrations whose document holds hand-written provider knowledge, so a caller can tell before spending a call on the metadata endpoint: 18 of ~216 qualify, and for the rest the endpoint returns what was inferred from the same scripts a search already hands back. Carry the flag onto search results, where the model first meets a slug, so get_hub_integration is aimed at the few instead of guessed at. A hub that predates the flag omits it and nothing is marked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: stop the documented mark reading as a reason not to call The flag says the hub additionally holds provider knowledge checked against the live API. It is not a signal to skip the lookup elsewhere: that call still returns the resource type and the usage-ranked examples, neither of which a search result carries, and neither guessable. The prompt said to read a script instead when the mark is absent, which traded those for a guess on the ~198 undocumented integrations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: keep the benchmark's documented flag true to what it serves The eval hub reported Baremetrics and Holded as undocumented while its metadata endpoint handed back their authored notes, so a case could teach the model the flag means nothing. Derive it from both fixture sources, and pin the agreement. Also trims the documentedIntegrations comment to the four lines AGENTS.md allows, keeping the case-folding constraint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: keep the benchmark hub mostly undocumented, like the real one Adding holded and baremetrics to the documented set resolved the fixture's contradiction the wrong way: it left five of six integrations marked, against the live hub's 18 of ~216, and claimed authored notes for two the hub reports as having none. Drop the notes instead. Their auth and endpoints are in their shipped scripts, which is what the cases that use them are about. Also drops the last two places still describing the flag as a reason to spend or skip the metadata call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor: let the model name the integration instead of guessing it from the query Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor: word web search guidance conditionally instead of tracking provider capability Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: open the hub metadata route to job tokens and share one integrations fetch Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: record a copied hub script's source where write_script keeps it Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test: pin that a query narrows to the integration it was given Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: match integration suggestions on the name the hub curates Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: name an integration the hub leaves unnamed from the local word table Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 AlexRV12andClaude Opus 5
|
c8c06d8f79 |
feat(ai-chat): tell the chat which kind of app it is looking at (#11208)
* feat(ai-chat): tell the chat which kind of app it is looking at Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018r5ppn8YmHsM1MaAwY7xgB * refactor(ai-chat): drop a stale comment and hoist the prefix local Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018r5ppn8YmHsM1MaAwY7xgB * test(ai-chat): add a global eval for refusing to edit a drag-and-drop app Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018r5ppn8YmHsM1MaAwY7xgB * test(ai-chat): make the app-kind eval require editing the code app too Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018r5ppn8YmHsM1MaAwY7xgB --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 GuilhemandClaude Opus 5
|
6e1ef93f32 |
feat: let test_run_flow name the conversation of a chat-mode test run (#11198)
* feat: let test_run_flow name the conversation of a chat-mode test run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: refuse a non-UUID conversation_id and mint chat test-run ids in one place Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(ai-evals): add a chat-flow follow-up case and mock flow preview runs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: expect the conversation id argument on the manager's flow test bridge Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(ai-evals): require the chat-flow follow-up runs to share one conversation id Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: name the test_run_flow argument memory_id after the run parameter Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AlexRV12andClaude Opus 5
|
e954d33613 |
feat(ai-chat): add list_workers and list_data_metrics global tools (#11143)
* feat(ai-chat): add list_workers and list_data_metrics global tools Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ai-evals): keep the generated eval workspace id valid for the backend Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(ai-chat): trim the worker truncation note and added comments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * fix(ai-chat): only hedge an empty worker list when workers can be hidden Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * fix(ai-chat): show the empty-worker message in the tool card Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * fix(ai-chat): stop advertising workers in the api catalog tool Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * test(ai-evals): move the workspace id guard beside its function Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * fix(ai-chat): keep the empty metric note true for every filter shape Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * fix(ai-chat): resolve the caller role before judging an empty worker list Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * fix(ai-evals): collapse doubled hyphens in the generated workspace id Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * docs(ai-chat): describe what the ducklake tools and the worker hedge do Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * fix(ai-chat): name the missing lake when a metric table filter has none Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * fix(ai-chat): strip the ducklake scheme from the table retry hint Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq * test(ai-evals): add a global case for the declared measure path Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gi6UeLKJB9UaSWtjDCVaMq --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AlexRV12andClaude Opus 5
|
5bb37ca338 |
feat(ai-chat): merge get_job_logs and get_flow_run_details into get_run (#11172)
* feat(ai-chat): merge get_job_logs and get_flow_run_details into get_run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N1arUSfbbVQSq2DsjamxKF * fix(ai-chat): tell an unreadable log apart from an empty one in get_run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N1arUSfbbVQSq2DsjamxKF * fix(ai-chat): keep get_run's step tree optional when the read resolves undefined Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N1arUSfbbVQSq2DsjamxKF * fix(ai-chat): match each WINDMILL_TOO_BIG marker to the field it elides Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N1arUSfbbVQSq2DsjamxKF --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AlexRV12andClaude Opus 5
|
b50de89479 |
feat: run a deployed flow through the chat's argument form (#11085)
* feat: run a deployed flow through the chat's argument form Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vHs7Jr4UDSbUe2KGjugMw * refactor: drop the unread dynselect helper from the deployed flow run form Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vHs7Jr4UDSbUe2KGjugMw * fix: skip the preprocessor when the chat runs a deployed flow Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vHs7Jr4UDSbUe2KGjugMw * test: pin run_flow steering with ai_evals cases Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vHs7Jr4UDSbUe2KGjugMw * refactor: inline the deployed flow schema and trim the eval draft check Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vHs7Jr4UDSbUe2KGjugMw * docs: correct the stale draft-validation comment on the flow test-run eval Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015vHs7Jr4UDSbUe2KGjugMw --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AlexRV12andClaude Opus 5
|
a6abf2c8a7 |
feat: run and test scripts from the AI chat through an argument form (#11001)
* fix: disable a dynamic input when its schema field is disabled Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERed9zo2oJjpSczzMayaNh * refactor: extract the run form's argument hygiene into job_args Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERed9zo2oJjpSczzMayaNh * feat: give Tabs an opt-in sliding selection indicator Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERed9zo2oJjpSczzMayaNh * refactor: share the chat's scroll-fade measurement Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERed9zo2oJjpSczzMayaNh * feat: give the chat a run-form contract and incremental job output Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERed9zo2oJjpSczzMayaNh * feat: run and test a script from the chat through an argument form Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERed9zo2oJjpSczzMayaNh * feat: carry a chat run's card and job across saves and reloads Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERed9zo2oJjpSczzMayaNh * feat: render a chat run as a tool call row with its form, logs and result Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERed9zo2oJjpSczzMayaNh * feat: open a pending run form in the sessions preview pane Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERed9zo2oJjpSczzMayaNh * test: benchmark running a deployed script from the chat Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ERed9zo2oJjpSczzMayaNh * feat: offer a test run's dynamic options from the draft it previews Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: say that a test run's dynselect helper executes on form display * fix: send a schema default the model omitted when yolo skips the form * fix: infer a test run's schema when the stored one declares no properties * docs: tighten the note on the form's mount-time helper job * fix: apply a nested schema default the bypass posture counts as answered * fix: apply a declared default to a null value and an optional nested field * fix: check required fields inside a supplied optional object before bypassing * fix: read required args as own properties before bypassing the form * fix: stop the turn from the run form's action row in the preview panel * refactor: drop the run-form prediction and share its secret minting * refactor: prefill a proposed secret instead of emptying the field * docs: correct the comments the run-form prediction left behind * fix: keep a proposed secret out of the chat's stored messages * docs: say what a literal secret argument now does * test: restore the copilotInfo export the aiStore mock omits * docs: cut the run form's helper-script note to its constraints * refactor: settle a run form from one entry and fetch a job's logs once * fix: separate colliding secret paths, gate plan mode, keep polled logs * fix: mint before the form opens, skip empty fields, show what ran * revert: mint a run form's secrets at submit, not before it opens * fix: settle a cancelled run card on the form's arguments, not the proposal * fix: settle a stopped run form like a cancelled one, and keep an empty secret empty * fix: snapshot a run's arguments before minting its secrets --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
GuilhemandClaude Opus 5
|
e39dd7eb12 |
docs: teach agents to pass a resource as $res:<path> in run arguments (#10927)
* docs: teach agents to pass a resource as $res:<path> in run arguments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XjRARL7JA7xm772iJP4mJk * docs: extend run-argument rule to in-editor chats, fix run-as wording Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XjRARL7JA7xm772iJP4mJk * docs: tighten resource run-argument rule after review - Drop the false rationale that "$var:" only works inside a resource value from the write_variable description and its runtime rejection message; keep the rule (a variable cannot reference itself). - MCP resource-argument description: the title fallback renders "No title", so say the title is only a label rather than that it can be empty. Guard the real-newline fix with asserts in the existing enrichment test. - Eval: assert the full "$res:f/evals/global/github_main" value as one prefix so a wrong path with a right prefix fails. - resources.md: narrow "a trigger's payload" to its configured static args. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XjRARL7JA7xm772iJP4mJk * docs: scope the run-argument rule to global chat, add an exact eval matcher The ai_evals A/B on the two in-editor modes showed no effect: script mode sonnet 5/5 both with and without the description, flow mode sonnet 5/5 and haiku 5/5 on the baseline alone. A flow's input schema already carries `format: resource-<type>`, so those modes have a signal global mode does not give. Revert both files to keep the tool schemas free of a description that buys nothing per iteration; global mode keeps it, where haiku goes 0/5 -> 5/5. Add `stringEqualsAnyOf` to toolCallArgs and use it for the resource reference: nothing in the eval resolves the value, so a prefix match accepted a near-miss path like `$res:f/evals/global/github_main_backup`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XjRARL7JA7xm772iJP4mJk * docs: address cubic review — CLI wording, mock resource getter - `-d --data` help on all four run/preview commands: give $res: and $var: their own clauses instead of a parenthetical that read as if a resource were a kind of variable. - Mock backend: `getBenchmarkResource` now resolves AI-provider seeds as well as plain ones, so it agrees with `existsResource` and `listResource` — both report either kind, and a case that listed a resource and then read it by path got a row it could not fetch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XjRARL7JA7xm772iJP4mJk --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
GuilhemandClaude Opus 5
|
449b1a6933 |
fix: ground the chat's AI agent provider in the workspace's models (#10774)
* fix: ground the chat's AI agent provider in the workspace's models Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: never reject an AI agent model the catalog could not confirm Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: benchmark AI agent provider grounding in ai_evals Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: only reject an AI agent model an exhaustive listing rules out Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep instance-level AI settings out of the workspace provider catalog Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep untrusted model ids out of the chat's context Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: carry completeness on the model listing itself Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct two comments left behind by the catalog rework Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: bound the model listing and verify the default against it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: honour a workspace default a filtered listing names Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: recognise a workspace default past the prompt's model cap Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep an aliasing provider's unlisted model ids permissive Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AlexRV12andClaude Opus 5
|
850b028778 |
feat: advertise the pinned artifact version in get_preview_status (#10691)
* test: let global evals seed the session's preview tabs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: advertise the pinned artifact version in get_preview_status Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: reject ambiguous preview-tab and artifact eval fixtures Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
GuilhemandClaude Opus 5
|
eb238e3f0b |
fix: stop the AI chat destroying secret variables on edit (#10616)
* fix: stop the AI chat destroying secret variables on edit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: clear stale staged secret values and state the draft-staging rule Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: condense the pending-secret invariant to its field Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: refuse empty and oauth-managed secret values, keep drawer-staged ones in the draft Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: resolve a variable deploy's secret from one draft snapshot Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: make the variable draft the single source of a staged secret Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: drop stale in-memory secret invariants from comments and the eval Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop null account/expires_at leaking into variable drafts and diffs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: report when a variable deploy leaves the secret value unchanged Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: scope the variable-value readability claims to the chat Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct the secret-draft invariant in the diff masking comment Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: record why a non-secret value is resent on a partial update Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop "Load secret value" discarding a staged secret The audit-logged load writes the deployed secret into the draft row the variable drawer shares with the AI chat, so offering it while that row already stages a value silently replaces it — and the deploy that follows carries the old value with no sign the staged one was lost. The gate that hid the action already existed but keyed on `isEncryptedDraftValue`, which only holds once a draft has round-tripped through the server. A value staged in the same tab is still plaintext, so it slipped through. Key on "anything staged" instead; clearing stays explicit via Reset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: extend the variable draft's empty-value sentinel past secrets Two gaps in the chat's variable write path, both from treating "the draft cannot carry this value" as meaning only "the value is secret". `variableToDraftState` drops the value of an OAuth-managed variable so a refreshed live token is never pinned into a draft, leaving '' behind. The deploy body resent that '' verbatim for a non-secret one, wiping the token the refresh flow owns. The sentinel now covers every value the draft is not allowed to hold, which also removes the divergence from `VariableEditor.save` and the shared deployer. Making a variable secret when it holds no value produced a secret draft staging '', a deploy body with no `value`, and the backend's "cannot change is_secret without updating value too" — the sibling create path already answers that case with guidance, so answer it here too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: gate the Secret toggle's secret load on the staged value too The toggle calls `onLoadSecret` on every change so an is_secret flip has a value to send, but that load overwrites the shared draft row — the same discard the button gate just closed, reached by a different control. It now loads only when the row stages nothing, which is exactly when the flip needs a value fetched. With a value already staged there is one to send, and it is the one the user or the chat put there. Blocking the load costs the side effect that used to mask a worse bug: for a deployed variable, the load replaced an `$encrypted:` marker with real plaintext before save. Without it, un-securing a marker would store the marker string as the value, since the deploy endpoints only decrypt it while is_secret stays true. So the toggle is disabled outright while a marker is staged — Reset first. That closes the marker case for draft-only variables as well, where no load could ever have masked it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
GuilhemandClaude Fable 5
|
0f62891d43 |
fix(ai): stop teaching nonexistent while-loop iter.value state-carrying (#10345)
* fix(ai): stop teaching nonexistent while-loop iter.value state-carrying Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ai): scope while-loop results guidance to cross-iteration reads only Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ai): drop unverified wmill state-helper fallback from while-loop guidance Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ai): document supported cross-iteration results state in while loops Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ai): rescope while-loop fast-path rule and add results-carrying example Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
 centdix
|
ae088fd032 | stabilize global ai eval smoke path (#9745) | ||
|
centdixandClaude Opus 4.8
|
74a2329d2e |
feat(copilot): improve global-mode path selection + add path-selection evals (#9698)
* test: add global-mode path-selection eval cases with seeded user Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(copilot): guide global-mode path selection with injected folder list Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): tailor global-mode folder guidance for workspace admins Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): type folders_read; isolate global-eval user from store Addresses PR review: - Add folders_read to the User/whoami openapi schema and UserExt; the global prompt builder and eval harness now read it typed instead of via inline casts (regen the client to pick it up). - prepareGlobalSystemMessage takes an explicit user; the eval harness passes it rather than mutating the process-global userStore, removing the concurrency race (path cases no longer need --verbose). - Rewrite the path-selection case comment as a current invariant. - Add buildFolderGuidance unit tests in core.test.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.8
|
4296a6ae1f |
feat(ai-chat): cap read_app_file + search_app grep tool to bound context in large raw apps (#9653)
* docs: add global AI chat context-optimization plan for raw apps Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai-evals): add global raw-app debugging cases on a large fixture Adds a ~20-file analytics_dashboard raw-app fixture (incl. a 5k-line data module and a planted wrong-totals bug), two global cases (read-heavy debug + small-edit baseline), app-seed support in the mock backend, directory-fixture loading, and a decorateHelpers seam so read-dedupe is measurable. Records tokenUsage for before/ after comparison of the read-tool optimization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-chat): cap and dedupe read_app_file to bound context in large apps read_app_file now defaults to a head slice (1500 lines / 50k chars) with offset/ limit to page further, and skips resending a file whose earlier read is still in context (per-conversation ledger keyed off the originating tool-call id, so it self-heals after compaction). Bounds the file-content portion of global-chat context when working in large raw apps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai-evals): add read-heavy raw-app debug case (large data module) global-test31 induces the model to inspect the 5k-line seedData module, exercising the read_app_file cap/offset path. Baseline ~262k tokens vs ~200k with the cap+dedupe change (-24%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: record A+B benchmark results and fixed-overhead finding Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): clearer read_app_file past-EOF message + unit tests for cap/dedupe Addresses local-review nits: out-of-range offset now reports 'offset N is past the end of the file' instead of a backwards 'lines 11-10' label; adds unit coverage for the slicing (line cap, offset/limit window, char budget, past-EOF) and re-read dedupe (hit + miss-when-not-retained). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-chat): char-level paging + per-range dedupe for read_app_file Adds char_offset/char_limit so minified/long-line files can be paged within a line window, keys the re-read ledger by range (so reading different ranges no longer collides), and dedupes on the full-file hash (a cached range stub is invalidated when any byte of the file changes, not just the returned range). Tests updated for the char-slice behavior plus single-line capping, char paging, and out-of-window change detection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai-chat): add read_app_file context micro-benchmark + re-read eval case Adds a deterministic micro-benchmark (no LLM) that drives read_app_file through a realistic big-project read pattern (large file, re-read, minified bundle, paging) and asserts the cap+dedupe cut returned context >50% vs the old whole-file behavior — isolating the feature's effect from model nondeterminism and guarding against silent weakening. Adds global-test32, a cross-file consistency investigation that revisits overlapping files so re-read dedupe is exercised in a real run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai-evals): clarify test32 measures the read cap, not dedupe Verified: sonnet and haiku both read each file once per conversation and retain it, so test32 never triggers read_app_file re-read dedupe. Dedupe is measured deterministically by the micro-benchmark instead. Comment corrected to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-chat): drop read_app_file re-read dedupe, ship the cap only Benchmarking showed the per-conversation re-read dedupe never fires in practice: across sonnet/opus/gpt-5.5/haiku, every model reads each file once per conversation and keeps it in context (0 within-conversation re-reads). It was a correct but unused guard, so this removes the ledger, full-file hash, retention predicate, the AIChatManager wiring, and the eval decorateHelpers seam — keeping the read cap + offset/limit/char paging (A), which is the lever that actually bounds context. The micro-benchmark is now cap-only; test32 is kept as a multi-file read-load case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-chat): add search_app grep tool for global raw-app chat (experimental) Client-side grep over a raw app's frontend files and inline runnables (literal, case-insensitive, optional file_glob/context_lines/max_matches, head-capped). Completes the list -> search -> ranged-read triad. Includes the eval A/B gate (WMILL_AI_EVAL_DISABLE_SEARCH_APP), unit tests + micro-benchmark, and a find-all-usages eval case (global-test33). Experimental: A/B benchmarking shows it is not an unconditional win — it helps on find-all-usages but adds agentic iterations on navigable apps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai-evals): accept search_app as a valid file-inspection tool in raw-app cases Add requiredToolsAnyOf alternatives-group to ToolValidationSpec and switch global-test29..32 to it so a model that locates files via search_app instead of read_app_file no longer false-fails the tool assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: remove stale ai-chat context-optimization planning doc Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-chat): drop read_app_file char paging for a hard char cap The char_offset/char_limit params guarded minified files (a single line over the char budget) but were effectively unused in benchmarks. Remove them and the in-window char paging; keep the hard 50k-char budget and, when a read hits it, tell the model to narrow the line limit (or treat the file as unreadable if a single line exceeds the budget). Proper long-line handling is left as a TODO. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-chat): bake search_app context to 1 line, clarify query is literal Drop the context_lines param (models varied it to little effect) for a fixed SEARCH_APP_CONTEXT_LINES=1, and cap on matching lines instead of pushed rows so max_matches stays accurate with context always on. Sharpen the query description to state it is a literal (non-regex) substring and to suggest the call form (e.g. formatCurrency() to hit call sites and skip formatCurrencyPrecise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-chat): widen baked search_app context to 2 lines Models that set the old context_lines param leaned to 2; match the lean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): count every file with a match in search_app header Move fileHadMatch ahead of the render cap so files whose matches fall past max_matches are still counted (with a regression test). Also swap the raw NUL globstar sentinel for a printable escape (the NUL bytes made core.ts read as binary to grep) and reword two comments to describe current constraints instead of drafting history. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): drop redundant input echoes from app tool results read_app_file and search_app no longer prefix results with the tool name or echo back the caller's own inputs (file path, query, file_glob) — the model already has them from the call args, and the unbounded query echo could push the search result past its output budget. Keeps the useful signals (line range, match/file counts, truncation) and the actionable advice. Also reword max_matches to 'matching lines' since it caps lines (each expands to context rows). Unit tests updated to the new format. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.8
|
cfe5119035 |
feat(ai): add list_runs and get_job_logs tools to global chat mode (#9488)
* feat(ai): add list_runs and get_job_logs tools to global chat mode Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai): always suppress ansi hint in get_job_logs, drop misnamed param Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai_evals): add global list_runs and get_job_logs eval cases Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(ai): trim get_job_logs description and format global core Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai): surface list_runs/get_job_logs output as tool result The tools set showDetails but never set message.result, so the details panel rendered "No result yet" even on success. Set result in setToolStatus (logs go in result for get_job_logs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.8
|
343368fb5e |
test: add datatable tool coverage to global AI evals (#9398)
* test: add datatable tool coverage to global ai_evals (stage 0+1) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: add seeded datatable difficulty-ladder global ai_evals (stage 2) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: skipJudge datatable evals and make stringIncludesAnyOf existential Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: make ai_evals datatable mock reflect SQL writes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
centdix
|
f300a716a9 | test: add global chat resource variable schedule evals (#9379) | ||
|
centdix
|
9e7eaf3684 | feat: inject active editor into global chat (#9361) | ||
|
centdix
|
e29dfbaa87 |
test: add global chat eval coverage (#9320)
* test: improve global chat eval parity * test: add human-style global chat evals |
||
|
centdix
|
7a7d246a6e |
test: add global ai eval mode (#9129)
* feat: add global ai eval mode * fix: improve global eval validation feedback |
||
|
centdixandClaude Opus 4.5
|
b883f9a9d2 |
feat: add ai chat schedule and trigger tools (#8961)
* feat: add ai chat schedule and trigger tools * refactor: use zod for ai chat workspace tools * refactor: let ai provide runnable target fields * refactor: generate ai chat workspace tool schemas * fix: add object type to composed tool schemas * fix: avoid top-level trigger schema unions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: block undeployed workspace ai tools Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: inject ai workspace tool target Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add ai evals for workspace tools Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: make workspace tool eval prompts realistic Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: surface workspace tool errors * fix: show workspace tool success details * fix: describe workspace tool path format * fix: clarify workspace path examples * fix: tighten workspace tool validation * fix: align workspace tool prompts * chore: mark generated chat schemas * chore: mark generated cli skills --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
483fb1fb9a |
perf: reduce app ai chat token usage (#8928)
* test: add app chat token usage evals Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * perf: make app file listing metadata only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * perf: reduce app datatable prompt context Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add app datatable persistence eval Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: fix file manager rename app eval Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: remove selected app context eval cases Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: address app eval review feedback Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
46b2915a9d |
feat: improve app evals and localized app edits (#8863)
* chore: record app benchmark baseline Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: strengthen app benchmark persistence checks Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: seed inventory tracker benchmark case Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add deterministic app diagnostics Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add app chat patch_file tool Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add app session id micro-edit case Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: narrow app patch file content Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: stop gating app evals on lint Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
 
|
49844eb240 |
fix: encourage subflow reuse in AI chat flow builder prompt (#8839)
* docs: encourage subflow reuse in AI chat flow builder prompt Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add workspace flow reuse benchmark Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: centdix <farhadg110@gmail.com> |
||
|
centdixandClaude Opus 4.5
|
b39671d933 |
feat: add compact json patch tool to flow chat (#8840)
* fix: use compact json for flow patches Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: improve flow eval harness Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: record flow benchmark history Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: preserve schema in set flow json Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * style: clean set flow json schema guard Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: clean flow patch review followups Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
d3cb0c6220 |
fix: improve flow chat and benchmark coverage (#8825)
* fix: support special flow modules in evals Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: extract shared flow helper logic Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: make special flow tools openai-compatible Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: improve flow eval prompts and validation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: relax flow benchmark overfits Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: record updated flow benchmark history Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: address flow review findings Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: source flow chat special module prompt Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: narrow rawscript helper return type Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: dedupe flow chat prompt guidance Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: relax flow test10 validation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
f1e84cb088 |
chore: add backend preview validation to ai evals (#8827)
* feat: add backend preview validation to ai evals Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: refresh shared preview workspace assets Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: harden shared backend preview validation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdix
|
cdcc56461b | feat: add black-box ai eval benchmarks (#8618) |