Files
windmill/ai_evals/adapters/frontend
GuilhemandClaude Opus 5 e39dd7eb12 docs: teach agents to pass a resource as $res:<path> in run arguments (#10927)
* docs: teach agents to pass a resource as $res:<path> in run arguments

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XjRARL7JA7xm772iJP4mJk

* docs: extend run-argument rule to in-editor chats, fix run-as wording

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XjRARL7JA7xm772iJP4mJk

* docs: tighten resource run-argument rule after review

- Drop the false rationale that "$var:" only works inside a resource value
  from the write_variable description and its runtime rejection message; keep
  the rule (a variable cannot reference itself).
- MCP resource-argument description: the title fallback renders "No title",
  so say the title is only a label rather than that it can be empty. Guard the
  real-newline fix with asserts in the existing enrichment test.
- Eval: assert the full "$res:f/evals/global/github_main" value as one prefix
  so a wrong path with a right prefix fails.
- resources.md: narrow "a trigger's payload" to its configured static args.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XjRARL7JA7xm772iJP4mJk

* docs: scope the run-argument rule to global chat, add an exact eval matcher

The ai_evals A/B on the two in-editor modes showed no effect: script mode
sonnet 5/5 both with and without the description, flow mode sonnet 5/5 and
haiku 5/5 on the baseline alone. A flow's input schema already carries
`format: resource-<type>`, so those modes have a signal global mode does not
give. Revert both files to keep the tool schemas free of a description that
buys nothing per iteration; global mode keeps it, where haiku goes 0/5 -> 5/5.

Add `stringEqualsAnyOf` to toolCallArgs and use it for the resource reference:
nothing in the eval resolves the value, so a prefix match accepted a near-miss
path like `$res:f/evals/global/github_main_backup`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XjRARL7JA7xm772iJP4mJk

* docs: address cubic review — CLI wording, mock resource getter

- `-d --data` help on all four run/preview commands: give $res: and $var:
  their own clauses instead of a parenthetical that read as if a resource
  were a kind of variable.
- Mock backend: `getBenchmarkResource` now resolves AI-provider seeds as well
  as plain ones, so it agrees with `existsResource` and `listResource` — both
  report either kind, and a case that listed a resource and then read it by
  path got a row it could not fetch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XjRARL7JA7xm772iJP4mJk

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-03 11:29:54 +02:00
..