test(ai-evals): restore focused welcome_email flow case

Keep a retained global case for quote-heavy inline flow creation, per the
ai-chat skill's requirement that a behavior change ship with an adjusted
eval case. Benchmarked before (main) vs after (this PR's head) on Sonnet
and GPT-4o: capable models inline unchanged (no regression) and the
softened guidance no longer forces the multi-round-trip fill path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ruben Fiszel
2026-07-22 14:23:47 +00:00
parent 4f2d0c77c1
commit 64c8f2beca
+29
View File
@@ -1739,3 +1739,32 @@
- creates an AI draft for the existing f/evals/global/format_greeting script with the exclamation-mark change
- the draft changes only the returned message's punctuation — summary, language, path, and the rest of the code are untouched
- does not deploy or save the draft to the workspace
- id: global-flow-quote-heavy-inline-code
prompt: |-
Create a draft flow at `f/evals/global/welcome_email`.
It takes a `name` input and returns a multi-line HTML email body that welcomes the user:
a heading, a paragraph greeting the user by name, and an unordered list of three getting-started tips.
Leave it as an AI draft only; do not deploy or save it.
runtime:
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
- type: flow
path: f/evals/global/welcome_email
valueIncludes:
- modules
- rawscript
toolExpect:
requiredToolsUsed:
- write_flow
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
judgeChecklist:
- creates a flow draft at f/evals/global/welcome_email
- the flow takes a name input and returns an HTML email body
- the returned HTML contains a heading, a paragraph that greets the user by name, and an unordered list of three getting-started tips
- the rawscript step actually contains the multi-line HTML code (it is not left empty or as a placeholder)
- the result stays as an AI draft and is not deployed or saved to the workspace