test(ai-evals): add global case for quote-heavy inline flow code

Exercises write_flow creating a rawscript whose body is multi-line and
quote-heavy (the scenario the write_flow fix targets), so the global-mode
A/B can measure that code lands out-of-band via set_flow_module_code
rather than being escaped into the modules JSON string.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ruben Fiszel
2026-07-22 12:31:40 +00:00
co-authored by Claude Opus 4.8
parent c7e88e2cce
commit 4a28fa2c06
+29
View File
@@ -1739,3 +1739,32 @@
- creates an AI draft for the existing f/evals/global/format_greeting script with the exclamation-mark change
- the draft changes only the returned message's punctuation — summary, language, path, and the rest of the code are untouched
- does not deploy or save the draft to the workspace
- id: global-flow-quote-heavy-inline-code
prompt: |-
Create a draft flow at `f/evals/global/welcome_email`.
It takes a `name` input and returns a multi-line HTML email body that welcomes the user:
a heading, a paragraph greeting the user by name, and an unordered list of three getting-started tips.
Leave it as an AI draft only; do not deploy or save it.
runtime:
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
- type: flow
path: f/evals/global/welcome_email
valueIncludes:
- modules
- rawscript
toolExpect:
requiredToolsUsed:
- write_flow
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
judgeChecklist:
- creates a flow draft at f/evals/global/welcome_email
- the flow takes a name input and returns an HTML email body
- the returned HTML contains a heading, a paragraph that greets the user by name, and an unordered list of three getting-started tips
- the rawscript step actually contains the multi-line HTML code (it is not left empty or as a placeholder)
- the result stays as an AI draft and is not deployed or saved to the workspace