From 4f2d0c77c1c4f42b60e99ea38bec39dd8ec10a9e Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Wed, 22 Jul 2026 14:04:30 +0000 Subject: [PATCH] test(ai-evals): drop global quote-heavy inline-code case The manual global-mode A/B (Sonnet, Gemini 3 flash/pro, GPT-4o) showed no pass-rate delta: the GPT-5 inline-escaping failure this change targets does not reproduce on any available model, so the case guards nothing measurable. Keep the unit tests as the regression guard instead. Co-Authored-By: Claude Opus 4.8 (1M context) --- ai_evals/cases/global.yaml | 29 ----------------------------- 1 file changed, 29 deletions(-) diff --git a/ai_evals/cases/global.yaml b/ai_evals/cases/global.yaml index eb7ad5901d..e2676c0b2a 100644 --- a/ai_evals/cases/global.yaml +++ b/ai_evals/cases/global.yaml @@ -1739,32 +1739,3 @@ - creates an AI draft for the existing f/evals/global/format_greeting script with the exclamation-mark change - the draft changes only the returned message's punctuation — summary, language, path, and the rest of the code are untouched - does not deploy or save the draft to the workspace - -- id: global-flow-quote-heavy-inline-code - prompt: |- - Create a draft flow at `f/evals/global/welcome_email`. - It takes a `name` input and returns a multi-line HTML email body that welcomes the user: - a heading, a paragraph greeting the user by name, and an unordered list of three getting-started tips. - Leave it as an AI draft only; do not deploy or save it. - runtime: - maxTurns: 10 - validate: - draftCountExactly: 1 - requiredDrafts: - - type: flow - path: f/evals/global/welcome_email - valueIncludes: - - modules - - rawscript - toolExpect: - requiredToolsUsed: - - write_flow - forbiddenToolsUsed: - - deploy_workspace_item - - delete_workspace_item - judgeChecklist: - - creates a flow draft at f/evals/global/welcome_email - - the flow takes a name input and returns an HTML email body - - the returned HTML contains a heading, a paragraph that greets the user by name, and an unordered list of three getting-started tips - - the rawscript step actually contains the multi-line HTML code (it is not left empty or as a placeholder) - - the result stays as an AI draft and is not deployed or saved to the workspace