mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
test(ai-evals): restore focused welcome_email flow case
Keep a retained global case for quote-heavy inline flow creation, per the ai-chat skill's requirement that a behavior change ship with an adjusted eval case. Benchmarked before (main) vs after (this PR's head) on Sonnet and GPT-4o: capable models inline unchanged (no regression) and the softened guidance no longer forces the multi-round-trip fill path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1739,3 +1739,32 @@
|
||||
- creates an AI draft for the existing f/evals/global/format_greeting script with the exclamation-mark change
|
||||
- the draft changes only the returned message's punctuation — summary, language, path, and the rest of the code are untouched
|
||||
- does not deploy or save the draft to the workspace
|
||||
|
||||
- id: global-flow-quote-heavy-inline-code
|
||||
prompt: |-
|
||||
Create a draft flow at `f/evals/global/welcome_email`.
|
||||
It takes a `name` input and returns a multi-line HTML email body that welcomes the user:
|
||||
a heading, a paragraph greeting the user by name, and an unordered list of three getting-started tips.
|
||||
Leave it as an AI draft only; do not deploy or save it.
|
||||
runtime:
|
||||
maxTurns: 10
|
||||
validate:
|
||||
draftCountExactly: 1
|
||||
requiredDrafts:
|
||||
- type: flow
|
||||
path: f/evals/global/welcome_email
|
||||
valueIncludes:
|
||||
- modules
|
||||
- rawscript
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- write_flow
|
||||
forbiddenToolsUsed:
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
judgeChecklist:
|
||||
- creates a flow draft at f/evals/global/welcome_email
|
||||
- the flow takes a name input and returns an HTML email body
|
||||
- the returned HTML contains a heading, a paragraph that greets the user by name, and an unordered list of three getting-started tips
|
||||
- the rawscript step actually contains the multi-line HTML code (it is not left empty or as a placeholder)
|
||||
- the result stays as an AI draft and is not deployed or saved to the workspace
|
||||
|
||||
Reference in New Issue
Block a user