mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-20 00:02:19 +00:00
centdix
b0ddcf31e4
* ci: add path-gated AI agent integration tests workflow Runs integration_tests/ai_agent_tests against real LLM providers (Anthropic/OpenAI/Google) only when AI-agent backend code or the tests change, since runs make paid LLM calls. Adds a conftest fixture that skips provider-parametrized cases whose API keys are absent, so CI exercises only the providers it has secrets for. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: add path-gated ai_evals global-mode smoke workflow Runs the global AI chat eval (global-test1) across one cheap model per provider (Anthropic/OpenAI/Google/DeepSeek) only when the eval harness or copilot chat code change, since runs make paid LLM calls. Builds Windmill CE from source as the AI proxy; global tools/drafts run in the Vitest bridge. Gates on the deterministic draft pipeline (run succeeded + produced a draft + used write_script), not the variable LLM judge score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: run AI smokes on PR ready-for-review instead of every push Switch the pull_request trigger from `synchronize` (every commit) to `ready_for_review`, with a job guard skipping draft PRs, so the paid LLM runs only fire when a PR is marked ready to merge (plus push-to-main and manual dispatch). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai_evals): lazily load cli mode so non-cli evals skip the cli toolchain The entrypoint eagerly imported modes/cli, which pulls the wmill CLI guidance modules and their JSR deps (@cliffy/*). Global/flow/script/app runs then crashed with "Cannot find module '@cliffy/ansi/colors'" when the cli workspace deps were not installed. Import createCliModeRunner dynamically inside runCliBenchmark instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai_agent): raise low max_completion_tokens to OpenAI's 16 minimum OpenAI's /v1/responses rejects max_output_tokens < 16 with a 400, failing test_low_max_tokens for openai. 16 still exercises a truncated response. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: run ai_evals workflow on Node 22 for the frontend undici 8.x dep The Vitest bridge loads frontend/node_modules/undici@8.x, which requires Node >=22.19; Node 20 failed with "webidl.util.markAsUncloneable is not a function" when loading vitest.config.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai_evals): run frontend evals autonomously + give global-test1 more turns Frontend evals (flow/script/app/global) ran the production chat prompt, which assumes an interactive human — so cheaper models burned their turn budget asking for confirmation, waiting for approval, or presenting a plan, sometimes hitting maxTurns without producing a draft. Append a shared autonomy note in baseEvalRunner (the path all frontend modes share, mirroring cli mode): act directly on clear requests; only ask on genuinely ambiguous ones (preserving the askUserQuestion cases). Also raise global-test1's maxTurns 8 -> 10 so a model that over-explores still converges. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(ai_evals): watch draft/prompt deps outside copilot/ The global eval runs production frontend code in-process, so the smoke's behavior depends on files outside frontend/src/lib/components/copilot/**: the draft model (userDraft.svelte.ts, userDraftDbSyncer.svelte.ts), script inference (infer.ts), and the chat system prompts ($system_prompts -> system_prompts/auto-generated). Add them to both push and PR path filters so a change there actually triggers the smoke that gates on draft production. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: skip direct provider tests without credentials * feat: add ai evals skip judge flag * fix: simplify ai evals ci gate * fix: simplify ai evals smoke gate * fix: handle ai eval workflow triggers --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1224 lines
41 KiB
YAML
1224 lines
41 KiB
YAML
- id: global-test1-script-create
|
|
prompt: |-
|
|
Create a draft Bun script at `f/evals/global/greet_user`.
|
|
It should take a string `name` input and return `Hello, ${name}!`.
|
|
Leave it as an AI draft only; do not deploy or save it.
|
|
runtime:
|
|
maxTurns: 10
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: script
|
|
path: f/evals/global/greet_user
|
|
language: bun
|
|
valueIncludes:
|
|
- name
|
|
- Hello
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_script
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- creates a Bun script draft at f/evals/global/greet_user
|
|
- the script accepts a name input
|
|
- the script returns a greeting containing Hello, the provided name, and an exclamation mark
|
|
- the result stays as an AI draft and is not deployed or saved to the workspace
|
|
|
|
- id: global-test2-script-edit-existing
|
|
prompt: |-
|
|
Update the existing workspace script at `f/evals/global/format_greeting`.
|
|
Keep it as a Bun script, but change the greeting so the provided name is uppercased and the returned message ends with an exclamation mark.
|
|
Leave the result as an AI draft only; do not deploy or save it.
|
|
initial: ai_evals/fixtures/frontend/global/initial/format_greeting_script.json
|
|
runtime:
|
|
maxTurns: 8
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: script
|
|
path: f/evals/global/format_greeting
|
|
language: bun
|
|
valueIncludes:
|
|
- toUpperCase
|
|
- "!"
|
|
toolExpect:
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- creates an AI draft for the existing f/evals/global/format_greeting script
|
|
- preserves the script as Bun
|
|
- uppercases the provided name in the greeting
|
|
- returns a message ending with an exclamation mark
|
|
- does not deploy or save the draft to the workspace
|
|
|
|
- id: global-test3-flow-create
|
|
prompt: |-
|
|
Create a draft flow at `f/evals/global/sum_numbers`.
|
|
It should take two numeric inputs, `a` and `b`, and return their sum.
|
|
Leave it as an AI draft only; do not deploy or save it.
|
|
runtime:
|
|
maxTurns: 8
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: flow
|
|
path: f/evals/global/sum_numbers
|
|
valueIncludes:
|
|
- modules
|
|
- rawscript
|
|
- flow_input.a
|
|
- flow_input.b
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_flow
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
toolCallArgs:
|
|
- tool: write_flow
|
|
field: modules
|
|
stringStartsWithAnyOf:
|
|
- "["
|
|
judgeChecklist:
|
|
- creates a flow draft at f/evals/global/sum_numbers
|
|
- the flow accepts numeric inputs a and b
|
|
- the flow returns the sum of a and b
|
|
- the result stays as an AI draft and is not deployed or saved to the workspace
|
|
|
|
- id: global-test4-multi-artifact-notification-job
|
|
prompt: |-
|
|
Set up a draft stale-trial notification job.
|
|
Create a Bun script at `f/evals/global/check_stale_trials` that accepts `max_age_days`, uses mocked inline trial account data, and returns the stale trial account IDs.
|
|
Also create a weekday 09:00 UTC schedule at `f/evals/global/check_stale_trials_weekday` for that script with `max_age_days` set to 14.
|
|
Add an HTTP POST trigger at `f/evals/global/check_stale_trials_manual` with route path `evals/check-stale-trials` that runs the same script manually.
|
|
Leave everything as AI drafts only; do not deploy or save anything to the workspace.
|
|
runtime:
|
|
maxTurns: 12
|
|
validate:
|
|
draftCountExactly: 3
|
|
requiredDrafts:
|
|
- type: script
|
|
path: f/evals/global/check_stale_trials
|
|
language: bun
|
|
valueIncludes:
|
|
- max_age_days
|
|
- trial
|
|
- type: schedule
|
|
path: f/evals/global/check_stale_trials_weekday
|
|
valueIncludes:
|
|
- f/evals/global/check_stale_trials
|
|
- UTC
|
|
- "14"
|
|
- type: trigger
|
|
triggerKind: http
|
|
path: f/evals/global/check_stale_trials_manual
|
|
valueIncludes:
|
|
- evals/check-stale-trials
|
|
- f/evals/global/check_stale_trials
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_script
|
|
- write_schedule
|
|
- write_trigger
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- creates a Bun script draft for stale trial accounts
|
|
- creates a weekday 09:00 UTC schedule draft for the script with max_age_days set to 14
|
|
- creates an HTTP POST trigger draft with route path evals/check-stale-trials for the same script
|
|
- leaves all artifacts as drafts only and does not deploy
|
|
|
|
- id: global-test5-existing-flow-inline-code-edit
|
|
prompt: |-
|
|
Update the existing flow at `f/evals/global/process_invoice`.
|
|
Only change the `calculate_total` inline code so it applies 8% tax and returns an object containing `subtotal`, `tax`, and `total`.
|
|
Leave the updated flow as an AI draft only; do not deploy or save it.
|
|
initial: ai_evals/fixtures/frontend/global/initial/process_invoice_flow.json
|
|
runtime:
|
|
maxTurns: 10
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: flow
|
|
path: f/evals/global/process_invoice
|
|
valueIncludes:
|
|
- calculate_total
|
|
- tax
|
|
- total
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- read_workspace_item
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- reads the existing process_invoice flow before editing it
|
|
- updates the calculate_total inline code to apply 8% tax
|
|
- returns subtotal, tax, and total from the updated flow logic
|
|
- leaves the result as an AI draft only
|
|
|
|
- id: global-test6-secret-variable-draft
|
|
prompt: |-
|
|
Create a secret variable draft at `f/evals/global/slack_bot_token`.
|
|
Use the placeholder value `xoxb-redacted-test-token` and description `Slack bot token for eval notifications`.
|
|
Do not create any resource or deploy anything.
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: variable
|
|
path: f/evals/global/slack_bot_token
|
|
valueIncludes:
|
|
- Slack bot token
|
|
- "true"
|
|
forbiddenDrafts:
|
|
- type: resource
|
|
path: f/evals/global/slack_bot_token
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_variable
|
|
forbiddenToolsUsed:
|
|
- write_resource
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
toolCallArgs:
|
|
- tool: write_variable
|
|
field: value
|
|
stringStartsWithAnyOf:
|
|
- xoxb-redacted-test-token
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- creates exactly one secret variable draft at f/evals/global/slack_bot_token
|
|
- uses the requested placeholder value and description
|
|
- does not create a resource or deploy anything
|
|
|
|
- id: global-test7-ambiguous-app-asks-question
|
|
prompt: |-
|
|
Create a new raw app for triaging support tickets.
|
|
runtime:
|
|
maxTurns: 4
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- askUserQuestion
|
|
forbiddenToolsUsed:
|
|
- init_app
|
|
- write_app_file
|
|
- write_app_runnable
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
|
|
- id: global-test8-human-script-infer-path-language
|
|
prompt: |-
|
|
I need a small helper that formats a customer-facing welcome line.
|
|
It should take a person's name and return "Welcome aboard, <name>!".
|
|
Please just stage it as a draft for now.
|
|
runtime:
|
|
maxTurns: 8
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: script
|
|
valueIncludes:
|
|
- Welcome aboard
|
|
- name
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_script
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- creates a single script draft for a welcome-line helper
|
|
- accepts a person's name as input
|
|
- returns a message containing Welcome aboard, the provided name, and an exclamation mark
|
|
- chooses a reasonable workspace path and script language without needing the user to specify them
|
|
- leaves the result as an AI draft only
|
|
|
|
- id: global-test9-human-weekday-trial-job
|
|
prompt: |-
|
|
Can you set up a draft daily job that checks a few hard-coded trial accounts and returns the ones whose trial has ended?
|
|
It should run every weekday morning around 9 in UTC with a 30 day cutoff.
|
|
Keep it as draft work only.
|
|
runtime:
|
|
maxTurns: 10
|
|
validate:
|
|
draftCountExactly: 2
|
|
requiredDrafts:
|
|
- type: script
|
|
pathIncludes:
|
|
- trial
|
|
valueIncludes:
|
|
- trial
|
|
- "30"
|
|
- type: schedule
|
|
pathIncludes:
|
|
- trial
|
|
valueIncludes:
|
|
- UTC
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_script
|
|
- write_schedule
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- creates a script draft that checks hard-coded trial accounts
|
|
- returns the accounts whose trial has ended based on a 30 day cutoff
|
|
- creates a schedule draft for weekday mornings around 09:00 UTC
|
|
- links the schedule to the generated script
|
|
- leaves both artifacts as drafts only
|
|
|
|
- id: global-test10-human-secret-variable
|
|
prompt: |-
|
|
I need a placeholder Slack bot token stored securely for future notification work.
|
|
Use xoxb-redacted-test-token and note that it is for eval notifications.
|
|
Only prepare a draft.
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: variable
|
|
pathIncludes:
|
|
- slack
|
|
valueIncludes:
|
|
- eval notifications
|
|
- "true"
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_variable
|
|
forbiddenToolsUsed:
|
|
- write_resource
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
toolCallArgs:
|
|
- tool: write_variable
|
|
field: value
|
|
stringStartsWithAnyOf:
|
|
- xoxb-redacted-test-token
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- creates a single secret variable draft for the Slack bot token placeholder
|
|
- uses the requested placeholder value
|
|
- includes a note or description that it is for eval notifications
|
|
- does not create a resource or deploy anything
|
|
|
|
- id: global-test11-human-existing-flow-informal-edit
|
|
prompt: |-
|
|
There is an invoice processing flow in this workspace.
|
|
Can you adjust its total calculation so it adds 8% tax and returns subtotal, tax, and total?
|
|
Keep the change as a draft.
|
|
initial: ai_evals/fixtures/frontend/global/initial/process_invoice_flow.json
|
|
runtime:
|
|
maxTurns: 10
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: flow
|
|
pathIncludes:
|
|
- invoice
|
|
valueIncludes:
|
|
- calculate_total
|
|
- tax
|
|
- total
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- read_workspace_item
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- finds and edits the existing invoice processing flow without the user providing its exact path
|
|
- updates the total calculation to apply 8% tax
|
|
- returns subtotal, tax, and total from the updated flow logic
|
|
- leaves the result as an AI draft only
|
|
|
|
- id: global-test12-current-live-script-edit
|
|
prompt: |-
|
|
The script I have open formats greetings.
|
|
Can you update this script so it uppercases the name before greeting them and ends with an exclamation mark?
|
|
Keep it as draft work.
|
|
initial: ai_evals/fixtures/frontend/global/initial/current_greeting_live_script.json
|
|
runtime:
|
|
maxTurns: 8
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: script
|
|
path: f/evals/global/current_greeting
|
|
language: bun
|
|
valueIncludes:
|
|
- toUpperCase
|
|
- "!"
|
|
forbiddenDrafts:
|
|
- type: script
|
|
path: f/evals/global/format_greeting
|
|
- type: script
|
|
path: f/evals/global/format_greeting_archive
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- read_workspace_item
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- resolves "this script" to the active live editor script instead of another similarly named workspace script
|
|
- updates the greeting logic to uppercase the provided name
|
|
- returns a greeting ending with an exclamation mark
|
|
- leaves the result as a draft only
|
|
|
|
- id: global-test13-current-live-flow-edit
|
|
prompt: |-
|
|
I have the invoice flow open.
|
|
In the current flow, update the total calculation to add 8% tax and return subtotal, tax, and total.
|
|
Keep the change as a draft.
|
|
initial: ai_evals/fixtures/frontend/global/initial/current_invoice_live_flow.json
|
|
runtime:
|
|
maxTurns: 10
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: flow
|
|
path: f/evals/global/current_invoice_flow
|
|
valueIncludes:
|
|
- calculate_total
|
|
- tax
|
|
- total
|
|
forbiddenDrafts:
|
|
- type: flow
|
|
path: f/evals/global/process_invoice
|
|
- type: flow
|
|
path: f/evals/global/process_refund
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- read_workspace_item
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- resolves "current flow" to the active live editor flow
|
|
- does not edit the similarly named deployed invoice or refund flows
|
|
- updates the calculate_total logic to apply 8% tax
|
|
- returns subtotal, tax, and total from the updated flow logic
|
|
- leaves the result as a draft only
|
|
|
|
- id: global-test14-current-without-live-editor-asks-question
|
|
prompt: |-
|
|
Please update this script so it returns `ok`.
|
|
Keep it as a draft.
|
|
runtime:
|
|
maxTurns: 4
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
forbiddenToolsUsed:
|
|
- write_script
|
|
- edit_script
|
|
- write_flow
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- asks which script to update when the user refers to "this script" without selected or active editor context
|
|
- does not guess a path or create a new script draft
|
|
|
|
- id: global-test15-human-postgres-resource
|
|
prompt: |-
|
|
I'm wiring the eval reporting database into this workspace.
|
|
Can you stage a Postgres connection for it in the shared evals/global folder?
|
|
Use host `reports-db.internal`, port 5432, database `evals_reporting`, user `report_reader`, and password `pg-redacted-reporting-password`.
|
|
Keep the credentials safe.
|
|
This is just draft work for now.
|
|
runtime:
|
|
maxTurns: 10
|
|
validate:
|
|
draftCountExactly: 2
|
|
requiredDrafts:
|
|
- type: variable
|
|
pathStartsWith: f/evals/global/
|
|
pathIncludes:
|
|
- evals
|
|
- global
|
|
- report
|
|
- password
|
|
valueIncludes:
|
|
- "true"
|
|
- report
|
|
- type: resource
|
|
pathStartsWith: f/evals/global/
|
|
pathIncludes:
|
|
- evals
|
|
- global
|
|
- report
|
|
valueIncludes:
|
|
- postgres
|
|
- reports-db.internal
|
|
- "5432"
|
|
- evals_reporting
|
|
- report_reader
|
|
- "$var:"
|
|
valueExcludes:
|
|
- pg-redacted-reporting-password
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_variable
|
|
- search_resource_types
|
|
- write_resource
|
|
forbiddenToolsUsed:
|
|
- write_schedule
|
|
- write_trigger
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
toolCallArgs:
|
|
- tool: write_variable
|
|
field: value
|
|
stringStartsWithAnyOf:
|
|
- pg-redacted-reporting-password
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- creates a Postgres resource draft for the eval reporting database
|
|
- creates a secret variable draft for the database password
|
|
- puts the drafts in sensible eval/global reporting-related paths
|
|
- uses the requested host, port, database, and user
|
|
- references the secret variable from the resource instead of embedding the password
|
|
- leaves the work as a draft only
|
|
|
|
- id: global-test16-human-visible-variable
|
|
prompt: |-
|
|
We keep reusing a 30 day trial cutoff in eval notification jobs.
|
|
Can you stage that as a normal workspace variable in the shared evals/global folder, with a short description so people know what it controls?
|
|
It is not a secret.
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: variable
|
|
pathStartsWith: f/evals/global/
|
|
pathIncludes:
|
|
- evals
|
|
- global
|
|
- trial
|
|
valueIncludes:
|
|
- "30"
|
|
- "false"
|
|
- trial
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_variable
|
|
forbiddenToolsUsed:
|
|
- write_resource
|
|
- write_schedule
|
|
- write_trigger
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- creates exactly one non-secret variable draft for the trial cutoff
|
|
- stores the value 30
|
|
- chooses a sensible eval/global path related to trials or notifications
|
|
- includes a useful description of what the value controls
|
|
- does not create resources, schedules, triggers, or deployed workspace changes
|
|
|
|
- id: global-test17-human-schedule-existing-helper
|
|
prompt: |-
|
|
The workspace already has a report digest helper.
|
|
Can you stage a weekday 8:30 AM UTC run for it with `dry_run` turned on?
|
|
I only want the schedule draft for review.
|
|
initial: ai_evals/fixtures/frontend/global/initial/report_digest_script.json
|
|
runtime:
|
|
maxTurns: 8
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: schedule
|
|
pathIncludes:
|
|
- digest
|
|
valueIncludes:
|
|
- f/evals/global/send_report_digest
|
|
- UTC
|
|
- dry_run
|
|
- "true"
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- list_workspace_items
|
|
- write_schedule
|
|
forbiddenToolsUsed:
|
|
- write_script
|
|
- write_flow
|
|
- write_resource
|
|
- write_variable
|
|
- write_trigger
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
judgeChecklist:
|
|
- finds the existing report digest helper rather than creating a new script or flow
|
|
- creates one schedule draft for that helper
|
|
- schedules it for weekdays around 08:30 UTC
|
|
- passes dry_run as true
|
|
- leaves only the schedule draft for review
|
|
|
|
- id: global-test18-human-slack-resource-with-secret
|
|
prompt: |-
|
|
I'm preparing Slack notifications for eval failures.
|
|
Can you stage a Slack connection in the shared evals/global folder?
|
|
The bot token is `xoxb-redacted-test-token`; keep it safe.
|
|
Don't deploy anything yet.
|
|
runtime:
|
|
maxTurns: 8
|
|
validate:
|
|
draftCountExactly: 2
|
|
requiredDrafts:
|
|
- type: variable
|
|
pathStartsWith: f/evals/global/
|
|
pathIncludes:
|
|
- evals
|
|
- global
|
|
- slack
|
|
- token
|
|
valueIncludes:
|
|
- "true"
|
|
- type: resource
|
|
pathStartsWith: f/evals/global/
|
|
pathIncludes:
|
|
- evals
|
|
- global
|
|
- slack
|
|
valueIncludes:
|
|
- slack
|
|
- "$var:"
|
|
valueExcludes:
|
|
- xoxb-redacted-test-token
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_variable
|
|
- search_resource_types
|
|
- write_resource
|
|
forbiddenToolsUsed:
|
|
- write_schedule
|
|
- write_trigger
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
toolCallArgs:
|
|
- tool: write_variable
|
|
field: value
|
|
stringStartsWithAnyOf:
|
|
- xoxb-redacted-test-token
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- creates a secret variable draft for the Slack bot token placeholder
|
|
- creates a Slack resource draft that references the secret variable instead of embedding the token
|
|
- keeps both drafts under a sensible eval/global Slack-related path
|
|
- does not create schedules, triggers, or deployed workspace changes
|
|
|
|
- id: global-test19-datatable-not-configured-asks-to-set-up
|
|
prompt: |-
|
|
Here are two newsletter signups: alice@example.com and bob@example.com.
|
|
Save them into a workspace data table for me.
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- list_datatables
|
|
forbiddenToolsUsed:
|
|
- exec_datatable_sql
|
|
- write_script
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- checks which data tables exist in the workspace before acting
|
|
- recognizes that no data table is configured in this workspace
|
|
- explains that a data table must first be set up by the user in the workspace settings (Data Tables) and is not created via SQL
|
|
- does not run SQL, write a script, or invent a data table to work around the missing configuration
|
|
- tells the user to configure a data table and then try again
|
|
|
|
- id: global-test20-datatable-no-hallucinated-main
|
|
prompt: |-
|
|
Pull the latest rows from the orders table in our data table so I can see recent orders.
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- list_datatables
|
|
forbiddenToolsUsed:
|
|
- exec_datatable_sql
|
|
- write_script
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- checks which data tables exist in the workspace before querying
|
|
- recognizes that no data table is configured in this workspace
|
|
- does not assume a data table named "main" (or any other name) exists
|
|
- does not run SQL against a guessed data table or fabricate order rows
|
|
- tells the user they need to set up a data table in the workspace settings first
|
|
|
|
- id: global-test21-datatable-list-summarize
|
|
prompt: |-
|
|
What tables do we have in our workspace data table? Just give me the list.
|
|
initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- list_datatables
|
|
forbiddenToolsUsed:
|
|
- get_datatable_table_schema
|
|
- exec_datatable_sql
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- lists the tables available in the workspace data table (orders and customers)
|
|
- answers from the data table listing rather than fabricating table names
|
|
- does not fetch column details or run SQL just to produce a table list
|
|
|
|
- id: global-test22-datatable-inspect-columns
|
|
prompt: |-
|
|
What columns does the orders table have in our workspace data table?
|
|
initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- get_datatable_table_schema
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- inspects the orders table schema in the workspace data table
|
|
- reports the orders columns (such as id, customer_id, total, status, created_at)
|
|
- answers from the retrieved schema rather than guessing the columns
|
|
|
|
- id: global-test23-datatable-query-select
|
|
prompt: |-
|
|
Show me the orders in our workspace data table, including their status and total.
|
|
initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- exec_datatable_sql
|
|
forbiddenToolsUsed:
|
|
- write_script
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
toolCallArgs:
|
|
- tool: exec_datatable_sql
|
|
field: sql
|
|
stringIncludesAnyOf:
|
|
- select
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- runs a SELECT query against the orders table in the workspace data table
|
|
- reports the orders returned by the query back to the user instead of fabricating data
|
|
- does not tell the user to set up a data table, since one already exists
|
|
|
|
- id: global-test24-datatable-create-table
|
|
prompt: |-
|
|
Add a new table called refunds to our workspace data table, with an order id and a refund amount.
|
|
initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- exec_datatable_sql
|
|
forbiddenToolsUsed:
|
|
- write_script
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
toolCallArgs:
|
|
- tool: exec_datatable_sql
|
|
field: sql
|
|
stringIncludesAnyOf:
|
|
- create table
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- creates the refunds table with a plain CREATE TABLE statement on the data table
|
|
- includes an order id and a refund amount column
|
|
- treats creating the table as a normal SQL statement and does not claim a separate registration step is needed
|
|
- does not write a script to create the table
|
|
|
|
- id: global-test25-datatable-mutate-rows
|
|
prompt: |-
|
|
Mark order number 2 as shipped in our workspace data table.
|
|
initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
|
|
runtime:
|
|
# Headroom for inspect-schema -> UPDATE -> verify; the in-memory engine now
|
|
# persists the write, so verification confirms on the first try (no loop).
|
|
maxTurns: 12
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- exec_datatable_sql
|
|
forbiddenToolsUsed:
|
|
- write_script
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
toolCallArgs:
|
|
- tool: exec_datatable_sql
|
|
field: sql
|
|
stringIncludesAnyOf:
|
|
- update
|
|
- insert into
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- runs an UPDATE on the orders table setting the status of order id 2 to shipped
|
|
- targets only order number 2 rather than rewriting the whole table
|
|
- confirms the change back to the user
|
|
|
|
- id: global-test26-datatable-script-sdk
|
|
prompt: |-
|
|
Write a script that reads our workspace data table and returns the total revenue across all orders.
|
|
Leave it as an AI draft only; do not deploy or save it.
|
|
initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
|
|
runtime:
|
|
maxTurns: 10
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: script
|
|
valueIncludes:
|
|
- wmill.datatable(
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- get_instructions
|
|
- write_script
|
|
forbiddenToolsUsed:
|
|
- exec_datatable_sql
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
# The judge has no datatable SDK reference and wrongly penalizes correct
|
|
# wmill.datatable() tagged-template usage, so rely on the deterministic checks:
|
|
# required get_instructions + write_script, forbidden exec_datatable_sql, and a
|
|
# draft that contains wmill.datatable(.
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- writes a script (not a chat-time SQL execution) that reads the workspace data table at runtime
|
|
- uses the wmill.datatable() SDK to query the orders table and sum the order totals
|
|
- returns the total revenue from the script
|
|
- leaves the result as an AI draft and does not deploy or save it
|
|
|
|
- id: global-test27-list-recent-runs
|
|
prompt: |-
|
|
What are the most recent runs in this workspace? Give me a quick rundown of what ran and whether it succeeded.
|
|
initial: ai_evals/fixtures/frontend/global/initial/jobs_seed.json
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- list_runs
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
- write_script
|
|
# Read-only job inspection produces no draft, so the global judge (which only
|
|
# sees the drafts artifact) would score it empty — validate via tool use.
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- lists the recent runs from the workspace rather than fabricating them
|
|
- summarizes each run's path and success/failure from the returned data
|
|
|
|
- id: global-test28-fetch-failed-run-logs
|
|
prompt: |-
|
|
The run with id 01920000-0000-7000-8000-0000000000f1 failed. Pull its logs and tell me what went wrong.
|
|
initial: ai_evals/fixtures/frontend/global/initial/jobs_seed.json
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- get_job_logs
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
- write_script
|
|
toolCallArgs:
|
|
- tool: get_job_logs
|
|
field: id
|
|
stringIncludesAnyOf:
|
|
- 01920000-0000-7000-8000-0000000000f1
|
|
# Same as above — no draft is produced, so rely on the deterministic tool-use
|
|
# and argument checks rather than the judge.
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- fetches the logs for the requested job id
|
|
- explains the failure from the returned logs (connection refused to the upstream API)
|
|
|
|
# --- Documentation search (search_docs) ---
|
|
# Pure product-knowledge questions: the assistant should consult the docs via
|
|
# search_docs and answer conversationally, not draft or mutate anything. No
|
|
# draft is produced, so the global judge is skipped and we validate tool use.
|
|
|
|
- id: global-docs-ai-agent-step
|
|
prompt: |-
|
|
Does Windmill support a flow step where an LLM decides which of my scripts to call based on the input?
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- search_docs
|
|
forbiddenToolsUsed:
|
|
- write_script
|
|
- write_flow
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
|
|
- id: global-docs-retry-step
|
|
prompt: |-
|
|
How does automatic retry work for a flow step that calls a flaky API?
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- search_docs
|
|
forbiddenToolsUsed:
|
|
- write_script
|
|
- write_flow
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
|
|
- id: global-docs-key-value-store
|
|
prompt: |-
|
|
Can I use a Redis-style key-value store from my Windmill scripts, and how?
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- search_docs
|
|
forbiddenToolsUsed:
|
|
- write_script
|
|
- write_flow
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
|
|
- id: global-docs-cron-schedule-format
|
|
prompt: |-
|
|
How do Windmill's cron schedules work, and what format does the schedule expression use?
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- search_docs
|
|
forbiddenToolsUsed:
|
|
- write_script
|
|
- write_flow
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
|
|
# --- Raw app on a large project (context-usage benchmark) ---
|
|
# These cases run against the deliberately large `analytics_dashboard` raw-app
|
|
# fixture (~20 frontend files incl. a 5k-line data module, plus backend runnables).
|
|
# They exist to measure how much context the global chat consumes when working in a
|
|
# big raw app: test29 is a read-heavy debugging hunt, test30 is a small edit baseline.
|
|
# tokenUsage is recorded per run, so the same cases re-run after a read-tool change
|
|
# (the read_app_file cap + offset/limit paging) quantify the optimization. skipJudge:
|
|
# the judge only sees the drafts artifact and cannot run the app, so we validate
|
|
# deterministically.
|
|
|
|
- id: global-test29-raw-app-debug-large
|
|
prompt: |-
|
|
The analytics dashboard app at `f/evals/global/analytics_dashboard` has a bug:
|
|
the Revenue Summary tile shows a total that is lower than the per-order line
|
|
totals and the per-region breakdown. Track down what is computing revenue
|
|
incorrectly and fix it. Keep the change as an AI draft only; do not deploy or
|
|
save it.
|
|
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
|
|
runtime:
|
|
maxTurns: 20
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: app
|
|
path: f/evals/global/analytics_dashboard
|
|
valueIncludes:
|
|
- "return order.unitPrice * order.quantity"
|
|
toolExpect:
|
|
requiredToolsAnyOf:
|
|
# Inspecting the app's files is satisfied by either reading them directly
|
|
# or grepping for the revenue calculation.
|
|
- [read_app_file, search_app]
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- inspects the dashboard app's files to locate the revenue calculation
|
|
- fixes the per-order revenue so it multiplies unit price by quantity
|
|
- leaves the result as an AI draft and does not deploy or save it
|
|
|
|
- id: global-test30-raw-app-small-edit-large
|
|
prompt: |-
|
|
In the dashboard app at `f/evals/global/analytics_dashboard`, change the main
|
|
page heading from "Operations Console" to "Revenue Overview". Leave everything
|
|
else unchanged. Keep it as an AI draft only; do not deploy or save it.
|
|
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
|
|
runtime:
|
|
maxTurns: 10
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: app
|
|
path: f/evals/global/analytics_dashboard
|
|
valueIncludes:
|
|
- "Revenue Overview"
|
|
toolExpect:
|
|
requiredToolsAnyOf:
|
|
# Inspecting the app's files is satisfied by reading them directly or
|
|
# grepping for the target with search_app.
|
|
- [read_app_file, search_app]
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- renames the main page heading to Revenue Overview
|
|
- does not change other dashboard behavior
|
|
- leaves the result as an AI draft only
|
|
|
|
- id: global-test31-raw-app-debug-inspect-data
|
|
prompt: |-
|
|
The raw app dashboard at `f/evals/global/analytics_dashboard` is reporting
|
|
revenue totals that look too low. Inspect the app's files — both the sample
|
|
order data module and the revenue calculation — to work out whether the bug is
|
|
in the data or in the calculation, then fix the actual cause. Keep the change as
|
|
an AI draft only; do not deploy or save it.
|
|
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
|
|
runtime:
|
|
maxTurns: 22
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: app
|
|
path: f/evals/global/analytics_dashboard
|
|
valueIncludes:
|
|
- "return order.unitPrice * order.quantity"
|
|
toolExpect:
|
|
requiredToolsAnyOf:
|
|
# Inspecting the app's files is satisfied by reading them directly or
|
|
# grepping for the target with search_app.
|
|
- [read_app_file, search_app]
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- inspects both the sample order data module and the revenue aggregation logic
|
|
- identifies the per-order revenue bug and fixes it to multiply unit price by quantity
|
|
- leaves the result as an AI draft only
|
|
|
|
- id: global-test32-raw-app-cross-file-consistency
|
|
prompt: |-
|
|
The raw app dashboard at `f/evals/global/analytics_dashboard` shows revenue
|
|
totals that disagree between the Revenue Summary tile, the orders table, and the
|
|
regional breakdown. Investigate how each of those computes revenue, work out
|
|
which calculation is wrong, and fix it. Keep the change as an AI draft only; do
|
|
not deploy or save it.
|
|
# Cross-file investigation: forces the model through several overlapping files
|
|
# (the summary's aggregation helper, the orders table, the regional breakdown) —
|
|
# a realistic multi-file read load that exercises the read_app_file cap.
|
|
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
|
|
runtime:
|
|
maxTurns: 24
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: app
|
|
path: f/evals/global/analytics_dashboard
|
|
valueIncludes:
|
|
- "return order.unitPrice * order.quantity"
|
|
toolExpect:
|
|
requiredToolsAnyOf:
|
|
# Inspecting the app's files is satisfied by reading them directly or
|
|
# grepping for the target with search_app.
|
|
- [read_app_file, search_app]
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- inspects the revenue calculation behind the summary tile, the orders table, and the regional breakdown
|
|
- identifies that the per-order revenue helper omits quantity and fixes it to multiply unit price by quantity
|
|
- leaves the result as an AI draft only
|
|
|
|
- id: global-test33-raw-app-rename-across-files
|
|
prompt: |-
|
|
In the dashboard app at `f/evals/global/analytics_dashboard`, rename the
|
|
`formatCurrency` helper to `formatMoney` everywhere it is defined, imported, and
|
|
called. Leave the separate `formatCurrencyPrecise` helper exactly as it is. Keep
|
|
the change as an AI draft only; do not deploy or save it.
|
|
# Find-all-usages rename: formatCurrency is defined once and called in 6 places
|
|
# spread across 4 component files (and imported in 4). Locating every usage is the
|
|
# exact task search_app is meant to make cheap — one grep returns all file:line
|
|
# rows instead of reading each component whole. valueExcludes "formatCurrency("
|
|
# asserts the definition and all call sites were renamed while tolerating the
|
|
# preserved formatCurrencyPrecise (which is never followed by "(").
|
|
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
|
|
runtime:
|
|
maxTurns: 22
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: app
|
|
path: f/evals/global/analytics_dashboard
|
|
valueIncludes:
|
|
- "export function formatMoney"
|
|
- "formatMoney("
|
|
valueExcludes:
|
|
- "formatCurrency("
|
|
toolExpect:
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- renames the formatCurrency definition, imports, and all call sites to formatMoney
|
|
- leaves the unrelated formatCurrencyPrecise helper unchanged
|
|
- leaves the result as an AI draft only
|
|
|
|
# --- Path selection (u/<user> vs f/<folder>) ---
|
|
# These cases assert how the assistant picks a workspace path when the user gives
|
|
# none: a bare name defaults to the personal scope `u/<user>/`, an existing folder
|
|
# whose purpose matches is used, a non-admin targets a writable folder and never a
|
|
# read-only one, and shared intent with no matching folder asks rather than invents.
|
|
# Each depends on the seeded `user` fixture (username / is_admin / folders /
|
|
# folders_read) so the prompt's folder guidance and `u/{username}` are well-formed.
|
|
|
|
- id: global-path1-bare-name-defaults-to-personal
|
|
prompt: |-
|
|
Stage a quick draft helper that takes a string and returns it trimmed of
|
|
leading and trailing whitespace. Just keep it as a draft.
|
|
initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
|
|
runtime:
|
|
maxTurns: 8
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: script
|
|
pathStartsWith: u/admin/
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_script
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- stages a single script draft for a trim helper
|
|
- defaults the path to the current user's personal scope (u/admin/...) since no path or folder was given
|
|
- does not invent an f/<folder> path
|
|
- leaves the result as a draft only
|
|
|
|
- id: global-path2-match-existing-folder
|
|
prompt: |-
|
|
Draft a flow for the marketing team's weekly campaign report.
|
|
It should take a week number and return a short summary string.
|
|
Keep it as a draft only.
|
|
initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
|
|
runtime:
|
|
maxTurns: 10
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: flow
|
|
pathStartsWith: f/marketing/
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_flow
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- drafts a flow for the marketing campaign report
|
|
- places it in the existing marketing folder (f/marketing/...) rather than the personal scope or an invented folder
|
|
- leaves the result as a draft only
|
|
|
|
- id: global-path3-shared-intent-unknown-folder-asks
|
|
prompt: |-
|
|
Put together a draft onboarding checklist flow for the People Ops team to use
|
|
when a new hire joins. Keep it as a draft.
|
|
initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
|
|
runtime:
|
|
maxTurns: 6
|
|
validate:
|
|
draftCountExactly: 0
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- askUserQuestion
|
|
forbiddenToolsUsed:
|
|
- write_flow
|
|
- write_script
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- recognizes the request implies shared/team work but names no existing folder (none of marketing/data_engineering/shared_utils fit People Ops)
|
|
- asks which folder to use instead of guessing or inventing one
|
|
- does not create a draft until the folder is known
|
|
|
|
- id: global-path4-nonadmin-avoids-readonly-folder
|
|
prompt: |-
|
|
Draft a small flow that returns today's date as an ISO string, and stage it in
|
|
one of our shared team folders. Keep it as a draft.
|
|
initial: ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json
|
|
runtime:
|
|
maxTurns: 10
|
|
validate:
|
|
draftCountExactly: 1
|
|
requiredDrafts:
|
|
- type: flow
|
|
pathStartsWith: f/team_a/
|
|
forbiddenDrafts:
|
|
- type: flow
|
|
pathStartsWith: f/team_b/
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- write_flow
|
|
forbiddenToolsUsed:
|
|
- deploy_workspace_item
|
|
- delete_workspace_item
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- drafts a flow that returns the current date as an ISO string
|
|
- places it in team_a (writable by this non-admin user) and not team_b (read-only)
|
|
- leaves the result as a draft only
|