- id: global-test1-script-create
  prompt: |-
    Create a draft Bun script at `f/evals/global/greet_user`.
    It should take a string `name` input and return `Hello, ${name}!`.
    Leave it as an AI draft only; do not deploy or save it.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        path: f/evals/global/greet_user
        language: bun
        valueIncludes:
          - name
          - Hello
  toolExpect:
    requiredToolsUsed:
      - write_script
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - creates a Bun script draft at f/evals/global/greet_user
    - the script accepts a name input
    - the script returns a greeting containing Hello, the provided name, and an exclamation mark
    - the result stays as an AI draft and is not deployed or saved to the workspace

- id: global-test2-script-edit-existing
  prompt: |-
    Update the existing workspace script at `f/evals/global/format_greeting`.
    Keep it as a Bun script, but change the greeting so the provided name is uppercased and the returned message ends with an exclamation mark.
    Leave the result as an AI draft only; do not deploy or save it.
  initial: ai_evals/fixtures/frontend/global/initial/format_greeting_script.json
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        path: f/evals/global/format_greeting
        language: bun
        valueIncludes:
          - toUpperCase
          - "!"
  toolExpect:
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - creates an AI draft for the existing f/evals/global/format_greeting script
    - preserves the script as Bun
    - uppercases the provided name in the greeting
    - returns a message ending with an exclamation mark
    - does not deploy or save the draft to the workspace

- id: global-test3-flow-create
  prompt: |-
    Create a draft flow at `f/evals/global/sum_numbers`.
    It should take two numeric inputs, `a` and `b`, and return their sum.
    Leave it as an AI draft only; do not deploy or save it.
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: flow
        path: f/evals/global/sum_numbers
        valueIncludes:
          - modules
          - rawscript
          - flow_input.a
          - flow_input.b
  toolExpect:
    requiredToolsUsed:
      - write_flow
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: write_flow
        field: modules
        stringStartsWithAnyOf:
          - "["
  judgeChecklist:
    - creates a flow draft at f/evals/global/sum_numbers
    - the flow accepts numeric inputs a and b
    - the flow returns the sum of a and b
    - the result stays as an AI draft and is not deployed or saved to the workspace

- id: global-test4-multi-artifact-notification-job
  prompt: |-
    Set up a draft stale-trial notification job.
    Create a Bun script at `f/evals/global/check_stale_trials` that accepts `max_age_days`, uses mocked inline trial account data, and returns the stale trial account IDs.
    Also create a weekday 09:00 UTC schedule at `f/evals/global/check_stale_trials_weekday` for that script with `max_age_days` set to 14.
    Add an HTTP POST trigger at `f/evals/global/check_stale_trials_manual` with route path `evals/check-stale-trials` that runs the same script manually.
    Leave everything as AI drafts only; do not deploy or save anything to the workspace.
  runtime:
    maxTurns: 12
  validate:
    draftCountExactly: 3
    requiredDrafts:
      - type: script
        path: f/evals/global/check_stale_trials
        language: bun
        valueIncludes:
          - max_age_days
          - trial
      - type: schedule
        path: f/evals/global/check_stale_trials_weekday
        valueIncludes:
          - f/evals/global/check_stale_trials
          - UTC
          - "14"
      - type: trigger
        triggerKind: http
        path: f/evals/global/check_stale_trials_manual
        valueIncludes:
          - evals/check-stale-trials
          - f/evals/global/check_stale_trials
  toolExpect:
    requiredToolsUsed:
      - write_script
      - write_schedule
      - write_trigger
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - creates a Bun script draft for stale trial accounts
    - creates a weekday 09:00 UTC schedule draft for the script with max_age_days set to 14
    - creates an HTTP POST trigger draft with route path evals/check-stale-trials for the same script
    - leaves all artifacts as drafts only and does not deploy

- id: global-test5-existing-flow-inline-code-edit
  prompt: |-
    Update the existing flow at `f/evals/global/process_invoice`.
    Only change the `calculate_total` inline code so it applies 8% tax and returns an object containing `subtotal`, `tax`, and `total`.
    Leave the updated flow as an AI draft only; do not deploy or save it.
  initial: ai_evals/fixtures/frontend/global/initial/process_invoice_flow.json
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: flow
        path: f/evals/global/process_invoice
        valueIncludes:
          - calculate_total
          - tax
          - total
  toolExpect:
    requiredToolsUsed:
      - read_workspace_item
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - reads the existing process_invoice flow before editing it
    - updates the calculate_total inline code to apply 8% tax
    - returns subtotal, tax, and total from the updated flow logic
    - leaves the result as an AI draft only

- id: global-test6-secret-variable-draft
  prompt: |-
    Create a secret variable draft at `f/evals/global/slack_bot_token`.
    Use the placeholder value `xoxb-redacted-test-token` and description `Slack bot token for eval notifications`.
    Do not create any resource or deploy anything.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: variable
        path: f/evals/global/slack_bot_token
        valueIncludes:
          - Slack bot token
          - "true"
    forbiddenDrafts:
      - type: resource
        path: f/evals/global/slack_bot_token
  toolExpect:
    requiredToolsUsed:
      - write_variable
    forbiddenToolsUsed:
      - write_resource
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: write_variable
        field: value
        stringStartsWithAnyOf:
          - xoxb-redacted-test-token
  skipJudge: true
  judgeChecklist:
    - creates exactly one secret variable draft at f/evals/global/slack_bot_token
    - uses the requested placeholder value and description
    - does not create a resource or deploy anything

- id: global-test7-ambiguous-app-asks-question
  prompt: |-
    Create a new raw app for triaging support tickets.
  runtime:
    maxTurns: 4
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - askUserQuestion
    forbiddenToolsUsed:
      - init_app
      - write_app_file
      - write_app_runnable
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true

- id: global-test8-human-script-infer-path-language
  prompt: |-
    I need a small helper that formats a customer-facing welcome line.
    It should take a person's name and return "Welcome aboard, <name>!".
    Please just stage it as a draft for now.
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        valueIncludes:
          - Welcome aboard
          - name
  toolExpect:
    requiredToolsUsed:
      - write_script
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - creates a single script draft for a welcome-line helper
    - accepts a person's name as input
    - returns a message containing Welcome aboard, the provided name, and an exclamation mark
    - chooses a reasonable workspace path and script language without needing the user to specify them
    - leaves the result as an AI draft only

- id: global-test9-human-weekday-trial-job
  prompt: |-
    Can you set up a draft daily job that checks a few hard-coded trial accounts and returns the ones whose trial has ended?
    It should run every weekday morning around 9 in UTC with a 30 day cutoff.
    Keep it as draft work only.
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 2
    requiredDrafts:
      - type: script
        pathIncludes:
          - trial
        valueIncludes:
          - trial
          - "30"
      - type: schedule
        pathIncludes:
          - trial
        valueIncludes:
          - UTC
  toolExpect:
    requiredToolsUsed:
      - write_script
      - write_schedule
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - creates a script draft that checks hard-coded trial accounts
    - returns the accounts whose trial has ended based on a 30 day cutoff
    - creates a schedule draft for weekday mornings around 09:00 UTC
    - links the schedule to the generated script
    - leaves both artifacts as drafts only

- id: global-test10-human-secret-variable
  prompt: |-
    I need a placeholder Slack bot token stored securely for future notification work.
    Use xoxb-redacted-test-token and note that it is for eval notifications.
    Only prepare a draft.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: variable
        pathIncludes:
          - slack
        valueIncludes:
          - eval notifications
          - "true"
  toolExpect:
    requiredToolsUsed:
      - write_variable
    forbiddenToolsUsed:
      - write_resource
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: write_variable
        field: value
        stringStartsWithAnyOf:
          - xoxb-redacted-test-token
  skipJudge: true
  judgeChecklist:
    - creates a single secret variable draft for the Slack bot token placeholder
    - uses the requested placeholder value
    - includes a note or description that it is for eval notifications
    - does not create a resource or deploy anything

- id: global-test11-human-existing-flow-informal-edit
  prompt: |-
    There is an invoice processing flow in this workspace.
    Can you adjust its total calculation so it adds 8% tax and returns subtotal, tax, and total?
    Keep the change as a draft.
  initial: ai_evals/fixtures/frontend/global/initial/process_invoice_flow.json
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: flow
        pathIncludes:
          - invoice
        valueIncludes:
          - calculate_total
          - tax
          - total
  toolExpect:
    requiredToolsUsed:
      - read_workspace_item
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - finds and edits the existing invoice processing flow without the user providing its exact path
    - updates the total calculation to apply 8% tax
    - returns subtotal, tax, and total from the updated flow logic
    - leaves the result as an AI draft only

- id: global-test12-current-live-script-edit
  prompt: |-
    The script I have open formats greetings.
    Can you update this script so it uppercases the name before greeting them and ends with an exclamation mark?
    Keep it as draft work.
  initial: ai_evals/fixtures/frontend/global/initial/current_greeting_live_script.json
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        path: f/evals/global/current_greeting
        language: bun
        valueIncludes:
          - toUpperCase
          - "!"
    forbiddenDrafts:
      - type: script
        path: f/evals/global/format_greeting
      - type: script
        path: f/evals/global/format_greeting_archive
  toolExpect:
    requiredToolsUsed:
      - read_workspace_item
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - resolves "this script" to the active live editor script instead of another similarly named workspace script
    - updates the greeting logic to uppercase the provided name
    - returns a greeting ending with an exclamation mark
    - leaves the result as a draft only

- id: global-test13-current-live-flow-edit
  prompt: |-
    I have the invoice flow open.
    In the current flow, update the total calculation to add 8% tax and return subtotal, tax, and total.
    Keep the change as a draft.
  initial: ai_evals/fixtures/frontend/global/initial/current_invoice_live_flow.json
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: flow
        path: f/evals/global/current_invoice_flow
        valueIncludes:
          - calculate_total
          - tax
          - total
    forbiddenDrafts:
      - type: flow
        path: f/evals/global/process_invoice
      - type: flow
        path: f/evals/global/process_refund
  toolExpect:
    requiredToolsUsed:
      - read_workspace_item
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - resolves "current flow" to the active live editor flow
    - does not edit the similarly named deployed invoice or refund flows
    - updates the calculate_total logic to apply 8% tax
    - returns subtotal, tax, and total from the updated flow logic
    - leaves the result as a draft only

- id: global-test14-current-without-live-editor-asks-question
  prompt: |-
    Please update this script so it returns `ok`.
    Keep it as a draft.
  runtime:
    maxTurns: 4
  validate:
    draftCountExactly: 0
  toolExpect:
    forbiddenToolsUsed:
      - write_script
      - edit_script
      - write_flow
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - asks which script to update when the user refers to "this script" without selected or active editor context
    - does not guess a path or create a new script draft

- id: global-test15-human-postgres-resource
  prompt: |-
    I'm wiring the eval reporting database into this workspace.
    Can you stage a Postgres connection for it in the shared evals/global folder?
    Use host `reports-db.internal`, port 5432, database `evals_reporting`, user `report_reader`, and password `pg-redacted-reporting-password`.
    Keep the credentials safe.
    This is just draft work for now.
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 2
    requiredDrafts:
      - type: variable
        pathStartsWith: f/evals/global/
        pathIncludes:
          - evals
          - global
          - report
          - password
        valueIncludes:
          - "true"
          - report
      - type: resource
        pathStartsWith: f/evals/global/
        pathIncludes:
          - evals
          - global
          - report
        valueIncludes:
          - postgres
          - reports-db.internal
          - "5432"
          - evals_reporting
          - report_reader
          - "$var:"
        valueExcludes:
          - pg-redacted-reporting-password
  toolExpect:
    requiredToolsUsed:
      - write_variable
      - search_resource_types
      - write_resource
    forbiddenToolsUsed:
      - write_schedule
      - write_trigger
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: write_variable
        field: value
        stringStartsWithAnyOf:
          - pg-redacted-reporting-password
  skipJudge: true
  judgeChecklist:
    - creates a Postgres resource draft for the eval reporting database
    - creates a secret variable draft for the database password
    - puts the drafts in sensible eval/global reporting-related paths
    - uses the requested host, port, database, and user
    - references the secret variable from the resource instead of embedding the password
    - leaves the work as a draft only

- id: global-test16-human-visible-variable
  prompt: |-
    We keep reusing a 30 day trial cutoff in eval notification jobs.
    Can you stage that as a normal workspace variable in the shared evals/global folder, with a short description so people know what it controls?
    It is not a secret.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: variable
        pathStartsWith: f/evals/global/
        pathIncludes:
          - evals
          - global
          - trial
        valueIncludes:
          - "30"
          - "false"
          - trial
  toolExpect:
    requiredToolsUsed:
      - write_variable
    forbiddenToolsUsed:
      - write_resource
      - write_schedule
      - write_trigger
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - creates exactly one non-secret variable draft for the trial cutoff
    - stores the value 30
    - chooses a sensible eval/global path related to trials or notifications
    - includes a useful description of what the value controls
    - does not create resources, schedules, triggers, or deployed workspace changes

- id: global-test17-human-schedule-existing-helper
  prompt: |-
    The workspace already has a report digest helper.
    Can you stage a weekday 8:30 AM UTC run for it with `dry_run` turned on?
    I only want the schedule draft for review.
  initial: ai_evals/fixtures/frontend/global/initial/report_digest_script.json
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: schedule
        pathIncludes:
          - digest
        valueIncludes:
          - f/evals/global/send_report_digest
          - UTC
          - dry_run
          - "true"
  toolExpect:
    requiredToolsUsed:
      - list_workspace_items
      - write_schedule
    forbiddenToolsUsed:
      - write_script
      - write_flow
      - write_resource
      - write_variable
      - write_trigger
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - finds the existing report digest helper rather than creating a new script or flow
    - creates one schedule draft for that helper
    - schedules it for weekdays around 08:30 UTC
    - passes dry_run as true
    - leaves only the schedule draft for review

- id: global-test18-human-slack-resource-with-secret
  prompt: |-
    I'm preparing Slack notifications for eval failures.
    Can you stage a Slack connection in the shared evals/global folder?
    The bot token is `xoxb-redacted-test-token`; keep it safe.
    Don't deploy anything yet.
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 2
    requiredDrafts:
      - type: variable
        pathStartsWith: f/evals/global/
        pathIncludes:
          - evals
          - global
          - slack
          - token
        valueIncludes:
          - "true"
      - type: resource
        pathStartsWith: f/evals/global/
        pathIncludes:
          - evals
          - global
          - slack
        valueIncludes:
          - slack
          - "$var:"
        valueExcludes:
          - xoxb-redacted-test-token
  toolExpect:
    requiredToolsUsed:
      - write_variable
      - search_resource_types
      - write_resource
    forbiddenToolsUsed:
      - write_schedule
      - write_trigger
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: write_variable
        field: value
        stringStartsWithAnyOf:
          - xoxb-redacted-test-token
  skipJudge: true
  judgeChecklist:
    - creates a secret variable draft for the Slack bot token placeholder
    - creates a Slack resource draft that references the secret variable instead of embedding the token
    - keeps both drafts under a sensible eval/global Slack-related path
    - does not create schedules, triggers, or deployed workspace changes

- id: global-test19-datatable-not-configured-asks-to-set-up
  prompt: |-
    Here are two newsletter signups: alice@example.com and bob@example.com.
    Save them into a workspace data table for me.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - list_datatables
    forbiddenToolsUsed:
      - exec_datatable_sql
      - write_script
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - checks which data tables exist in the workspace before acting
    - recognizes that no data table is configured in this workspace
    - explains that a data table must first be set up by the user in the workspace settings (Data Tables) and is not created via SQL
    - does not run SQL, write a script, or invent a data table to work around the missing configuration
    - tells the user to configure a data table and then try again

- id: global-test20-datatable-no-hallucinated-main
  prompt: |-
    Pull the latest rows from the orders table in our data table so I can see recent orders.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - list_datatables
    forbiddenToolsUsed:
      - exec_datatable_sql
      - write_script
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - checks which data tables exist in the workspace before querying
    - recognizes that no data table is configured in this workspace
    - does not assume a data table named "main" (or any other name) exists
    - does not run SQL against a guessed data table or fabricate order rows
    - tells the user they need to set up a data table in the workspace settings first

- id: global-test21-datatable-list-summarize
  prompt: |-
    What tables do we have in our workspace data table? Just give me the list.
  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - list_datatables
    forbiddenToolsUsed:
      - get_datatable_table_schema
      - exec_datatable_sql
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - lists the tables available in the workspace data table (orders and customers)
    - answers from the data table listing rather than fabricating table names
    - does not fetch column details or run SQL just to produce a table list

- id: global-test22-datatable-inspect-columns
  prompt: |-
    What columns does the orders table have in our workspace data table?
  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - get_datatable_table_schema
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - inspects the orders table schema in the workspace data table
    - reports the orders columns (such as id, customer_id, total, status, created_at)
    - answers from the retrieved schema rather than guessing the columns

- id: global-test23-datatable-query-select
  prompt: |-
    Show me the orders in our workspace data table, including their status and total.
  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - exec_datatable_sql
    forbiddenToolsUsed:
      - write_script
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: exec_datatable_sql
        field: sql
        stringIncludesAnyOf:
          - select
  skipJudge: true
  judgeChecklist:
    - runs a SELECT query against the orders table in the workspace data table
    - reports the orders returned by the query back to the user instead of fabricating data
    - does not tell the user to set up a data table, since one already exists

- id: global-test24-datatable-create-table
  prompt: |-
    Add a new table called refunds to our workspace data table, with an order id and a refund amount.
  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - exec_datatable_sql
    forbiddenToolsUsed:
      - write_script
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: exec_datatable_sql
        field: sql
        stringIncludesAnyOf:
          - create table
  skipJudge: true
  judgeChecklist:
    - creates the refunds table with a plain CREATE TABLE statement on the data table
    - includes an order id and a refund amount column
    - treats creating the table as a normal SQL statement and does not claim a separate registration step is needed
    - does not write a script to create the table

- id: global-test25-datatable-mutate-rows
  prompt: |-
    Mark order number 2 as shipped in our workspace data table.
  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
  runtime:
    # Headroom for inspect-schema -> UPDATE -> verify; the in-memory engine now
    # persists the write, so verification confirms on the first try (no loop).
    maxTurns: 12
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - exec_datatable_sql
    forbiddenToolsUsed:
      - write_script
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: exec_datatable_sql
        field: sql
        stringIncludesAnyOf:
          - update
          - insert into
  skipJudge: true
  judgeChecklist:
    - runs an UPDATE on the orders table setting the status of order id 2 to shipped
    - targets only order number 2 rather than rewriting the whole table
    - confirms the change back to the user

- id: global-test26-datatable-script-sdk
  prompt: |-
    Write a script that reads our workspace data table and returns the total revenue across all orders.
    Leave it as an AI draft only; do not deploy or save it.
  initial: ai_evals/fixtures/frontend/global/initial/datatable_orders_seed.json
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        valueIncludes:
          - wmill.datatable(
  toolExpect:
    requiredToolsUsed:
      - get_instructions
      - write_script
    forbiddenToolsUsed:
      - exec_datatable_sql
      - deploy_workspace_item
      - delete_workspace_item
  # The judge has no datatable SDK reference and wrongly penalizes correct
  # wmill.datatable() tagged-template usage, so rely on the deterministic checks:
  # required get_instructions + write_script, forbidden exec_datatable_sql, and a
  # draft that contains wmill.datatable(.
  skipJudge: true
  judgeChecklist:
    - writes a script (not a chat-time SQL execution) that reads the workspace data table at runtime
    - uses the wmill.datatable() SDK to query the orders table and sum the order totals
    - returns the total revenue from the script
    - leaves the result as an AI draft and does not deploy or save it

- id: global-test27-list-recent-runs
  prompt: |-
    What are the most recent runs in this workspace? Give me a quick rundown of what ran and whether it succeeded.
  initial: ai_evals/fixtures/frontend/global/initial/jobs_seed.json
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - list_runs
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
      - write_script
  # Read-only job inspection produces no draft, so the global judge (which only
  # sees the drafts artifact) would score it empty — validate via tool use.
  skipJudge: true
  judgeChecklist:
    - lists the recent runs from the workspace rather than fabricating them
    - summarizes each run's path and success/failure from the returned data

- id: global-test28-fetch-failed-run-logs
  prompt: |-
    The run with id 01920000-0000-7000-8000-0000000000f1 failed. Pull its logs and tell me what went wrong.
  initial: ai_evals/fixtures/frontend/global/initial/jobs_seed.json
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - get_job_logs
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
      - write_script
    toolCallArgs:
      - tool: get_job_logs
        field: id
        stringIncludesAnyOf:
          - 01920000-0000-7000-8000-0000000000f1
  # Same as above — no draft is produced, so rely on the deterministic tool-use
  # and argument checks rather than the judge.
  skipJudge: true
  judgeChecklist:
    - fetches the logs for the requested job id
    - explains the failure from the returned logs (connection refused to the upstream API)

# --- Page navigation (open_page) ---
# The assistant should take the user to a Windmill page (Runs/Schedules) with the
# right filters via open_page, rather than describing where to click or dumping the
# data. No draft is produced, so the global judge is skipped and we validate the
# tool call and its arguments.

- id: global-openpage1-runs-failed-of-script
  prompt: |-
    Take me to the failed runs of the script at f/evals/global/greet_user so I can see what's going wrong.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - open_page
    forbiddenToolsUsed:
      - write_script
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: open_page
        field: page
        stringIncludesAnyOf:
          - runs
      - tool: open_page
        field: status
        stringIncludesAnyOf:
          - failure
      - tool: open_page
        field: path
        stringIncludesAnyOf:
          - f/evals/global/greet_user
  skipJudge: true
  judgeChecklist:
    - opens the Runs page filtered to the failed runs of f/evals/global/greet_user
    - applies both the failure status and the script path as filters
    - does not write, deploy, or delete anything

- id: global-openpage2-runs-of-schedule
  prompt: |-
    Open the runs page filtered to the jobs triggered by the schedule f/evals/global/nightly_digest.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - open_page
    forbiddenToolsUsed:
      - write_script
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: open_page
        field: page
        stringIncludesAnyOf:
          - runs
      - tool: open_page
        field: schedule_path
        stringIncludesAnyOf:
          - f/evals/global/nightly_digest
  skipJudge: true
  judgeChecklist:
    - opens the Runs page filtered to jobs triggered by the f/evals/global/nightly_digest schedule
    - passes the schedule path as the filter
    - does not write, deploy, or delete anything

- id: global-openpage3-open-schedule
  prompt: |-
    Open the schedule f/evals/global/nightly_digest so I can review and edit it.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - open_page
    forbiddenToolsUsed:
      - write_schedule
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: open_page
        field: page
        stringIncludesAnyOf:
          - schedules
      - tool: open_page
        field: open
        stringIncludesAnyOf:
          - f/evals/global/nightly_digest
  skipJudge: true
  judgeChecklist:
    - opens the Schedules page and targets the f/evals/global/nightly_digest schedule for editing
    - passes the schedule path so the editor opens on it
    - does not write, deploy, or delete anything

- id: global-openpage4-workspace-settings-tab
  prompt: |-
    Take me to the Git sync configuration for this workspace.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - open_page
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: open_page
        field: page
        stringIncludesAnyOf:
          - workspace_settings
      - tool: open_page
        field: tab
        stringIncludesAnyOf:
          - git_sync
  skipJudge: true
  judgeChecklist:
    - opens the Workspace settings page on the git_sync tab
    - does not write, deploy, or delete anything

- id: global-openpage5-audit-logs-user
  prompt: |-
    Open the audit logs filtered to actions performed by the user admin.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - open_page
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: open_page
        field: page
        stringIncludesAnyOf:
          - audit_logs
      - tool: open_page
        field: username
        stringIncludesAnyOf:
          - admin
  skipJudge: true
  judgeChecklist:
    - opens the Audit logs page filtered to the admin user
    - does not write, deploy, or delete anything

- id: global-openpage6-triggers-kind
  prompt: |-
    Take me to the Kafka triggers for this workspace.
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - open_page
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: open_page
        field: page
        stringIncludesAnyOf:
          - triggers
      - tool: open_page
        field: trigger_kind
        stringIncludesAnyOf:
          - kafka
  skipJudge: true
  judgeChecklist:
    - opens the Kafka triggers page
    - does not write, deploy, or delete anything

- id: global-openpage7-compare-review
  prompt: |-
    Create a TypeScript script draft at f/evals/global/compare_review_demo that returns the string "ok" (no need to test it), then open the review page so I can look over the pending change and deploy it myself.
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 1
  toolExpect:
    requiredToolsUsed:
      - write_script
      - open_page
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: open_page
        field: page
        stringIncludesAnyOf:
          - compare
      # The eval chat is untracked (no modified-items mask), so the model must scope
      # the review by passing the item it changed explicitly — an omitted mask would
      # preselect every pending change in the workspace.
      - tool: open_page
        field: items
        stringIncludesAnyOf:
          - f/evals/global/compare_review_demo
  skipJudge: true
  judgeChecklist:
    - creates the script draft, then opens the Compare & Deploy review page instead of deploying itself
    - preselects only the created script on the review page
    - does not deploy or delete anything

- id: global-closepage1-close-runs-tab
  prompt: |-
    You just opened the runs page for me in the side panel. Close that tab, I'm done looking at it.
  runtime:
    maxTurns: 6
    sessionChat: true
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - close_page
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: close_page
        field: match
        stringIncludesAnyOf:
          - runs
  skipJudge: true
  judgeChecklist:
    - closes the runs preview tab in the side panel
    - does not write, deploy, or delete anything

# --- Documentation search (search_docs) ---
# Pure product-knowledge questions: the assistant should consult the docs via
# search_docs and answer conversationally, not draft or mutate anything. No
# draft is produced, so the global judge is skipped and we validate tool use.

- id: global-docs-ai-agent-step
  prompt: |-
    Does Windmill support a flow step where an LLM decides which of my scripts to call based on the input?
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - search_docs
    forbiddenToolsUsed:
      - write_script
      - write_flow
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true

- id: global-docs-retry-step
  prompt: |-
    How does automatic retry work for a flow step that calls a flaky API?
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - search_docs
    forbiddenToolsUsed:
      - write_script
      - write_flow
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true

- id: global-docs-key-value-store
  prompt: |-
    Can I use a Redis-style key-value store from my Windmill scripts, and how?
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - search_docs
    forbiddenToolsUsed:
      - write_script
      - write_flow
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true

- id: global-docs-cron-schedule-format
  prompt: |-
    How do Windmill's cron schedules work, and what format does the schedule expression use?
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - search_docs
    forbiddenToolsUsed:
      - write_script
      - write_flow
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true

# --- Raw app on a large project (context-usage benchmark) ---
# These cases run against the deliberately large `analytics_dashboard` raw-app
# fixture (~20 frontend files incl. a 5k-line data module, plus backend runnables).
# They exist to measure how much context the global chat consumes when working in a
# big raw app: test29 is a read-heavy debugging hunt, test30 is a small edit baseline.
# tokenUsage is recorded per run, so the same cases re-run after a read-tool change
# (the read_app_file cap + offset/limit paging) quantify the optimization. skipJudge:
# the judge only sees the drafts artifact and cannot run the app, so we validate
# deterministically.

- id: global-test29-raw-app-debug-large
  prompt: |-
    The analytics dashboard app at `f/evals/global/analytics_dashboard` has a bug:
    the Revenue Summary tile shows a total that is lower than the per-order line
    totals and the per-region breakdown. Track down what is computing revenue
    incorrectly and fix it. Keep the change as an AI draft only; do not deploy or
    save it.
  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
  runtime:
    maxTurns: 20
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: app
        path: f/evals/global/analytics_dashboard
        valueIncludes:
          - "return order.unitPrice * order.quantity"
  toolExpect:
    requiredToolsAnyOf:
      # Inspecting the app's files is satisfied by either reading them directly
      # or grepping for the revenue calculation.
      - [read_app_file, search_app]
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - inspects the dashboard app's files to locate the revenue calculation
    - fixes the per-order revenue so it multiplies unit price by quantity
    - leaves the result as an AI draft and does not deploy or save it

- id: global-test30-raw-app-small-edit-large
  prompt: |-
    In the dashboard app at `f/evals/global/analytics_dashboard`, change the main
    page heading from "Operations Console" to "Revenue Overview". Leave everything
    else unchanged. Keep it as an AI draft only; do not deploy or save it.
  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: app
        path: f/evals/global/analytics_dashboard
        valueIncludes:
          - "Revenue Overview"
  toolExpect:
    requiredToolsAnyOf:
      # Inspecting the app's files is satisfied by reading them directly or
      # grepping for the target with search_app.
      - [read_app_file, search_app]
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - renames the main page heading to Revenue Overview
    - does not change other dashboard behavior
    - leaves the result as an AI draft only

- id: global-test31-raw-app-debug-inspect-data
  prompt: |-
    The raw app dashboard at `f/evals/global/analytics_dashboard` is reporting
    revenue totals that look too low. Inspect the app's files — both the sample
    order data module and the revenue calculation — to work out whether the bug is
    in the data or in the calculation, then fix the actual cause. Keep the change as
    an AI draft only; do not deploy or save it.
  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
  runtime:
    maxTurns: 22
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: app
        path: f/evals/global/analytics_dashboard
        valueIncludes:
          - "return order.unitPrice * order.quantity"
  toolExpect:
    requiredToolsAnyOf:
      # Inspecting the app's files is satisfied by reading them directly or
      # grepping for the target with search_app.
      - [read_app_file, search_app]
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - inspects both the sample order data module and the revenue aggregation logic
    - identifies the per-order revenue bug and fixes it to multiply unit price by quantity
    - leaves the result as an AI draft only

- id: global-test32-raw-app-cross-file-consistency
  prompt: |-
    The raw app dashboard at `f/evals/global/analytics_dashboard` shows revenue
    totals that disagree between the Revenue Summary tile, the orders table, and the
    regional breakdown. Investigate how each of those computes revenue, work out
    which calculation is wrong, and fix it. Keep the change as an AI draft only; do
    not deploy or save it.
  # Cross-file investigation: forces the model through several overlapping files
  # (the summary's aggregation helper, the orders table, the regional breakdown) —
  # a realistic multi-file read load that exercises the read_app_file cap.
  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
  runtime:
    maxTurns: 24
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: app
        path: f/evals/global/analytics_dashboard
        valueIncludes:
          - "return order.unitPrice * order.quantity"
  toolExpect:
    requiredToolsAnyOf:
      # Inspecting the app's files is satisfied by reading them directly or
      # grepping for the target with search_app.
      - [read_app_file, search_app]
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - inspects the revenue calculation behind the summary tile, the orders table, and the regional breakdown
    - identifies that the per-order revenue helper omits quantity and fixes it to multiply unit price by quantity
    - leaves the result as an AI draft only

- id: global-test33-raw-app-rename-across-files
  prompt: |-
    In the dashboard app at `f/evals/global/analytics_dashboard`, rename the
    `formatCurrency` helper to `formatMoney` everywhere it is defined, imported, and
    called. Leave the separate `formatCurrencyPrecise` helper exactly as it is. Keep
    the change as an AI draft only; do not deploy or save it.
  # Find-all-usages rename: formatCurrency is defined once and called in 6 places
  # spread across 4 component files (and imported in 4). Locating every usage is the
  # exact task search_app is meant to make cheap — one grep returns all file:line
  # rows instead of reading each component whole. valueExcludes "formatCurrency("
  # asserts the definition and all call sites were renamed while tolerating the
  # preserved formatCurrencyPrecise (which is never followed by "(").
  initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
  runtime:
    maxTurns: 22
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: app
        path: f/evals/global/analytics_dashboard
        valueIncludes:
          - "export function formatMoney"
          - "formatMoney("
        valueExcludes:
          - "formatCurrency("
  toolExpect:
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - renames the formatCurrency definition, imports, and all call sites to formatMoney
    - leaves the unrelated formatCurrencyPrecise helper unchanged
    - leaves the result as an AI draft only

# --- Path selection (u/<user> vs f/<folder>) ---
# These cases assert how the assistant picks a workspace path when the user gives
# none: a bare name defaults to the personal scope `u/<user>/`, an existing folder
# whose purpose matches is used, a non-admin targets a writable folder and never a
# read-only one, and shared intent with no matching folder asks rather than invents.
# Each depends on the seeded `user` fixture (username / is_admin / folders /
# folders_read) so the prompt's folder guidance and `u/{username}` are well-formed.

- id: global-path1-bare-name-defaults-to-personal
  prompt: |-
    Stage a quick draft helper that takes a string and returns it trimmed of
    leading and trailing whitespace. Just keep it as a draft.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        pathStartsWith: u/admin/
  toolExpect:
    requiredToolsUsed:
      - write_script
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - stages a single script draft for a trim helper
    - defaults the path to the current user's personal scope (u/admin/...) since no path or folder was given
    - does not invent an f/<folder> path
    - leaves the result as a draft only

- id: global-path2-match-existing-folder
  prompt: |-
    Draft a flow for the marketing team's weekly campaign report.
    It should take a week number and return a short summary string.
    Keep it as a draft only.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: flow
        pathStartsWith: f/marketing/
  toolExpect:
    requiredToolsUsed:
      - write_flow
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - drafts a flow for the marketing campaign report
    - places it in the existing marketing folder (f/marketing/...) rather than the personal scope or an invented folder
    - leaves the result as a draft only

- id: global-path3-shared-intent-unknown-folder-asks
  prompt: |-
    Put together a draft onboarding checklist flow for the People Ops team to use
    when a new hire joins. Keep it as a draft.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - askUserQuestion
    forbiddenToolsUsed:
      - write_flow
      - write_script
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - recognizes the request implies shared/team work but names no existing folder (none of marketing/data_engineering/shared_utils fit People Ops)
    - asks which folder to use instead of guessing or inventing one
    - does not create a draft until the folder is known

- id: global-path4-nonadmin-avoids-readonly-folder
  prompt: |-
    Draft a small flow that returns today's date as an ISO string, and stage it in
    one of our shared team folders. Keep it as a draft.
  initial: ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: flow
        pathStartsWith: f/team_a/
    forbiddenDrafts:
      - type: flow
        pathStartsWith: f/team_b/
  toolExpect:
    requiredToolsUsed:
      - write_flow
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - drafts a flow that returns the current date as an ISO string
    - places it in team_a (writable by this non-admin user) and not team_b (read-only)
    - leaves the result as a draft only

- id: global-test-pipeline-create-node
  prompt: |-
    Set up the first step of a data pipeline at `f/evals/global/orders_ingest`.
    On a schedule, it should pull raw orders and land them in a managed DuckLake
    table so later steps can build on it. Keep it as an AI draft only — don't
    deploy or save it.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        path: f/evals/global/orders_ingest
        valueIncludes:
          - pipeline
    forbiddenDrafts:
      - type: flow
        pathStartsWith: f/evals/global/
  toolExpect:
    requiredToolsUsed:
      - write_script
    forbiddenToolsUsed:
      - write_flow
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - builds a data pipeline node as a script (not a flow)
    - marks the script as a pipeline member with the pipeline annotation in the script's comment syntax (`-- pipeline` for a DuckDB/SQL node, not `// pipeline`)
    - declares a schedule trigger and writes its output to a managed DuckLake table
    - leaves the result as an AI draft and does not deploy or save it

- id: global-test-pipeline-two-node-chain
  prompt: |-
    Build a small data pipeline in the `f/evals/global` folder: one step that
    ingests orders into a DuckLake table, and a second step that reads that table
    and writes a daily order-count rollup table. Wire the second step to run off
    the first step's output. Keep everything as drafts — don't deploy.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
  runtime:
    maxTurns: 14
  validate:
    draftCountAtLeast: 2
    forbiddenDrafts:
      - type: flow
        pathStartsWith: f/evals/global/
  toolExpect:
    requiredToolsUsed:
      - write_script
    forbiddenToolsUsed:
      - write_flow
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - creates two data pipeline nodes as scripts (not a flow) in f/evals/global
    - both scripts carry the pipeline annotation in their comment syntax (`-- pipeline` for DuckDB/SQL nodes, not `// pipeline`)
    - the first ingests orders into a DuckLake table
    - the second reads that same table and writes a daily rollup, wired to the first step's output asset
    - leaves both as AI drafts without deploying

- id: global-path5-create-folder-then-draft
  prompt: |-
    Create a new shared folder called "analytics" for our data work, then draft a
    script in it that returns the current timestamp as an ISO string. Keep the
    script as a draft.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        pathStartsWith: f/analytics/
  toolExpect:
    requiredToolsUsed:
      - create_folder
      - write_script
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  skipJudge: true
  judgeChecklist:
    - creates a new shared folder named "analytics" via create_folder
    - drafts a script placed in that folder (f/analytics/...) returning an ISO timestamp
    - leaves the script as a draft only

- id: global-artifact-plan-create
  prompt: |-
    I'm about to build a customer onboarding flow, but first I want a short written plan I can review and iterate on before any code.
    Draft a markdown plan with a title, a one-sentence summary, and three or four bullet steps.
    Keep it as something I can reopen and revise later — don't build the flow itself yet.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
  runtime:
    maxTurns: 6
    sessionChat: true
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - create_artifact
    forbiddenToolsUsed:
      - write_flow
      - write_script
      - deploy_workspace_item
  judgeChecklist:
    - saves the plan as a markdown artifact via create_artifact rather than only replying inline
    - the artifact content has a title, a one-line summary, and three or four bullet steps for onboarding
    - does not create a flow or script draft yet

- id: global-npm1-script-search-package
  prompt: |-
    Find a good npm package for parsing RSS/Atom feeds and use it to create a draft Bun script
    at `f/evals/global/fetch_feed`. The script should take a string `url` input and return the
    feed title along with the titles of the 5 most recent items. Tell me which package you
    picked and link its documentation. Leave it as an AI draft only; do not deploy.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
  runtime:
    maxTurns: 12
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        path: f/evals/global/fetch_feed
        language: bun
        valueIncludes:
          - url
  toolExpect:
    requiredToolsUsed:
      - search_npm_packages
      - write_script
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
  judgeChecklist:
    - searches npm for a feed-parsing package before writing the code
    - the draft Bun script imports the chosen npm package rather than hand-rolling an XML/RSS parser
    - the script takes a url input and returns the feed title plus the recent item titles
    - the reply names the chosen package and links its documentation
    - the result stays as an AI draft and is not deployed

- id: global-dbschema1-postgres-resource-tables
  prompt: |-
    I have a postgres resource at f/data/reports_pg in this workspace.
    What tables does that database have?
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
  runtime:
    maxTurns: 12
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - get_db_schema
    forbiddenToolsUsed:
      - write_script
      - write_resource
      - test_run_script
      - deploy_workspace_item
    toolCallArgs:
      - tool: get_db_schema
        field: resourcePath
        stringIncludesAnyOf:
          - f/data/reports_pg
  skipJudge: true
  judgeChecklist:
    - fetches the schema through get_db_schema with the resource path f/data/reports_pg
    - when the lookup fails, tells the user instead of inventing table names
    - does not write scripts or resources to answer a read-only question

# --- API catalog (search_api_endpoints / call_api_get / call_api_endpoint) ---
# The harness serves the catalog and the executed calls itself (mock
# listMcpTools + benchmark fetch handlers in adapters/frontend), so these cases
# do not require an mcp-enabled eval backend.

- id: global-test30-api-catalog-workers
  prompt: |-
    Are any workers online right now? Give me a quick summary of the workers and when they last pinged.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - search_api_endpoints
      - call_api_get
    forbiddenToolsUsed:
      - call_api_endpoint
      - write_script
      - deploy_workspace_item
    toolCallArgs:
      - tool: call_api_get
        field: name
        stringIncludesAnyOf:
          - listWorkers
  # Read-only workspace inspection produces no draft; validate via tool use.
  skipJudge: true
  judgeChecklist:
    - discovers the workers endpoint through the API catalog instead of guessing or fabricating
    - reports worker status from the returned data

- id: global-test31-draft-test-run-not-deployed
  prompt: |-
    Update `f/evals/global/format_greeting` so the provided name is uppercased in the greeting, then run it with name "ada" to check it works.
    Keep it as an AI draft only; do not deploy or save it.
  initial: ai_evals/fixtures/frontend/global/initial/format_greeting_script.json
  runtime:
    maxTurns: 10
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        path: f/evals/global/format_greeting
        language: bun
        valueIncludes:
          - toUpperCase
  toolExpect:
    requiredToolsUsed:
      - test_run_script
    forbiddenToolsUsed:
      - call_api_endpoint
      - deploy_workspace_item
      - delete_workspace_item
  # The judge only sees the drafts artifact and cannot observe runs, so it always
  # docks the prompt's "run it" requirement — validate deterministically instead:
  # draft content via valueIncludes, the test run via toolExpect (test_run_script
  # required, call_api_endpoint forbidden).
  skipJudge: true
  judgeChecklist:
    - creates an AI draft of f/evals/global/format_greeting with the name uppercased in the greeting
    - does not deploy or save the draft

- id: global-test32-delete-via-dedicated-tool
  prompt: |-
    Delete the deployed script `f/evals/global/send_report_digest` from the workspace — we don't use it anymore.
  initial: ai_evals/fixtures/frontend/global/initial/report_digest_script.json
  runtime:
    maxTurns: 6
  validate:
    draftCountExactly: 0
  toolExpect:
    requiredToolsUsed:
      - delete_workspace_item
    forbiddenToolsUsed:
      - call_api_endpoint
      - search_api_endpoints
      - write_script
  # Deletion produces no draft; validate via tool use.
  skipJudge: true
  judgeChecklist:
    - deletes the deployed script via delete_workspace_item rather than a raw API endpoint

- id: global-draft-diff-report
  prompt: |-
    Update the existing workspace script at `f/evals/global/format_greeting` so the returned message ends with an exclamation mark, keeping everything else the same.
    Then show me exactly what your draft changes compared to the deployed version.
    Leave the result as an AI draft only; do not deploy or save it.
  initial: ai_evals/fixtures/frontend/global/initial/format_greeting_script.json
  runtime:
    maxTurns: 8
  validate:
    draftCountExactly: 1
    requiredDrafts:
      - type: script
        path: f/evals/global/format_greeting
        language: bun
        valueIncludes:
          - "!"
  toolExpect:
    requiredToolsUsed:
      - diff
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      - tool: diff
        field: path
        stringIncludesAnyOf:
          - f/evals/global/format_greeting
  judgeChecklist:
    - creates an AI draft for the existing f/evals/global/format_greeting script with the exclamation-mark change
    - the draft changes only the returned message's punctuation — summary, language, path, and the rest of the code are untouched
    - does not deploy or save the draft to the workspace

- id: global-resource-manual-credentials
  prompt: |-
    Set up a resource for our production Postgres database at `f/evals/global/prod_db` (host db.internal.example.com, port 5432, database `orders`, user `app`).
    I don't want to paste the password into this chat — prepare everything so I can enter it myself.
    Leave it as an AI draft only; do not deploy or save it.
  initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
  runtime:
    maxTurns: 12
  validate:
    draftCountAtLeast: 1
    requiredDrafts:
      - type: resource
        path: f/evals/global/prod_db
        valueIncludes:
          - db.internal.example.com
          - orders
  toolExpect:
    requiredToolsUsed:
      - write_resource
      - open_page
    forbiddenToolsUsed:
      - deploy_workspace_item
      - delete_workspace_item
    toolCallArgs:
      # The model may land the user in the resource's drawer or in the drawer of
      # the secret variable it created for the password — both are correct.
      - tool: open_page
        field: page
        stringIncludesAnyOf:
          - resources
          - variables
      - tool: open_page
        field: open
        stringIncludesAnyOf:
          - prod_db
          - password
  judgeChecklist:
    - creates a postgres resource draft at f/evals/global/prod_db with the provided host, port, database, and user
    - the password is left for the user to provide (empty, a placeholder, or a secret variable reference) — no invented password value presented as real
    - does not deploy or save anything to the workspace
