windmill/ai_evals/cases/flow.yaml

- id: flow-test0-sum-two-numbers
  prompt: |-
    Create a flow that takes two numbers, `a` and `b`, and returns their sum.
    Keep it simple and use a single step named `sum_numbers`.
  expected: ai_evals/fixtures/frontend/flow/expected/test0_sum_two_numbers.json
  runtime:
    backendPreview:
      args:
        a: 4
        b: 5
  toolExpect:
    requiredToolsUsed:
      - test_run_flow
  judgeChecklist:
    - "the flow takes `a` and `b` as inputs"
    - "the main step is named `sum_numbers`"
    - the flow returns the sum of the two numbers

- id: flow-test1-reuse-existing-script
  prompt: |-
    I need a flow that adds two numbers.
    If there is already a script in the workspace that does that, reuse it instead of rewriting the logic.
    The flow should take `a` and `b` as inputs and use a single step named `sum_numbers`.
  initial: ai_evals/fixtures/frontend/flow/initial/test1_reuse_existing_script_initial.json
  expected: ai_evals/fixtures/frontend/flow/expected/test1_reuse_existing_script.json
  runtime:
    backendPreview:
      args:
        a: 2
        b: 3
  toolExpect:
    requiredToolsUsed:
      - test_run_flow
  judgeChecklist:
    - "the flow takes `a` and `b` as inputs"
    - "the main step is named `sum_numbers`"
    - the flow reuses the existing workspace script instead of rewriting the addition logic

- id: flow-test2-call-existing-subflow
  prompt: |-
    Create a parent flow that adds two numbers by reusing an existing flow in the workspace if one already exists.
    The parent flow should take `a` and `b` as inputs and delegate the calculation instead of inlining it.
    Use a single step named `call_add_numbers`.
  initial: ai_evals/fixtures/frontend/flow/initial/test2_call_existing_subflow_initial.json
  expected: ai_evals/fixtures/frontend/flow/expected/test2_call_existing_subflow.json
  runtime:
    backendPreview:
      args:
        a: 7
        b: 8
  toolExpect:
    requiredToolsUsed:
      - test_run_flow
  judgeChecklist:
    - "the parent flow takes `a` and `b` as inputs"
    - "the main step is named `call_add_numbers`"
    - the parent flow delegates to an existing workspace subflow instead of inlining the addition logic

- id: flow-test13-prefer-existing-workspace-flow
  prompt: |-
    Create a parent flow that adds two numbers by reusing an existing flow from the workspace if one fits.
    A reusable script may also be available, but for this task prefer the existing flow rather than calling a script directly or rewriting the logic inline.
    The parent flow should take `a` and `b` as inputs and use a single top-level step named `call_add_numbers_flow`.
  initial: ai_evals/fixtures/frontend/flow/initial/test13_prefer_existing_workspace_flow_initial.json
  expected: ai_evals/fixtures/frontend/flow/expected/test13_prefer_existing_workspace_flow.json
  validate:
    exactTopLevelStepIds:
      - call_add_numbers_flow
    topLevelStepTypes:
      - id: call_add_numbers_flow
        type: flow
    moduleRules:
      - id: call_add_numbers_flow
        requiredInputTransforms:
          - type: javascript
            expr: flow_input.a
          - type: javascript
            expr: flow_input.b
  runtime:
    backendPreview:
      args:
        a: 10
        b: 5
  judgeChecklist:
    - "the parent flow takes `a` and `b` as inputs"
    - "the main step is named `call_add_numbers_flow`"
    - the parent flow reuses the existing workspace flow as a subflow
    - the parent flow does not call the standalone workspace script directly
    - the parent flow does not inline the addition logic

- id: flow-test3-branchone-routing
  prompt: |-
    Create a flow that routes incoming support requests based on the customer's tier.
    The input should contain a string field named `tier`.
    Free, pro, and enterprise requests should go to different queues, and unknown tiers should fall back to a default queue.
    Name the main routing step `route_by_tier`.
  expected: ai_evals/fixtures/frontend/flow/expected/test3_branchone_routing.json
  judgeChecklist:
    - "the input schema includes a string field named `tier`"
    - "the main routing step is named `route_by_tier`"
    - free requests go to a free queue
    - pro requests go to a pro queue
    - enterprise requests go to an enterprise queue
    - unknown tiers fall back to a default queue

- id: flow-test4-order-processing-loop
  prompt: |-
    Build an order-processing flow.

    The input should include an order with:
    - an `items` array containing `name`, `price`, and `quantity`
    - `customer_email`
    - `shipping_address`

    The flow should:
    - validate that every item has a positive price and quantity
    - calculate the order total with 8% tax
    - check inventory for each item using placeholder availability data
    - create a shipment if everything is in stock, otherwise create a backorder
    - send a confirmation using placeholder email logic
    - return a final order summary with the status
  validate:
    schemaAnyOf:
      - requiredPaths:
          - order
          - order.items
          - order.customer_email
          - order.shipping_address
      - requiredPaths:
          - items
          - customer_email
          - shipping_address
    resolveResultsRefs: true
  judgeChecklist:
    - the flow validates that every item has a positive price and quantity
    - the flow calculates the order total with 8% tax
    - the flow checks inventory for each item using placeholder availability data
    - the flow creates a shipment if everything is in stock, otherwise a backorder
    - the flow sends a confirmation using placeholder email logic
    - the flow returns a final order summary with the resulting status

- id: flow-test5-parallel-data-pipeline
  prompt: |-
    Create a data-processing flow for three external data sources.

    It should:
    - load a small placeholder configuration listing the three sources
    - fetch placeholder records from each source
    - clean and validate each source's records
    - combine everything into one dataset
    - compute an overall quality score
    - store the result differently depending on the score:
      - 90 or above goes to the primary database
      - 70 to 89 goes to a secondary database with a warning
      - below 70 goes to quarantine and triggers an alert
    - return a processing report with total records, quality score, and destination
  judgeChecklist:
    - the flow loads a placeholder configuration listing three external sources
    - the flow fetches placeholder records from each source
    - the flow cleans and validates each source's records
    - the flow combines everything into one dataset
    - the flow computes an overall quality score
    - scores of 90 or above go to the primary database
    - scores from 70 to 89 go to a secondary database with a warning
    - scores below 70 go to quarantine and trigger an alert
    - the final report includes total records, quality score, and destination

- id: flow-test6-ai-agent-tools
  prompt: |-
    Create a customer support flow.

    The input should include `customer_id` and `query_text`.
    The flow should load the customer's profile and order history, then use an AI assistant to help with the request.
    The assistant should be able to:
    - look up orders
    - check refund eligibility
    - search FAQs
    - open a support ticket when needed

    After that, log the interaction and return the assistant's response.
  judgeChecklist:
    - "the input schema includes `customer_id` and `query_text`"
    - the flow loads the customer's profile and order history
    - the flow uses an AI assistant step
    - the assistant can look up orders
    - the assistant can check refund eligibility
    - the assistant can search FAQs
    - the assistant can open a support ticket
    - the flow logs the interaction
    - the final output returns the assistant response

- id: flow-test7-simple-modification
  prompt: |-
    Update this flow so it validates processed data before saving it.

    After `process_data`, add a `validate_data` step that checks the data array is not empty.
    If the array is empty, the flow should surface the message `No data to save` and prevent saving.
    If validation passes, let the save continue normally.
    Update `save_results` so it uses the validation outcome instead of bypassing it.
  initial: ai_evals/fixtures/frontend/flow/initial/test5_initial.json
  runtime:
    maxTurns: 8
  validate:
    topLevelStepIds:
      - fetch_data
      - process_data
      - validate_data
    topLevelStepOrder:
      - fetch_data
      - process_data
      - validate_data
    topLevelStepTypes:
      - id: fetch_data
        type: rawscript
      - id: process_data
        type: rawscript
      - id: validate_data
        type: rawscript
  judgeChecklist:
    - the updated flow keeps the original fetch and process steps intact
    - "a `validate_data` step is added after `process_data`"
    - "`validate_data` checks that the processed data array is not empty"
    - "when processed data is empty, the flow surfaces the message `No data to save` and does not save results"
    - "`save_results` uses the validation outcome instead of reading `results.process_data` directly"
    - "exact field names or wrapper object shape for the validation result are not important"

- id: flow-test8-branching-in-loop
  prompt: |-
    Update the order-processing logic inside `loop_orders` so different order types are handled differently.

    For `express`, mark the order as priority and use a shipping cost of $15.99.
    For `standard`, use a shipping cost of $5.99.
    For `pickup`, mark it as no shipping required with a cost of $0.
    Keep the existing processing as a fallback for unknown order types.
    Each path should return the orderId, shipping cost, and shipping type.
  initial: ai_evals/fixtures/frontend/flow/initial/test6_initial.json
  judgeChecklist:
    - "the existing `loop_orders` flow still handles per-order processing"
    - exact branching topology is not required as long as `loop_orders` handles the order types correctly
    - express orders are marked as priority and use a shipping cost of 15.99
    - standard orders use a shipping cost of 5.99
    - pickup orders use a shipping cost of 0 and are treated as no shipping required
    - unknown order types still follow a fallback path
    - "each processed order returns `orderId`, `shippingCost`, and `shippingType`"

- id: flow-test9-parallel-refactor
  prompt: |-
    Refactor this flow so the enrichment work no longer runs one step at a time.

    `enrich_price`, `enrich_inventory`, and `enrich_reviews` should run independently.
    Each one should return a fallback value if it fails.
    Update `combine_data` so it merges the enrichment results and sets a `hasFallbacks` flag when any fallback was used.
    Keep `get_item` as the first step and `return_result` as the last step.
  initial: ai_evals/fixtures/frontend/flow/initial/test7_initial.json
  validate:
    topLevelStepIds:
      - get_item
      - combine_data
      - return_result
    topLevelStepOrder:
      - get_item
      - combine_data
      - return_result
    topLevelStepTypeCountsAtLeast:
      - type: branchall
        count: 1
    topLevelStepTypes:
      - id: get_item
        type: rawscript
      - id: combine_data
        type: rawscript
      - id: return_result
        type: rawscript
    moduleRules:
      - id: enrich_price
      - id: enrich_inventory
      - id: enrich_reviews
  judgeChecklist:
    - "the updated flow keeps `get_item` as the first step"
    - "the updated flow keeps `return_result` as the last step"
    - "`enrich_price`, `enrich_inventory`, and `enrich_reviews` run independently rather than sequentially"
    - each enrichment path returns a fallback value if it fails
    - "`combine_data` merges the enrichment results"
    - "`combine_data` sets `hasFallbacks` when any fallback was used"

- id: flow-test10-while-loop-counter
  prompt: |-
    Create a flow that keeps incrementing a counter until it reaches a target value.
    The input should include a number field named `target`.
    Use a top-level loop step named `count_until_target`.
    Inside it, use a single step named `increment_counter` that increments the current counter.
    The loop should stop once the counter reaches `target`.
    After the loop, add a top-level step named `return_final_counter` that returns the last counter value.
  validate:
    exactTopLevelStepIds:
      - count_until_target
      - return_final_counter
    topLevelStepOrder:
      - count_until_target
      - return_final_counter
    topLevelStepTypes:
      - id: count_until_target
        type: whileloopflow
      - id: return_final_counter
        type: rawscript
    moduleRules:
      - id: count_until_target
        hasStopAfterIf: true
        hasStopAfterAllItersIf: false
        exactImmediateChildStepIds:
          - increment_counter
        immediateChildStepTypes:
          - id: increment_counter
            type: rawscript
    moduleFieldRules:
      - id: count_until_target
        path: stop_after_if.expr
        equals: result >= flow_input.target
  judgeChecklist:
    - "the input schema includes a number field named `target`"
    - "the top-level while loop step is named `count_until_target`"
    - "`count_until_target` contains a single increment step named `increment_counter`"
    - "`count_until_target` uses module-level `stop_after_if` to stop when the counter reaches `target`"
    - "`increment_counter` uses `flow_input.iter.value` or an equivalent loop-state expression and falls back to `0` on the first iteration"
    - "`return_final_counter` returns the final counter value"

- id: flow-test11-preprocessor-and-failure-handler
  prompt: |-
    Create an event-processing flow for a string payload.

    Before the main processing runs, trim the payload and reject empty strings.
    The main step should be named `process_event` and return a simple success object.
    If anything fails, return a compact error object with the error message and the failing step id.
  expected: ai_evals/fixtures/frontend/flow/expected/test11_preprocessor_failure.json
  validate:
    requireSpecialModules:
      - preprocessor_module
      - failure_module
  judgeChecklist:
    - the flow trims the payload before the main processing runs
    - the flow rejects empty payload strings
    - "the main step is named `process_event`"
    - "`process_event` returns a simple success object"
    - failures return a compact error object with the error message and failing step id

- id: flow-test12-approval-step
  prompt: |-
    Create a purchase approval flow.

    The input should include `requester_email` and `amount`.
    Add an approval step named `request_approval` that pauses the flow and asks the approver for a comment.
    One approval should be enough to continue.
    After approval, add a final step named `finalize_purchase` that returns an approved status object.
  validate:
    topLevelStepIds:
      - request_approval
      - finalize_purchase
    topLevelStepOrder:
      - request_approval
      - finalize_purchase
    topLevelStepTypes:
      - id: finalize_purchase
        type: rawscript
    schemaRequiredPaths:
      - requester_email
      - amount
    requireSuspendSteps:
      - id: request_approval
        requiredEvents: 1
        resumeRequiredStringFieldAnyOf:
          - comment
          - approver_comment
  judgeChecklist:
    - "the flow includes an approval step named `request_approval`"
    - "`request_approval` pauses the flow and asks the approver for a comment"
    - one approval is enough to continue
    - "the flow includes a final step named `finalize_purchase`"
    - "`finalize_purchase` returns an approved status object after approval"

- id: flow-test13-loop-resilience-toggle
  prompt: |-
    Update `loop_orders` so it can process orders in parallel.
    If one order fails, the rest should still continue.
    Keep the existing order-fetching and summary steps the same.
  initial: ai_evals/fixtures/frontend/flow/initial/test6_initial.json
  validate:
    exactTopLevelStepIds:
      - get_orders
      - loop_orders
      - summarize
    topLevelStepTypes:
      - id: loop_orders
        type: forloopflow
    moduleFieldRules:
      - id: loop_orders
        path: value.parallel
        equals: true
      - id: loop_orders
        path: value.skip_failures
        equals: true
  judgeChecklist:
    - "the flow keeps `get_orders` before `loop_orders` and `summarize` after it"
    - "`loop_orders` processes orders in parallel"
    - "a failure in one order does not stop the remaining orders from being processed"

- id: flow-test14-modify-existing-special-modules
  prompt: |-
    Update this event-processing flow for a string payload.

    Before `process_event` runs, trim the payload and reject empty strings.
    If anything fails, return a compact error object with the error message and the failing step id.
    Keep `process_event` as the main step.
  initial: ai_evals/fixtures/frontend/flow/initial/test11_initial.json
  expected: ai_evals/fixtures/frontend/flow/expected/test11_preprocessor_failure.json
  validate:
    requireSpecialModules:
      - preprocessor_module
      - failure_module
  judgeChecklist:
    - the updated flow trims the payload before the main processing runs
    - the updated flow rejects empty payload strings
    - "the existing `process_event` step remains the main step"
    - failures return a compact error object with the error message and failing step id

- id: flow-test15-create-current-flow-schedule
  prompt: |-
    Update this flow by adding a final step named `return_schedule_status`.
    It should return an object with `scheduled: true` and the order summary from `results.summarize_orders`.
    Also create an enabled daily schedule named `order_processing_daily` for the current flow.
    It should run every day at 07:30 UTC with empty args.
    Do not ask me for the flow path.
  initial: ai_evals/fixtures/frontend/flow/initial/scheduled_order_flow.json
  validate:
    topLevelStepIds:
      - return_schedule_status
  toolExpect:
    requiredToolsUsed:
      - test_run_flow
      - create_schedule
    toolCallArgs:
      - tool: create_schedule
        field: path
        stringStartsWithAnyOf:
          - f/
          - u/
        stringMustNotStartWithAnyOf:
          - schedules/
  skipJudge: true
  judgeChecklist:
    - "the flow includes a final top-level step named `return_schedule_status`"
    - "`return_schedule_status` returns `scheduled: true` and the order summary"

- id: flow-test16-create-current-flow-http-trigger
  prompt: |-
    Update this flow by adding a final step named `webhook_response`.
    It should return an object with `ok: true` and the order summary from `results.summarize_orders`.
    Also create a public POST HTTP endpoint named `order_processing_webhook` for the current flow.
    Use route path `ai-evals/order-processing` and no authentication.
    Do not ask me for the flow path.
  initial: ai_evals/fixtures/frontend/flow/initial/scheduled_order_flow.json
  validate:
    topLevelStepIds:
      - webhook_response
  toolExpect:
    requiredToolsUsed:
      - test_run_flow
      - create_trigger
    toolCallArgs:
      - tool: create_trigger
        field: path
        stringStartsWithAnyOf:
          - f/
          - u/
        stringMustNotStartWithAnyOf:
          - schedules/
  skipJudge: true
  judgeChecklist:
    - "the flow includes a final top-level step named `webhook_response`"
    - "`webhook_response` returns `ok: true` and the order summary"