- id: flow-test0-sum-two-numbers prompt: |- Create a flow that takes two numbers, `a` and `b`, and returns their sum. Keep it simple and use a single step named `sum_numbers`. expected: ai_evals/fixtures/frontend/flow/expected/test0_sum_two_numbers.json runtime: backendPreview: args: a: 4 b: 5 toolExpect: requiredToolsUsed: - test_run_flow judgeChecklist: - "the flow takes `a` and `b` as inputs" - "the main step is named `sum_numbers`" - the flow returns the sum of the two numbers - id: flow-test1-reuse-existing-script prompt: |- I need a flow that adds two numbers. If there is already a script in the workspace that does that, reuse it instead of rewriting the logic. The flow should take `a` and `b` as inputs and use a single step named `sum_numbers`. initial: ai_evals/fixtures/frontend/flow/initial/test1_reuse_existing_script_initial.json expected: ai_evals/fixtures/frontend/flow/expected/test1_reuse_existing_script.json runtime: backendPreview: args: a: 2 b: 3 toolExpect: requiredToolsUsed: - test_run_flow judgeChecklist: - "the flow takes `a` and `b` as inputs" - "the main step is named `sum_numbers`" - the flow reuses the existing workspace script instead of rewriting the addition logic - id: flow-test2-call-existing-subflow prompt: |- Create a parent flow that adds two numbers by reusing an existing flow in the workspace if one already exists. The parent flow should take `a` and `b` as inputs and delegate the calculation instead of inlining it. Use a single step named `call_add_numbers`. initial: ai_evals/fixtures/frontend/flow/initial/test2_call_existing_subflow_initial.json expected: ai_evals/fixtures/frontend/flow/expected/test2_call_existing_subflow.json runtime: backendPreview: args: a: 7 b: 8 toolExpect: requiredToolsUsed: - test_run_flow judgeChecklist: - "the parent flow takes `a` and `b` as inputs" - "the main step is named `call_add_numbers`" - the parent flow delegates to an existing workspace subflow instead of inlining the addition logic - id: flow-test13-prefer-existing-workspace-flow prompt: |- Create a parent flow that adds two numbers by reusing an existing flow from the workspace if one fits. A reusable script may also be available, but for this task prefer the existing flow rather than calling a script directly or rewriting the logic inline. The parent flow should take `a` and `b` as inputs and use a single top-level step named `call_add_numbers_flow`. initial: ai_evals/fixtures/frontend/flow/initial/test13_prefer_existing_workspace_flow_initial.json expected: ai_evals/fixtures/frontend/flow/expected/test13_prefer_existing_workspace_flow.json validate: exactTopLevelStepIds: - call_add_numbers_flow topLevelStepTypes: - id: call_add_numbers_flow type: flow moduleRules: - id: call_add_numbers_flow requiredInputTransforms: - type: javascript expr: flow_input.a - type: javascript expr: flow_input.b runtime: backendPreview: args: a: 10 b: 5 judgeChecklist: - "the parent flow takes `a` and `b` as inputs" - "the main step is named `call_add_numbers_flow`" - the parent flow reuses the existing workspace flow as a subflow - the parent flow does not call the standalone workspace script directly - the parent flow does not inline the addition logic - id: flow-test3-branchone-routing prompt: |- Create a flow that routes incoming support requests based on the customer's tier. The input should contain a string field named `tier`. Free, pro, and enterprise requests should go to different queues, and unknown tiers should fall back to a default queue. Name the main routing step `route_by_tier`. expected: ai_evals/fixtures/frontend/flow/expected/test3_branchone_routing.json judgeChecklist: - "the input schema includes a string field named `tier`" - "the main routing step is named `route_by_tier`" - free requests go to a free queue - pro requests go to a pro queue - enterprise requests go to an enterprise queue - unknown tiers fall back to a default queue - id: flow-test4-order-processing-loop prompt: |- Build an order-processing flow. The input should include an order with: - an `items` array containing `name`, `price`, and `quantity` - `customer_email` - `shipping_address` The flow should: - validate that every item has a positive price and quantity - calculate the order total with 8% tax - check inventory for each item using placeholder availability data - create a shipment if everything is in stock, otherwise create a backorder - send a confirmation using placeholder email logic - return a final order summary with the status validate: schemaAnyOf: - requiredPaths: - order - order.items - order.customer_email - order.shipping_address - requiredPaths: - items - customer_email - shipping_address resolveResultsRefs: true judgeChecklist: - the flow validates that every item has a positive price and quantity - the flow calculates the order total with 8% tax - the flow checks inventory for each item using placeholder availability data - the flow creates a shipment if everything is in stock, otherwise a backorder - the flow sends a confirmation using placeholder email logic - the flow returns a final order summary with the resulting status - id: flow-test5-parallel-data-pipeline prompt: |- Create a data-processing flow for three external data sources. It should: - load a small placeholder configuration listing the three sources - fetch placeholder records from each source - clean and validate each source's records - combine everything into one dataset - compute an overall quality score - store the result differently depending on the score: - 90 or above goes to the primary database - 70 to 89 goes to a secondary database with a warning - below 70 goes to quarantine and triggers an alert - return a processing report with total records, quality score, and destination judgeChecklist: - the flow loads a placeholder configuration listing three external sources - the flow fetches placeholder records from each source - the flow cleans and validates each source's records - the flow combines everything into one dataset - the flow computes an overall quality score - scores of 90 or above go to the primary database - scores from 70 to 89 go to a secondary database with a warning - scores below 70 go to quarantine and trigger an alert - the final report includes total records, quality score, and destination - id: flow-test6-ai-agent-tools prompt: |- Create a customer support flow. The input should include `customer_id` and `query_text`. The flow should load the customer's profile and order history, then use an AI assistant to help with the request. The assistant should be able to: - look up orders - check refund eligibility - search FAQs - open a support ticket when needed After that, log the interaction and return the assistant's response. judgeChecklist: - "the input schema includes `customer_id` and `query_text`" - the flow loads the customer's profile and order history - the flow uses an AI assistant step - the assistant can look up orders - the assistant can check refund eligibility - the assistant can search FAQs - the assistant can open a support ticket - the flow logs the interaction - the final output returns the assistant response - id: flow-test7-simple-modification prompt: |- Update this flow so it validates processed data before saving it. After `process_data`, add a `validate_data` step that checks the data array is not empty. If the array is empty, the flow should surface the message `No data to save` and prevent saving. If validation passes, let the save continue normally. Update `save_results` so it uses the validation outcome instead of bypassing it. initial: ai_evals/fixtures/frontend/flow/initial/test5_initial.json runtime: maxTurns: 8 validate: topLevelStepIds: - fetch_data - process_data - validate_data topLevelStepOrder: - fetch_data - process_data - validate_data topLevelStepTypes: - id: fetch_data type: rawscript - id: process_data type: rawscript - id: validate_data type: rawscript judgeChecklist: - the updated flow keeps the original fetch and process steps intact - "a `validate_data` step is added after `process_data`" - "`validate_data` checks that the processed data array is not empty" - "when processed data is empty, the flow surfaces the message `No data to save` and does not save results" - "`save_results` uses the validation outcome instead of reading `results.process_data` directly" - "exact field names or wrapper object shape for the validation result are not important" - id: flow-test8-branching-in-loop prompt: |- Update the order-processing logic inside `loop_orders` so different order types are handled differently. For `express`, mark the order as priority and use a shipping cost of $15.99. For `standard`, use a shipping cost of $5.99. For `pickup`, mark it as no shipping required with a cost of $0. Keep the existing processing as a fallback for unknown order types. Each path should return the orderId, shipping cost, and shipping type. initial: ai_evals/fixtures/frontend/flow/initial/test6_initial.json judgeChecklist: - "the existing `loop_orders` flow still handles per-order processing" - exact branching topology is not required as long as `loop_orders` handles the order types correctly - express orders are marked as priority and use a shipping cost of 15.99 - standard orders use a shipping cost of 5.99 - pickup orders use a shipping cost of 0 and are treated as no shipping required - unknown order types still follow a fallback path - "each processed order returns `orderId`, `shippingCost`, and `shippingType`" - id: flow-test9-parallel-refactor prompt: |- Refactor this flow so the enrichment work no longer runs one step at a time. `enrich_price`, `enrich_inventory`, and `enrich_reviews` should run independently. Each one should return a fallback value if it fails. Update `combine_data` so it merges the enrichment results and sets a `hasFallbacks` flag when any fallback was used. Keep `get_item` as the first step and `return_result` as the last step. initial: ai_evals/fixtures/frontend/flow/initial/test7_initial.json validate: topLevelStepIds: - get_item - combine_data - return_result topLevelStepOrder: - get_item - combine_data - return_result topLevelStepTypeCountsAtLeast: - type: branchall count: 1 topLevelStepTypes: - id: get_item type: rawscript - id: combine_data type: rawscript - id: return_result type: rawscript moduleRules: - id: enrich_price - id: enrich_inventory - id: enrich_reviews judgeChecklist: - "the updated flow keeps `get_item` as the first step" - "the updated flow keeps `return_result` as the last step" - "`enrich_price`, `enrich_inventory`, and `enrich_reviews` run independently rather than sequentially" - each enrichment path returns a fallback value if it fails - "`combine_data` merges the enrichment results" - "`combine_data` sets `hasFallbacks` when any fallback was used" - id: flow-test10-while-loop-counter prompt: |- Create a flow that keeps incrementing a counter until it reaches a target value. The input should include a number field named `target`. Use a top-level loop step named `count_until_target`. Inside it, use a single step named `increment_counter` that increments the current counter. The loop should stop once the counter reaches `target`. After the loop, add a top-level step named `return_final_counter` that returns the last counter value. validate: exactTopLevelStepIds: - count_until_target - return_final_counter topLevelStepOrder: - count_until_target - return_final_counter topLevelStepTypes: - id: count_until_target type: whileloopflow - id: return_final_counter type: rawscript moduleRules: - id: count_until_target hasStopAfterIf: true hasStopAfterAllItersIf: false exactImmediateChildStepIds: - increment_counter immediateChildStepTypes: - id: increment_counter type: rawscript moduleFieldRules: - id: count_until_target path: stop_after_if.expr equals: result >= flow_input.target judgeChecklist: - "the input schema includes a number field named `target`" - "the top-level while loop step is named `count_until_target`" - "`count_until_target` contains a single increment step named `increment_counter`" - "`count_until_target` uses module-level `stop_after_if` to stop when the counter reaches `target`" - "`increment_counter` uses `flow_input.iter.value` or an equivalent loop-state expression and falls back to `0` on the first iteration" - "`return_final_counter` returns the final counter value" - id: flow-test11-preprocessor-and-failure-handler prompt: |- Create an event-processing flow for a string payload. Before the main processing runs, trim the payload and reject empty strings. The main step should be named `process_event` and return a simple success object. If anything fails, return a compact error object with the error message and the failing step id. expected: ai_evals/fixtures/frontend/flow/expected/test11_preprocessor_failure.json validate: requireSpecialModules: - preprocessor_module - failure_module judgeChecklist: - the flow trims the payload before the main processing runs - the flow rejects empty payload strings - "the main step is named `process_event`" - "`process_event` returns a simple success object" - failures return a compact error object with the error message and failing step id - id: flow-test12-approval-step prompt: |- Create a purchase approval flow. The input should include `requester_email` and `amount`. Add an approval step named `request_approval` that pauses the flow and asks the approver for a comment. One approval should be enough to continue. After approval, add a final step named `finalize_purchase` that returns an approved status object. validate: topLevelStepIds: - request_approval - finalize_purchase topLevelStepOrder: - request_approval - finalize_purchase topLevelStepTypes: - id: finalize_purchase type: rawscript schemaRequiredPaths: - requester_email - amount requireSuspendSteps: - id: request_approval requiredEvents: 1 resumeRequiredStringFieldAnyOf: - comment - approver_comment judgeChecklist: - "the flow includes an approval step named `request_approval`" - "`request_approval` pauses the flow and asks the approver for a comment" - one approval is enough to continue - "the flow includes a final step named `finalize_purchase`" - "`finalize_purchase` returns an approved status object after approval" - id: flow-test13-loop-resilience-toggle prompt: |- Update `loop_orders` so it can process orders in parallel. If one order fails, the rest should still continue. Keep the existing order-fetching and summary steps the same. initial: ai_evals/fixtures/frontend/flow/initial/test6_initial.json validate: exactTopLevelStepIds: - get_orders - loop_orders - summarize topLevelStepTypes: - id: loop_orders type: forloopflow moduleFieldRules: - id: loop_orders path: value.parallel equals: true - id: loop_orders path: value.skip_failures equals: true judgeChecklist: - "the flow keeps `get_orders` before `loop_orders` and `summarize` after it" - "`loop_orders` processes orders in parallel" - "a failure in one order does not stop the remaining orders from being processed" - id: flow-test14-modify-existing-special-modules prompt: |- Update this event-processing flow for a string payload. Before `process_event` runs, trim the payload and reject empty strings. If anything fails, return a compact error object with the error message and the failing step id. Keep `process_event` as the main step. initial: ai_evals/fixtures/frontend/flow/initial/test11_initial.json expected: ai_evals/fixtures/frontend/flow/expected/test11_preprocessor_failure.json validate: requireSpecialModules: - preprocessor_module - failure_module judgeChecklist: - the updated flow trims the payload before the main processing runs - the updated flow rejects empty payload strings - "the existing `process_event` step remains the main step" - failures return a compact error object with the error message and failing step id - id: flow-test15-create-current-flow-schedule prompt: |- Update this flow by adding a final step named `return_schedule_status`. It should return an object with `scheduled: true` and the order summary from `results.summarize_orders`. Also create an enabled daily schedule named `order_processing_daily` for the current flow. It should run every day at 07:30 UTC with empty args. Do not ask me for the flow path. initial: ai_evals/fixtures/frontend/flow/initial/scheduled_order_flow.json validate: topLevelStepIds: - return_schedule_status toolExpect: requiredToolsUsed: - test_run_flow - create_schedule toolCallArgs: - tool: create_schedule field: path stringStartsWithAnyOf: - f/ - u/ stringMustNotStartWithAnyOf: - schedules/ skipJudge: true judgeChecklist: - "the flow includes a final top-level step named `return_schedule_status`" - "`return_schedule_status` returns `scheduled: true` and the order summary" - id: flow-test16-create-current-flow-http-trigger prompt: |- Update this flow by adding a final step named `webhook_response`. It should return an object with `ok: true` and the order summary from `results.summarize_orders`. Also create a public POST HTTP endpoint named `order_processing_webhook` for the current flow. Use route path `ai-evals/order-processing` and no authentication. Do not ask me for the flow path. initial: ai_evals/fixtures/frontend/flow/initial/scheduled_order_flow.json validate: topLevelStepIds: - webhook_response toolExpect: requiredToolsUsed: - test_run_flow - create_trigger toolCallArgs: - tool: create_trigger field: path stringStartsWithAnyOf: - f/ - u/ stringMustNotStartWithAnyOf: - schedules/ skipJudge: true judgeChecklist: - "the flow includes a final top-level step named `webhook_response`" - "`webhook_response` returns `ok: true` and the order summary"