mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-13 16:02:59 +00:00
76 lines
3.2 KiB
YAML
76 lines
3.2 KiB
YAML
- id: script-test1-greet-user
|
|
prompt: |-
|
|
Update the current Bun script so it takes the existing `name` input and returns a plain greeting string like `Hello, Alice!`.
|
|
Do not wrap the result in an object or array.
|
|
Keep it simple and do not add external dependencies.
|
|
initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
|
|
expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- test_run_script
|
|
judgeChecklist:
|
|
- uses the existing `name` input
|
|
- returns a plain greeting string
|
|
- does not wrap the result in an object or array
|
|
|
|
# An in-house API has no hub resource type: the connection still comes in as one resource, not
|
|
# a token parameter or a variable read inside the code.
|
|
- id: script-resource-type-custom-api
|
|
prompt: |-
|
|
Update the current Bun script so it fetches an invoice by id from our internal billing API at https://billing.acme.internal/api/v1. It authenticates with a bearer token.
|
|
initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
|
|
judgeChecklist:
|
|
- the script takes the billing API's base URL and token together as one resource-typed parameter, not a plain string token or a variable read inside the code
|
|
- the script fetches an invoice by id with a bearer token
|
|
- the custom resource type's name has the c_ prefix (e.g. `RT.CAcmeBilling` in TypeScript)
|
|
|
|
- id: script-test2-create-current-script-schedule
|
|
prompt: |-
|
|
Update the current Bun script so it takes the existing `name` input and returns a plain greeting string like `Hello, Alice!`.
|
|
Also create an enabled daily schedule named `greet_user_daily` for the current script.
|
|
It should run every day at 09:00 UTC and pass `{ "name": "Alice" }` as args.
|
|
Do not ask me for the script path.
|
|
initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
|
|
expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- test_run_script
|
|
- create_schedule
|
|
toolCallArgs:
|
|
- tool: create_schedule
|
|
field: path
|
|
stringStartsWithAnyOf:
|
|
- f/
|
|
- u/
|
|
stringMustNotStartWithAnyOf:
|
|
- schedules/
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- uses the existing `name` input
|
|
- returns a plain greeting string
|
|
|
|
- id: script-test3-create-current-script-http-trigger
|
|
prompt: |-
|
|
Update the current Bun script so it takes the existing `name` input and returns a plain greeting string like `Hello, Alice!`.
|
|
Also create a public POST HTTP endpoint named `greet_user_webhook` for the current script.
|
|
Use route path `ai-evals/greet-user` and no authentication.
|
|
Do not ask me for the script path.
|
|
initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
|
|
expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- test_run_script
|
|
- create_trigger
|
|
toolCallArgs:
|
|
- tool: create_trigger
|
|
field: path
|
|
stringStartsWithAnyOf:
|
|
- f/
|
|
- u/
|
|
stringMustNotStartWithAnyOf:
|
|
- schedules/
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- uses the existing `name` input
|
|
- returns a plain greeting string
|