mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-15 16:03:02 +00:00
hugocasaandClaude Opus 5.5
88c3cc0ced
* feat: guide the AI when no resource type fits a service Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: report a failed resource type search and model a synced instance in evals Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: keep named resource types first and list only related names on search failure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: point the sync note at the hub sync button only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor: list the instance's resource types when the search finds nothing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: let the user create a resource type instead of syncing the hub Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: make the linked-variable resource the default, not the only option Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: add a cli eval for a service without a hub resource type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: keep the resource type search fresh and serve it without embeddings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: let the global chat create a resource type after the user confirms it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: simulate create_resource_type in global evals and fix the in-house API checklists Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: reuse unchanged custom resource type vectors and match names case-insensitively Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: add the unsynced note whatever the resource type search returns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: carry the no-type rule into flow inputs and script parameters, and ask the chat to relay the sync Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: tell script chat to hand the user the schema it cannot create Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: skip the resource type fingerprint query when no index is loaded Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: index resource types locally when the hub's embeddings cannot be fetched Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: match short service names in the resource type text fallback Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: name custom resource types with the c_ prefix the Resources page uses Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: require the c_ prefix in in-house API evals and a real sync mention Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: cover workspace scoping of the resource type text search Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: say the form adds the c_ prefix unless an admin turns it off Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: refresh resource type names and the resources tab after the chat creates a type Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf: fingerprint resource types without sorting every schema Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: tell the global chat in each message when the hub's types are not synced Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: send users who cannot deploy to an admin for a resource type, and have the cli list types first Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YR13nJWziUSWktZwhC7dx * refactor: move the resource type search index changes to their own pull request Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015YR13nJWziUSWktZwhC7dx --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
76 lines
3.2 KiB
YAML
76 lines
3.2 KiB
YAML
- id: script-test1-greet-user
|
|
prompt: |-
|
|
Update the current Bun script so it takes the existing `name` input and returns a plain greeting string like `Hello, Alice!`.
|
|
Do not wrap the result in an object or array.
|
|
Keep it simple and do not add external dependencies.
|
|
initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
|
|
expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- test_run_script
|
|
judgeChecklist:
|
|
- uses the existing `name` input
|
|
- returns a plain greeting string
|
|
- does not wrap the result in an object or array
|
|
|
|
# An in-house API has no hub resource type: the connection still comes in as one resource, not
|
|
# a token parameter or a variable read inside the code.
|
|
- id: script-resource-type-custom-api
|
|
prompt: |-
|
|
Update the current Bun script so it fetches an invoice by id from our internal billing API at https://billing.acme.internal/api/v1. It authenticates with a bearer token.
|
|
initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
|
|
judgeChecklist:
|
|
- the script takes the billing API's base URL and token together as one resource-typed parameter, not a plain string token or a variable read inside the code
|
|
- the script fetches an invoice by id with a bearer token
|
|
- the custom resource type's name has the c_ prefix (e.g. `RT.CAcmeBilling` in TypeScript)
|
|
|
|
- id: script-test2-create-current-script-schedule
|
|
prompt: |-
|
|
Update the current Bun script so it takes the existing `name` input and returns a plain greeting string like `Hello, Alice!`.
|
|
Also create an enabled daily schedule named `greet_user_daily` for the current script.
|
|
It should run every day at 09:00 UTC and pass `{ "name": "Alice" }` as args.
|
|
Do not ask me for the script path.
|
|
initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
|
|
expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- test_run_script
|
|
- create_schedule
|
|
toolCallArgs:
|
|
- tool: create_schedule
|
|
field: path
|
|
stringStartsWithAnyOf:
|
|
- f/
|
|
- u/
|
|
stringMustNotStartWithAnyOf:
|
|
- schedules/
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- uses the existing `name` input
|
|
- returns a plain greeting string
|
|
|
|
- id: script-test3-create-current-script-http-trigger
|
|
prompt: |-
|
|
Update the current Bun script so it takes the existing `name` input and returns a plain greeting string like `Hello, Alice!`.
|
|
Also create a public POST HTTP endpoint named `greet_user_webhook` for the current script.
|
|
Use route path `ai-evals/greet-user` and no authentication.
|
|
Do not ask me for the script path.
|
|
initial: ai_evals/fixtures/frontend/script/initial/test1_empty_bun.json
|
|
expected: ai_evals/fixtures/frontend/script/expected/test1_greet_user.json
|
|
toolExpect:
|
|
requiredToolsUsed:
|
|
- test_run_script
|
|
- create_trigger
|
|
toolCallArgs:
|
|
- tool: create_trigger
|
|
field: path
|
|
stringStartsWithAnyOf:
|
|
- f/
|
|
- u/
|
|
stringMustNotStartWithAnyOf:
|
|
- schedules/
|
|
skipJudge: true
|
|
judgeChecklist:
|
|
- uses the existing `name` input
|
|
- returns a plain greeting string
|