test: require the c_ prefix in in-house API evals and a real sync mention

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
hugocasa
2026-10-09 10:09:59 +02:00
co-authored by Claude Opus 5.5
parent 9247790e53
commit 589a069228
4 changed files with 9 additions and 1 deletions
+1
View File
@@ -99,6 +99,7 @@
- creates the script at f/evals/get_invoice.ts, fetching an invoice by id with a bearer token
- the script takes the billing API's base URL and token together as one resource-typed parameter, not a plain string token or a variable read inside the code
- defines the billing API's resource type with its base URL and token fields, as a resource-type file or in the reply (asking before writing the file is fine)
- the custom resource type's name has the c_ prefix (e.g. `RT.CAcmeBilling` in TypeScript, `c_acme_billing.resource-type.json`)
- id: flow-reuse-existing-script
prompt: |-
+1
View File
@@ -24,6 +24,7 @@
judgeChecklist:
- the flow takes the billing API's base URL and token together as one resource-typed input, not a plain string token or a variable read inside the code
- the flow fetches an invoice by id with a bearer token
- the custom resource type's name has the c_ prefix (e.g. `resource-c_acme_billing`)
- id: flow-test1-reuse-existing-script
prompt: |-
+6 -1
View File
@@ -781,8 +781,12 @@
- create_resource_type
- deploy_workspace_item
assistantExpect:
# Plain substring test: a bare "sync" would also match "async".
requiredMentionsAnyOf:
- - sync
- - " sync "
- sync resource types
- synced
- syncing
judgeChecklist:
- the script takes the Confluence connection as one resource-typed parameter rather than separate string credentials or a variable read inside the code
- the script creates a Confluence page from a space key, a title and a body
@@ -812,6 +816,7 @@
judgeChecklist:
- the script takes the billing API's base URL and token together as one resource-typed parameter, not a plain string token or a variable read inside the code
- the script fetches an invoice by id with a bearer token
- the custom resource type's name has the c_ prefix (e.g. `RT.CAcmeBilling` in TypeScript, `resource-c_acme_billing` in the schema)
- id: global-test19-datatable-not-configured-asks-to-set-up
prompt: |-
+1
View File
@@ -22,6 +22,7 @@
judgeChecklist:
- the script takes the billing API's base URL and token together as one resource-typed parameter, not a plain string token or a variable read inside the code
- the script fetches an invoice by id with a bearer token
- the custom resource type's name has the c_ prefix (e.g. `RT.CAcmeBilling` in TypeScript)
- id: script-test2-create-current-script-schedule
prompt: |-