mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
centdix
8196857c8f
* feat: add workflow-as-code skill
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: make system prompt freshness self-contained
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Revert "fix: make system prompt freshness self-contained"
This reverts commit 7d2fde9585.
* fix: refresh wac generated guidance
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: add wac cli eval cases
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: align wac prompt imports
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
264 lines
9.6 KiB
YAML
264 lines
9.6 KiB
YAML
- id: bun-hello-script
|
|
prompt: |-
|
|
Create a Windmill Bun script at `f/evals/hello.ts`.
|
|
It should take a `name` input and return a greeting object like `{ greeting: "Hello, Alice!" }`.
|
|
expected: ai_evals/fixtures/cli/expected/bun-hello-script
|
|
cliExpect:
|
|
requiredSkills:
|
|
- write-script-bun
|
|
requiredSkillsBeforeFirstMutation:
|
|
- write-script-bun
|
|
forbiddenSkills:
|
|
- write-script-python3
|
|
- write-flow
|
|
orderedAssistantMentions:
|
|
- wmill generate-metadata
|
|
- wmill sync push
|
|
orderedProposedCommands:
|
|
- wmill generate-metadata
|
|
- wmill sync push
|
|
forbiddenExecutedCommands:
|
|
- ^wmill generate-metadata
|
|
- ^wmill sync push
|
|
judgeChecklist:
|
|
- creates the requested Bun script at f/evals/hello.ts
|
|
- takes a name input
|
|
- returns an object containing the greeting
|
|
|
|
- id: bun-hello-flow
|
|
prompt: |-
|
|
Create a Windmill flow at `f/evals/hello__flow`.
|
|
It should take a `name` input and return a greeting object like `{ greeting: "Hello, Alice!" }`.
|
|
Put the step code in `hello.ts`.
|
|
expected: ai_evals/fixtures/cli/expected/bun-hello-flow
|
|
cliExpect:
|
|
requiredSkills:
|
|
- write-flow
|
|
requiredSkillsBeforeFirstMutation:
|
|
- write-flow
|
|
forbiddenSkills:
|
|
- write-script-python3
|
|
orderedAssistantMentions:
|
|
- wmill generate-metadata
|
|
- wmill sync push
|
|
orderedProposedCommands:
|
|
- wmill generate-metadata
|
|
- wmill sync push
|
|
forbiddenExecutedCommands:
|
|
- ^wmill generate-metadata
|
|
- ^wmill sync push
|
|
judgeChecklist:
|
|
- creates the requested flow folder with flow.yaml and hello.ts
|
|
- wires the name input into the flow step
|
|
- returns the greeting object
|
|
|
|
- id: python-add-numbers-script
|
|
prompt: |-
|
|
Add a Windmill Python script at `f/evals/add_numbers.py`.
|
|
It should take `a` and `b` as inputs and return `{ "total": a + b }`.
|
|
expected: ai_evals/fixtures/cli/expected/python-add-numbers-script
|
|
cliExpect:
|
|
requiredSkills:
|
|
- write-script-python3
|
|
requiredSkillsBeforeFirstMutation:
|
|
- write-script-python3
|
|
forbiddenSkills:
|
|
- write-script-bun
|
|
- write-flow
|
|
orderedAssistantMentions:
|
|
- wmill generate-metadata
|
|
- wmill sync push
|
|
orderedProposedCommands:
|
|
- wmill generate-metadata
|
|
- wmill sync push
|
|
forbiddenExecutedCommands:
|
|
- ^wmill generate-metadata
|
|
- ^wmill sync push
|
|
judgeChecklist:
|
|
- creates the requested Python script at f/evals/add_numbers.py
|
|
- takes `a` and `b` as inputs
|
|
- returns an object with total equal to a plus b
|
|
|
|
- id: bun-hello-script-uppercase
|
|
prompt: |-
|
|
Update `f/evals/hello.ts` so it accepts an optional `uppercase` boolean.
|
|
Keep returning `{ greeting: ... }`, but when `uppercase` is true the greeting should be uppercased before returning it.
|
|
initial: ai_evals/fixtures/cli/initial/bun-hello-script-uppercase
|
|
expected: ai_evals/fixtures/cli/expected/bun-hello-script-uppercase
|
|
judgeChecklist:
|
|
- updates the existing hello.ts file rather than creating a new script
|
|
- accepts an optional uppercase boolean input
|
|
- keeps returning an object with greeting
|
|
- uppercases the greeting when uppercase is true
|
|
|
|
- id: bun-hello-flow-punctuation
|
|
prompt: |-
|
|
Update the existing flow in `f/evals/hello__flow` so it also accepts an optional `punctuation` input.
|
|
The greeting should use that punctuation and default to `!` when it is missing.
|
|
initial: ai_evals/fixtures/cli/initial/bun-hello-flow-punctuation
|
|
expected: ai_evals/fixtures/cli/expected/bun-hello-flow-punctuation
|
|
judgeChecklist:
|
|
- updates the existing hello flow instead of creating a new one
|
|
- adds an optional punctuation input to the flow
|
|
- updates the step code so the returned greeting uses punctuation
|
|
- defaults punctuation to an exclamation mark when omitted
|
|
|
|
- id: flow-reuse-existing-script
|
|
prompt: |-
|
|
There is already a reusable greeting script at `f/lib/format_greeting.ts`.
|
|
Create a flow at `f/evals/reuse_greeting__flow` that takes a `name` input and reuses that existing script instead of duplicating the logic inline.
|
|
initial: ai_evals/fixtures/cli/initial/flow-reuse-existing-script
|
|
expected: ai_evals/fixtures/cli/expected/flow-reuse-existing-script
|
|
cliExpect:
|
|
requiredSkills:
|
|
- write-flow
|
|
requiredSkillsBeforeFirstMutation:
|
|
- write-flow
|
|
orderedAssistantMentions:
|
|
- wmill generate-metadata
|
|
- wmill sync push
|
|
orderedProposedCommands:
|
|
- wmill generate-metadata
|
|
- wmill sync push
|
|
forbiddenExecutedCommands:
|
|
- ^wmill generate-metadata
|
|
- ^wmill sync push
|
|
judgeChecklist:
|
|
- creates the requested flow at f/evals/reuse_greeting__flow
|
|
- reuses the existing script from f/lib by path
|
|
- does not duplicate the greeting logic in a new inline script
|
|
- wires the name input into the reused script
|
|
|
|
- id: wac-typescript-order-workflow
|
|
prompt: |-
|
|
Create a Windmill Workflow-as-Code TypeScript script at `f/evals/order_workflow.ts`.
|
|
It should take an `orderId` string, load the order in a durable task, checkpoint a processing timestamp with `step`, and return `{ orderId, processedAt, status }`.
|
|
cliExpect:
|
|
requiredSkills:
|
|
- write-workflow-as-code
|
|
requiredSkillsBeforeFirstMutation:
|
|
- write-workflow-as-code
|
|
forbiddenSkills:
|
|
- write-flow
|
|
- write-script-bun
|
|
- write-script-python3
|
|
judgeChecklist:
|
|
- creates the requested TypeScript WAC script at f/evals/order_workflow.ts
|
|
- uses the Workflow-as-Code SDK from windmill-client
|
|
- wraps the entrypoint with workflow
|
|
- uses a durable task for loading the order
|
|
- uses step to checkpoint the processing timestamp
|
|
- does not create an OpenFlow flow.yaml or flow folder
|
|
|
|
- id: wac-python-approval-workflow
|
|
prompt: |-
|
|
Create a Windmill Workflow-as-Code Python script at `f/evals/approval_workflow.py`.
|
|
It should take a `request_id` string, prepare an approval summary in a task, create resume URLs inside a durable step, wait for approval, and return the approval result.
|
|
cliExpect:
|
|
requiredSkills:
|
|
- write-workflow-as-code
|
|
requiredSkillsBeforeFirstMutation:
|
|
- write-workflow-as-code
|
|
forbiddenSkills:
|
|
- write-flow
|
|
- write-script-bun
|
|
- write-script-python3
|
|
judgeChecklist:
|
|
- creates the requested Python WAC script at f/evals/approval_workflow.py
|
|
- imports Workflow-as-Code helpers from wmill
|
|
- decorates an async entrypoint with @workflow
|
|
- uses @task for the approval summary work
|
|
- gets resume URLs inside step before waiting for approval
|
|
- uses wait_for_approval
|
|
- does not create an OpenFlow flow.yaml or flow folder
|
|
|
|
- id: wac-not-openflow-disambiguation
|
|
prompt: |-
|
|
Create this as Workflow-as-Code, not an OpenFlow YAML flow: a TypeScript script at `f/evals/fanout_workflow.ts`.
|
|
It should take an array of customer IDs, process each customer with a WAC task, run the independent customer tasks in parallel, and return the collected results.
|
|
cliExpect:
|
|
requiredSkills:
|
|
- write-workflow-as-code
|
|
requiredSkillsBeforeFirstMutation:
|
|
- write-workflow-as-code
|
|
forbiddenSkills:
|
|
- write-flow
|
|
- write-script-bun
|
|
- write-script-python3
|
|
judgeChecklist:
|
|
- creates the requested TypeScript script at f/evals/fanout_workflow.ts
|
|
- treats the request as Workflow-as-Code rather than an OpenFlow flow
|
|
- uses workflow for the script entrypoint
|
|
- uses task for each customer processing unit
|
|
- runs independent customer tasks in parallel
|
|
- does not create a flow folder or flow.yaml
|
|
|
|
- id: cli-job-debug-guidance
|
|
prompt: |-
|
|
A Windmill job failed.
|
|
Tell me exactly which `wmill` commands to run to inspect the job details, logs, and final result for job ID `123`.
|
|
Do not modify any files.
|
|
cliExpect:
|
|
requiredSkills:
|
|
- cli-commands
|
|
workspaceUnchanged: true
|
|
orderedProposedCommands:
|
|
- wmill job get 123
|
|
- wmill job logs 123
|
|
- wmill job result 123
|
|
forbiddenProposedCommands:
|
|
- wmill sync push
|
|
forbiddenExecutedCommands:
|
|
- ^wmill job get
|
|
- ^wmill job logs
|
|
- ^wmill job result
|
|
judgeChecklist:
|
|
- does not modify the workspace
|
|
- recommends commands to inspect the job details
|
|
- recommends commands to inspect the job logs
|
|
- recommends commands to inspect the final result
|
|
|
|
- id: cli-sync-pull-guidance
|
|
prompt: |-
|
|
I want to review remote workspace changes before editing locally.
|
|
Tell me the first `wmill` command I should run.
|
|
Do not modify any files.
|
|
cliExpect:
|
|
requiredSkills:
|
|
- cli-commands
|
|
workspaceUnchanged: true
|
|
requiredProposedCommands:
|
|
- wmill sync pull
|
|
forbiddenProposedCommands:
|
|
- wmill sync push
|
|
forbiddenExecutedCommands:
|
|
- ^wmill sync pull
|
|
- ^wmill sync push
|
|
judgeChecklist:
|
|
- does not modify the workspace
|
|
- recommends using sync pull before making local edits
|
|
- does not recommend pushing first
|
|
|
|
- id: cli-script-deploy-guidance
|
|
prompt: |-
|
|
I already modified a Windmill script locally and now want the next CLI commands to prepare it and deploy it.
|
|
Tell me the commands to run, in order.
|
|
Do not modify any files.
|
|
cliExpect:
|
|
requiredSkills:
|
|
- cli-commands
|
|
workspaceUnchanged: true
|
|
orderedAssistantMentions:
|
|
- wmill generate-metadata
|
|
- wmill sync push
|
|
orderedProposedCommands:
|
|
- wmill generate-metadata
|
|
- wmill sync push
|
|
forbiddenExecutedCommands:
|
|
- ^wmill generate-metadata
|
|
- ^wmill sync push
|
|
judgeChecklist:
|
|
- does not modify the workspace
|
|
- recommends generate-metadata before sync push
|
|
- presents the commands in order
|