starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-09-07 00:01:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
bebd8bef8ba86f0cc016db3827ed1d8ee8c577fc
windmill/ai_evals/cases/app.yaml
T
GuilhemClaude Opus 5Ruben Fiszel
 574775d50c fix: teach the AI the raw-app job bindings, the SDK reference and the draft/deployed split (#10754) 
* feat: teach the AI the raw-app job bindings and the draft/deployed split

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: scope the raw-app deploy advice to the referenced item, and stop kind-conversion from stranding fields

The draft/deployed guidance added in the previous commit was read as "deploy the
app too": the agent asked for both the flow and the app and routed a one-item
dependency through the review-and-deploy page. Only the referenced flow or
script has to exist deployed — the preview runs the app's draft — so the prompts,
the `write_app_runnable` warning and the testing rule now say to offer that one
deploy and leave the app a draft.

`buildPersistedRunnable` spread the existing runnable when rewriting it, so
converting a path runnable to inline left `runType`/`path` behind (and the
reverse left `inlineScript`). `isRunnableByName` matches the inline branch
first, so an app "wired to a flow" silently ran stale inline code.

`test_run_app_runnable` now fills ctx-bound inputs with `$ctx:<prop>` the way
RawAppBackgroundRunner does, so a ctx argument no longer arrives missing.

The SDK-reference rationale claimed WM_TOKEN may be unset, that a missing base
URL falls back to localhost, and that a job token is scoped enough to 403 a
hand-rolled REST call. None of the three is true, and it shipped to every
write-script prompt; the text now only says the client configures itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: address review round on the raw-app AI instructions

The eval case could pass on the exact answer it exists to reject. Every
`requiredMentionsAnyOf` alternative but one was flow-agnostic, so "the app must
be deployed" satisfied "must be deployed". All alternatives now name the flow,
and a unit test pins that the app-only phrasing fails.

`instanceLine` asserted "self-hosted Community Edition" outside the browser,
where `isCloudHosted()` reads false and the license store is unset — so every
global eval was told that regardless of what it pointed at. It is now emitted
only under BROWSER.

`assistantExpect.forbiddenMentions` defaulted a missing `assistantText` to "",
which passes every entry forever on a mode whose runner does not report it.
It now fails with that as the reason.

`buildPersistedRunnable` carried `schema` across a retarget, so a path runnable
pointed at a new flow kept the previous item's schema and `genWmillTs` typed
`backend.<key>(args)` from the wrong inputs. It survives only while kind and
path both match.

The SDK header claimed "a function that is not listed below does not exist".
`windmill-client` also exports the generated services, and the Python client
exposes `Windmill.get`/`.post`, so an endpoint without a helper had no legal
move. Each language now names its own escape hatch.

`getAppInstructions` said the attached reference carries the TypeScript SDK even
when `language: "python3"` had swapped in the Python one — on the very sentence
telling the model to make that call.

The kind-conversion comment claimed a hybrid runnable "silently runs stale
inline code". It does not: `isRunnableByName`, `isRunnableByPath`,
`convertPersistedToBackendRunnable` and `rawAppPolicy.processRunnable` all
dispatch on `type` alone. The leftovers contradict the runnable's kind rather
than override it, which is what the comment now says.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: address round-2 review nits on the raw-app AI instructions

`flow is deployed` was satisfied both by "once the flow is deployed, the button
works" and by a hallucinated "done — the flow is deployed", which eval mode makes
impossible and the drafts-only judge cannot see. Every alternative now states an
outstanding obligation, and two more real phrasings ("will need to be deployed")
are accepted so a correct answer is not failed on wording.

Condenses the three comment blocks that ran past the four-line limit in
AGENTS.md, and drops two claims inside them that no longer hold: the
`testRunAppRunnable` doc said it runs a runnable the way the app's own frontend
does (it is the editor preview, which a deployed app's stored policy does not
match), and `undeployedRunnableTargets` described its argument as the write
tool's raw input when the call site passes the persisted runnable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: report the real cause when a test run fails, and label the app-runnable card

Driving `test_run_app_runnable` in a live session surfaced two defects the
API-level check could not see.

`executeTestRun` built its failure message from `error.message`, which the
generated client leaves as the bare status text while the server's message sits
in `body`. A path runnable aimed at an undeployed flow reported "Not Found"
instead of "Not found: flow not found at name u/admin/current_time" — dropping
the one diagnostic the run exists to produce. `formatToolError`, in the same
file and written for exactly this, now does it. This also applies to
test_run_script and test_run_flow, which had the same loss.

The completion card read "Flow test completed successfully" for an app runnable,
because `contextName` doubles as the jobs-tray kind and a path runnable pointing
at a flow really does queue a flow job. A `completionName` override now names
what ran without changing the kind.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test: pin the deploy expectation against wrong answers, not just correct ones

`deploying the flow` was satisfied by "done deploying the flow" — a deploy the
agent only claims to have made, which eval mode makes impossible and the
drafts-only judge cannot see. Replaced with the prospective forms, and dropped
the same reading from the workflow variant.

Three review rounds each found this same class of hole in the phrasing list, so
the list is now exercised against the wrong answers themselves rather than
eyeballed: naming the app as what needs deploying, claiming the deploy is
already done, claiming to have deployed the flow, and saying nothing about
deploying all have to fail, while four real correct phrasings have to pass. The
test reads the case out of global.yaml, so a future edit to the alternatives is
checked by it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test: drop the tense-neutral deploy alternatives and cover completed claims

A gerund after a preposition carries no tense, so `before`/`after`/`by deploying
the flow` all match a deploy the agent only claims to have made ("after
deploying the flow, I clicked the button and it returns the greeting") just as
the bare gerund did. All three are gone rather than swapped for whichever reads
least badly, and the two completed-deploy phrasings are now negative fixtures.
The remaining alternatives are imperative or obligational, which a claim of
having already deployed cannot satisfy.

Condenses the two comments this list carries: the YAML block to four lines, and
the test's rationale to the durable constraint about substring matching.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: encrypt sensitive inputs when test-running an app runnable

`test_run_app_runnable` sent `force_viewer_static_fields` but not
`force_viewer_sensitive_inputs`, which every other preview path derives from
the runnable's `sensitive` user fields. That list is the only thing driving the
encryption loop in apps.rs, so testing a runnable with a sensitive input wrote
the real value into the job's args in plaintext, readable by anyone with run
access to the workspace.

Verified against a running EE instance. With the list, `api_key` is stored as
`$encrypted:mvqtSRI9…` and the sentinel appears nowhere in the job record;
without it, the sentinel is readable in run details. A non-sensitive field is
left plaintext either way.

The tool claims parity with the editor preview, so it uses that same filter
(`type == 'user' && sensitive`) and omits the field entirely when empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2026-08-20 11:17:33 +02:00

356 lines
13 KiB
YAML
Raw Blame History

- id: app-test1-counter-create
prompt: |-
Create a simple counter app with increment and decrement buttons.
judgeChecklist:
- shows the current count in the UI
- includes an increment button
- includes a decrement button
- clicking the buttons updates the count correctly
- id: app-test2-counter-reset
prompt: |-
Add a reset button that sets the counter back to 0
initial: ai_evals/fixtures/frontend/app/initial/test1_counter_app
judgeChecklist:
- adds a reset control to the existing counter app
- clicking reset sets the count back to 0
- keeps the existing increment and decrement behavior working
- id: app-test3-shopping-cart-quantity
prompt: |-
Add a quantity selector (+ and - buttons) to each cart item so users can adjust quantities without removing and re-adding items
initial: ai_evals/fixtures/frontend/app/initial/shopping_cart
judgeChecklist:
- each cart item has visible plus and minus quantity controls
- users can increase quantity without re-adding the product
- users can decrease quantity from the cart UI
- cart totals stay in sync with quantity changes
- id: app-test4-shopping-cart-discount
prompt: |-
Add a discount code input field in the cart.
When the code "SAVE10" is entered, apply a 10% discount to the total
initial: ai_evals/fixtures/frontend/app/initial/shopping_cart
judgeChecklist:
- adds a discount code input to the cart
- recognizes the code SAVE10
- applies a 10 percent discount to the displayed total
- keeps the rest of the cart behavior intact
- id: app-test5-file-manager-search
prompt: |-
Add a search bar in the toolbar that filters files and folders by name as the user types
initial: ai_evals/fixtures/frontend/app/initial/file_manager
judgeChecklist:
- adds a search input in the toolbar
- filters files and folders by name as the user types
- updates the visible file list from the search query
- keeps the rest of the file manager usable
- id: app-test6-file-manager-rename-save-cancel
prompt: |-
Improve the existing inline rename flow for files and folders.
When renaming, show explicit Save and Cancel buttons next to the name input.
Pressing Enter should save, pressing Escape should cancel, and Cancel should restore the original name without calling rename.
Keep the existing backend rename behavior for successful saves.
initial: ai_evals/fixtures/frontend/app/initial/file_manager
validate:
requiredFrontendPaths:
- /index.tsx
- /components/FileItem.tsx
requiredFrontendFileContent:
- path: /components/FileItem.tsx
includes:
- Save
- Cancel
- Escape
forbiddenAppContent:
- onBlur={handleRename}
judgeChecklist:
- keeps the existing visible rename action in the file list
- shows explicit Save and Cancel controls while editing a name
- pressing Enter saves the new name through the existing rename behavior
- pressing Escape or Cancel exits rename mode and restores the original name without saving
- id: app-test7-file-manager-select-all
prompt: |-
Add a "Select All" checkbox in the file list header and individual checkboxes for each file.
Add a "Delete Selected" button that appears when items are selected
initial: ai_evals/fixtures/frontend/app/initial/file_manager
judgeChecklist:
- adds a select-all control in the file list header
- adds per-item selection controls
- shows a delete-selected action only when there is a selection
- deleting selected items updates the visible list
- id: app-test8-inventory-tracker-search-delete
prompt: |-
Update this inventory tracker app so users can search items by name or sku and delete existing items.
Keep the existing add-item flow and datatable-backed persistence working.
initial: ai_evals/fixtures/frontend/app/initial/inventory_tracker
validate:
requiredFrontendPaths:
- /index.tsx
requiredBackendRunnableKeys:
- listInventory
- addInventory
- deleteInventory
requiredBackendRunnableTypes:
- key: listInventory
type: inline
- key: addInventory
type: inline
- key: deleteInventory
type: inline
requiredDatatables:
- datatableName: main
schema: public
table: inventory_items
judgeChecklist:
- keeps the existing add-item form working
- adds a search input that filters inventory by name or sku
- adds a delete action for existing inventory items
- deleting an inventory item updates the visible list
- keeps inventory persistence working through the existing datatable-backed app setup
- id: app-test9-recipe-book-search-delete
prompt: |-
Update this recipe book app so users can search recipes by name and delete existing recipes.
Keep the existing add-recipe flow and datatable-backed persistence working.
initial: ai_evals/fixtures/frontend/app/initial/recipe_book
validate:
requiredFrontendPaths:
- /index.tsx
requiredBackendRunnableKeys:
- listRecipes
- addRecipe
- deleteRecipe
requiredBackendRunnableTypes:
- key: listRecipes
type: inline
- key: addRecipe
type: inline
- key: deleteRecipe
type: inline
requiredDatatables:
- datatableName: main
schema: public
table: recipes
judgeChecklist:
- keeps the existing add-recipe form working
- adds a search input that filters recipes by name
- adds a delete action for existing recipes
- deleting a recipe updates the visible list
- keeps recipe persistence working through the existing datatable-backed app setup
- id: app-datatable-persistent-notes
prompt: |-
Build a notes app that persists notes in the existing datatable table.
Inspect the existing datatable and table schema before writing code.
Use the existing main/public.notes table, and do not create any new tables.
Create backend runnables named exactly listNotes, addNote, and deleteNote.
The UI should list notes, add a note with title/body, and delete notes.
Do not use localStorage, sessionStorage, IndexedDB, or in-memory-only persistence.
initial: ai_evals/fixtures/frontend/app/initial/notes_datatable
runtime:
maxTurns: 10
validate:
requiredFrontendPaths:
- /index.tsx
requiredFrontendFileContent:
- path: /index.tsx
includes:
- backend.listNotes
- backend.addNote
- backend.deleteNote
requiredBackendRunnableKeys:
- listNotes
- addNote
- deleteNote
requiredBackendRunnableTypes:
- key: listNotes
type: inline
- key: addNote
type: inline
- key: deleteNote
type: inline
requiredBackendRunnableContent:
- key: listNotes
includes:
- wmill.datatable
- select
- notes
- key: addNote
includes:
- wmill.datatable
- insert
- notes
- key: deleteNote
includes:
- wmill.datatable
- delete
- notes
datatableTableCountExactly: 1
requiredDatatables:
- datatableName: main
schema: public
table: notes
requiredToolsUsed:
- list_datatables
- get_datatable_table_schema
forbiddenAppContent:
- localStorage
- sessionStorage
- indexedDB
judgeChecklist:
- creates a notes UI that lists notes from the backend
- adds notes through backend datatable persistence
- deletes notes through backend datatable persistence
- reuses the existing main/public.notes table without creating new tables
- does not use browser storage or in-memory-only persistence as the source of truth
- id: app-test10-session-id-no-crypto
prompt: |-
Update `generateSessionId` so it no longer uses `crypto.randomUUID()`.
Make it return a handmade string id built from the current time and random characters.
Keep the existing sessionStorage-based chat session behavior unchanged.
initial: ai_evals/fixtures/frontend/app/initial/session_id_chat
runtime:
maxTurns: 4
validate:
requiredFrontendPaths:
- /index.tsx
requiredBackendRunnableKeys:
- a
requiredBackendRunnableTypes:
- key: a
type: inline
judgeChecklist:
- generateSessionId no longer calls crypto.randomUUID
- generateSessionId returns a handmade string id without using crypto
- getSessionId still stores and reuses chat_session_id in sessionStorage
- the existing new chat and send message session behavior remains wired up
- id: app-token-baseline-large-app-small-edit
prompt: |-
Change the main heading from "Analytics Console" to "Operations Console".
Keep the existing filtering, summary loading, and backend calls unchanged.
initial: ai_evals/fixtures/frontend/app/initial/token_heavy_context
runtime:
maxTurns: 8
validate:
requiredFrontendPaths:
- /index.tsx
requiredBackendRunnableKeys:
- loadAnalytics
- refreshSummary
judgeChecklist:
- changes the visible main heading to Operations Console
- keeps the existing summary loading behavior wired to loadAnalytics
- keeps the existing filter input and metric list behavior intact
- id: app-token-many-datatable-context
prompt: |-
Add a short note under the dashboard heading that says "Using existing analytics tables".
Do not create any new tables.
initial: ai_evals/fixtures/frontend/app/initial/token_heavy_datatables
runtime:
maxTurns: 8
appContext:
additional:
- type: datatable
datatableName: main
schema: analytics
table: event_log_01
- type: datatable
datatableName: main
schema: analytics
table: event_log_02
- type: datatable
datatableName: main
schema: analytics
table: event_log_03
- type: datatable
datatableName: main
schema: analytics
table: event_log_04
- type: datatable
datatableName: main
schema: analytics
table: event_log_05
- type: datatable
datatableName: main
schema: analytics
table: event_log_06
- type: datatable
datatableName: main
schema: analytics
table: event_log_07
- type: datatable
datatableName: main
schema: analytics
table: event_log_08
- type: datatable
datatableName: main
schema: operations
table: ops_record_13
- type: datatable
datatableName: main
schema: operations
table: ops_record_14
validate:
requiredFrontendPaths:
- /index.tsx
datatableCountAtLeast: 1
datatableTableCountAtLeast: 18
judgeChecklist:
- adds the note Using existing analytics tables under or near the heading
- does not create new datatable tables
- keeps the configured datatable references available in the app artifact
- id: app-token-large-datatable-discovery
prompt: |-
Build a read-only dashboard page that reuses the existing analytics datatable tables.
Show a simple summary of which existing tables are available, and do not create any new tables.
initial: ai_evals/fixtures/frontend/app/initial/token_heavy_datatables
runtime:
maxTurns: 8
validate:
requiredFrontendPaths:
- /index.tsx
datatableCountAtLeast: 1
datatableTableCountAtLeast: 18
judgeChecklist:
- reuses the existing datatable configuration rather than creating new tables
- presents a read-only dashboard or summary of available analytics data
- keeps the configured datatable references available in the app artifact
# GIT-967, app mode: asked to track a long-running job, the agent hand-wrote a
# runnable that fetched the jobs REST API — guessing at WM_TOKEN and a base URL
# until it fell back to localhost — instead of using backendAsync + getJob/waitJob,
# which the generated ./wmill bindings already provide.
- id: app-long-job-progress
prompt: |-
Add a "Generate report" button. Building the report takes a few minutes, so as soon
as the user clicks it the app should show the run's job id and keep updating its
status until it finishes, then display the result.
runtime:
maxTurns: 22
validate:
forbiddenAppContent:
- BASE_INTERNAL_URL
- WM_BASE_URL
- WM_TOKEN
- localhost:8000
- getResultMaybe
- jobs/list
- getWorkspaceToken
- getBaseUrl
judgeChecklist:
- adds a Generate report button that starts the report
- shows the run's job id as soon as the run starts
- keeps the status updating while the run is in flight and shows the result when it completes
- starts the run with backendAsync and tracks it with getJob, waitJob or streamJob — all three are real exports of the generated ./wmill module, alongside backend and backendAsync
- does not write a backend runnable that polls job status or lists jobs itself
- does not call the Windmill API with fetch from the frontend, and does not read WM_TOKEN, BASE_INTERNAL_URL or WM_BASE_URL anywhere
Reference in New Issue View Git Blame Copy Permalink