windmill/docs/ai-agent-evals.md

AI agent evals

A reusable AI agent (docs/reusable-ai-agents.md) can be run on its own, against a curated set of cases — the inputs it is expected to keep handling.

Three words, and no fourth:

• a case is one input the agent should handle, held in a dataset;
• a run (stored as an experiment) is one execution of a whole dataset: a single flow job that answers every case, which is what the UI labels "Run N";
• each case is answered as one iteration of that run.

The surface is a dialog, two screens deep: this agent's runs across every dataset it has been measured on (one row per run, a badge per scorer), and, on opening a run, one table — a row per case, a column per scorer, the cell being that scorer's verdict — with a case's detail beside it. Editing a dataset is a drawer over both. It opens from where an agent already is: the agent card at the top of an AI agent step's inputs in the flow editor, and the ai_agent row on /resources.

Evals belong to a saved agent. A dataset and its runs hang off an ai_agent resource, so they outlive the step being renamed, copied or deleted, and two runs are comparable because they name the same thing. A step whose agent is written inline has nothing to hang them on: what stands in its place is Save as reusable agent, which is the only setup evals ask for.

What runs

A run is one flow: a loop over the dataset's cases, each iteration answering its case and then scoring the answer. Pushed as a RawFlow, so the agent step is the same vehicle ModuleTest.svelte uses to test an agent step and a case exercises the production branch of ai_executor.rs rather than a parallel one.

One flow rather than one job per case, because a run outlives the tab that started it: a dataset of two hundred cases against a slow provider takes long enough that nobody watches it, and only a worker can notice that the last case finished. Scoring is therefore a step, not something the client does afterwards, and the run is one thing to watch, cancel, or point a schedule at. Each iteration is read back by node id (get_result_and_success_by_id_from_flow), so an answer is fetched without walking the loop's status.

The loop is parallel with a bounded parallelism and skip_failures: a dataset is a burst of calls to one provider, and one case failing is one cell of the run rather than the end of it.

The cases are the loop's static iterator, so they live in the flow's value, which is stored once. Passing them as an argument would put a copy of the whole dataset in every iteration's arguments.

Whichever state of the agent is chosen — what is deployed, the edits in progress, or a past version — its configuration is fixed once, when the run is opened, and inlined into the step every case runs. A linked step would resolve the resource when each case reaches it, so a deploy part-way through a run would be executed by the cases after it while every row still named the version the run started against. One run measures one configuration; the cost is that a run does not exercise the linked branch the production step takes. The edits and a past version could not be run any other way: a reference resolves to what is deployed, which is exactly what neither of them is.

A saved agent's and a past version's configuration are never taken from the request: both are read from the workspace by the path they name, and a subject carrying one is refused. The edits in progress are the one kind the request has to carry — they exist only in the editor — and the run records what it was handed, inlined into its flow and hashed, so it is reproducible and attributable to "this version plus these edits"; what the server cannot assert about them is that they derive from that version.

Each iteration is an agent step, then a payload step, then one step per scorer. The payload step exists because the flow cannot see what it needs to: the agent's own result carries the answer and every message, but each tool call's arguments, result, status, duration and schema belong to the job that ran it. The step reads them back through GET /ai_evals/run_payload, whose one argument is the iteration's own job id, and hands the scorers exactly what a scorer receives anywhere else. A scorer therefore measures the agent's latency and not its own: the payload reports the agent step's duration, never the iteration's.

The edits are the transforms as authored, expressions included. One that reads results.<step>.x or a flow_input the case does not supply resolves to nothing here, the same way it would in any run of that step outside its flow.

A linked agent is not fully self-contained: a host flow can override its tools' inputs through the step's tool_inputs. A run does not reproduce that wiring — the agent runs with its own authored defaults — so an agent whose behaviour depends on one flow's overrides is measured here without them.

A case carries no conversation: one question and the answer it should produce, so a run starts from the agent's own memory configuration and nothing is replayed into it.

Where results live

Results are jobs. A run's logs, trajectory, tool-call child jobs, permissions and retention are already v2_job / v2_job_completed and the flow status's agent_actions, and none of that is stored a second time.

What the table itself is made of is the exception: each cell's answer, its outcome and every scorer's verdict are copied into the run's own rows the first time they can be read. Jobs have their own retention, and a recorded run is meant to still read as the run it was long after the jobs that produced it are gone.

The pane shows the answer and nothing else of a job; the trajectory is the run page's, and job_id is the way there. For a recorded row the answer is read off the row rather than out of the job, because job_id is the whole iteration — the agent and then the scorers that measured it — and its result is the last scorer's verdict, not the answer.

What makes a job findable again is stamped on it at push:

• runnable_path is the agent's own path, so the existing script_path_start job filter answers "every run of this agent" with no new state.
• _eval in the flow's args records {subject: {kind, path, version}, dataset, experiment_id}, and every iteration inherits it, so a job opened cold from the runs page explains itself. Which case an iteration ran is in its own iter.value, which is also how a cell finds its job again. Extra flow inputs are inert — the agent step reads only user_message/user_attachments.

Versioning

subject.version is the agent's version number: how many times it has been saved. It is counted per resource rather than read off resource_version.id, which is one identity sequence for the whole table — an agent saved nine times reads v4 … v24 under it, and the gaps count writes in workspaces the reader cannot see. The id stays how a version is addressed, by the history routes and by restore; the number is what a version is called, and what runs are named and compared by.

The number is stored on the row rather than counted when read, because both ways of deleting versions take the oldest: the monitor's trim past MAX_RESOURCE_VERSIONS, and clearing a history down to its current value. Counting the survivors would renumber under either, so a run recorded against v3 would later name a different version.

For an agent run the version names the configuration the run read when it opened, which is the one every case executes. Pinning an older version is a subject kind of its own — an agent_version run says which version to read, where an agent run reads whatever is deployed at the moment it starts.

A version captures the resource, not its transitive closure. Two byte-identical versions can behave differently because a $var:/$res: they reference changed underneath them, so a recorded version is necessary for attribution but not sufficient.

Experiments

Running a dataset produces an experiment: every case executed against one subject, with a row per case. The experiment records the exact case set it ran, by value — a dataset keeps changing, and a result set that cannot say which inputs produced it is not reproducible.

What a run is called

An experiment is Run N: run_number is allocated per (dataset, agent path) when the run is opened, once, and never reused. Stored rather than counted at read time, so a run keeps the name it was given as history is pruned around it. There is no user-given label.

Numbering is per agent and not per subject kind, so runs of what is deployed and runs of the edits on top of it share one sequence: "Run 7" means one thing, and which of the two ran it is what the run says beside its number.

A run is permanent

Every run is written once and then only ever read: there is no writable experiment, no partial rerun, and no cell that can be edited after the fact. A run in which some cells came from one version and some from another would not be worth comparing, and running the dataset is the only way a run appears.

• One experiment holds one subject, and one agent keeps one history. The experiment list is filtered to the agent the pane was opened on, across both kinds, so a dataset shared by two agents never shows one agent's runs when the other is opened.
• A version is per cell (eval_experiment_case.subject_version), not per experiment. The subject is resolved once when the run is opened and every cell is stamped from it, so the column is uniform today; it is per cell so that a run which one day executes cell by cell can say so rather than averaging two versions silently.
• Edits are dated by their hash, not by a version, because editing moves nothing a version could record. Each cell carries subject_draft_hash, the hash of the configuration it ran (canonicalised: key order is not meaningful and serde_json preserves insertion order).
• An agent's unsaved edits are their own subject. Their runs are keyed under agent_draft, so a number produced by edits is never quietly read as the deployed agent's. The run dialog offers them only when it was opened from the editing card, preselected there; from anywhere else the agent is what is deployed.

The table asks what version the agent is on when it opens and whenever the tab regains focus — a small subject_state read — rather than polling for it; the results endpoint reports the same version but collects the run as it goes, so it is polled only while a run is in flight, one pass at a time. An agent saved in another tab while this one stays focused is noticed on the next focus or the next run.

A run of an older version is history and says so (Run 14 · v23 beside an agent on v24); nothing flags it, since that would flag every past run the moment anything is deployed. A run whose edits were later deployed is a run of that version, and the results endpoint recognises and restamps it (see "What a run says it ran"). The hash each run carries is recorded, not shown.

In the flow editor, editing a linked agent forks the configuration into the step and clears the link; the step is the only copy of the edits until Save changes, Cancel or Discard (docs/reusable-ai-agents.md). Evals open from the agent card in both of its states: from the editing card they run the edits as the step holds them when Run is pressed; from the linked card they run the deployed agent, the same reading a linked step makes at run time. The agent's own resource draft — the one the resource editor writes — is never read by evals. The card also names the version a run is recorded against (v24, and v24 beside an unsaved-changes badge for edits on top of it), read from the resource's newest history entry since the resource itself does not carry its version.

Scoring

Scoring is not a second act with a button of its own: Run produces an answer and then scores it. Each iteration of the run's flow scores its own answer as a step, and the numbers are harvested into rows when results are read.

A run's cells are therefore measured by the scorers as they stood when it ran, and never again. A scorer edited or added afterwards has no cell in the runs that predate it: rescoring a run in place would make a permanent run editable. The one thing read through the present is the pass line — pass_if is applied when a score is read, so moving it re-reads every run with no model call.

The columns themselves are the dataset's current scorers, so the table stays comparable across the runs it lists rather than growing a column per run. Removing a scorer therefore takes its column off the runs already recorded as well: the rows it produced are not deleted, but nothing renders them, and adding the scorer back mints a new column that fills from the next run on. The removal asks first, and says that.

Two things are deliberately absent. Rescoring stored answers under edited scorers would need a run of its own that reuses a parent run's answers and is attributed to the version that produced them, so it never reads as the agent having answered again. A result cache keyed on (agent configuration, case, scorer definition) would assert the agent is deterministic, which it is not, so it has to be an explicit choice with its own answer to what a run means when half of it was computed last week.

A score is a number, and optionally a line through it

Every scorer returns a number between 0 and 1 — both templates say so, and the mean and the pass rate read it as a fraction; a scorer returning anything outside that range has its result recorded as an error rather than counted, and a pass_if threshold is held to the same range. Pass or fail is not a second kind of score: a column carries an optional pass_if, and a case scoring at or above it counts as a pass. A boolean scorer is one that returns 0 or 1 with the line at 0.5.

A column with a threshold reports a pass rate beside its mean and marks each cell; a column without one is a plain number and is not dressed up as a verdict.

The line is deliberately outside the score's definition hash: where it sits is an interpretation of a score rather than part of producing it, so moving it re-reads every run already recorded with nothing re-run. It is set when the column is added and changed later under Scorer settings in the dataset drawer, which is also where the column is named.

The name is this dataset's own name for the scorer, seeded from the summary given when it was added. It is a copy, not a link: the script or judge agent keeps whatever it is called, so renaming a column here does not rename anything a second dataset shows. Reading it live from the runnable would cost a fetch per column and leave a column blank for anyone who cannot read what it points at.

What a scorer receives

An agent is judged on its behaviour, so the final answer is the smaller half of the evidence. Every scorer — a judge prompt or a script — is handed the same EvalRun, built from the job the run already stored:

field	from
input, expected	the case as the experiment recorded it
output	the agent step's own result
tool_calls	every message carrying an agent_action, in order, with the arguments, result, error and duration of the job that call ran
tools	the tools that were called, with the schema of the script version that ran
metrics	steps, duration_ms, and the provider's usage when it reported any

Tool results are truncated at 4 KiB with truncated: true, so a large one cannot swamp a judge's context, and a check that reads a truncated result can say so rather than failing on the missing tail. A tool whose schema could not be resolved carries null, and a scorer validating arguments must treat that as unchecked rather than as a failure. There is no cost field: Windmill keeps no provider price table — the script template takes a rate as an argument instead.

The kinds

Two, and both are runnables:

kind	is	receives
agent	an ai_agent resource used as a judge	the run, rendered as a message
script	a workspace script	run, with input, output and expected also spelled out

Keeping every scorer a runnable is what makes columns comparable: each has a path, a version, and code you can open. There is no third kind stored as configuration on the dataset — a judge's model and grading prompt live on the agent resource, so editing a judge is editing that agent, and the column is not something you edit at all. Editing a column is editing the runnable it points at, so the dataset drawer opens it in place: a script in the script editor, a judge in the resource editor.

Adding a scorer chooses the kind before the form opens: a judge is created next to the dataset from the model you pick and a grading prompt that starts at the default; a script is created from the template and opened in the editor. Both are named by a summary of what they score, which becomes the column header and, prefixed with the dataset, the path.

A reason is worth returning: it is what the cell shows on hover, together with the per-assertion checks, so a number that looks wrong can be read rather than re-derived from the trajectory.

A scorer may return a bare number, a boolean, or {score, reason, checks}; a judge's answer arrives under output, sometimes as a string holding one of those, and often as a markdown code fence around it, which is still read. comment is read as reason, so a scorer written for another platform keeps its rationale. Anything with no number in it is left empty rather than guessed at, and means skip the empty ones — a missing score counted as zero would read as a regression.

{score: null} is the one exception, and it means the scorer read the case and had nothing to measure on it: a column asking whether sources were cited has no verdict on a case with nothing to cite. The cell shows n/a and is left out of the column's mean and pass rate, which is not the same as the scorer failing — that is an error, and the column reports it as one. Written out rather than merely absent, since a scorer that returns nothing at all is a scorer that is broken.

What a run says it ran

A run is v15 when it ran the deployed agent and v15 + edits when it ran that version with undeployed changes on top.

An agent_draft run records the version it is an edit of, because "the draft" is not attributable without saying which deployed state it is a draft of. It also stops being a draft by itself: the agent is hashed as deployed, in the same shape a draft is hashed in, so a run whose configuration was later saved is recognised as the version it became. Edit, run, deploy, and the run you made reads as v16 rather than staying an edit of v15 forever.

That recognition is written, not derived. When the hashes match, the run's subject is rewritten to agent at that version, once, keeping the hash it is founded on. Deriving it on every read would make the answer expire: it would only ever mean "this ran what is deployed right now", so the next deployment would send a run that already read v16 back to v15 + edits. The write goes to the unrestricted pool alongside the scores harvested in the same read, and nothing in it comes from the caller — the hash is the proof, and a run of a configuration that was never deployed simply stays an edit.

It follows that the resolution needs someone to look: a run is stamped by the first results read after its configuration is deployed. A run whose configuration was deployed and then replaced without anyone opening the table keeps saying + edits, which is the honest answer when the only evidence is a hash that matches nothing deployed.

Reusing a scorer

The add form lists the scorers this workspace already uses, most recently edited dataset first, read out of the datasets' own scorers rather than stored anywhere new. It is filtered twice, both times by what the caller can read: the datasets are read through user_db, so a scorer only appears if the dataset carrying it does; then the runnables themselves are checked the same way, so a script or agent the caller cannot open is never suggested.

A scorer is a column

A scorer is stored on the dataset as {id, name?, pass_if?, kind, path}, with the id assigned once and never reused: on a write, an incoming id is kept only when it names a column the dataset already holds, and anything else is minted, so a column that was removed cannot come back under its old id and inherit the scores recorded against it. That id is what makes a column the same column across experiments when the scorer is renamed or its definition edited, and a delta is only ever computed between two scores carrying the same id. Two scorers pointing at the same script are two columns.

A score is keyed (experiment_id, ordinal, scorer_id), not baked into the experiment, so a frozen experiment can gain a score without becoming mutable in any way that matters: what is frozen is which runs are in it.

Each score also records the definition that produced it — the kind, the path, and the script hash or resource version that actually ran, so a path alone cannot hide an edit. When two scores of one column carry different definitions the delta is still shown, marked: hiding the number would force model calls just to see anything, and showing it unmarked would let a change of judge read as a change of agent.

The surface

Opening evals selects the dataset this agent was last worked in, remembered per agent in localStorage and only restored while it still exists and is still readable; no run is opened for you. The picker lists this agent's own datasets first and everyone else's below, sorted rather than filtered, since running one dataset against a second agent is a comparison the picker exists for.

A dataset is named the way a script is: a summary of what the cases are for, from which the path follows, prefixed with the agent so it sorts with the agent's own. With no summary the fallback is <agent>_dataset1, taking the next free number. One path segment rather than a folder under the agent, because a Windmill path is <kind>/<owner>/<name> and the picker that edits it cannot express a deeper one.

The dataset is edited in a drawer over the table: the summary and the path, the scorers, then the cases in a grid. Every way of managing a scorer is in that drawer — adding, renaming, moving its pass line, opening the runnable behind it, removing it; the column header over a run's table reports and does not edit, since a run is permanent. Creating a dataset is the same drawer with no cases yet, reached from the dataset named on a row of the runs list and from the run dialog. Renaming moves the dataset, and its cases and its runs follow through the foreign keys. The drawer edits a working copy and writes it in one request when Save is pressed — the rename, the summary and the cases together — so a rename the server refuses leaves the cases as they were, and a half-finished edit is never what the next run executes. A row's panel in the results table is read-only and shows the case as the run executed it, not as the dataset holds it now; deleting a case is in the drawer, and asks first.

A case is its message, what it expects, and nothing else; the message is what identifies it. expected is what a scorer compares an answer against: plain text, or JSON when the answer has structure.

The runs list is one row per run of this agent, newest first, whichever dataset it was of: the run's number and what executed it (v24, v24 + edits, or a pinned v18), how many cases, one badge per scorer, the dataset, and when. Each badge is the headline that column reports — a pass rate where the column has a line, the mean where it does not — read through the thresholds as they are now. A column that never scored a run reads —; a run still going spins. The badges are named and resolved server-side: a list spanning datasets cannot hold every dataset's scorers to look a column's name up, so the name and the kind ride along with the number, and the thresholds are joined in per (run, column) — one grouped query over eval_score rather than a read of each run's cells. A run whose scores are still in its flow is read out of it by the list itself, capped per call and skipped for runs already collected, so the steady state is one query.

Run asks two questions: which state of the agent (v24 (latest deployed) as it is saved when you press Run; a past version as it was then; v24 + edits (current) running the step's edits as they are when you press Run, offered and preselected only from the editing card), and which dataset, with an edit button on the row and a way to start a new one without leaving. A pinned version reproduces the configuration, not the world around it: $var: and $res: references inside it still resolve at run time. The run that was just started opens straight away.

The results table's rows are the dataset's cases, in dataset order, each carrying its result in the selected experiment when it has one, so a dataset that has never been run is not an empty table. A case the experiment ran but the dataset no longer holds keeps its row at the end: the run happened, and deleting the case does not unmake it. Each column's mean sits under its header, with its delta beside it when a baseline is selected.

Picking a baseline adds a per-scorer delta to every cell and to each column's mean, and counts the cells that regressed. Every delta names its scorer; there is no single number for a dataset, since averaging a judge with an exact match would invent one. Rows are joined by case id, so a case added after the baseline ran has no delta rather than counting as a change, and a column the baseline was never scored with reports that rather than a difference that does not exist.

Storage

Datasets, cases and experiments are rows:

table	holds
eval_dataset	one dataset, addressed by a workspace path, and the scorers that are its columns
eval_case	one case: its inputs and the answer it was expected to produce
eval_experiment	one run over a dataset, against one subject; written once, then only read
eval_experiment_case	the case set it executed, the job each case became, and the version or draft hash each ran against
eval_score	one scorer's verdict on one run, with the definition that produced it

An experiment records its cases by value instead of pointing at eval_case, because a dataset keeps changing and a result set that cannot say which inputs produced it is not reproducible. For the same reason case_id is a plain column rather than a foreign key: deleting a case must not rewrite the history of the runs that used it.

Deleting a dataset takes its cases, its experiments, their recorded case sets and every score with it through the foreign keys. The jobs those experiments produced are left alone — they are jobs, with their own retention.

A case is text: a message and an expected answer. Attachments are S3 references rather than inline bytes, so nothing in a case is meant to be large, and three caps keep it that way — 256 KiB per case, 16 MiB and 1 000 cases per dataset — all refused at the API rather than truncated. A run scores every case by every scorer, so a dataset also holds at most 20 scorers, refused the same way.

Permissions

A dataset is permissioned like any other path-addressed object: row-level security on eval_dataset decides who may see it (readers of its folder, u/<self>, a group, or an extra_perms grant) and who may change it. Operators cannot write at all. Recording an experiment counts as a write, since it persists into the dataset.

Cases are the contents of a dataset rather than objects in their own right. eval_case carries a read policy derived from its dataset (see_parent_dataset) and write policies that check the dataset is writable — eval_dataset_writable, one function holding the same disjunction the dataset's own write policies use, so a read-only grant can list a dataset's cases but not edit them. A dataset and its cases therefore move in one user_db transaction, governed by the same policies, and a rename is checked against the destination path the same way. The experiment tables are the exception: their rows are written both by a launch (which holds dataset write) and by the harvest (which holds only read of the run it copies onto its rows), so they carry read policies only and are written on the unrestricted pool after the API has checked the right access.

Why an experiment is recorded before it is launched

Launching picks the run job's id up front, writes the experiment, its case set and a pending score per cell in one transaction, and only then queues the flow. Queueing first and recording afterwards leaves a window in which a flow is running that no experiment accounts for, that nothing will collect and that a retry would silently duplicate. In this order, a launch that dies before the push leaves an experiment naming a job that never started — a run that did not run — and a push that fails deletes it, because one failed push is the whole run.

The dataset's foreign key guards a delete that races the assembly: the transaction fails, and at that point nothing has been queued. It does not cover a delete that lands after this transaction commits and before the flow is queued, which cascades the experiment away while the run still starts.

How a cell finds its job, and its score

The flow engine mints the iteration job ids, so a case is recorded before it has one. Three things fill the gap, each copied out of the flow the first time it can be read:

• Which iteration ran which case. The case is what the loop iterates over, so it is in the iteration's own arguments by construction: args -> 'iter' -> 'value' ->> 'case_id' matches the cell, whatever order the iterations finish in.
• What the agent answered. The agent step's result and outcome, copied onto the cell as soon as that step is done — which is well before the iteration around it, since the scorers are still reading it.
• What the scorers returned. Each scorer step's result is read out of the iteration's flow status into the pending row that was written for it at launch.

All three are written once, when they first become readable, and every later read is of the rows. A job that was retained away before anything read it leaves the cell saying so, rather than looking like a case still being answered.

The flow itself cannot write them: it runs on workers that know nothing about these tables. So two things call the collector. A run's flow ends with a step that calls POST /ai_evals/experiments/collect on itself, which is what records a run nobody watched finish. Reading a run collects it too, which covers the run whose flow never reached that step: one cancelled part-way, or started while nothing served the nativets tag.

That step is bookkeeping, so it is continue_on_error: a run whose every case answered and scored does not become a failed job because the call did not land.