13 Commits

Include renames

Author	SHA1	Message	Date
Ruben Fiszelandwindmill-internal-app[bot]	cfcfe298dd	feat(ai-chat): make reusable skills ai_skill resources you select per workspace (#10914) ... * feat(ai-chat): make reusable skills ai_skill resources you select per workspace * chore: pin the ee ref to the skill telemetry counters * fix: address review findings on skill authoring, import and migration * fix: enforce skill selection in read_skill and stop imports clobbering resources * feat: carry format_extension from the hub into synced resource types * fix: let an edit set or clear a resource type's format_extension * fix: regenerate the sqlx cache and close the review round findings * fix: close the round-2 findings on folder ACLs, cached sync and truncation * refactor: make the skills migration non-destructive and use design-system inputs * fix: close the round-4 findings on folder owners, startup sync and truncation * fix: clear obsolete extensions, guard folder owners, and report skipped skills * fix: honor explicit-null extensions and report same-type migration conflicts * fix: scope skill actions to the committed workspace and paginate the listing * fix: keep the drawer scoped to the live workspace and surface truncation * fix: discard a skills refresh for a workspace the chat has left * chore: update ee-repo-ref to 6efe7a73c745c2e1377a34498523c00d89010a3d This commit updates the EE repository reference after PR #764 was merged in windmill-ee-private. Previous ee-repo-ref: 55998c142bc72edd08532748af1974b16035658d New ee-repo-ref: 6efe7a73c745c2e1377a34498523c00d89010a3d Automated by sync-ee-ref workflow. --------- Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-09-01 12:51:27 +00:00
hugocasaandClaude Fable 5	9c557859c5	feat: AI agent evals: datasets, scored runs and comparison (#10633) ... * feat: eval datasets and standalone runs for reusable AI agents Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: agent eval drawer with case editor, runs and capture entry points Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: document AI agent eval datasets and standalone runs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: say how many eval cases the list is not showing Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address review findings on eval datasets - keep an edited case's conversation and tool inputs: serde(flatten) silently drops Box<RawValue> fields, so the update payload is spelled out - remount the case editor per case so one case's turns cannot leak into another - require jobs:read / flow_conversations:read on the capture endpoints, which UserDB does not gate by token scope - take the dataset lock in create and update so a delete cannot be undone by a concurrent metadata write, and delete cases before metadata - load more cases beyond the first page, and stop capping the agent picker - record that the version stamp is taken at enqueue, not at resolution Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-2 review findings on eval datasets - block operators from dataset and case writes - pass the editor's operating workspace through the drawer and the capture request, instead of assuming the navigation workspace - discard superseded case-list responses so switching datasets cannot land the previous dataset's cases - reject a dataset without a case_id (or vice versa) rather than running an inline case under a dangling association - run unsaved edits inline instead of silently running the stored case - surface the API error body on a failed run - fetch dataset metadata concurrently when listing - $bindable() without a default on the optional open prop - correct the permission and enqueue-time-version wording in the docs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: run an untouched saved case by reference again The editor writes back keys the stored case omits, so comparing the raw objects reported every unedited case as edited: the run went inline and lost the dataset/case stamp its history depends on. Compare a normalized form, and pin it with a test. Also scope the history query to the drawer's workspace and drop superseded responses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: show a dataset's cases as a table, and fix round-4 review findings The case list showed one case at a time with no overview. It is now a table with the case, where it was captured from, and its last run — the last-run column is a single jobs query on the path stamp rather than a request per row. Review fixes in the same file: - keep the edit baseline on the selected case rather than looking it up in the loaded page, so a case beyond page 1 is not treated as unedited and run stale - release the loading state when a superseded case load returns early - reload every loaded page after a write instead of collapsing to page 1 - last remaining 'resolved to' wording in the version tooltip Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: run a dataset as an experiment, with scorers as runnables An experiment runs every case of a dataset against one subject and records the exact case set it executed, so a result set stays reproducible while the dataset keeps changing. Each case runs as its own small flow — the agent, then a step per scorer — so a case keeps the run stamp, history query and trajectory view a single run already has, and scorers need no orchestration of their own. Results are read back per step by node id rather than by walking a nested loop's status. A scorer is any runnable taking (input, output, expected): a script, a flow, or a reusable agent used as a judge. A judge is prompted with the case and the answer as one JSON message; a script or flow receives them as named arguments. Scores accept a bare number, a boolean or {score}. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: results table for an experiment, with scorer columns One row per case: status, the agent's answer, and a column per scorer, with the mean per scorer above the table and a link into each case's run for its trajectory. Averages skip cases a scorer produced no number for — counting a missing score as zero would read as a regression. The drawer's left pane becomes Cases / Results, and Results carries the scorer picker and Run dataset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: compare an experiment against a baseline Per-scorer deltas on each row and on the mean, and a filter down to the rows that regressed. Rows join by case id, so a case added after the baseline ran has no delta instead of counting as a change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-5 review findings on experiments - match scorers by label when diffing two experiments; joining by array position subtracted one scorer from another whenever the scorer sets differed - report a row's status from the case job, not the agent step, so a case whose scorer failed no longer reads as a success - delete a dataset's experiments with it: they hold copies of its cases, and a recreated dataset of the same path would have exposed them - select the experiment that Run dataset just started instead of leaving the table on the previous one - expected is scored now, so stop describing it as having no consumer Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-6 review findings on experiments - hold the dataset lock across an experiment launch, so a delete landing between reading the cases and writing the experiment cannot recreate the deleted dataset's inputs - match scorers between experiments on kind and path, not on label: labels default to a path's last segment, so f/a/quality and f/b/quality compared against each other - average mean deltas over the cases both runs scored; comparing each run's own average reported a regression from a case the baseline never ran, with no regressed row to point at - openapi: the row status is the job's, which is also canceled/skipped; runEval takes scorers; the update-case body no longer advertises source, which the handler deliberately ignores - record why the experiment prefix cannot reach a sibling dataset Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-7 review findings on experiments - release the dataset lock for the push loop and retake it for the write, re-checking the dataset still exists: holding it across the whole launch made every capture and case edit on that dataset 409 until the last job queued - assemble experiment results with bounded concurrency; a 100-case, 3-scorer experiment was 400 sequential lookups, each itself several queries - clear the baseline when it becomes the selected experiment, which was comparing a run against itself and reporting zero deltas - take the header mean over the same cases as its delta while comparing, so the two numbers beside each other describe the same set - a canceled or skipped case is no longer the same grey dot as a running one Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-8 review findings on experiments - verify the dataset's identity, not just its existence, before recording an experiment: the path can be deleted and recreated during the push loop, and the experiment holds copies of the old dataset's cases - give the recording lock a longer budget than a case edit, since its jobs are already queued and giving up strands them, and say so when it fails - keep score lookups sequential within a case: nesting two bounded streams multiplied into 32 in-flight queries against a 50-connection pool - clear a baseline that no longer belongs to the loaded experiments, so switching datasets does not leave comparison mode on with nothing to compare - keep a scorer's own mean when the baseline never ran it, instead of blanking a column full of numbers - EvalCaseDraft.expected no longer claims nothing scores it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: do not trust an experiment's job ids, and require write to record one Experiment objects live in workspace object storage, which a script can write directly, and results are read on the unrestricted pool — so a forged experiment naming another flow job returned output the jobs API would have refused. Only jobs this server stamped with that experiment's id are read now. Also from round 9: - recording an experiment requires write on the dataset, not read: it persists into the dataset's namespace and its shared list - clear the results table when the selection changes and surface a failed load, instead of labelling the previous experiment's numbers as the new one's - a storage fault is no longer reported as a deleted dataset - the lock-timeout message at the recording site no longer says to retry, which would run the whole dataset again on top of the jobs already queued - ExperimentRow.status documents canceled and skipped Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: bind the experiment trust check to the requested dataset The previous check matched jobs on the experiment id alone, which the stored object supplies — so copying another dataset's experiment JSON under a readable key carried its jobs' output along with it. A job is now only read if it was stamped for this experiment *and* for the dataset the caller's read access was checked against, and an experiment that names a different dataset is not served from this key at all. Also from round 10: - add the .sqlx entry for that query; without it every SQLX_OFFLINE build failed - serve results over GET: as POST the route-scope middleware classified a read as ai_evals:write, locking read-only tokens out of their own results - clear the selected and baseline experiments synchronously when the dataset changes, so the previous dataset's id is not requested under the new one Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-11 review findings on experiments and scorers - give scorers the whole case input, not just the message: an answer that came from attachments or a replayed conversation could not be judged on it - accept a judge's boolean and structured {score} answers, including stringified ones, and pin every documented scorer shape with a test - record an experiment for the cases that did launch when a later push fails, instead of leaving those jobs running with nothing to attribute them to - do not capture a preview parent's synthetic runnable_path as a host flow; the saved case could not be rerun - clear the case table before loading a dataset and surface a failed load, so a failure cannot leave the previous dataset's cases under the new name - keep the results table through a refresh of the same experiment - exclude flow-step jobs from the per-case last-run lookup - drop case sets from the experiment list, which is only used to pick a run - report a database failure at the recording lock as itself, not as contention Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-12 review findings on capture and run history - load flow_node.flow for flownode parents: an agent inside a deployed branch or loop captured without its agent, host flow or tool bindings - decide host_flow_path by whether the path resolves to a flow, not by job kind: excluding previews wholesale also dropped the flow editor's step test, whose path is real - page the per-case last-run lookup by created_before until the loaded cases are covered; one page of 200 reported older cases as never run - do not record an experiment when nothing launched - only attach the case input to a job when a scorer will read it - keep the case table through a save; only a different dataset clears it - drop the superseded duplicate comment on the score parser Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop refetching run history on every case write Reading the case list before the first await made the whole job-history query a dependency of it, so every save, delete and Load more refetched up to 1000 job rows and blanked the column. Read untracked instead. - an empty Last run cell now distinguishes never-ran from not-found-within the page bound, which the comment already claimed and the cell did not - reloading a dataset no longer replaces a populated table with a skeleton - keep the score-parser comment that describes every shape it handles Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: keep eval datasets in Postgres instead of object storage Datasets, cases and experiments become rows (`eval_dataset`, `eval_case`, `eval_experiment`, `eval_experiment_case`) rather than objects under a `wmill_eval_datasets/` prefix. What a run produced is still the job's: only case inputs and an experiment's case snapshot are stored. This removes the machinery the object store needed: - The advisory lock and the read-modify-write of a per-dataset JSONL. A case is a row, so there is nothing to serialize. - The launch-time identity check on the dataset. The foreign key makes a concurrent delete fail the transaction instead. - The trust guard on an experiment's job ids, which existed because a script can write workspace object storage directly and could forge an experiment naming somebody else's job. An experiment now chooses every job id and records itself before pushing anything, so a launch that dies partway leaves a recorded case whose job is missing rather than a running job nothing accounts for; cases that never reached the queue are removed again. Row-level security on `eval_dataset` is the authority on who may read or write a dataset, so `extra_perms` grants work and the rule is not mirrored in Rust. Cases and experiments carry a read policy derived from their dataset and no write policy: they are written on the unrestricted pool after the dataset row itself has been asked, with `SELECT ... FOR UPDATE`, whether the caller may write it. Cases are capped at 256 KiB each and 10 000 per dataset, refused rather than truncated. Attachments are S3 references, not inline bytes, so a case that approaches either cap is a mistake rather than a use case. Evals no longer need the `parquet` feature or a configured workspace object storage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * style: align the eval drawer with the design system - Scorer chips are `Badge`s rather than a hand-rolled bordered span, and the section header is a `Label` with its tooltip, as are the case editor's fields (which also gets the label colour right). - The results table showed status as a coloured bullet, which says nothing to a colour-blind reader. It now carries the same icons the runs table uses, with the status as its accessible name. - Feedback colours move to the `-500` shades the brand guidelines name. - The conversation JSON error uses `TextInput`'s `error` prop for the border and the caption style for the message, as elsewhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: author an expected answer, tags and attachments on a case Every scorer is handed `(input, output, expected)`, but nothing could produce an `expected` except a conversation capture: the case editor had no field for it and a captured run left it empty. So: - The editor gains Expected, Tags and a read-only list of the attachments a captured case carries. Expected is plain text, or JSON when the answer has structure. - Capturing from an AI agent run keeps what that run answered, which is the only moment a reference answer exists for free. The results table also laid itself out by content, so a long answer pushed the scores — the numbers the table exists for — off the edge of the pane. It is fixed-layout now, with the text columns bounded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: expected is captured from a run and can be authored Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: link a saved agent when inserting an ai agent step "AI Agent" in the step picker was a leaf that always created a blank step, so reusing a saved agent meant inserting a blank one, opening its step input and linking it there. It is a category now, like Flow and AI Sandbox, listing the workspace's `ai_agent` resources next to a blank option, filtered by the picker's own search. A picked agent produces a step that is already linked rather than one linked afterwards: `agent` set, no tools, and only the flow-local `user_message`/`user_attachments` transforms. Seeding the brain keys there would leave transforms a linked step never reads and that `AgentResourceBar` strips on its next link change. Each `on:new` forwarder rebuilds the insert detail field by field instead of spreading it, so a new field is dropped unless the forwarder names it. `agentPath` is typed on both `GraphEventHandlers.insert` and `FlowGraphV2`'s `onInsert` so the next one to forget it fails the check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: restore the link on cancel and simplify the agent bar Cancel on an agent edit forked the step into a standalone copy, which is the opposite of what the word means and needed a paragraph under the card to explain. It discards the edits and re-links the step now, leaving the agent untouched; diverging from an agent is Unlink's job, on the linked card. This flow's `tool_inputs` survive the round trip as overrides, so Cancel no longer folds them into the tools the way Unlink does. Linking a step to a saved agent happens in the step picker at insert time, so the bar's own resource picker is gone and "Save as agent" is the one action left. Its `+` button was a trap besides: it opened the generic resource form, where an agent would have to be written as raw JSON. The card itself was `surface-secondary`, the sections token, so in dark mode it was darker than the pane and read as a sunken well rather than an elevated card. It uses `surface-tertiary` as the brand table prescribes, its tool chips are `Badge`s, and the editing card no longer overflows the pane and clips its own buttons. The remaining tooltip follows the inline `Label` convention rather than sitting in a flex row whose gap stacked on the trigger's own margin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: rework the AI agent evals surface into one table Evals become a single pane: a dataset of cases, one column per scorer, one row per case, with the run being looked at chosen from the toolbar. Runs are permanent. Running the whole dataset opens one; running a single case records nothing at all — it is a job, and looking at what it did is not a claim that it belongs in the history. Its result and its scores sit over the row until they are saved as a run, which carries the cases that were not rerun and the scoring jobs themselves, so the number that is saved is the number that was looked at. A scorer is a runnable: a judge agent or a script, created in one click and edited in place. Scores carry a reason and per-assertion checks, shown on hover with a rescore button. What ran is always named. A run records the agent version, or — for a configuration that is not deployed — a hash of it, so a table can say that its numbers describe an agent that no longer exists: those rows dim and the table offers to rerun. An agent's draft can be run directly instead of the deployed value, and once those edits are deployed the runs that made them are recognised as that version. A step with no agent of its own is evaluable too, and saving it as an agent moves its history onto it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: keep an agent's in-progress edits on the agent Editing a linked agent forks it into the step, which is what makes the edits runnable there — but the agent is what is being edited, so that is where the unsaved state belongs. The edit is mirrored into the agent's own resource draft as it is made. It then survives leaving the flow, shows the agent as drafted wherever it appears, and is what evals run when asked to run the draft rather than what is deployed. Deploying or cancelling clears it; opening Edit without changing anything does not mark the agent as drafted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: shape the evals surface around a saved agent Evals hang off an `ai_agent` resource, so the surface is now only ever about one: the `draft` subject kind, the standalone-step subject and the move that carried a step's history onto a newly saved agent are gone. - A run is permanent and numbered per agent. Running a single case is a trial: it answers in the panel and never touches the table. - "Run scorers only" opens a run of its own that reuses the answers of the run you are looking at, so a scorer added later measures what already ran without calling the agent again. - A draft run whose configuration is later deployed is stamped, once, to the version it became, so its label stops reading `v23 + edits` forever. - A scorer can carry a pass threshold, read off the scores already recorded. - The table is the case, its answer and one number per scorer; datasets are created and edited in a drawer; a run that executed an earlier state of the current draft says so above the table, in one line. - Which agent a step is, whether it is being edited, and which version it is on is a strip above the step's tabs, because it is true of every tab. - Capturing a case from a step test or a conversation is dropped, and with it the `memory` override on a linked step that nothing set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: run past versions of an agent, and number versions per resource The evals home becomes one table of every run of the agent, whichever dataset each is of, with one badge per scorer. A list spanning datasets cannot hold every dataset's scorers to look a name up, so a score carries its name and kind with its number, and thresholds are joined in per run and column. Run now asks what to run: the latest agent, resolved when the run executes as a flow step does, any past version, or the unsaved edits. Pinning is a subject kind of its own, since a linked step resolves the resource live and inlining is the only way to run a version that is no longer current. Scorers move into the edit-dataset drawer. The column header over a run reports and nothing else: a run is permanent, and a control there that changed the columns would edit the past from the one place that must not. Adding one offers four ways rather than two, writing and reusing being different jobs, and both new kinds open with a summary filled in. Versions are numbered per resource. `resource_version.id` is one identity sequence for the whole table, so an agent saved nine times read v4 ... v24, and the gaps counted writes in workspaces the reader cannot see. The id stays how a version is addressed; the new number is what it is called, in the resource history drawer as well as here. It is assigned on write rather than counted on read because trimming past the cap and clearing a history both take the oldest rows, and counting the survivors would renumber a version a run already names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: read the dataset a remembered selection names Reopening the evals modal restored the last dataset from storage as a bare path, without reading the row it names. Every "is this already the one?" test compared against that selection, so all of them short-circuited and the dataset was never loaded: editing it opened a drawer with no summary, no scorers and no cases. The remembered path is now brought into context the same way any other choice is, and the tests compare against the dataset that is loaded rather than the one that is selected, so a selection can no longer stand for a read that did not happen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: give dialogs a trail in their header A dialog deep enough to navigate had nowhere to say where you were: the header held a fixed title, and the way back was a control each body placed for itself, somewhere in a toolbar that moves with everything else the toolbar holds. The header is the one part of the surface that does not move, which is where the trail belongs. `Modal` takes an optional `trail` of levels below its title, rendered as a breadcrumb whose ancestors are the way back. Declarative on purpose: callers of this depth already hold the state that says where they are, so the dialog reads it rather than owning a stack they would have to push and pop in step with it. Escape follows the trail. Leaving a level is what someone deep in a dialog means by it, and closing the whole surface throws away the navigating they did to get there; at the root it closes as before. That only works if a dialog can tell it is the surface being addressed, so `Disposable` now answers `isTopmost()` and the dialog asks before acting: it keeps Escape for itself, so nothing else was arbitrating between it and a drawer opened from inside it, and both were acting on one key press. Evals is the first caller: its runs list is the root, a run is a level in it, and the back button that used to sit above the table is gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: portal dialogs out of wherever they were opened from A dialog rendered in place inherits whatever the calling component happens to sit inside. One `transform`, `filter` or `overflow` anywhere above it makes its `fixed` positioning resolve against that ancestor instead of the viewport, and a surface meant to cover the app is then confined to a box it never asked for: the nav rail paints over it and its own edges are clipped. Drawers have always portalled for this reason. Dialogs only did so when an enclosing pane claimed them, and rendered in place otherwise, so the same screen could show a drawer over everything and a dialog trapped behind the nav. They now portal the same way: to the pane when one claims it, to `body` otherwise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: make the dialog's title the first step of its trail The trail listed levels below the title, so a dialog one level deep read "Evals > All runs > Run 20 · v6": three steps for two places, the first two of them the same place under different names. The title is the root, so it is the root's own segment, and the trail a dialog is given is now the whole path with that segment at its head. Its height stopped moving too. A heading carries a line-height of its own, so a header holding only an h3 stood six pixels shorter than one holding segments as well, and the dialog's whole top edge stepped as you navigated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: sharpen the evals controls around where you are standing Each screen now offers what belongs to it. The list starts runs; a run is a record, so it offers only the one thing that acts on the record itself, which is measuring the answers it already stored. Starting a fresh run from inside one asked which agent and which dataset from the screen least about either, and scoring an existing run was offered from the list, where there is no run to score. Which run and what it is read against are one question asked twice, so they sit together rather than at opposite ends of a row. Choosing what to run is now a toggle over the two states worth naming, the draft and the saved agent, with every earlier version one click further: running an old version is deliberate, and a list made all three look alike. The draft is read when the dialog opens rather than taken from the caller's polled copy, which could be seconds behind an agent edited a moment ago and would leave the option out exactly when it is the reason for opening the dialog. The dataset field carries its path under it and its edit button on hover, as a resource picker does, so the closed field says what the open list said. Edits waiting on an agent are a "draft" here as everywhere else in Windmill, rather than "+ edits". The dialog runs an evaluation rather than "the agent", which is what it was already called everywhere it is recorded. An agent being edited keeps its evals button on a line of its own, clear of the decision to save or discard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: settle the evals controls on the patterns Windmill already has The version choice uses ToggleButtonMore, as the AI provider picker does: the two states worth naming stay in the group, the rest are behind the overflow menu, and the one you pick joins the group rather than appearing in a second control below it. The deployed one says which version it resolves to. A run offers nothing to start. Scoring an existing run again was the last thing left there, and it was one button explaining a distinction that the run and the dataset already make between them. The warning that a run executed an earlier draft is about the run on screen, so it goes when the run does rather than following you back to the list, and it sits against the table instead of inside a frame of its own. A dataset just created stays open for its scorers and cases: those are what a dataset is, they can only be added to one that exists, and closing on create sent you to find it again to add them. Scorer settings are a cog rather than a word, now that the row holds three actions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: close the gap in the version toggle and say what naming a dataset does The overflow trigger is not a pill, so the room it reserves showed as a gap between it and the button before it; it is pulled in by that much. The dataset field gets its clear button, which is also the slot the edit button is positioned against, so the two now sit where a resource picker puts them. Naming a new dataset said nothing about what happens next, and the drawer looked like it was missing the rest of itself. It says so instead: a scorer and a case both belong to a dataset, so there is nothing to attach either to until this one exists, and creating it leaves the drawer open on them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: choose a dataset's scorers while naming it A scorer is a reference to a runnable, not a child of the dataset, so it needs the dataset's name but not its row. The list is collected in the drawer while the dataset is being named and sent with the create, which already accepts one, so a dataset arrives holding the columns that were chosen for it rather than being made empty and then edited to hold them. Cases stay where they were: a case *is* a row of the dataset, so there is nothing for it to be a row of until one exists. The drawer says which of the two is which instead of leaving the screen looking like it is missing the rest of itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: level the version toggle and name the dataset in its own field The overflow trigger stands a row taller than a toggle button, so the group grew to its height and left the sunken background showing under every pill beside it. Every child of the group is the same height now, which is why the AI provider picker never had the band: it sizes them all alike. The dataset field says the summary with the path after it rather than carrying the path on a line below. The list stacks the two, which a one-line field cannot do, so it says both the other way round. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: tidy the evals forms and the run's own controls Picking a scorer that exists chooses between two sources rather than showing both: the ones already measuring something, and everything else in the workspace. The first list says what each is called with its path under it and what it already measures on the right, instead of three columns that were the same path truncated three ways whenever a scorer had no name of its own. A dataset's drawer says what it is for on the page rather than under an icon, and its summary is sized like the field beneath it. The run's own row lines up with the table under it, the warning above that table is spaced off the rule rather than sitting on it, and adding a case is gone from a run: a run is a record of cases that were answered, so curating them from it is editing what it measured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: create a dataset holding the cases written for it Creating a dataset takes the cases to create it with, so one can be assembled in a single act instead of made empty and then filled in. The drawer holds them while the dataset is being named, gives them ids of its own to be edited by, and sends them with the create. Every case is checked before the dataset is written. `eval_case` grants users no write, so the rows cannot be inserted in the transaction that creates the dataset under the caller's own policies; validating first is what keeps "created holding these cases" from becoming "created, holding some of them", and the rows that do follow go in one transaction of their own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: name the button for what it opens, and say what each version is Starting an evaluation asks which state of the agent and which dataset, and both cost a provider bill, so a button that read as spending one on the way past was lying about the click. It opens something, and says so. Running one case from the panel keeps its own name and its play icon, because that one does run on click. The version options say what they are rather than what they are not: what a flow step would or would not run is a fact about somewhere else, and someone choosing what to evaluate is not standing in it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: give the editing card two rows and mark evals as beta At the width of a step panel the card's one row wrapped: the line naming the agent, the line saying what saving does, and the two buttons deciding the edits' fate all fought for it. Deciding gets a row of its own, and evals sits against the line it is about, since evals of an agent being edited run the edits. Evals is named wherever it is offered. It read as a word in one state of the card and as an icon in the other, which is two things to recognise for one door. The dialog carries a beta badge against its own name, before any level below it: every way in lands there, so it is said once and stays put as you navigate. The version toggle spells out which is which. Both are the agent at v2 and the difference between them is the whole choice, so it is worth the width. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: name a new dataset, and lay the scorer's settings out like a step's inputs A new dataset arrives called "Dataset 1", which the path follows as it follows any summary: a dataset with none was one every table could only call by its path, and the two seeds are what the summary rule already produces. Scorer settings put each field's description between its label and its input, where a step's inputs put theirs, and its inputs are the size the rest of the drawer uses. The runnable behind the column is a link to it with its kind's icon, since it is a resource of its own and the one thing about it these fields cannot change. The line explaining that a pass line re-reads recorded scores went: the threshold is a number to set, and how it is applied is not a decision being made here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: curate a dataset in the drawer and save it in one act The drawer holds the cases while they are edited and writes them when it is saved: added, changed and dropped, whichever it is. Typing no longer writes, so a set is never half saved while someone is still deciding what is in it, and Save means the same thing whether the dataset exists yet or not. A case panel offers reading rather than acting. Running one case now and editing one from a run were the last two ways to change a record from the screen showing it, and the machinery behind the first went with it. The answer is rendered as the prose it is, under what it is: the case's result, whichever run is selected above it. The rest is what the run's table was doing to its own edges: a column name is clipped to its column rather than running into the next, the table squares off against an open panel, and that panel closes with the run it belonged to. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: one border above a table, and a link to the run's job The row above the table drew a bottom border and the table draws its own top edge, so every table sat under two lines. The row keeps its spacing and the table keeps its edge. A column header no longer spins while its scores arrive: the cells under it are where the numbers are missing, and they say so themselves. The beta badge is the height of the word beside it rather than of the line it sits on. A run is one flow and therefore one job, so the run says where that job is: what it is doing, what it cost and what it logged are all there rather than reconstructed from the table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: stream scores as each scorer finishes, and show them per case A scorer runs after the agent inside the case's own iteration, so its verdict can be read as soon as its step is done. Waiting for the iteration to end held every column of a case back until the last of them finished, which is why answers arrived one at a time and scores all at once. Reading a job that is still running needs one guard: a module with nothing in it is a step that has not run, not one that produced nothing, and recording the second makes a failure that never goes away. The panel beside the table shows what each column made of the case and why. The reason a judge gave was stored and never shown, which is the half of a score that says anything. It stops repeating the question the header already asks, and a case still running reads as waiting rather than as an answer that says "Running". A run is a number beside a dataset, so the list puts the two together. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: score a case with every scorer at once The scorers of a case read the answer and never each other, so they ran one after another for no reason: measuring a case now takes as long as its slowest column rather than as long as all of them. Each is a branch of its own, kept from failing the others, so a judge that errors costs its own column and no more. An iteration is three steps again — answer, payload, scores — rather than one per scorer, and each branch is named for the column it produces, so the graph of a run says which scorer did what instead of spelling out an id. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: read a judge's score out of the JSON it nearly wrote A judge quoting the agent inside its own reason writes those quotes unescaped, which is invalid JSON and also the most ordinary sentence for it to produce. The whole verdict was being thrown away over it, so a column that had a number reported having none. The number and the reason are now read straight out of such text. Deliberately not a second JSON parser: it finds the two keys and takes what follows, which is what survives a quote in the middle of a sentence. A case still running says so with a spinner rather than with the word "Running" sitting where its answer goes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: ask a judge for a shape instead of trusting it to write one A new judge carries an output schema, so the provider holds it to `{score, reason}` rather than the prompt asking it to. Windmill already delivers a schema whichever way the model takes it, a tool for Claude and Bedrock and the native parameter elsewhere, so there is no list of models to keep here. An agent with no runs offers its first one where the first row would be, rather than from a toolbar above a table that has nothing in it. Starting a run no longer picks a dataset for you. It fell back to whichever came first, which on an agent that has never run means offering another agent's set as though it were the obvious one; and with no dataset at all it says so and offers the one move there is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: report a column that failed throughout, and hold the run dialog The runs overview dropped any column that produced no number, so a judge that failed on every case of a run vanished from the row and read as a column nobody had asked for. The aggregate now reports every column that has cells, with the count of the ones it failed on, and the badge says "failed" where there is nothing to average. A column with no cells at all is still left out: that one was added after the run and has nothing to say about it. Creating a dataset closes the drawer rather than turning it into an edit of what it just made: scorers and cases already ship with the create, so there is nothing left to stay open for. Reached from the run dialog, it gives the screen back with the new dataset selected, and the dialog keeps the version you had already chosen. Also: - the case panel's job link moves to the panel's own header, where its scope is: the job is the whole iteration, not the answer it sat over - one action in the scorer drawer's header, as its neighbours have. The reuse list picks rather than adds, and says which dataset each column already measures - adding a case is the last row of the list it lands in - the pane shows what it has read rather than an empty state it has not earned yet, and its rows say they open - the linked agent card loses a border it had inside another one * fix: keep the linked agent card's outline The card is a thing inside the step's inputs rather than a section of them, and the outline is what says so. Only the rule inside it goes: the detail it separates is already set apart by being detail. * refactor: fit the eval surface to the shipped design * feat: give a nested dialog a back control and the runs list its own moves Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: put a dialog's description under its title Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: fold a dialog's back control into the crumb it returns to Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: edit a dataset's cases as a table rather than a list beside a form Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: edit a dataset's cases in the grid the data tables are edited in Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: edit a grid cell of prose in place, and cap a dataset at one page Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: keep the cell editor's styles beside it, not in the vendored theme Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: keep an empty cell empty and cap the editor's growth Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: name the step that assembles a run for the scorers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: run the payload step natively, and say so when nothing serves that tag Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: report an answer as answered while its scorers are still running Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: let a scorer say a case is not one it measures Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: score the answer, and leave a case with no expected answer unmeasured Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: split the evals backend into modules Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: record what a run produced so it outlives its jobs * fix: read only the agent step's own tool jobs into the payload * fix: pin a run's configuration and give the judge the attachments * feat: write a dataset's cases in one transaction * chore: refresh the sqlx cache for the eval queries * fix: drop results a newer selection has superseded * fix: keep a draft the agent editor never opened on * feat: let a run record what it produced instead of waiting to be read * fix: serialize the replacements of a dataset's cases * fix: stop the poller from superseding a read slower than its interval * chore: refresh the sqlx cache * fix: keep a failed read from settling a cell as a case with no answer * fix: hold the case grid while its save is in flight * fix: keep a failed collect step from failing the run it recorded * chore: refresh the sqlx cache * fix: commit an open cell into the save that reads it * refactor: size the eval buttons with unifiedSize * docs: describe a run as the one flow it is * fix: show a run's recorded rows when part of it cannot be collected * refactor: size the remaining PR-added buttons with unifiedSize * fix: save the dataset name that was submitted, not the one typed after * fix: force an open cell into the save that was pressed for it * fix: refuse to score a run whose evidence could not be read * fix: hold one lock over a dataset's case count and its writes * fix: keep one unreadable run from costing the whole runs list * refactor: drop the banned bindable-default from the eval props * fix: hold the scorer controls while the dataset is written * fix: read only the caller's own draft of an agent * docs: say in the contract that a run pins its configuration * fix: say a scorer did not run rather than blaming a missing answer * feat: resume the agent draft you already had when you press Edit * refactor: build the trail and dataset controls from Button * fix: clear the open-cell flag when the drawer reopens * chore: refresh the sqlx cache * fix: read a run's configuration and its version from one snapshot * fix: refuse a dataset path or summary the column cannot hold * refactor: handle the agent draft the way the resource editor does * fix: run only a configuration the launch actually read * docs: bound dataset path and summary where they are submitted * fix: surface a stalled agent draft instead of claiming it is kept * fix: stop claiming a draft holds edits a failed write never sent * fix: word a missing score only once the run says whether the case answered * fix: let a breadcrumb crumb shrink so its truncation applies * docs: describe where an agent's unsaved edits live and what drops them * fix: keep harvesting scores when the run cannot yet word a missing one * fix: report a refused draft write the card was reading as a save * fix: drop the refused draft write when the server copy is taken instead * refactor: build the scorer and dataset pickers from the design system * fix: say what removing a scorer column actually does * fix: drop a refused draft write wherever the server copy is read * fix: let a picker row be as tall as the two lines it holds * docs: record what removing a scorer column does to recorded runs * fix: send a queued draft write before reopening, and drop only what it refuses * refactor: write the agent draft at commit points instead of mirroring keystrokes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: run an agent's edits from the step instead of keeping them as a draft Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: make the diff badge keyboard operable and refuse an edits run without its edits Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: drop the dataset icon from the scorer picker rows Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: size the evals buttons like the rest of windmill and call a run of edits edits Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: count a brain expression as an edit of the linked agent * fix: cap scorers per dataset and report a launched run as launched * fix: harvest scores in one read, refuse duplicate case ids, allow group paths * fix: mint scorer ids server-side, save a dataset edit in one request, check attachments * fix: write a dataset edit and its cases in one transaction * fix: atomic dataset create/edit, reset eval pane per agent, stable pending scorer ids * refactor: govern eval_case writes by RLS so a dataset edit is one transaction * fix: pin launch snapshot, order case locks, cap dataset size, guard stale load * fix: cap dataset bytes on single-case writes, reset run-dialog flag on load failure * feat: migrate eval datasets on username change, settle unspawned cases, drop unused case endpoints * fix: resolve scorer scripts as the caller and pin their hash; migrate scorer paths on rename * fix: bound a failed tool call's error to the payload truncation cap * fix: pin scorer hash as a hex string, reject missing judges, migrate eval authorship * fix: record an out-of-range scorer result as an error, not a score * fix: resolve judges in one caller-scoped read, pin deployed scripts, bound pass_if * fix: settle unspawned cases only when the run completes, and their score cells too * feat: reassign eval datasets and their path references when offboarding a user * fix: use the regex backreference in offboarding eval path rewrites * fix: register eval datasets in offboarding registries, keep resource-version param name * refactor: name the resource-version path param id, since it is the row id not the version * fix: validate dataset paths canonically, clone eval data on fork, surface eval load and launch failures * docs: note MCP tool results are not yet surfaced to eval scorers * fix: show the eval error state on any load failure, not only an empty dataset list * fix: preserve eval case order across a batched save Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * docs: scope the eval launch delete-safety guarantee to the assembly window Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: only offer deployed scripts as eval scorers, drop unbuilt rescore claim Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: enforce 0-1 scorer threshold in the settings drawer and clear stale eval load errors Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: scope subject version/hash reads to the caller and keep a 0 pass threshold Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: select the saved dataset when creating or renaming from the Run dialog Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: gate eval dataset rename on path ownership, not just write access Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * fix: tolerate a malformed agent config when resolving the deployed label Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LVjewUvXFEjNLLw7kxz41h * refactor: trim eval code and comments, fix shared select and modal paths Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: drop the rename warning when editing an eval dataset path Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: add eval dataset delete, keep summary on partial edits, settle resultless scorer cells Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: cover parseThreshold and subjectLabel Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: hold dataset Save during a scorer write, derive draft_hash only from the carried draft Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-24 11:08:56 +02:00
Ruben FiszelandClaude Opus 5	633d7bcb2e	feat: add trigger_history table with source tracking (#10696) ... * feat: add trigger_history table with source tracking Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: gate trigger history reads on scopes and harden its writers Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: filter trigger history scopes in SQL and match the cleared-handler diff Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: record a trigger restore from the trashbin in its history Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: record bulk http trigger creates and document the recording boundary Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: lock the trigger row when capturing its history preimage Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: only record an auto-disable that actually flipped the schedule Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: state the auto-disable invariant once instead of at four call sites Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: render trigger history changes as a structured field diff Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: make a server-initiated disable atomic with its history row Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: note that the auto-disable savepoint takes no pool connection Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: note the flow fallback is the last chance to disable Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: never leave a trigger enabled because its history row failed Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: retry the disable history row instead of dropping it on first failure Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: use the design-system Button for the change-value expander Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: hold the trigger row lock across its disable history row Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the history-loss alert out of the listener cancellation race Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: read the history workspace through the trigger-workspace seam Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-14 17:57:11 +02:00
Ruben FiszelandClaude Opus 5	c69f08073a	fix: add apps:run to the token scope picker and confine path-scoped app tokens (#10428) ... * fix: expose apps:run in the token scope picker and let apps:write grant it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: confine path-scoped app run/write tokens to the app they name Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: let apps:run read back its own app's S3 files, condense scope comments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: let apps:write mint apps:run and extend run read-back to app S3 display routes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: drop stale embed-token wording from the app S3 helper summary Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 11:38:05 +02:00
3 people	68debab877	feat(triggers): add AMQP (RabbitMQ) trigger via lapin (#10230) ... * feat(triggers): add AMQP (RabbitMQ) trigger using the lapin library Fixes WIN-2214 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(triggers): defer AMQP cross-workspace deploy pending utils-internal publish Revert the amqp_trigger additions to the shared windmill-utils-internal TriggerDeployKind and the frontend cross-workspace deploy adapter: the frontend installs the published npm package, which lacks the new kind until a release is cut. AMQP create/edit/delete/list/sync/capture are unaffected (they use local types); only cross-workspace deploy/merge of AMQP triggers waits on the package bump. Also document the at-most-once ack in the consumer loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): address AMQP review — at-least-once ack, workspace cascade, contracts - ack AMQP deliveries only after successful dispatch; nack+requeue on failure - add ON DELETE CASCADE workspace FK so amqp_trigger rows are cleaned on workspace deletion (and the listener stops) - fix the /amqp_triggers/test OpenAPI body and add amqp_trigger to WorkspaceDiffRow.kind - register AMQP in the generated workspace trigger tool (create_trigger) - drop banned $bindable defaults on optional props in the config section - add build_uri unit tests (encoding, ports, vhost) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): stop AMQP poison-message loop and reconnect on transient drops Chaos testing against a live RabbitMQ broker showed the previous nack(requeue) + immediate re-poll spun a tight redelivery loop (~1000 critical-error reports/sec) on a poison message, and any connection blip permanently disabled the trigger (lapin has no built-in reconnect). - on dispatch failure: nack+requeue then stop consuming; the listener framework re-lists the trigger after its ping goes stale (~15s), backing redelivery off to that cadence instead of a tight loop (verified: rate dropped from ~1000/s to ~1 per ~26s, message preserved) - on connection/stream error: stop and let the framework reconnect instead of disabling; persistent failures are still disabled via get_consumer (verified: a forced connection close now auto-reconnects and resumes) - finish the AI create-trigger action wiring for AMQP: add amqp to CreatedResourceTriggerKind, the action-card registry, and the drawer registry so the result card renders and its "Open" action works Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP frontend registries and defer merge rows - add amqp to capturableTriggerTypes (so AmqpCapture mounts), the Runs jobTriggerKinds filter, and CLOUD_DISABLED_TRIGGER_TYPES - wire AMQP into global AI chat mode: TRIGGER_KINDS, the request union, writeTriggerSchema, triggerServices, and the draft adapter - stop emitting actionable AMQP fork-comparison rows (revert amqp_trigger from TRIGGER_OR_SCHEDULE_TABLES) since cross-workspace deploy is deferred until windmill-utils-internal is published — avoids a deploy that fails with "Unknown kind: amqp_trigger" - use design-system TextInput instead of raw <input> in the config section Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP session/draft registries and constrain prefetch - add amqp to the session-deploy, draft-compare, preview-router, and copilot workspace-item registries so AMQP drafts/deploys/nav/path resolution work - include amqp_count in the MoveDrawer attached-trigger rename warning - replace the raw prefetch <input> with a design-system TextInput bounded to an integer 1-65535 (backend u16) and block save on invalid values Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): make AMQP disconnect/reconnect consistent with the Kafka trigger lapin, like rdkafka, has no transparent reconnect, so the AMQP listener now mirrors the Kafka trigger's explicit reconnect loop instead of relying on the framework re-list (which disabled the trigger once get_consumer failed on a sustained outage): - get_consumer returns cheaply; consume owns a (re)connect loop that retries with a 30s backoff, reports a critical error every 10 failed attempts, and reports a recovered critical error once it reconnects — never disabling the trigger on a connectivity failure - a consumer/stream error breaks out to reconnect rather than disabling - dispatch failure still nacks+requeues (at-least-once) with a short backoff to avoid a tight poison-message loop, keeping the connection alive Verified against a live RabbitMQ broker: killing the broker keeps the trigger enabled and retrying (attempt N), and restarting it auto-reconnects (logs "reconnected after N attempts") and resumes dispatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP capture registries and constrain prefetch contract - add the 'amqp' case to triggerKindToTriggerType so opening the AMQP editor from a capture button no longer throws "Unknown TriggerKind: amqp" - register AmqpIcon in CaptureTable's icon map and add an AMQP entry to the script/flow CaptureButton menu - bound the OpenAPI prefetch_count to an integer 1-65535 (matches the Rust u16) and regenerate clients/prompts - require a non-empty exchange name when the exchange binding is enabled - build_uri: fall back to "/" on a blank vhost and bracket IPv6 hosts (+ tests) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(triggers): wire AMQP into pipeline graph, git-sync, and preprocessor types - asset_graph: discover attached amqp_trigger rows and emit an AMQP TriggerEdge so AMQP triggers render (and can be opened/deleted) on the data-pipeline canvas - frontend pipeline graph: add amqp to NativeTriggerKind, the add-trigger menu, node presentation, event-trigger set, annotation keywords, and the editor/service registrations - git-sync: add the amqp_trigger include pattern (+ test) so an AMQP git-sync deployment stages only its .amqp_trigger.* file, not an unrelated same-path object - preprocessor starters: add the AMQP event to the generated TS/Python/PHP trigger event types (kind/payload/exchange/routing_key/queue_name/redelivered/ delivery_tag) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): finish AMQP pipeline/parser wiring, prefetch validation, source lists - fix a stray edit that corrupted the pre-existing MqttTriggerEditor import ($lib/... path) in PipelineTriggerEditors.svelte - reject prefetch_count = 0 server-side in validate_config (RabbitMQ treats 0 as unlimited) and defensively skip basic_qos(0) in build_consumer (covers the capture path that bypasses CRUD validation) - recognize `// on amqp` in the canonical parser (TriggerSpec::Amqp) and add amqp to the CLI non-autorun/event-trigger sets so a pipeline cascade never runs an AMQP-only node as a manual root without an event - add amqp to the preprocessor intro lists and both pipeline AI instructions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): reject zero AMQP prefetch in all paths and finish guidance lists - extract a shared validate_amqp_options used by both CRUD validate_config and build_consumer, so capture configs (which bypass CRUD validation) also reject prefetch 0 instead of silently connecting with an unlimited buffer (+ unit tests for 0/1/65535/None) - add AMQP to the main script-writing preprocessor-sources prompt and the CLI triggers-skill guidance list Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(triggers): de-duplicate AMQP prefetch comment and fix GET response text - keep the zero-prefetch rationale only on the shared validate_amqp_options doc; drop the redundant call-site comments - correct the getAmqpTrigger OpenAPI 200 description ("deleted" -> "retrieved") Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to aaa6cb89b05b76139252c64f057e53b94d12ac60 This commit updates the EE repository reference after PR #680 was merged in windmill-ee-private. Previous ee-repo-ref: 5da5fd65aca9594b2611837a52e4677b544b0380 New ee-repo-ref: aaa6cb89b05b76139252c64f057e53b94d12ac60 Automated by sync-ee-ref workflow. * chore(migrations): consolidate the four AMQP migrations into one The table and the three enum ADD VALUE statements (trigger_kind, job_trigger_kind, draft_kind) are one atomic feature. ALTER TYPE ... ADD VALUE runs inside the migration transaction on PG >= 14 (Windmill's minimum) since the amqp_trigger table doesn't reference those enum types, so they can share a single migration instead of four. Verified applying cleanly in a single transaction on a fresh DB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-21 15:10:29 +00:00
Ruben Fiszel	fd51d40f12	feat(pipelines): catalog declared measures and dimensions (#10190)	2026-07-21 07:34:44 +02:00
3 people	9d61e4e59e	feat: self-host docs search for chat, mcp, cli; drop inkeep (#9772) ... * feat: self-host docs search for chat, mcp and cli; remove inkeep Embed a vendored docs snapshot (llms.txt/llms-full.txt) in the backend and serve ranking + page rendering from GET /api/docs/{search,page}. The AI chat, the MCP searchDocs/readDocsPage tools, and 'wmill docs' all consume it, so docs search works with no runtime egress and is no longer EE-gated. Removes the inkeep proxy. EE companion deletes inkeep_ee.rs (ee-repo-ref bumped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: name read_docs_page param `url` instead of `path` search_docs returns each hit's `Source` URL, so the read tool now takes a `url` argument to match — the AI/MCP loop reads "search gives a Source URL, read takes that url" rather than copying a `Source:` URL into a `path` slot. A bare `/docs/...` path is still accepted and canonicalized before lookup. Regenerated openapi-deref, the MCP endpoint tools, and the frontend client. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: add scheduled workflow to refresh the vendored docs snapshot The backend embeds docs_snapshot/*.gz at build time, so the in-product docs corpus is otherwise only as fresh as the last manual fetch.sh run. This adds a weekly (and manually dispatchable) job that re-runs fetch.sh, sanity-checks the result against truncation/garbage, and opens a PR via the internal app when the snapshot changed — so a human reviews the docs diff before it rides into the next release build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: make docs tool-result strings caller-neutral The search/page endpoints back three differently-named consumers (the AI chat `read_docs_page` tool, the MCP `readDocsPage` tool, and the `wmill docs` CLI), so the shared rendered text shouldn't name one of them. Refer to "the docs page-reading tool" and its `url` argument instead, and add tests pinning the caller-neutral follow-up guidance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: point ee-repo-ref at inkeep-removal companion rebased on EE main The companion branch now carries only the inkeep_ee.rs deletion on top of EE main (was based on the native-job-retry EE line, which polluted the EE PR diff). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(docs): expose docs:read in token catalog; precompute lowercased corpus Addresses two review nits on the self-hosted docs PR: - docs:read was enforced (ScopeDomain::Docs) but missing from the token scope catalog (token.rs ALL_SCOPES), so it couldn't be selected when creating a standard scoped token in the UI — leaving scope-restricted CLI/MCP docs use effectively ungrantable. Add a read-only "Documentation" group (no write surface) and a test asserting it is exposed. - search ran page.body.to_lowercase() on the whole corpus per query. Lowercase body/title/description once at parse time (into the OnceLock corpus) and scan the precomputed copies instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: update ee-repo-ref to 27a4f41b8e5603d6e444efcfc420bd1c44a07eed This commit updates the EE repository reference after PR #630 was merged in windmill-ee-private. Previous ee-repo-ref: c7ec3a0c2fa38d4cb5e50bf0265eef4710de4860 New ee-repo-ref: 27a4f41b8e5603d6e444efcfc420bd1c44a07eed Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-06-25 16:32:07 +02:00
centdixandClaude Opus 4.8	6f4017d694	feat(ai-chat): workspace AI chat skills (SKILL.md upload + read_skill tool) (#9648) ... * feat(ai-chat): workspace ai_skill table + CRUD API Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-chat): AI Skills workspace settings tab with SKILL.md upload Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-chat): advertise skills in global system prompt + read_skill tool Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-chat): move custom skills into AI settings (paste or folder) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-chat): cap folder import (depth<=3, max 50 skills, confirm dialog) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(ai-chat): give import folder its own labeled subsection Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): resolve svelte-check never-narrowing in skills preview Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address ai skills review issues * fix: validate ai skills and reload workspace list * fix(ai-chat): spec-align skill validation and cap skills per workspace Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): reject duplicate skill uploads, audit skill names Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): sync deref openapi specs with skill validation rules Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-23 00:16:13 +02:00
3 people	d6c642b170	feat: add Azure Event Grid triggers (#8888) ... * feat: add Azure Event Grid triggers (EE) Introduces a new enterprise trigger kind `azure` that supports three modes via a single unified trigger type: - basic_push: Azure Event Grid basic — custom topics, system topics (Storage, Resource Manager, Key Vault, etc.), domains (push only) - namespace_push: Event Grid Namespace topics (CloudEvents over HTTP push) - namespace_pull: Event Grid Namespace topics (HTTP pull with lock-token ack/reject for dead-lettering) Auth uses a Service Principal resource (tenant_id, client_id, client_secret, subscription_id). Subscriptions are created in CloudEvents 1.0 schema so the push webhook handler and the pull listener share one payload parser. Backend - New crate `windmill-trigger-azure` (OSS stubs + EE impl symlinked from windmill-ee-private) - Migration `azure_trigger` table with CHECK constraints enforcing mode/columns coherence - `TriggerKind::Azure`, `JobTriggerKind::Azure`, `DeployedObject::AzureTrigger` variants - Push route `/api/azure/w/{workspace}/*path` handles classic Event Grid SubscriptionValidation handshake and CloudEvents 1.0 abuse-protection OPTIONS handshake - Optional inbound JWT validation (audience check only for v1) - Feature flag `azure_trigger` propagated through windmill-api, windmill-store (resource helper), and added to ee_core Frontend - `triggers/azure/` editor with mode toggle (basic/namespace-push/ namespace-pull) and per-mode config (topic ARM id / namespace + topic name / subscription / filters / push auth / pull options) - Registered in icon map, display names, save functions, badge, wrapper, editor, add-trigger menu OpenAPI - `AzureTrigger`, `AzureTriggerData`, `AzureMode`, `AzureSubscriptionMode`, `AzureDeliveryConfig`, `TestAzureConnection` schemas; `/azure_triggers/*` endpoints; client regenerated Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to eaa7c3a9cb37a9ccc93f10a2535d929365acd2d8 This commit updates the EE repository reference after PR #541 was merged in windmill-ee-private. Previous ee-repo-ref: 9689014e8c12c36c1059fd8fa5758d550b8b8bc9 New ee-repo-ref: eaa7c3a9cb37a9ccc93f10a2535d929365acd2d8 Automated by sync-ee-ref workflow. * feat(azure-trigger): secret-auth push, ARM discovery, capture isolation, CLI + parity Frontend: - Split mode selector into Namespace/Basic + Pull/Push - ARM resource dropdowns (namespaces, Basic topics, namespace topics) populated from the service principal; cascade with stale-selection reset on SP / edition change - Remove stale authenticate toggle + audience input (server-managed push_auth_config has replaced them) - Azure listing page: "Create from template" button; "Also delete Azure subscription" toggle in the delete modal; simplified trigger label falling back to path - AzureCapture.svelte: "Test subscription name" with -wm-capture suffix - CompareWorkspaces.svelte: wire Azure for fork/compare - Drop Trigger-deployed/event-loss warning (capture subscription is isolated with -wm-capture) Backend: - Shared-secret push auth (see EE crate for detail) - JSONB push_auth_config column (renamed from delivery_config), #[serde(skip)] so clients/CLI/exports never see it - Drop redundant enabled column; mode supersedes - Azure capture infra: AzureTriggerConfig + set_azure_trigger_config + azure_payload route + TriggerKind::Azure arm; PT15M queue TTL on capture subscriptions so they bound storage after tab close - Granular ACLs, users offboarding, trash, git-sync deployed-object: all include azure_trigger CLI: - Add azure to TRIGGER_TYPES, pushObj dispatch, getTypeStrFromPath, trigger commands (get/update/create/list/template), sync delete switch + regex; e2e test for `trigger new --kind azure` - system_prompts: SCHEMA_MAPPINGS + schema_names include AzureTrigger; auto-generated/* regenerated Skill: - .claude/skills/adding-a-trigger/ checklist covering every file that needs editing when wiring a new trigger type (learned from this PR) ee-repo-ref bumped to b0e490cbf3724b7b64c6a5b010e3bdf24acd873c. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(azure-trigger): ci — ShareModal Kind + regenerated system_prompts - frontend/src/lib/components/ShareModal.svelte: add 'azure_trigger' to the Kind type so the listing page's "Permissions" action compiles (ts2345 — caught by npm_check on CI, missed by fast-check locally). - system_prompts/auto-generated/: regenerate to drop the stale delivery_config / AzureDeliveryConfig fields from the Azure schema (check-freshness on CI). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(azure-trigger): use workspace constant_time_eq crate Drop hand-rolled constant-time compare in favour of the workspace constant_time_eq crate (same one used by http_trigger_auth). ee-repo-ref bumped to 9659382d47286e7f7f66d01b6f5dd8d4ed34848b. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(azure-trigger): pass placeholder + disabled via inputProps `TextInput`'s `placeholder` and `disabled` go through its `inputProps` prop — CI's `npm run check` caught the stale top-level passing that `npm run check:fast` missed. Align with the DefaultEmailConfigSection pattern. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(azure-trigger): correct LATEST_GIT_SYNC_SCRIPT_PATH version to 28213 The hub deploy of the azure-aware sync-script is version 28213, not 28214. Backend was pinning a non-existent hub script, which broke the git_sync_e2e suite (every deploy's sync step 404'd). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(azure-trigger): add azure_triggers to token scope selector + skill - windmill-api/src/token.rs: `build_trigger_scope_domains` was missing `("azure_triggers", "Azure Event Grid")`, so the CreateToken UI's scope selector didn't surface azure_triggers:read/write. Backend already had `ScopeDomain::AzureTriggers` wired (scopes.rs), this just exposes it. - .claude/skills/adding-a-trigger/SKILL.md: capture both scope-related files under the hardcoded-arrays section so future triggers don't miss the UI surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(adding-a-trigger-skill): clarify token.rs scope effect Not a regression — nothing was working before. Skipping TRIGGER_DOMAINS just means the scope works via API/CLI but has no UI checkbox. * docs(adding-a-trigger-skill): trim token.rs bullet * fix(azure-trigger): regen openapi-deref + swap textarea for TextInput - Run build_openapi.sh to regenerate openapi-deref.{yaml,json} with the 12 azure_triggers paths + schemas. These files are served by the runtime (include_str! in windmill-api/src/lib.rs) to external SDK consumers; without this regen the new endpoints wouldn't be advertised. - Replace the raw <textarea> for event type filters with the design-system TextInput in textarea mode (frontend/CLAUDE.md bans raw HTML elements). Addresses cubic + claude PR review items. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-04-23 16:30:18 +00:00
centdixandClaude Opus 4.5	aea74445a3	fix: add flow conversation token scope (#8903) ... * fix: add flow conversation token scope Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: make flow conversations scope plural Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: update flow chat service import Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-04-22 13:42:22 +00:00
dieriba	6418c4bcc6	feat: nextcloud native triggers (#6797)	2026-01-26 16:49:52 +00:00
3 people	36bbde6239	feat: email triggers (#6548) ... * feat: email triggers * Change down migration to drop email_trigger table * email triggers UI * bug fix * Apply suggestion from @ellipsis-dev[bot] Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * cli and git sync * Revert "cli and git sync" This reverts commit 220fd50d13. --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev>	2025-09-09 21:04:23 +00:00
dieribaandHugoCasa	5f364100f3	feat: granular token scopes (#6093) ... * base * add scopes in the UI * remove legacy scope, unified create token into a components * fix layout, convert old scope to new scope * update scope * update ui, and clear scope * remove desc * almsot there * fix path * fix delete and scope resource path matching * update scope * update scope and error message * nits and fix logic * nits and fix * added multiple resource and jobs scope for script and flow * add check_scope for endpoint * clean front and improve fronetend code * fix resource validation logic and backward compatibility with old scope * fix frontend state and scope checks logic * update scopes * fix height * nits: better_naming * fix route * fix add missing import * fix import and move fn * update repo ref * fix import * fix query and nits * nits * fix ,missing import * fix * revert add admin protection * handle run action correctly * fix * add check scopes to some endpoint * fix and nits * nits * remove unused import * nits * add chevron when domain is exapanded * fix border bottom * nits adds resource path button * nits * fix * nits * nits * nits * fix merge * fix * UI nits * update repo ref * add lazy static * update scopes --------- Co-authored-by: HugoCasa <hugo@casademont.ch>	2025-07-17 15:18:40 +00:00