mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-25 00:01:55 +00:00
Ruben Fiszel
e41440b344
* test(ai-evals): guard implicit trigger/schedule intent in flow chat Investigation of WIN-2228 (does flow AI chat understand it should create a flow AND its associated triggers): the flow-editor chat already exposes create_schedule and create_trigger (10 kinds), both confirmation-gated, and an A/B eval shows the model already recognizes IMPLICIT trigger intent reliably (12/12 across two new cases on the current prompt) without naming a "schedule" or "trigger". Add two ai_evals flow cases that phrase the trigger intent implicitly, to guard that recognition against future prompt/tool regressions. These are not redundant with the existing explicit cases (flow-test15/16): a trial system prompt addition that spelled out a deployment prerequisite regressed the HTTP case from 6/6 to 2/6 (the model deferred instead of creating the trigger), which these cases caught. No prompt change ships: the addition showed no measured benefit over baseline and the fuller version regressed behavior. Fixes WIN-2228 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-chat): support email triggers in flow/script create_trigger The chat's create_trigger tool exposed 10 trigger kinds but not email, even though the backend supports email triggers and the chat's open-resource drawer was already wired for them (CreatedResourceActionDrawers, the 'email' CreatedResourceTriggerKind). So when asked to make a flow run on incoming email, the model had no email kind and substituted an HTTP trigger it mislabeled as email. Add email as a create_trigger kind (generator + regenerated zod schema + triggerConfigs → EmailTriggerService.createEmailTrigger). Email triggering only works once an instance superadmin has stood up an SMTP server and set the `email_domain` global setting, so guard the create path: read `email_domain` (readable by any authed user; returns null when unset) and, when it is not configured, return role-aware setup guidance instead of a failing create — pointing a superadmin to Instance settings and a regular user to ask a superadmin, both with the docs link. When configured, create the trigger and report the resulting inbound email address. userStore and the email-address helper are lazy-imported so the chat tools module does not drag in the heavy $lib/stores graph at load. Guarded by unit tests for both branches (shared.test.ts) and an ai_evals case (flow-test19); the model now calls create_trigger(kind=email) 3/3 on a natural "run when an email is received" prompt. Fixes WIN-2228 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): address codex review on email trigger + eval guards - [P1] Default `workspaced_local_part` on the email trigger request body before it is sent, not only when formatting the success address. The column is BOOLEAN NOT NULL, so a request omitting it (the model may) was rejected by the backend. Assert the defaulted `false` in the happy-path unit test. - [P2] Tighten the implicit-intent eval guards so they validate the requested configuration, not just tool selection + path prefix: flow-test17 now checks the cron time (07:30) and UTC timezone; flow-test18 checks kind=http, POST method, no auth, and the route path. Cases still pass 9/9 (sonnet). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>