* test(ai-evals): guard implicit trigger/schedule intent in flow chat
Investigation of WIN-2228 (does flow AI chat understand it should create a
flow AND its associated triggers): the flow-editor chat already exposes
create_schedule and create_trigger (10 kinds), both confirmation-gated, and
an A/B eval shows the model already recognizes IMPLICIT trigger intent
reliably (12/12 across two new cases on the current prompt) without naming a
"schedule" or "trigger".
Add two ai_evals flow cases that phrase the trigger intent implicitly, to
guard that recognition against future prompt/tool regressions. These are not
redundant with the existing explicit cases (flow-test15/16): a trial system
prompt addition that spelled out a deployment prerequisite regressed the HTTP
case from 6/6 to 2/6 (the model deferred instead of creating the trigger),
which these cases caught. No prompt change ships: the addition showed no
measured benefit over baseline and the fuller version regressed behavior.
Fixes WIN-2228
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): support email triggers in flow/script create_trigger
The chat's create_trigger tool exposed 10 trigger kinds but not email, even
though the backend supports email triggers and the chat's open-resource
drawer was already wired for them (CreatedResourceActionDrawers, the 'email'
CreatedResourceTriggerKind). So when asked to make a flow run on incoming
email, the model had no email kind and substituted an HTTP trigger it
mislabeled as email.
Add email as a create_trigger kind (generator + regenerated zod schema +
triggerConfigs → EmailTriggerService.createEmailTrigger). Email triggering
only works once an instance superadmin has stood up an SMTP server and set
the `email_domain` global setting, so guard the create path: read
`email_domain` (readable by any authed user; returns null when unset) and,
when it is not configured, return role-aware setup guidance instead of a
failing create — pointing a superadmin to Instance settings and a regular
user to ask a superadmin, both with the docs link. When configured, create
the trigger and report the resulting inbound email address.
userStore and the email-address helper are lazy-imported so the chat tools
module does not drag in the heavy $lib/stores graph at load.
Guarded by unit tests for both branches (shared.test.ts) and an ai_evals
case (flow-test19); the model now calls create_trigger(kind=email) 3/3 on a
natural "run when an email is received" prompt.
Fixes WIN-2228
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): address codex review on email trigger + eval guards
- [P1] Default `workspaced_local_part` on the email trigger request body
before it is sent, not only when formatting the success address. The
column is BOOLEAN NOT NULL, so a request omitting it (the model may) was
rejected by the backend. Assert the defaulted `false` in the happy-path
unit test.
- [P2] Tighten the implicit-intent eval guards so they validate the
requested configuration, not just tool selection + path prefix:
flow-test17 now checks the cron time (07:30) and UTC timezone;
flow-test18 checks kind=http, POST method, no auth, and the route path.
Cases still pass 9/9 (sonnet).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai): flow writer builds approval steps as scripts with getResumeUrls, not identity
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(ai-evals): accept rawscript or script for approval step type
The flow-writer prompt allows an approval step to be `type: rawscript`
or `type: script`, but the topLevelStepTypes check pinned an exact
`rawscript` match, so a valid `type: script` approval would fail
deterministically. Let the check accept a list of allowed types.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add global ai chat test tools
* fix: avoid session id in flow test preview
* test: cover global flow preview ids
* test: require script and flow test tools
* fix: harden global flow test fallback
* Revert "fix: harden global flow test fallback"
This reverts commit 97254ef33a.
* fix: fallback from inactive flow test hook
* fix: list nested flow steps in errors
* fix: use compact json for flow patches
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: improve flow eval harness
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: record flow benchmark history
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: preserve schema in set flow json
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* style: clean set flow json schema guard
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: clean flow patch review followups
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: support special flow modules in evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: extract shared flow helper logic
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: make special flow tools openai-compatible
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: improve flow eval prompts and validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: relax flow benchmark overfits
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* test: record updated flow benchmark history
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: address flow review findings
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: source flow chat special module prompt
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: narrow rawscript helper return type
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: dedupe flow chat prompt guidance
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: relax flow test10 validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add backend preview validation to ai evals
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: refresh shared preview workspace assets
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: harden shared backend preview validation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
GuilhemandClaude Fable 5
Ruben FiszelandClaude Opus 4.8
centdix
