7 Commits

Author	SHA1	Message	Date
centdix	3f5f211a22	add final context size metric to ai_evals harness (#9660) ... Record finalContextTokens per attempt: the input-token total of the last model request (input + cache-creation + cache-read), i.e. how full the context window ended up. Complements the cumulative tokenUsage.prompt, which conflates context size with loop-iteration count. Captured generically in the shared frontend runEval via the chat loop's lastIterationUsage, so it covers all frontend modes (global/flow/script/ app), plus CLI mode via the last assistant turn's usage. Aggregated as average and max over passed attempts and printed in the run summary. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 23:21:28 +02:00
centdix	a305a74e73	refactor: require ai evals proxy backend (#9119)	2026-05-12 13:44:00 +00:00
centdix	b883f9a9d2	feat: add ai chat schedule and trigger tools (#8961) ... * feat: add ai chat schedule and trigger tools * refactor: use zod for ai chat workspace tools * refactor: let ai provide runnable target fields * refactor: generate ai chat workspace tool schemas * fix: add object type to composed tool schemas * fix: avoid top-level trigger schema unions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: block undeployed workspace ai tools Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: inject ai workspace tool target Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add ai evals for workspace tools Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: make workspace tool eval prompts realistic Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: surface workspace tool errors * fix: show workspace tool success details * fix: describe workspace tool path format * fix: clarify workspace path examples * fix: tighten workspace tool validation * fix: align workspace tool prompts * chore: mark generated chat schemas * chore: mark generated cli skills --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-04-29 14:00:01 +00:00
centdix	fddd8e288f	fix: add proxy eval coverage for gemini schemas (#8897) ... * feat: add proxy transport for ai evals Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: strip propertyNames for gemini schemas Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: require explicit eval transport Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-04-21 16:03:44 +00:00
centdix	51b09ace45	feat: add empty inline script warnings to flow chat (#8853) ... * fix: seed empty inline flow scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: cap frontend eval chat turns * fix: roll back failed inline script seeding * refactor: simplify inline flow script warnings * refactor: share flow module traversal * refactor: make flow chat code edits explicit * fix: resolve ai tool review actions * refactor: remove dead flow rawscript helper --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-04-17 12:10:26 +00:00
centdix	f1e84cb088	chore: add backend preview validation to ai evals (#8827) ... * feat: add backend preview validation to ai evals Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: refresh shared preview workspace assets Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: harden shared backend preview validation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-04-15 15:11:25 +00:00
centdix	cdcc56461b	feat: add black-box ai eval benchmarks (#8618)	2026-04-13 14:05:46 +02:00