7 Commits

Author	SHA1	Message	Date
centdix	3f5f211a22	add final context size metric to ai_evals harness (#9660) ... Record finalContextTokens per attempt: the input-token total of the last model request (input + cache-creation + cache-read), i.e. how full the context window ended up. Complements the cumulative tokenUsage.prompt, which conflates context size with loop-iteration count. Captured generically in the shared frontend runEval via the chat loop's lastIterationUsage, so it covers all frontend modes (global/flow/script/ app), plus CLI mode via the last assistant turn's usage. Aggregated as average and max over passed attempts and printed in the run summary. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 23:21:28 +02:00
centdix	a305a74e73	refactor: require ai evals proxy backend (#9119)	2026-05-12 13:44:00 +00:00
centdix	483fb1fb9a	perf: reduce app ai chat token usage (#8928) ... * test: add app chat token usage evals Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * perf: make app file listing metadata only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * perf: reduce app datatable prompt context Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add app datatable persistence eval Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: fix file manager rename app eval Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: remove selected app context eval cases Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: address app eval review feedback Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-04-24 19:49:16 +00:00
centdix	fddd8e288f	fix: add proxy eval coverage for gemini schemas (#8897) ... * feat: add proxy transport for ai evals Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: strip propertyNames for gemini schemas Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: require explicit eval transport Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-04-21 16:03:44 +00:00
centdix	46b2915a9d	feat: improve app evals and localized app edits (#8863) ... * chore: record app benchmark baseline Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: strengthen app benchmark persistence checks Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: seed inventory tracker benchmark case Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add deterministic app diagnostics Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add app chat patch_file tool Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add app session id micro-edit case Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: narrow app patch file content Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: stop gating app evals on lint Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-04-20 11:56:35 +00:00
centdix	51b09ace45	feat: add empty inline script warnings to flow chat (#8853) ... * fix: seed empty inline flow scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: cap frontend eval chat turns * fix: roll back failed inline script seeding * refactor: simplify inline flow script warnings * refactor: share flow module traversal * refactor: make flow chat code edits explicit * fix: resolve ai tool review actions * refactor: remove dead flow rawscript helper --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-04-17 12:10:26 +00:00
centdix	cdcc56461b	feat: add black-box ai eval benchmarks (#8618)	2026-04-13 14:05:46 +02:00