Commit Graph

3 Commits

Author SHA1 Message Date
centdix 3f5f211a22 add final context size metric to ai_evals harness (#9660)
Record finalContextTokens per attempt: the input-token total of the last
model request (input + cache-creation + cache-read), i.e. how full the
context window ended up. Complements the cumulative tokenUsage.prompt,
which conflates context size with loop-iteration count.

Captured generically in the shared frontend runEval via the chat loop's
lastIterationUsage, so it covers all frontend modes (global/flow/script/
app), plus CLI mode via the last assistant turn's usage. Aggregated as
average and max over passed attempts and printed in the run summary.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 23:21:28 +02:00
centdix 434113b5fd tests: add cli eval behavior checks (#8899)
* feat: add cli eval behavior checks

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: harden cli eval command parsing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-04-21 16:17:40 +00:00
centdix cdcc56461b feat: add black-box ai eval benchmarks (#8618) 2026-04-13 14:05:46 +02:00