Commit Graph
4 Commits
Author SHA1 Message Date
hugocasa ede4103b00 feat: share AI guidance with the CLI skills and lint flow groups (#11397)
* feat: lint AI agent tool names and flow groups in wmill lint

* refactor: assemble chat and CLI AI guidance from one topic table

* feat: share flow groups, reuse and pipeline guidance with the CLI skills

* feat: share raw app, data table and secret guidance between chat and CLI

* docs: document the shared AI guidance source for contributors

* fix: keep wmill lint running on flows with malformed collections

* fix: reject skill descriptions that are not plain YAML text

* fix: tighten fence typos, script base scope and app prompt order

* fix: skip tool name checks on agent steps linked to a saved agent

* fix: catch any misspelled prompt fence and soften the tool name claim

* fix: align cli eval harness with the files and steps wmill init adds

* fix: drop cli eval checks that expect unrequested deploy commands

* fix: list ansible as mainless and c# Main in script base guidance
2026-09-29 14:47:42 +02:00
centdixandClaude Opus 4.8 3f5f211a22 add final context size metric to ai_evals harness (#9660)
Record finalContextTokens per attempt: the input-token total of the last
model request (input + cache-creation + cache-read), i.e. how full the
context window ended up. Complements the cumulative tokenUsage.prompt,
which conflates context size with loop-iteration count.

Captured generically in the shared frontend runEval via the chat loop's
lastIterationUsage, so it covers all frontend modes (global/flow/script/
app), plus CLI mode via the last assistant turn's usage. Aggregated as
average and max over passed attempts and printed in the run summary.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 23:21:28 +02:00
centdixandClaude Opus 4.5 434113b5fd tests: add cli eval behavior checks (#8899)
* feat: add cli eval behavior checks

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: harden cli eval command parsing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-04-21 16:17:40 +00:00
centdix cdcc56461b feat: add black-box ai eval benchmarks (#8618) 2026-04-13 14:05:46 +02:00