Commit Graph
9 Commits
Author SHA1 Message Date
8f7e0c7284 feat: add gpt-6.1-sol to the chat, agent steps and weekly ai evals (#11584)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2026-10-08 08:52:59 +02:00
hugocasaandClaude Opus 5.5 e50e1f5b31 run the cli ai evals on codex and deepseek (#11591)
* feat: run the cli ai evals on codex and deepseek

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: pick cli eval models from the harness and install codex

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: stop counting wmill in a file name as a cli eval mutation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: harden the codex and deepseek cli eval runtimes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep codex tool order and keep other keys from deepseek

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:27:40 +00:00
Ruben Fiszel f1d970bd1a ci: move CI AI models to opus 5.5, gpt-6.1 sol and deepseek v4.1 (#11461)
* ci: move CI AI models to opus 5.5, gpt-6 and deepseek v4.1 flash

* test: point the deepseek eval alias at v4.1 flash

* ci: run the codex review on gpt-6.1-sol with codex cli 0.159.3
2026-10-01 11:11:57 +02:00
hugocasaandClaude Opus 5.5 e117871bec weekly ai evals on current models, and claude 5.5/gpt-6 support (#11409)
* feat: run ai evals weekly on current models and post results to a dashboard

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: add current flagship models, a reasoning flag and claude 5.5 defaults

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: never send a reasoning disable claude 5.5 or gpt-6-astra reject, and treat gpt-6 as a reasoning model

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: address review on gpt-6 support, chat completions tools and model metadata

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: leave tiered gpt-6 unpriced and drop the off sentinel on gpt-5 and o-series

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: point the ai_evals readme at the model registry instead of copying it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor: encode the reasoning rules as per-family maps with a shared parity fixture

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep the chat completions tools rule open-ended past gpt-5.6 and scope the parity fixture

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 13:10:24 +02:00
centdix f300a716a9 test: add global chat resource variable schedule evals (#9379) 2026-05-30 10:02:36 +00:00
centdix 3345837574 chore: add gpt-5.5 eval model (#9377) 2026-05-29 15:33:27 +02:00
centdix fec4008696 fix: preserve ai reasoning content (#9208)
* fix: preserve ai reasoning content

* fix: avoid text-only reasoning replay

* feat: add deepseek ai eval models
2026-05-18 10:40:18 +00:00
centdix 7a7d246a6e test: add global ai eval mode (#9129)
* feat: add global ai eval mode

* fix: improve global eval validation feedback
2026-05-13 08:20:30 +00:00
centdix cdcc56461b feat: add black-box ai eval benchmarks (#8618) 2026-04-13 14:05:46 +02:00