mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-06 16:02:23 +00:00
bebd8bef8ba86f0cc016db3827ed1d8ee8c577fc
20
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
574775d50c |
fix: teach the AI the raw-app job bindings, the SDK reference and the draft/deployed split (#10754)
* feat: teach the AI the raw-app job bindings and the draft/deployed split Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: scope the raw-app deploy advice to the referenced item, and stop kind-conversion from stranding fields The draft/deployed guidance added in the previous commit was read as "deploy the app too": the agent asked for both the flow and the app and routed a one-item dependency through the review-and-deploy page. Only the referenced flow or script has to exist deployed — the preview runs the app's draft — so the prompts, the `write_app_runnable` warning and the testing rule now say to offer that one deploy and leave the app a draft. `buildPersistedRunnable` spread the existing runnable when rewriting it, so converting a path runnable to inline left `runType`/`path` behind (and the reverse left `inlineScript`). `isRunnableByName` matches the inline branch first, so an app "wired to a flow" silently ran stale inline code. `test_run_app_runnable` now fills ctx-bound inputs with `$ctx:<prop>` the way RawAppBackgroundRunner does, so a ctx argument no longer arrives missing. The SDK-reference rationale claimed WM_TOKEN may be unset, that a missing base URL falls back to localhost, and that a job token is scoped enough to 403 a hand-rolled REST call. None of the three is true, and it shipped to every write-script prompt; the text now only says the client configures itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address review round on the raw-app AI instructions The eval case could pass on the exact answer it exists to reject. Every `requiredMentionsAnyOf` alternative but one was flow-agnostic, so "the app must be deployed" satisfied "must be deployed". All alternatives now name the flow, and a unit test pins that the app-only phrasing fails. `instanceLine` asserted "self-hosted Community Edition" outside the browser, where `isCloudHosted()` reads false and the license store is unset — so every global eval was told that regardless of what it pointed at. It is now emitted only under BROWSER. `assistantExpect.forbiddenMentions` defaulted a missing `assistantText` to "", which passes every entry forever on a mode whose runner does not report it. It now fails with that as the reason. `buildPersistedRunnable` carried `schema` across a retarget, so a path runnable pointed at a new flow kept the previous item's schema and `genWmillTs` typed `backend.<key>(args)` from the wrong inputs. It survives only while kind and path both match. The SDK header claimed "a function that is not listed below does not exist". `windmill-client` also exports the generated services, and the Python client exposes `Windmill.get`/`.post`, so an endpoint without a helper had no legal move. Each language now names its own escape hatch. `getAppInstructions` said the attached reference carries the TypeScript SDK even when `language: "python3"` had swapped in the Python one — on the very sentence telling the model to make that call. The kind-conversion comment claimed a hybrid runnable "silently runs stale inline code". It does not: `isRunnableByName`, `isRunnableByPath`, `convertPersistedToBackendRunnable` and `rawAppPolicy.processRunnable` all dispatch on `type` alone. The leftovers contradict the runnable's kind rather than override it, which is what the comment now says. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-2 review nits on the raw-app AI instructions `flow is deployed` was satisfied both by "once the flow is deployed, the button works" and by a hallucinated "done — the flow is deployed", which eval mode makes impossible and the drafts-only judge cannot see. Every alternative now states an outstanding obligation, and two more real phrasings ("will need to be deployed") are accepted so a correct answer is not failed on wording. Condenses the three comment blocks that ran past the four-line limit in AGENTS.md, and drops two claims inside them that no longer hold: the `testRunAppRunnable` doc said it runs a runnable the way the app's own frontend does (it is the editor preview, which a deployed app's stored policy does not match), and `undeployedRunnableTargets` described its argument as the write tool's raw input when the call site passes the persisted runnable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: report the real cause when a test run fails, and label the app-runnable card Driving `test_run_app_runnable` in a live session surfaced two defects the API-level check could not see. `executeTestRun` built its failure message from `error.message`, which the generated client leaves as the bare status text while the server's message sits in `body`. A path runnable aimed at an undeployed flow reported "Not Found" instead of "Not found: flow not found at name u/admin/current_time" — dropping the one diagnostic the run exists to produce. `formatToolError`, in the same file and written for exactly this, now does it. This also applies to test_run_script and test_run_flow, which had the same loss. The completion card read "Flow test completed successfully" for an app runnable, because `contextName` doubles as the jobs-tray kind and a path runnable pointing at a flow really does queue a flow job. A `completionName` override now names what ran without changing the kind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: pin the deploy expectation against wrong answers, not just correct ones `deploying the flow` was satisfied by "done deploying the flow" — a deploy the agent only claims to have made, which eval mode makes impossible and the drafts-only judge cannot see. Replaced with the prospective forms, and dropped the same reading from the workflow variant. Three review rounds each found this same class of hole in the phrasing list, so the list is now exercised against the wrong answers themselves rather than eyeballed: naming the app as what needs deploying, claiming the deploy is already done, claiming to have deployed the flow, and saying nothing about deploying all have to fail, while four real correct phrasings have to pass. The test reads the case out of global.yaml, so a future edit to the alternatives is checked by it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: drop the tense-neutral deploy alternatives and cover completed claims A gerund after a preposition carries no tense, so `before`/`after`/`by deploying the flow` all match a deploy the agent only claims to have made ("after deploying the flow, I clicked the button and it returns the greeting") just as the bare gerund did. All three are gone rather than swapped for whichever reads least badly, and the two completed-deploy phrasings are now negative fixtures. The remaining alternatives are imperative or obligational, which a claim of having already deployed cannot satisfy. Condenses the two comments this list carries: the YAML block to four lines, and the test's rationale to the durable constraint about substring matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: encrypt sensitive inputs when test-running an app runnable `test_run_app_runnable` sent `force_viewer_static_fields` but not `force_viewer_sensitive_inputs`, which every other preview path derives from the runnable's `sensitive` user fields. That list is the only thing driving the encryption loop in apps.rs, so testing a runnable with a sensitive input wrote the real value into the job's args in plaintext, readable by anyone with run access to the workspace. Verified against a running EE instance. With the list, `api_key` is stored as `$encrypted:mvqtSRI9…` and the sentinel appears nowhere in the job record; without it, the sentinel is readable in run details. A non-sensitive field is left plaintext either way. The tool claims parity with the editor preview, so it uses that same filter (`type == 'user' && sensitive`) and omits the field entirely when empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
 AlexRV12andClaude Opus 5
|
850b028778 |
feat: advertise the pinned artifact version in get_preview_status (#10691)
* test: let global evals seed the session's preview tabs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: advertise the pinned artifact version in get_preview_status Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: reject ambiguous preview-tab and artifact eval fixtures Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
AlexRV12andClaude Opus 5
|
caa189868c |
feat(ai-sessions): add plan mode (#10057)
* feat(sessions): let an opener name the artifact version to show A tab already remembers the version a reader pinned, and re-pointing it keeps that pin. Plan mode needs the two intents that leaves out: a plan card scrolled up the transcript wants the version it proposed, and a plan going up for approval wants the current text with no pin at all. `ArtifactVersionTarget` is those two alongside the existing one: a number, `'latest'`, or omitted. Omitted still cannot double as `'latest'` — every artifact tool re-opens the document it just wrote, so taking that as a request to move would yank a reader out of the version they chose on every edit the agent makes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(copilot): add the plan-mode gate and tag plan-mode-safe tools Plan mode is a read-only posture, so something has to decide which tools it may still run. `Tool.planModeSafe` is that tag, and processToolCall fails closed on it: untagged means mutating means blocked. Deriving it from `requiresConfirmation` was not an option — unconfirmed mutating tools exist, and a posture that leaks one is not a posture. The gate runs twice per call. Before `validateBeforeConfirmation`, so a validator cannot reach out while planning; and again after the confirmation wait, because plan mode can be entered while a mutating tool's card is already pending, and that approval must not carry it through. Arguments are read one field at a time rather than through a parse of the whole call. `change_note` is optional and cosmetic, and a model that sends it as `null` would otherwise fail the object parse and take the plan down with it — the user being told there was no plan to approve, which is false. Also here, because refusing a call well needs them: a validator may now return the row the user reads and the result the model gets separately, a tool may word its own cancellation, and a tool may start work when its card appears rather than when it is approved. The gate is consulted before any of them. `shouldAutoAcceptToolConfirmations` is asked about the tool by name, because skipping the confirmation wait is itself an answer on the user's behalf and one tool must not be answered for. Deciding that without the name would put the exception out of reach of the only path that needs it. The gate stays inert until a chat supplies `isPlanModeActive`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(copilot): give a session one versioned plan document The plan the user agrees to has to survive `/clear`, so it belongs to the session rather than the conversation, and a session holds exactly one. Its id is the session's, so the primary key is the constraint — there is no second row to mint, no index to maintain and no schema change at all. Every write reads the row it is about to replace inside the transaction that replaces it. Read outside, two tabs both see version N, both stamp N+1, and the later write silently drops the earlier one's text and its snapshot; IndexedDB serialises readwrite transactions over a store, so read and write together cannot interleave. Approval takes the same route but patches only the pointer: an approval computed while another tab was revising must not carry this tab's older content back over the newer text. Approval is `approvedVersion`, a pointer at a version, never a flag. Below the current version means the newest text is a proposal the user has not agreed to; absent means nothing here was ever approved. Only exit_plan_mode can leave the pointer behind, since every write outside plan mode carries it forward — an amendment the user's posture already trusts is still the agreed plan. Declining writes nothing at all: the refused proposal stands as the newest version, with the agreed one still in history. Nor can create_artifact confer approval. It asks for no confirmation, so the model writing a plan document is not the user agreeing to one; a plan written there holds the session's slot as a draft until a decision lands on it. That is also why the approved version is exempt from pruning. A plan approved at v1 and then planned against for twenty more rounds would otherwise lose the very version that stands as agreed, and with it the card that opens it, the banner offering it back, and read_artifact at that version. It is excluded from the pruning candidates rather than added on top, so the budget is unchanged and what survives simply stops being contiguous. The write reports whether the database took it. Most callers still degrade like the reads do, but a plan cannot: returning one the database refused would let the user approve and execute against a document that disappears on reload — a refused plan write raises instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(copilot): add plan mode — the posture and its two tools enter_plan_mode asks to hold work; exit_plan_mode hands over a plan and, on approval, gives the posture back to whatever preceded it. Both carry `planModeSafe`, since a posture with no exit is a trap. Only the transition the current posture allows is offered, so there is no tool for leaving a posture the chat is not in. A planning round runs from entering plan mode to the proposal the user decides on. It remembers only the write it made, because nothing it does is undone — and that write is shared between the card's confirmation hook and the tool's `fn`, so the plan is on screen while the user is deciding whether to approve it rather than after. The round is identified by an epoch bumped on *entering*, not by the conversation. A chat rotation mid-approval must still let that approval hand the posture back; a round the user has since left and re-entered must not, or approving the old plan would drop them out of a read-only posture they just chose. Saving a proposal revises the session's plan document and creates one only when there is none — both halves in a single transaction, so a second tab proposing at the same moment revises the row this one wrote rather than racing it. Persistence failures hold the posture. Approval is reported only once both the proposal and the approval pointer are durable, so a plan the database refused cannot unblock mutating tools. The failure is reported from `fn` and no earlier: the write settles while the card is still waiting to be confirmed, and clearing that card from underneath the wait would take away the only control that resolves it. An auto-accepting posture answers for the user through one predicate, asked by every path that answers: the pending-card sweep, the confirmation itself, and the decision to skip the wait at all. enter_plan_mode never qualifies: YOLO means "stop asking and run it", and a call from a tool set snapshotted before the switch must not answer that with a read-only posture — whether its card is already pending or has yet to be registered. Plan mode lives in its own controller with a narrow view of the chat it runs in: it reads that autonomy state and asks for the two changes it can cause, rather than owning any of it. Plan mode is offered only in a session chat, and a session chat is GLOBAL for its whole life. The gate reads that mode, so `changeMode` refuses to move one out of GLOBAL rather than resting the invariant on a picker being hidden. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(copilot): surface plan mode in the chat and the artifact list Plan mode is the only posture that refuses work, so the composer says so before the user types the request it is about to turn down: the mode pill is tinted whole rather than by its icon, and the empty placeholder carries the constraint in words. Teal, not the house green — green is the transcript's success colour a few rows up, and a mode signal in it would read as "this worked" rather than "this is held". A blocked tool renders as its own lean row naming the tool, not as an error: the call did what plan mode says it should, and "why can't it edit" is answered where it is asked. A plan card names the decision — proposed, approved, or not approved — and never the button, since a Stop and a posture switch resolve it too. Its button opens the version that card proposed, so a card far up the transcript still shows the plan it put forward rather than whatever the document has become since. The artifact list and the preview header both label the plan through one badge helper, so the two cannot disagree about what counts as one: a plan the user never approved keeps the plan icon and takes the neutral badge, leaving the teal to mean exactly one thing. In the viewer, an unapproved revision says so in a bar that cannot be scrolled past, with the version the user did agree to one click away. The autonomy picker became a table with one row per posture, so adding one touches a single place instead of four parallel switch statements. A version of a plan is read against the one the user approved, not against the newest: latest is only where the model happened to stop. So the approved version is never stale — its bar is teal and points forward to the draft rather than warning about it — the version in front of it is the draft, and anything behind it is history that is neither and takes no pill at all. The list opens a plan at the approved version for the same reason, which is what lets its pill say `plan` while an unapproved draft sits at the head. One helper answers all of it, so the list and the preview header cannot drift apart on what counts as the plan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(ai-evals): exercise plan mode end to end A case a unit test cannot stand in for: it starts in plan mode against the real gate and the real exit_plan_mode, and grades whether the model researches and hands over a usable plan instead of guessing at one. The checklist does not grade what the harness does for the model — exit_plan_mode writes the plan document itself, so "saves the plan as an artifact" would pass on any run where the tool is called at all. The eval store seeds artifacts with history and mirrors the store's own approval rules, so a rename cannot promote a proposal the user turned down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ai-evals): import the plan-mode messages from the module that owns them `PLAN_MODE_MESSAGES` moved to `planModeMessages.ts`; `planMode.ts` imports it without re-exporting. Under vitest, which runs the frontend adapters, the stale import resolved to `undefined` rather than failing to link, so `global-planmode1-hands-over-a-plan` threw on the approval message after the posture had already been dropped and the tool withdrawn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(copilot): state plan mode's constraint in neutral text The composer's two-tone placeholder becomes a plain "Read-only" beside the autonomy picker, next to where YOLO puts its own warning, and a blocked call's row drops the mode colour. Teal is left marking what the posture is — the badge, the version bars, the pill — rather than every call it refuses. ContextTextarea goes back to main with the accent: `placeholderAccent` had no other consumer, and the aria-label existed only because the accent blanked the native placeholder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(copilot): hold the plan header's verdict until the snapshot lands Opening a plan at the version its reader approved pins a version behind the head, and until that read resolves `shownVersion` is still the head — so the header wore the draft's badge and its orange "not approved" bar over the very case the pin exists to serve, then flipped. The header now says nothing while `restoringPin`, as the body already does. Judging `pinned` instead would print the approved signal over text that is still the draft, trading a true transient signal for a false one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(copilot): refuse a hand-over once plan mode has ended A response can carry two exit_plan_mode calls, and the tool list they run against is snapshotted before the first one restores the posture. The second then found the tool with plan mode already over: under YOLO every confirmation is answered for the user, so it wrote its own summary and stamped the user's approval on a plan no card had shown them. Refused in `validateBeforeConfirmation` rather than in `fn`, since `onConfirmationRequested` writes the document too. The maintenance path is untouched — a plan still gets revised outside the posture with update_artifact, which is what the tool's own description already tells the model to use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
0ea570570e |
feat(ai-sessions): CRUD markdown artifacts in sessions (#10046)
* feat: add IndexedDB persistence layer for AI-chat artifacts * feat: add reactive store for AI-chat artifacts * feat: add artifact chat tools and wire store lifecycle * feat: add markdown artifact viewer with source toggle * feat: surface session artifacts in the preview panel and chat list * feat: tell the copilot when to use artifacts in the session prompt * test(ai_evals): add artifact case and wire artifact helpers for session context * fix(copilot): keep in-memory artifacts across same-session resyncs Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: unify session composer edits/artifacts/jobs into a status line Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add an artifacts section to the session preview picker Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: share markdown prose presets and restyle the artifact viewer Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: unify session status popovers into one keyboard-navigable shell Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: reset first-block top margin in all markdown prose presets Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: open the preview picker on the artifacts branch for an active artifact Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: keep artifact picker scope independent of branch hydration state Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Guilhem Lemouel <guilhemlemouel@gmail.com> |
||
 centdixandClaude Opus 4.8
|
74a2329d2e |
feat(copilot): improve global-mode path selection + add path-selection evals (#9698)
* test: add global-mode path-selection eval cases with seeded user Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(copilot): guide global-mode path selection with injected folder list Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): tailor global-mode folder guidance for workspace admins Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(copilot): type folders_read; isolate global-eval user from store Addresses PR review: - Add folders_read to the User/whoami openapi schema and UserExt; the global prompt builder and eval harness now read it typed instead of via inline casts (regen the client to pick it up). - prepareGlobalSystemMessage takes an explicit user; the eval harness passes it rather than mutating the process-global userStore, removing the concurrency race (path cases no longer need --verbose). - Rewrite the path-selection case comment as a current invariant. - Add buildFolderGuidance unit tests in core.test.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.8
|
4296a6ae1f |
feat(ai-chat): cap read_app_file + search_app grep tool to bound context in large raw apps (#9653)
* docs: add global AI chat context-optimization plan for raw apps Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai-evals): add global raw-app debugging cases on a large fixture Adds a ~20-file analytics_dashboard raw-app fixture (incl. a 5k-line data module and a planted wrong-totals bug), two global cases (read-heavy debug + small-edit baseline), app-seed support in the mock backend, directory-fixture loading, and a decorateHelpers seam so read-dedupe is measurable. Records tokenUsage for before/ after comparison of the read-tool optimization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-chat): cap and dedupe read_app_file to bound context in large apps read_app_file now defaults to a head slice (1500 lines / 50k chars) with offset/ limit to page further, and skips resending a file whose earlier read is still in context (per-conversation ledger keyed off the originating tool-call id, so it self-heals after compaction). Bounds the file-content portion of global-chat context when working in large raw apps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai-evals): add read-heavy raw-app debug case (large data module) global-test31 induces the model to inspect the 5k-line seedData module, exercising the read_app_file cap/offset path. Baseline ~262k tokens vs ~200k with the cap+dedupe change (-24%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: record A+B benchmark results and fixed-overhead finding Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): clearer read_app_file past-EOF message + unit tests for cap/dedupe Addresses local-review nits: out-of-range offset now reports 'offset N is past the end of the file' instead of a backwards 'lines 11-10' label; adds unit coverage for the slicing (line cap, offset/limit window, char budget, past-EOF) and re-read dedupe (hit + miss-when-not-retained). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-chat): char-level paging + per-range dedupe for read_app_file Adds char_offset/char_limit so minified/long-line files can be paged within a line window, keys the re-read ledger by range (so reading different ranges no longer collides), and dedupes on the full-file hash (a cached range stub is invalidated when any byte of the file changes, not just the returned range). Tests updated for the char-slice behavior plus single-line capping, char paging, and out-of-window change detection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai-chat): add read_app_file context micro-benchmark + re-read eval case Adds a deterministic micro-benchmark (no LLM) that drives read_app_file through a realistic big-project read pattern (large file, re-read, minified bundle, paging) and asserts the cap+dedupe cut returned context >50% vs the old whole-file behavior — isolating the feature's effect from model nondeterminism and guarding against silent weakening. Adds global-test32, a cross-file consistency investigation that revisits overlapping files so re-read dedupe is exercised in a real run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai-evals): clarify test32 measures the read cap, not dedupe Verified: sonnet and haiku both read each file once per conversation and retain it, so test32 never triggers read_app_file re-read dedupe. Dedupe is measured deterministically by the micro-benchmark instead. Comment corrected to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-chat): drop read_app_file re-read dedupe, ship the cap only Benchmarking showed the per-conversation re-read dedupe never fires in practice: across sonnet/opus/gpt-5.5/haiku, every model reads each file once per conversation and keeps it in context (0 within-conversation re-reads). It was a correct but unused guard, so this removes the ledger, full-file hash, retention predicate, the AIChatManager wiring, and the eval decorateHelpers seam — keeping the read cap + offset/limit/char paging (A), which is the lever that actually bounds context. The micro-benchmark is now cap-only; test32 is kept as a multi-file read-load case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-chat): add search_app grep tool for global raw-app chat (experimental) Client-side grep over a raw app's frontend files and inline runnables (literal, case-insensitive, optional file_glob/context_lines/max_matches, head-capped). Completes the list -> search -> ranged-read triad. Includes the eval A/B gate (WMILL_AI_EVAL_DISABLE_SEARCH_APP), unit tests + micro-benchmark, and a find-all-usages eval case (global-test33). Experimental: A/B benchmarking shows it is not an unconditional win — it helps on find-all-usages but adds agentic iterations on navigable apps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai-evals): accept search_app as a valid file-inspection tool in raw-app cases Add requiredToolsAnyOf alternatives-group to ToolValidationSpec and switch global-test29..32 to it so a model that locates files via search_app instead of read_app_file no longer false-fails the tool assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: remove stale ai-chat context-optimization planning doc Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-chat): drop read_app_file char paging for a hard char cap The char_offset/char_limit params guarded minified files (a single line over the char budget) but were effectively unused in benchmarks. Remove them and the in-window char paging; keep the hard 50k-char budget and, when a read hits it, tell the model to narrow the line limit (or treat the file as unreadable if a single line exceeds the budget). Proper long-line handling is left as a TODO. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-chat): bake search_app context to 1 line, clarify query is literal Drop the context_lines param (models varied it to little effect) for a fixed SEARCH_APP_CONTEXT_LINES=1, and cap on matching lines instead of pushed rows so max_matches stays accurate with context always on. Sharpen the query description to state it is a literal (non-regex) substring and to suggest the call form (e.g. formatCurrency() to hit call sites and skip formatCurrencyPrecise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-chat): widen baked search_app context to 2 lines Models that set the old context_lines param leaned to 2; match the lean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): count every file with a match in search_app header Move fileHadMatch ahead of the render cap so files whose matches fall past max_matches are still counted (with a regression test). Also swap the raw NUL globstar sentinel for a printable escape (the NUL bytes made core.ts read as binary to grep) and reword two comments to describe current constraints instead of drafting history. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): drop redundant input echoes from app tool results read_app_file and search_app no longer prefix results with the tool name or echo back the caller's own inputs (file path, query, file_glob) — the model already has them from the call args, and the unbounded query echo could push the search result past its output budget. Keeps the useful signals (line range, match/file counts, truncation) and the actionable advice. Also reword max_matches to 'matching lines' since it caps lines (each expands to context rows). Unit tests updated to the new format. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.8
|
3f5f211a22 |
add final context size metric to ai_evals harness (#9660)
Record finalContextTokens per attempt: the input-token total of the last model request (input + cache-creation + cache-read), i.e. how full the context window ended up. Complements the cumulative tokenUsage.prompt, which conflates context size with loop-iteration count. Captured generically in the shared frontend runEval via the chat loop's lastIterationUsage, so it covers all frontend modes (global/flow/script/ app), plus CLI mode via the last assistant turn's usage. Aggregated as average and max over passed attempts and printed in the run summary. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
centdix
|
9e7eaf3684 | feat: inject active editor into global chat (#9361) | ||
|
centdix
|
fec4008696 |
fix: preserve ai reasoning content (#9208)
* fix: preserve ai reasoning content * fix: avoid text-only reasoning replay * feat: add deepseek ai eval models |
||
|
centdix
|
7a7d246a6e |
test: add global ai eval mode (#9129)
* feat: add global ai eval mode * fix: improve global eval validation feedback |
||
|
centdix
|
a305a74e73 | refactor: require ai evals proxy backend (#9119) | ||
|
centdixandClaude Opus 4.5
|
b883f9a9d2 |
feat: add ai chat schedule and trigger tools (#8961)
* feat: add ai chat schedule and trigger tools * refactor: use zod for ai chat workspace tools * refactor: let ai provide runnable target fields * refactor: generate ai chat workspace tool schemas * fix: add object type to composed tool schemas * fix: avoid top-level trigger schema unions Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: block undeployed workspace ai tools Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: inject ai workspace tool target Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add ai evals for workspace tools Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: make workspace tool eval prompts realistic Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: surface workspace tool errors * fix: show workspace tool success details * fix: describe workspace tool path format * fix: clarify workspace path examples * fix: tighten workspace tool validation * fix: align workspace tool prompts * chore: mark generated chat schemas * chore: mark generated cli skills --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
483fb1fb9a |
perf: reduce app ai chat token usage (#8928)
* test: add app chat token usage evals Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * perf: make app file listing metadata only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * perf: reduce app datatable prompt context Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add app datatable persistence eval Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: fix file manager rename app eval Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: remove selected app context eval cases Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: address app eval review feedback Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
434113b5fd |
tests: add cli eval behavior checks (#8899)
* feat: add cli eval behavior checks Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: harden cli eval command parsing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
fddd8e288f |
fix: add proxy eval coverage for gemini schemas (#8897)
* feat: add proxy transport for ai evals Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: strip propertyNames for gemini schemas Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: require explicit eval transport Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
46b2915a9d |
feat: improve app evals and localized app edits (#8863)
* chore: record app benchmark baseline Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: strengthen app benchmark persistence checks Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: seed inventory tracker benchmark case Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add deterministic app diagnostics Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add app chat patch_file tool Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add app session id micro-edit case Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: narrow app patch file content Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: stop gating app evals on lint Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
51b09ace45 |
feat: add empty inline script warnings to flow chat (#8853)
* fix: seed empty inline flow scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: cap frontend eval chat turns * fix: roll back failed inline script seeding * refactor: simplify inline flow script warnings * refactor: share flow module traversal * refactor: make flow chat code edits explicit * fix: resolve ai tool review actions * refactor: remove dead flow rawscript helper --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
b39671d933 |
feat: add compact json patch tool to flow chat (#8840)
* fix: use compact json for flow patches Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: improve flow eval harness Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: record flow benchmark history Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: preserve schema in set flow json Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * style: clean set flow json schema guard Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: clean flow patch review followups Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdixandClaude Opus 4.5
|
f1e84cb088 |
chore: add backend preview validation to ai evals (#8827)
* feat: add backend preview validation to ai evals Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: refresh shared preview workspace assets Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: harden shared backend preview validation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
centdix
|
cdcc56461b | feat: add black-box ai eval benchmarks (#8618) |