windmill/docs/app-mode-ai-chat-review.md

App Mode AI Chat Review

This note only tracks the highest-value next steps for making app-mode AI chat safer and more efficient.

Recommended Next Steps

1. Add confirmation for dangerous app tools.

   Require explicit user confirmation before file writes, file deletes, backend runnable writes, backend runnable deletes, and datatable SQL execution. Show a useful diff or exact SQL before applying the action.

2. Enforce datatable SQL safety in code.

   Do not rely on prompt instructions for SQL safety. Classify statements before execution, block DDL unless table creation is allowed, and require confirmation for DDL, DML, and row-returning reads that would expose data back to the model.

3. Keep default context demand-driven.

   Prefer selected context and targeted reads before broad discovery. Keep file listings metadata-only, avoid sending full datatable schemas by default, and keep SDK/reference material out of the base prompt unless it is requested or needed for the task.

4. Improve app context lifecycle.

   Treat @ context as per-message by default, with an explicit pinning affordance for context that should persist. Lazy-load file and runnable contents, and add a visible approximate context-size indicator so users can spot prompt bloat.

5. Refresh datatable context after mutations.

   Refresh table metadata after data-panel changes and after AI-created tables so follow-up tool calls and user-visible context do not use stale schema data.

6. Persist table creation policy explicitly.

   Store whether AI table creation is enabled as an explicit app setting instead of inferring it from the presence of datatable configuration.

7. Add focused eval coverage for these behaviors.

   Cover confirmation requirements, datatable SQL policy enforcement, selected context minimization, and stale-schema refresh behavior with targeted app-mode evals or lower-level tests where practical.