mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
centdix
b0ddcf31e4
* ci: add path-gated AI agent integration tests workflow Runs integration_tests/ai_agent_tests against real LLM providers (Anthropic/OpenAI/Google) only when AI-agent backend code or the tests change, since runs make paid LLM calls. Adds a conftest fixture that skips provider-parametrized cases whose API keys are absent, so CI exercises only the providers it has secrets for. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: add path-gated ai_evals global-mode smoke workflow Runs the global AI chat eval (global-test1) across one cheap model per provider (Anthropic/OpenAI/Google/DeepSeek) only when the eval harness or copilot chat code change, since runs make paid LLM calls. Builds Windmill CE from source as the AI proxy; global tools/drafts run in the Vitest bridge. Gates on the deterministic draft pipeline (run succeeded + produced a draft + used write_script), not the variable LLM judge score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: run AI smokes on PR ready-for-review instead of every push Switch the pull_request trigger from `synchronize` (every commit) to `ready_for_review`, with a job guard skipping draft PRs, so the paid LLM runs only fire when a PR is marked ready to merge (plus push-to-main and manual dispatch). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai_evals): lazily load cli mode so non-cli evals skip the cli toolchain The entrypoint eagerly imported modes/cli, which pulls the wmill CLI guidance modules and their JSR deps (@cliffy/*). Global/flow/script/app runs then crashed with "Cannot find module '@cliffy/ansi/colors'" when the cli workspace deps were not installed. Import createCliModeRunner dynamically inside runCliBenchmark instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai_agent): raise low max_completion_tokens to OpenAI's 16 minimum OpenAI's /v1/responses rejects max_output_tokens < 16 with a 400, failing test_low_max_tokens for openai. 16 still exercises a truncated response. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: run ai_evals workflow on Node 22 for the frontend undici 8.x dep The Vitest bridge loads frontend/node_modules/undici@8.x, which requires Node >=22.19; Node 20 failed with "webidl.util.markAsUncloneable is not a function" when loading vitest.config.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai_evals): run frontend evals autonomously + give global-test1 more turns Frontend evals (flow/script/app/global) ran the production chat prompt, which assumes an interactive human — so cheaper models burned their turn budget asking for confirmation, waiting for approval, or presenting a plan, sometimes hitting maxTurns without producing a draft. Append a shared autonomy note in baseEvalRunner (the path all frontend modes share, mirroring cli mode): act directly on clear requests; only ask on genuinely ambiguous ones (preserving the askUserQuestion cases). Also raise global-test1's maxTurns 8 -> 10 so a model that over-explores still converges. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(ai_evals): watch draft/prompt deps outside copilot/ The global eval runs production frontend code in-process, so the smoke's behavior depends on files outside frontend/src/lib/components/copilot/**: the draft model (userDraft.svelte.ts, userDraftDbSyncer.svelte.ts), script inference (infer.ts), and the chat system prompts ($system_prompts -> system_prompts/auto-generated). Add them to both push and PR path filters so a change there actually triggers the smoke that gates on draft production. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: skip direct provider tests without credentials * feat: add ai evals skip judge flag * fix: simplify ai evals ci gate * fix: simplify ai evals smoke gate * fix: handle ai eval workflow triggers --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windmill integration tests
Integration that are run on every push to the main branch (excluding tags)
The concept is the following:
- It pulls the latest published image with the version from (version.txt)[../version.txt] and deploys a Windmill stack using the (docker-compose.yml)[docker-compose.yml]
- It runs the tests using python testing framework (
python -m unittest -v test). All test classes exported in (__init__.py)[./test/__init__py] will be run - Then it upgrades the stack to the latest (unpublished) version of Windmill built for this commit. It expects a docker image with the tag
devto be present. - It re-runs the tests with the following env var:
WMILL_RUNNING_DEVset to1
Some tests behaves differently depending on WMILL_RUNNING_DEV. This way we can test that scripts/flows/schedules... deployed on a previous version of Windmill won't break with the upgrade
Some tests are skipped unless WMILL_RUNNING_DEV == 1, those are just regular integration tests that just needs to be run on the dev version.
Running locally
Running the tests locally is not easy given that we need to have a Docker image running the latest version of the code. However, the tests simply reaches Windmill API on http://localhost:8000, so it can easily be modified to run alongside a simple cargo run --features enterprise.
Note that Windmill Enterprise version is required, you'll need a license key for all the tests to run. It needs to be set to the environment variable: WM_LICENSE_KEY_CI
For example, to run identity_script_test.py, you can do the following:
In one terminal, run your local version of Windmill:
cargo run --features enterprise
Then in another tab, run the test:
python -m unittest -v test.TestIdentityScript
Some tests requires additional setup. This is the case of the SDK tests, which requires Windmill SDK package to be published to either private NPM registry or PyPI server. Custom setup logic can be found in (build.sh)[./build.sh].
TODOs
Add integration tests for:
- Test Python SDK
- Test job failures as well
- Test dedicated workers
- Error handlers and Recovery handlers for schedule
- Concurrency limits