mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
centdix
b0ddcf31e4
* ci: add path-gated AI agent integration tests workflow Runs integration_tests/ai_agent_tests against real LLM providers (Anthropic/OpenAI/Google) only when AI-agent backend code or the tests change, since runs make paid LLM calls. Adds a conftest fixture that skips provider-parametrized cases whose API keys are absent, so CI exercises only the providers it has secrets for. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: add path-gated ai_evals global-mode smoke workflow Runs the global AI chat eval (global-test1) across one cheap model per provider (Anthropic/OpenAI/Google/DeepSeek) only when the eval harness or copilot chat code change, since runs make paid LLM calls. Builds Windmill CE from source as the AI proxy; global tools/drafts run in the Vitest bridge. Gates on the deterministic draft pipeline (run succeeded + produced a draft + used write_script), not the variable LLM judge score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: run AI smokes on PR ready-for-review instead of every push Switch the pull_request trigger from `synchronize` (every commit) to `ready_for_review`, with a job guard skipping draft PRs, so the paid LLM runs only fire when a PR is marked ready to merge (plus push-to-main and manual dispatch). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai_evals): lazily load cli mode so non-cli evals skip the cli toolchain The entrypoint eagerly imported modes/cli, which pulls the wmill CLI guidance modules and their JSR deps (@cliffy/*). Global/flow/script/app runs then crashed with "Cannot find module '@cliffy/ansi/colors'" when the cli workspace deps were not installed. Import createCliModeRunner dynamically inside runCliBenchmark instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ai_agent): raise low max_completion_tokens to OpenAI's 16 minimum OpenAI's /v1/responses rejects max_output_tokens < 16 with a 400, failing test_low_max_tokens for openai. 16 still exercises a truncated response. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: run ai_evals workflow on Node 22 for the frontend undici 8.x dep The Vitest bridge loads frontend/node_modules/undici@8.x, which requires Node >=22.19; Node 20 failed with "webidl.util.markAsUncloneable is not a function" when loading vitest.config.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai_evals): run frontend evals autonomously + give global-test1 more turns Frontend evals (flow/script/app/global) ran the production chat prompt, which assumes an interactive human — so cheaper models burned their turn budget asking for confirmation, waiting for approval, or presenting a plan, sometimes hitting maxTurns without producing a draft. Append a shared autonomy note in baseEvalRunner (the path all frontend modes share, mirroring cli mode): act directly on clear requests; only ask on genuinely ambiguous ones (preserving the askUserQuestion cases). Also raise global-test1's maxTurns 8 -> 10 so a model that over-explores still converges. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(ai_evals): watch draft/prompt deps outside copilot/ The global eval runs production frontend code in-process, so the smoke's behavior depends on files outside frontend/src/lib/components/copilot/**: the draft model (userDraft.svelte.ts, userDraftDbSyncer.svelte.ts), script inference (infer.ts), and the chat system prompts ($system_prompts -> system_prompts/auto-generated). Add them to both push and PR path filters so a change there actually triggers the smoke that gates on draft production. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: skip direct provider tests without credentials * feat: add ai evals skip judge flag * fix: simplify ai evals ci gate * fix: simplify ai evals smoke gate * fix: handle ai eval workflow triggers --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AI Agents Integration Tests
Integration tests for Windmill AI agents using the preview_flow endpoint.
Quick Start
-
Create and activate a virtual environment:
cd integration_tests/ai_agent_tests python -m venv .venv source .venv/bin/activate # On Windows: .venv\Scripts\activate -
Install dependencies:
pip install -r requirements.txt -
Create
.envfile with your API keys:OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... GOOGLE_AI_API_KEY=... OPENROUTER_API_KEY=sk-or-... BEDROCK_API_KEY=... BEDROCK_IAM_ACCESS_KEY_ID=... BEDROCK_IAM_SECRET_ACCESS_KEY=... BEDROCK_SESSION_ACCESS_KEY_ID=... BEDROCK_SESSION_SECRET_ACCESS_KEY=... BEDROCK_SESSION_TOKEN=... # optional, only for session-token Bedrock test AWS_ACCESS_KEY_ID=... # optional, for env-fallback Bedrock test AWS_SECRET_ACCESS_KEY=... # optional, for env-fallback Bedrock test AWS_SESSION_TOKEN=... # optional, required for env-fallback if AWS_ACCESS_KEY_ID is ASIA... BEDROCK_REGION=us-east-1 # optional -
Run a single test to verify:
pytest test_basic_completion.py::TestOpenAI::test_openai_completion -v -s
Prerequisites
- A running Windmill instance (default:
http://localhost:8000) - Python 3.10+
- API keys for the providers you want to test
How It Works
Provider Setup
The setup_providers fixture automatically creates Windmill variables and resources from your environment variables:
| Provider | Variable Path | Resource Path | Resource Type |
|---|---|---|---|
| OpenAI | u/admin/openai_api_key |
u/admin/openai |
openai |
| Anthropic | u/admin/anthropic_api_key |
u/admin/anthropic |
anthropic |
| Google AI | u/admin/google_ai_api_key |
u/admin/googleai |
googleai |
| OpenRouter | u/admin/openrouter_api_key |
u/admin/openrouter |
openrouter |
| Bedrock (api key) | u/admin/bedrock_api_key |
u/admin/bedrock |
aws_bedrock |
| Bedrock (IAM) | u/admin/bedrock_iam_access_key_id, u/admin/bedrock_iam_secret_access_key |
u/admin/bedrock_iam |
aws_bedrock |
| Bedrock (IAM + session) | u/admin/bedrock_session_access_key_id, u/admin/bedrock_session_secret_access_key, u/admin/bedrock_session_token |
u/admin/bedrock_iam_session |
aws_bedrock |
| Bedrock (environment fallback) | none (uses worker/api AWS env) | u/admin/bedrock_env |
aws_bedrock |
Tools
Tools use rawscript with inline content instead of creating actual scripts. This means:
- No script creation/deployment needed
- Scripts are embedded directly in the flow definition
- Faster test execution
Running Tests
All commands assume you're in the integration_tests/ai_agent_tests directory with the venv activated.
Run all AI agent tests
pytest . -v -s
Run only basic completion tests
pytest test_basic_completion.py -v -s
Run only tool calling tests
pytest test_tool_calling.py -v -s
Run tests for a specific provider
# Only Anthropic tests
pytest . -v -s -k "anthropic"
# Only OpenAI tests
pytest . -v -s -k "openai"
Run a single test
# OpenAI basic completion
pytest test_basic_completion.py::TestOpenAI::test_openai_completion -v -s
# Anthropic basic completion
pytest test_basic_completion.py::TestAnthropic::test_anthropic_completion -v -s
# Parametrized test for one provider
pytest test_basic_completion.py::TestBasicCompletion::test_simple_prompt[openai] -v -s
Test Structure
conftest.py- Test fixtures and utilitiesAIAgentTestClient- HTTP client for preview_flowcreate_ai_agent_flow()- Creates AI agent flow definitionscreate_rawscript_tool()- Creates inline script toolssetup_providers- Sets up variables and resources
providers.py- Provider configurations (OpenAI, Anthropic, Google AI, Bedrock, OpenRouter)test_basic_completion.py- Basic AI completion teststest_tool_calling.py- Tool calling tests with inline scripts
Troubleshooting
Tests fail with authentication error
Make sure Windmill is running and you can login with the default credentials:
- Email:
admin@windmill.dev - Password:
changeme
Tests skip due to missing API keys
Ensure the environment variables are set correctly:
echo $OPENAI_API_KEY
Resource creation fails
If resources already exist with different values, you may need to delete them manually via the Windmill UI or API before running tests again.