mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-12 00:06:14 +00:00
Merge remote-tracking branch 'origin/main' into change-68b704f7
# Conflicts: # backend/ee-repo-ref.txt # backend/windmill-api-workspaces/src/workspaces.rs
This commit is contained in:
Diego Imbert
@@ -0,0 +1,40 @@
|
||||
---
|
||||
name: ai-chat
|
||||
description: Guidance for improving the Windmill AI chat (copilot), especially global mode — tools, prompts, and context-window discipline. Use when editing chat tools, system prompts, or tool-result shapes under frontend/src/lib/components/copilot/chat, or when changing how the chat manages its context window.
|
||||
---
|
||||
|
||||
## Always benchmark before and after
|
||||
|
||||
No context or behavior change ships without an `ai_evals` A/B on the affected mode.
|
||||
Add or adjust cases for exactly what you changed — see the `ai-evals` skill for
|
||||
authoring and the full run reference.
|
||||
|
||||
Run the affected mode **before** your change and **after**, same model(s), same cases.
|
||||
|
||||
## Measure the window first, and cumulative second
|
||||
|
||||
Optimize **`finalContextTokens`** (window occupancy — what drives overflow and
|
||||
compaction), then cumulative prompt tokens.
|
||||
|
||||
## Context discipline
|
||||
|
||||
The dominant fixed cost is per-iteration overhead: the system prompt **plus every
|
||||
tool schema** is re-sent on every loop iteration. So:
|
||||
|
||||
- **Every tool and every parameter is a permanent tax.** Justify each one and measure
|
||||
it; an extra "locate" round-trip can cost more than the reads it saves. Strip dead
|
||||
params rather than leaving them in the schema.
|
||||
- **Tool results return the minimum.** Never echo content the model already has. The
|
||||
canonical mistake: a write tool that returns the whole edited artifact right after
|
||||
the model authored it — return `{ success, message }` instead. When you touch a
|
||||
*shared* write helper (e.g. `finishAppDraftWrite` in `global/core.ts`), re-check
|
||||
this invariant for **all** the write tools routing through it — the echo has
|
||||
regressed before via a shared refactor.
|
||||
|
||||
## Prompts and tool descriptions are part of the surface
|
||||
|
||||
The system prompt and tool descriptions steer behavior as much as the tools
|
||||
themselves, and are benchmarkable the same way. A description that advertises
|
||||
truncation makes the model self-limit; the path-conventions block changes where
|
||||
drafts land. Treat prompt/description edits as real changes and A/B them — a
|
||||
pure-prompt change is a legitimate, measurable improvement.
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: ai-evals
|
||||
description: Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes.
|
||||
---
|
||||
|
||||
# AI evals — authoring and running benchmark cases
|
||||
|
||||
`ai_evals/` is a black-box benchmark runner for the Windmill AI generation modes:
|
||||
`flow`, `app`, `script`, `cli`, `global`. It always tests the **current** production
|
||||
prompts, tools, and guidance in this checkout. Each attempt runs the real production
|
||||
path, deterministic validation, then LLM judging.
|
||||
|
||||
The goal is to test current production guidance with realistic user requests — **not**
|
||||
to pin one exact implementation shape.
|
||||
|
||||
## Running benchmarks
|
||||
|
||||
```bash
|
||||
cd ai_evals
|
||||
bun install # first time; frontend modes also need `cd frontend && bun install`
|
||||
bun run cli -- models # list model aliases
|
||||
bun run cli -- cases global # list cases for a mode
|
||||
bun run cli -- run global global-test1-script-create --model sonnet
|
||||
```
|
||||
|
||||
Frontend modes (`flow`/`script`/`app`/`global`) route model calls through a Windmill
|
||||
backend's `/api/w/<ws>/ai/proxy`, so you need **any** reachable backend:
|
||||
|
||||
```bash
|
||||
WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1:<port> WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests \
|
||||
bun run cli -- run global <caseIds...> --models sonnet,gpt-5.5,gemini-3.1-pro-preview
|
||||
```
|
||||
|
||||
- **Reuse an existing workspace.** CE builds cap workspaces, so temp-workspace
|
||||
creation 400s ("reached workspace limit"). Always set
|
||||
`WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` (or any existing workspace) to
|
||||
reuse one. The only side effect of a run is upserting an `f/evals/ai/<provider>`
|
||||
resource there.
|
||||
- Provider keys live in `ai_evals/.env` and are auto-loaded by bun. The judge is a
|
||||
separate Anthropic call (default `claude-sonnet-4-6`) regardless of the model under
|
||||
test.
|
||||
|
||||
## Authoring core rules
|
||||
|
||||
1. Write prompts like a real user request.
|
||||
2. Prefer behavior, inputs, constraints, and outcomes over internal implementation.
|
||||
3. Keep deterministic validation narrow and hard.
|
||||
4. Put semantic expectations in `judgeChecklist`.
|
||||
5. Use `expected` fixtures only when exact structure really matters.
|
||||
|
||||
### Prompt writing
|
||||
|
||||
Prompts should sound like something a user would naturally ask. Do not write prompts
|
||||
as if the user knows Windmill internals unless the case explicitly tests a power-user
|
||||
workflow.
|
||||
|
||||
Good:
|
||||
- "Create a flow that routes support requests based on customer tier."
|
||||
- "Add a reset button that sets the counter back to 0."
|
||||
- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
|
||||
|
||||
Bad:
|
||||
- "Use `branchone` with 3 branches and a default branch."
|
||||
- "Create a `rawscript` step with this exact topology."
|
||||
- "This is a benchmark harness."
|
||||
|
||||
### Deterministic validation
|
||||
|
||||
Use deterministic checks only for hard failures: missing required files; unexpected
|
||||
extra files when the prompt says not to create them; syntax errors; unresolved flow
|
||||
refs; missing required special modules or suspend config; obvious corruption.
|
||||
|
||||
Do **not** encode one preferred implementation. Bad hard checks: exact step topology
|
||||
for a creation flow; exact branch structure when the prompt only asked for routing;
|
||||
exact input shape when multiple reasonable shapes are acceptable.
|
||||
|
||||
### Judge checklist
|
||||
|
||||
Every non-trivial case should have a `judgeChecklist` capturing user-visible behavior
|
||||
that must be present, important constraints, and key completion criteria — not
|
||||
low-level implementation details unless truly required.
|
||||
|
||||
Good: "the flow calculates the order total with 8% tax"; "the flow reuses the existing
|
||||
workspace script instead of rewriting the logic". Bad: "uses `branchone`"; "contains a
|
||||
`rawscript` node".
|
||||
|
||||
See `ai_evals/README.md` for the full case format, fields, and fixture details.
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
../../../.agents/skills/ai-chat/SKILL.md
|
||||
Symlink
+1
@@ -0,0 +1 @@
|
||||
../../../.agents/skills/ai-evals/SKILL.md
|
||||
@@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
|
||||
ENV GO_PATH=/usr/local/go/bin/go
|
||||
|
||||
# UV
|
||||
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
|
||||
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv
|
||||
|
||||
ENV TZ=Etc/UTC
|
||||
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
name: AI Agent Integration Tests
|
||||
|
||||
# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against
|
||||
# real LLM providers. Runs only when AI-agent backend code or the tests change,
|
||||
# because each run makes real (paid) LLM calls. To avoid spending on every commit,
|
||||
# the PR side triggers only when a PR is marked ready for review (out of draft) —
|
||||
# not on `synchronize` — plus push to main and manual dispatch.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "integration_tests/ai_agent_tests/**"
|
||||
- "backend/windmill-ai/**"
|
||||
- "backend/windmill-api/src/ai.rs"
|
||||
- "backend/windmill-worker/src/ai_executor.rs"
|
||||
- "backend/windmill-worker/src/ai/**"
|
||||
- "backend/windmill-worker/src/memory_common.rs"
|
||||
- "backend/windmill-common/src/flow_conversations.rs"
|
||||
- ".github/workflows/ai-agent-tests.yml"
|
||||
pull_request:
|
||||
types: [opened, reopened, ready_for_review]
|
||||
paths:
|
||||
- "integration_tests/ai_agent_tests/**"
|
||||
- "backend/windmill-ai/**"
|
||||
- "backend/windmill-api/src/ai.rs"
|
||||
- "backend/windmill-worker/src/ai_executor.rs"
|
||||
- "backend/windmill-worker/src/ai/**"
|
||||
- "backend/windmill-worker/src/memory_common.rs"
|
||||
- "backend/windmill-common/src/flow_conversations.rs"
|
||||
- ".github/workflows/ai-agent-tests.yml"
|
||||
|
||||
concurrency:
|
||||
group: ai-agent-tests-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
ai_agent_e2e:
|
||||
# Skip draft PRs; the `opened`/`reopened` types would otherwise fire while
|
||||
# still a draft. `ready_for_review` always arrives non-draft.
|
||||
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
|
||||
runs-on: ubicloud-standard-16
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16
|
||||
ports:
|
||||
- 5432:5432
|
||||
env:
|
||||
POSTGRES_DB: windmill
|
||||
POSTGRES_PASSWORD: changeme
|
||||
options: >-
|
||||
--health-cmd pg_isready --health-interval 10s --health-timeout 5s
|
||||
--health-retries 5
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
bun-version: 1.3.10
|
||||
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "20"
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
# CE build (no enterprise/license needed for AI agents). `quickjs` powers
|
||||
# flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool
|
||||
# test. Bun tool scripts run via the always-on worker (BUN_PATH).
|
||||
- name: Build Windmill
|
||||
working-directory: ./backend
|
||||
env:
|
||||
SQLX_OFFLINE: true
|
||||
CARGO_BUILD_JOBS: 12
|
||||
RUSTFLAGS: ""
|
||||
run: cargo build --features quickjs,mcp
|
||||
|
||||
- name: Start Windmill
|
||||
working-directory: ./backend
|
||||
env:
|
||||
DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
|
||||
BUN_PATH: bun
|
||||
NODE_BIN_PATH: node
|
||||
RUST_LOG: info
|
||||
run: |
|
||||
mkdir -p ../integration_tests/logs
|
||||
./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 &
|
||||
echo "Waiting for Windmill to be ready..."
|
||||
for i in $(seq 1 60); do
|
||||
if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
|
||||
echo "Windmill is ready"
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; }
|
||||
|
||||
- name: Run AI agent integration tests
|
||||
timeout-minutes: 20
|
||||
working-directory: ./integration_tests/ai_agent_tests
|
||||
env:
|
||||
WINDMILL_URL: http://localhost:8000
|
||||
# Only the providers we have org secrets for. Other providers
|
||||
# (Azure, Bedrock, OpenRouter) are skipped by conftest when their
|
||||
# keys are absent — see skip_provider_without_credentials.
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
|
||||
run: |
|
||||
python -m venv .venv
|
||||
.venv/bin/pip install -r requirements.txt
|
||||
# The S3/vision-attachment tests need MinIO large-file storage and
|
||||
# image-capable provider setup; out of scope for this cost-controlled
|
||||
# smoke. Add MinIO secrets + a storage service to enable them.
|
||||
.venv/bin/python -m pytest -v \
|
||||
--ignore=test_user_attachments.py \
|
||||
--ignore=test_user_images.py \
|
||||
--ignore=test_image_output.py
|
||||
|
||||
- name: Archive Windmill logs
|
||||
uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: ai-agent-tests-windmill-logs
|
||||
path: integration_tests/logs
|
||||
@@ -0,0 +1,166 @@
|
||||
name: AI Evals (global mode)
|
||||
|
||||
# Smoke-tests the production global AI chat proxy/frontend execution path via
|
||||
# the ai_evals harness, one case across one cheap model per provider. Runs only
|
||||
# when the eval harness or the global chat code change, since each run makes real
|
||||
# (paid) LLM calls. The backend is built from source purely as the AI proxy the
|
||||
# harness routes model calls through; the global tools/drafts run in-process in
|
||||
# the Vitest bridge against production frontend code. To avoid spending on every
|
||||
# commit, the PR side triggers only when a PR is marked ready for review (out of
|
||||
# draft) — not on `synchronize` — plus push to main and manual dispatch.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "ai_evals/**"
|
||||
- "backend/windmill-api/src/ai.rs"
|
||||
- "backend/windmill-ai/**"
|
||||
- "frontend/src/lib/components/copilot/**"
|
||||
# The eval harness runs production frontend code in-process; these are the
|
||||
# AI/draft-specific deps outside copilot/ that the global smoke exercises.
|
||||
- "frontend/src/lib/userDraft.svelte.ts"
|
||||
- "frontend/src/lib/userDraftDbSyncer.svelte.ts"
|
||||
- "frontend/src/lib/infer.ts"
|
||||
- ".github/workflows/ai-evals-test.yml"
|
||||
pull_request:
|
||||
types: [opened, reopened, ready_for_review]
|
||||
paths:
|
||||
- "ai_evals/**"
|
||||
- "backend/windmill-api/src/ai.rs"
|
||||
- "backend/windmill-ai/**"
|
||||
- "frontend/src/lib/components/copilot/**"
|
||||
# The eval harness runs production frontend code in-process; these are the
|
||||
# AI/draft-specific deps outside copilot/ that the global smoke exercises.
|
||||
- "frontend/src/lib/userDraft.svelte.ts"
|
||||
- "frontend/src/lib/userDraftDbSyncer.svelte.ts"
|
||||
- "frontend/src/lib/infer.ts"
|
||||
- ".github/workflows/ai-evals-test.yml"
|
||||
|
||||
concurrency:
|
||||
group: ai-evals-test-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
ai_evals_global:
|
||||
# Provider secrets are unavailable to forked and Dependabot PRs.
|
||||
if: >-
|
||||
github.event_name != 'pull_request' ||
|
||||
(
|
||||
github.event.pull_request.draft == false &&
|
||||
github.event.pull_request.head.repo.full_name == github.repository &&
|
||||
github.event.pull_request.user.login != 'dependabot[bot]'
|
||||
)
|
||||
runs-on: ubicloud-standard-16
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16
|
||||
ports:
|
||||
- 5432:5432
|
||||
env:
|
||||
POSTGRES_DB: windmill
|
||||
POSTGRES_PASSWORD: changeme
|
||||
options: >-
|
||||
--health-cmd pg_isready --health-interval 10s --health-timeout 5s
|
||||
--health-retries 5
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
bun-version: 1.3.10
|
||||
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
# Node 22.19+ is required by the frontend's undici 8.x, which the
|
||||
# Vitest bridge loads; Node 20 fails with markAsUncloneable.
|
||||
node-version: "22"
|
||||
|
||||
# CE build used only as the AI proxy (login, workspace, provider resource,
|
||||
# /ai/proxy). No worker execution or MCP needed — global tools/drafts run
|
||||
# in the Vitest bridge. quickjs matches the standard CE feature set.
|
||||
- name: Build Windmill (AI proxy)
|
||||
working-directory: ./backend
|
||||
env:
|
||||
SQLX_OFFLINE: true
|
||||
CARGO_BUILD_JOBS: 12
|
||||
RUSTFLAGS: ""
|
||||
run: cargo build --features quickjs
|
||||
|
||||
- name: Start Windmill
|
||||
working-directory: ./backend
|
||||
env:
|
||||
DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
|
||||
RUST_LOG: info
|
||||
run: |
|
||||
mkdir -p ../ai_evals/logs
|
||||
./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
|
||||
echo "Waiting for Windmill to be ready..."
|
||||
for i in $(seq 1 60); do
|
||||
if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
|
||||
echo "Windmill is ready"
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; }
|
||||
|
||||
- name: Install frontend deps + generate client
|
||||
working-directory: ./frontend
|
||||
run: |
|
||||
npm ci
|
||||
npm run generate-backend-client
|
||||
|
||||
- name: Run global AI evals
|
||||
timeout-minutes: 20
|
||||
working-directory: ./ai_evals
|
||||
env:
|
||||
WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
|
||||
WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
|
||||
# Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
|
||||
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
|
||||
DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
|
||||
run: |
|
||||
bun install
|
||||
mkdir -p results
|
||||
# One cheap model per provider (anthropic/openai/googleai/deepseek).
|
||||
fail=0
|
||||
for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do
|
||||
echo "::group::global-test1-script-create ($m)"
|
||||
if ! bun run cli -- run global global-test1-script-create \
|
||||
--model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then
|
||||
echo "$m: harness/proxy errored"
|
||||
fail=1
|
||||
echo "::endgroup::"
|
||||
continue
|
||||
fi
|
||||
# The CLI exits 0 when the harness records failed attempts, so gate
|
||||
# on execution-only pass counts while ignoring model output quality.
|
||||
if jq -e \
|
||||
'.attemptCount > 0 and .passedAttempts == .attemptCount' \
|
||||
"results/ci-$m.json" > /dev/null; then
|
||||
echo "$m: OK — proxy/frontend execution completed"
|
||||
else
|
||||
echo "$m: FAILED proxy/frontend execution"
|
||||
jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true
|
||||
fail=1
|
||||
fi
|
||||
echo "::endgroup::"
|
||||
done
|
||||
[ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; }
|
||||
|
||||
- name: Archive logs and results
|
||||
uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: ai-evals-global-logs
|
||||
path: |
|
||||
ai_evals/logs
|
||||
ai_evals/results
|
||||
@@ -58,7 +58,9 @@ jobs:
|
||||
|
||||
- uses: denoland/setup-deno@v2
|
||||
with:
|
||||
deno-version: v2.x
|
||||
# Pin to the Deno version shipped in the runtime image (Dockerfile) so CI
|
||||
# tests what production runs, instead of floating on the latest v2.x.
|
||||
deno-version: 2.2.1
|
||||
|
||||
- uses: actions/setup-go@v2
|
||||
with:
|
||||
@@ -74,7 +76,7 @@ jobs:
|
||||
|
||||
- uses: astral-sh/setup-uv@v6.2.1
|
||||
with:
|
||||
version: "0.9.25"
|
||||
version: "0.11.24"
|
||||
|
||||
- uses: shivammathur/setup-php@v2
|
||||
with:
|
||||
|
||||
@@ -50,7 +50,9 @@ jobs:
|
||||
dotnet-version: "9.0.x"
|
||||
- uses: denoland/setup-deno@v2
|
||||
with:
|
||||
deno-version: v2.x
|
||||
# Pin to the Deno version shipped in the runtime image (Dockerfile) so CI
|
||||
# tests what production runs, instead of floating on the latest v2.x.
|
||||
deno-version: 2.2.1
|
||||
- uses: actions/setup-go@v2
|
||||
with:
|
||||
go-version: 1.21.5
|
||||
@@ -62,7 +64,7 @@ jobs:
|
||||
node-version: "20"
|
||||
- uses: astral-sh/setup-uv@v6.2.1
|
||||
with:
|
||||
version: "0.9.25"
|
||||
version: "0.11.24"
|
||||
- uses: shivammathur/setup-php@v2
|
||||
with:
|
||||
php-version: "8.3"
|
||||
|
||||
@@ -37,6 +37,44 @@ jobs:
|
||||
with:
|
||||
fetch-depth: 1
|
||||
|
||||
# Make the EE source (the *_ee.rs files in the companion repo) available so the
|
||||
# reviewer can see EE-only code (e.g. windmill-queue/src/jobs_ee.rs), not just the
|
||||
# CE surface. The EE ref is read from the PR head's backend/ee-repo-ref.txt (via the
|
||||
# API, so it reflects the PR's EE pin regardless of which ref is checked out here).
|
||||
- name: Check EE access
|
||||
id: ee
|
||||
env:
|
||||
EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
PR_NUMBER: ${{ github.event.issue.number || github.event.pull_request.number }}
|
||||
run: |
|
||||
if [ -z "$EE_TOKEN" ] || [ -z "$PR_NUMBER" ]; then
|
||||
echo "available=false" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
HEAD_SHA=$(gh api "repos/${{ github.repository }}/pulls/$PR_NUMBER" --jq .head.sha)
|
||||
REF=$(gh api "repos/${{ github.repository }}/contents/backend/ee-repo-ref.txt?ref=$HEAD_SHA" --jq .content | base64 -d | tr -d '[:space:]')
|
||||
if [ -z "$REF" ]; then
|
||||
echo "available=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "available=true" >> "$GITHUB_OUTPUT"
|
||||
echo "ee_repo_ref=$REF" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Checkout EE repository
|
||||
if: steps.ee.outputs.available == 'true'
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
repository: windmill-labs/windmill-ee-private
|
||||
path: ./windmill-ee-private
|
||||
ref: ${{ steps.ee.outputs.ee_repo_ref }}
|
||||
token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }}
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Substitute EE code
|
||||
if: steps.ee.outputs.available == 'true'
|
||||
run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private
|
||||
|
||||
- name: Run Claude PR Action
|
||||
uses: anthropics/claude-code-action@v1
|
||||
with:
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
name: Refresh docs snapshot
|
||||
|
||||
# The backend embeds a vendored docs snapshot (backend/windmill-api/docs_snapshot/*.gz)
|
||||
# so in-product docs search works with no runtime egress. This job re-fetches it from
|
||||
# windmill.dev on a schedule and opens a PR when it changed, keeping the embedded docs
|
||||
# fresh independently of the release cadence (the binary embeds whatever is on the
|
||||
# source tree at build time, so a merged refresh rides into the next release build).
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 6 * * 1" # Mondays 06:00 UTC
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
refresh:
|
||||
runs-on: ubicloud
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
steps:
|
||||
- uses: actions/create-github-app-token@v2
|
||||
id: app
|
||||
with:
|
||||
app-id: ${{ vars.INTERNAL_APP_ID }}
|
||||
private-key: ${{ secrets.INTERNAL_APP_KEY }}
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
token: ${{ steps.app.outputs.token }}
|
||||
- name: Fetch + re-gzip docs snapshot
|
||||
run: cd backend/windmill-api/docs_snapshot && ./fetch.sh
|
||||
- name: Sanity-check the fetched corpus
|
||||
# curl -f in fetch.sh rejects HTTP errors, but not a valid-but-garbage 200
|
||||
# (truncated file, error page). Guard against embedding a broken snapshot.
|
||||
run: |
|
||||
cd backend/windmill-api/docs_snapshot
|
||||
test "$(wc -c < llms-full.txt.gz)" -gt 100000
|
||||
test "$(wc -c < llms.txt.gz)" -gt 1000
|
||||
pages=$(gzip -dc llms-full.txt.gz | grep -c '^Source:' || true)
|
||||
echo "pages in snapshot: $pages"
|
||||
test "${pages:-0}" -ge 200
|
||||
- uses: peter-evans/create-pull-request@v6
|
||||
with:
|
||||
token: ${{ steps.app.outputs.token }}
|
||||
branch: chore/refresh-docs-snapshot
|
||||
add-paths: backend/windmill-api/docs_snapshot/*.gz
|
||||
commit-message: "chore: refresh vendored docs snapshot"
|
||||
title: "chore: refresh vendored docs snapshot"
|
||||
body: |
|
||||
Automated refresh of the embedded docs snapshot
|
||||
(`backend/windmill-api/docs_snapshot/*.gz`) from windmill.dev.
|
||||
|
||||
Review the diff for unexpected churn (a bad upstream docs deploy would
|
||||
show up as a large drop in pages or content) before merging.
|
||||
+193
@@ -1,5 +1,198 @@
|
||||
# Changelog
|
||||
|
||||
## [1.742.0](https://github.com/windmill-labs/windmill/compare/v1.741.0...v1.742.0) (2026-06-28)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **apps:** add labels input to app editor deploy drawer ([#9828](https://github.com/windmill-labs/windmill/issues/9828)) ([da45e69](https://github.com/windmill-labs/windmill/commit/da45e699c8aefeede172c90769ef4f4b182fec0c))
|
||||
* column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) ([#9814](https://github.com/windmill-labs/windmill/issues/9814)) ([003a262](https://github.com/windmill-labs/windmill/commit/003a262a4e9d6c2a63ada01aa8429aea1fbb6031))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **audit:** don't read pg_authid from an elevated context in S3 export migration ([#9832](https://github.com/windmill-labs/windmill/issues/9832)) ([75ba81b](https://github.com/windmill-labs/windmill/commit/75ba81b2d27fb0722095780312064cb93d20287e))
|
||||
* close unauthenticated DAP debugger program-mode launch bypass ([#9829](https://github.com/windmill-labs/windmill/issues/9829)) ([c0768de](https://github.com/windmill-labs/windmill/commit/c0768de0acdf63eaba5fb97d04bfc64f2f03b93d))
|
||||
* redeploy older app version from deployment history ([#9826](https://github.com/windmill-labs/windmill/issues/9826)) ([c479afa](https://github.com/windmill-labs/windmill/commit/c479afab8ebceccbee050e923dc5c27a6712ea62))
|
||||
|
||||
## [1.741.0](https://github.com/windmill-labs/windmill/compare/v1.740.0...v1.741.0) (2026-06-26)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **ai-chat:** add create_folder tool to global chat ([#9819](https://github.com/windmill-labs/windmill/issues/9819)) ([44c25de](https://github.com/windmill-labs/windmill/commit/44c25de418612ab98341adb15d5671222b54367e))
|
||||
* **ai-chat:** hint /compact in context usage tooltip ([#9777](https://github.com/windmill-labs/windmill/issues/9777)) ([aadfb62](https://github.com/windmill-labs/windmill/commit/aadfb620c0b7dcd7e94367b761875e14ef9abe69))
|
||||
* **ai-chat:** let global chat edit the user's personal instructions ([#9771](https://github.com/windmill-labs/windmill/issues/9771)) ([3be2752](https://github.com/windmill-labs/windmill/commit/3be27521b05de33e48582e80c6651071f889f048))
|
||||
* **ai-chat:** surface raw apps in the @-mention context picker ([#9800](https://github.com/windmill-labs/windmill/issues/9800)) ([1602244](https://github.com/windmill-labs/windmill/commit/16022447c7b445be753b9545b10b4c67da0893d5))
|
||||
* capture managed-materialize output schema as asset metadata ([#2](https://github.com/windmill-labs/windmill/issues/2)a) ([#9812](https://github.com/windmill-labs/windmill/issues/9812)) ([ade74b2](https://github.com/windmill-labs/windmill/commit/ade74b297f6a03441e700a20ffc2d7291c8a85fd))
|
||||
* **sdk:** allow overriding worker tag when running jobs (WIN-2105) ([#9807](https://github.com/windmill-labs/windmill/issues/9807)) ([52fc7bf](https://github.com/windmill-labs/windmill/commit/52fc7bf94cf3f87f68d9dba9884944d87e7d5d57))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* apply step timeout to 'Test this step' preview ([#9810](https://github.com/windmill-labs/windmill/issues/9810)) ([d04062b](https://github.com/windmill-labs/windmill/commit/d04062bff58c9c4c79ce542a4321e71bcbcf0e98))
|
||||
* **flows:** reject corrupt step paths at deploy + atomic cache writes ([#9751](https://github.com/windmill-labs/windmill/issues/9751)) ([#9813](https://github.com/windmill-labs/windmill/issues/9813)) ([3cda447](https://github.com/windmill-labs/windmill/commit/3cda44762148bcd2ee5c0ea821db884950376ead))
|
||||
* **frontend:** clarify instance data table unavailable on cloud ([#9806](https://github.com/windmill-labs/windmill/issues/9806)) ([c3e8c78](https://github.com/windmill-labs/windmill/commit/c3e8c789ac05c9c28991d9ab6f2358f61fa87971))
|
||||
* hide GCS service account key behind a reveal in object storage settings ([#9815](https://github.com/windmill-labs/windmill/issues/9815)) ([0ec5061](https://github.com/windmill-labs/windmill/commit/0ec5061270749ed078e01f5a4bc7397a1755ca32))
|
||||
* ping job during volume setup to prevent false zombie restarts ([#9803](https://github.com/windmill-labs/windmill/issues/9803)) ([43bb676](https://github.com/windmill-labs/windmill/commit/43bb676dc5652cb06fe1414b8d3aacf295bae36b))
|
||||
* skipped suspend step no longer parks the flow forever ([#9821](https://github.com/windmill-labs/windmill/issues/9821)) ([40110bc](https://github.com/windmill-labs/windmill/commit/40110bc7158bc42c3d84bd4637a12b82fcd72a9a))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* **audit:** re-anchor S3 audit export on enable + opt-in backfill ([#9818](https://github.com/windmill-labs/windmill/issues/9818)) ([577ceee](https://github.com/windmill-labs/windmill/commit/577ceeee8679f054c6898d1a7889df30ab830f8f))
|
||||
|
||||
## [1.740.0](https://github.com/windmill-labs/windmill/compare/v1.739.0...v1.740.0) (2026-06-25)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **api:** add structured endpoint for flow logs ([#9797](https://github.com/windmill-labs/windmill/issues/9797)) ([ba768fe](https://github.com/windmill-labs/windmill/commit/ba768fee888682cb50142d6e76c0422c40307f46))
|
||||
* bounded-cascade selective execution for pipelines (UI + CLI) ([#9695](https://github.com/windmill-labs/windmill/issues/9695)) ([248540a](https://github.com/windmill-labs/windmill/commit/248540ac4d6e4ee9ee7c3e6f2cc822c63cc6426e))
|
||||
* data tests for ducklake pipeline materialization ([#9708](https://github.com/windmill-labs/windmill/issues/9708)) ([f6998ec](https://github.com/windmill-labs/windmill/commit/f6998ec54cba2507703790bf33427e7567d42c4b))
|
||||
* detect and guard against deploying stale drafts ([#9768](https://github.com/windmill-labs/windmill/issues/9768)) ([d865518](https://github.com/windmill-labs/windmill/commit/d8655189347f58df9d17e83dc55798baf7964279))
|
||||
* ducklake time-travel UX (snapshot history + AT VERSION reads) ([#9709](https://github.com/windmill-labs/windmill/issues/9709)) ([d131d75](https://github.com/windmill-labs/windmill/commit/d131d754e1fc9674abf5de383d2bc93596df9bd1))
|
||||
* self-host docs search for chat, mcp, cli; drop inkeep ([#9772](https://github.com/windmill-labs/windmill/issues/9772)) ([9d61e4e](https://github.com/windmill-labs/windmill/commit/9d61e4e59e4101de84217f7c7846f1aa94e84d89))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* allow hyphens in postgresql database name validation ([#9782](https://github.com/windmill-labs/windmill/issues/9782)) ([170cd79](https://github.com/windmill-labs/windmill/commit/170cd79aaf92152fc3c0f675f155853c7f0e5b25))
|
||||
* **debounce:** never supersede a running debounce survivor ([#9780](https://github.com/windmill-labs/windmill/issues/9780)) ([5549bdc](https://github.com/windmill-labs/windmill/commit/5549bdc67a5559a764616c44b1018543bc0568fe))
|
||||
* decrypt secret variables via external backend in common resolvers ([#9784](https://github.com/windmill-labs/windmill/issues/9784)) ([cd42c6c](https://github.com/windmill-labs/windmill/commit/cd42c6ca18261328055554788932c3fe876a4a5b))
|
||||
* enforce containment of python module dir for preview jobs ([#9704](https://github.com/windmill-labs/windmill/issues/9704)) ([88fca6a](https://github.com/windmill-labs/windmill/commit/88fca6a8c130b9e3b0f0cd410e422d4e074fc11f))
|
||||
* **frontend:** apply script editor timeout to preview/Test runs ([#9794](https://github.com/windmill-labs/windmill/issues/9794)) ([6664ce6](https://github.com/windmill-labs/windmill/commit/6664ce6dc0c5fbc283303148de06d6bb85e4acf7))
|
||||
* **frontend:** nested-loop "Test this step" resolves iter to innermost loop ([#9778](https://github.com/windmill-labs/windmill/issues/9778)) ([74ebfc6](https://github.com/windmill-labs/windmill/commit/74ebfc67f069047875db738926865bd4bd6fe9e9))
|
||||
* opt out of Deno minimum-dependency-age for private npm registries ([#9802](https://github.com/windmill-labs/windmill/issues/9802)) ([b28f974](https://github.com/windmill-labs/windmill/commit/b28f974e5069f635419d9ea56fad6a0e417894e8))
|
||||
* pass SSL cert env vars to `uv python install` ([#9790](https://github.com/windmill-labs/windmill/issues/9790)) ([962758c](https://github.com/windmill-labs/windmill/commit/962758c02de5f6d962c681fe9c39769b99429e8d))
|
||||
* **python:** re-verify wheel RECORD on local cache reuse (once per worker) ([#9775](https://github.com/windmill-labs/windmill/issues/9775)) ([6c71c33](https://github.com/windmill-labs/windmill/commit/6c71c33470e3ea547f3b994db829eb4d04882443))
|
||||
* **python:** serialize concurrent installs into shared wheel cache dir ([#9787](https://github.com/windmill-labs/windmill/issues/9787)) ([11d83ab](https://github.com/windmill-labs/windmill/commit/11d83ab1ec559be5d3263010228e6db65358e04b))
|
||||
* re-pin stale-draft fork base when restoring an app deployment ([#9792](https://github.com/windmill-labs/windmill/issues/9792)) ([b9711e5](https://github.com/windmill-labs/windmill/commit/b9711e5ace8585315a1c2b85bb25ac8dd7832d6f))
|
||||
* restore libargon2-1 for PHP runtime in server image ([#9795](https://github.com/windmill-labs/windmill/issues/9795)) ([e9cb806](https://github.com/windmill-labs/windmill/commit/e9cb80639b2dec63ede69fc3a4e3720bb1a3c319))
|
||||
* use transaction for parallel_monitor_lock DELETE in last-iteration path ([#9789](https://github.com/windmill-labs/windmill/issues/9789)) ([754cae9](https://github.com/windmill-labs/windmill/commit/754cae956ac8d431ddeb055453835e249cdd07b7))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes ([#9786](https://github.com/windmill-labs/windmill/issues/9786)) ([aa098c7](https://github.com/windmill-labs/windmill/commit/aa098c70c0271b2b1917749d1f607c0559cf04de))
|
||||
* eliminate dual-connection DB pool contention across worker, queue, and api ([#9798](https://github.com/windmill-labs/windmill/issues/9798)) ([0dbd9c1](https://github.com/windmill-labs/windmill/commit/0dbd9c1231b00d4693af68835fe1d9e7c8869b43))
|
||||
|
||||
## [1.739.0](https://github.com/windmill-labs/windmill/compare/v1.738.0...v1.739.0) (2026-06-24)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* add /compact session chat command ([#9764](https://github.com/windmill-labs/windmill/issues/9764)) ([83cc553](https://github.com/windmill-labs/windmill/commit/83cc5533ee92e59356a117eefeaf42dad23287f6))
|
||||
* add session chat slash commands ([#9748](https://github.com/windmill-labs/windmill/issues/9748)) ([24b95e9](https://github.com/windmill-labs/windmill/commit/24b95e9fe12ba4abdfe1ff6e9f9fe42cb2ded011))
|
||||
* **ai-chat:** add /clear session command to start a fresh conversation ([#9769](https://github.com/windmill-labs/windmill/issues/9769)) ([3fafac2](https://github.com/windmill-labs/windmill/commit/3fafac275d2100a6f89924040650cf959945d209))
|
||||
* **ai-chat:** context usage gauge + unified model settings menu ([#9763](https://github.com/windmill-labs/windmill/issues/9763)) ([2e020b2](https://github.com/windmill-labs/windmill/commit/2e020b2ccc7a649a5923bff72a98f07d4fc85381))
|
||||
* **apps:** show raw-app fork diffs as per-file tree items ([#9491](https://github.com/windmill-labs/windmill/issues/9491)) ([e98df38](https://github.com/windmill-labs/windmill/commit/e98df38ac43823ee85209a4b09cd70690469302d))
|
||||
* **frontend:** add filter submenu to collapsed AI sessions popover ([#9757](https://github.com/windmill-labs/windmill/issues/9757)) ([3d48ba7](https://github.com/windmill-labs/windmill/commit/3d48ba7738c3d3356539b5fc44a871f6b7f9d548))
|
||||
* **frontend:** restore raw app 'open preview in separate window' ([#9765](https://github.com/windmill-labs/windmill/issues/9765)) ([a116715](https://github.com/windmill-labs/windmill/commit/a116715c418c39d48a91e6c0b4484a31537dff38))
|
||||
* **frontend:** show approval wait as a distinct segment in flow timeline ([#9756](https://github.com/windmill-labs/windmill/issues/9756)) ([2a70ccc](https://github.com/windmill-labs/windmill/commit/2a70ccc38675c7c2353807a4f85764a8a35224e2))
|
||||
* scope AI sessions per workspace root with lifecycle reconcile ([#9734](https://github.com/windmill-labs/windmill/issues/9734)) ([42c5e7a](https://github.com/windmill-labs/windmill/commit/42c5e7a3fc9b74256de8806ec0d7b62e8bbf029c))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **ai-chat:** strip unclosed <summary> tag leaking into compaction summary ([#9750](https://github.com/windmill-labs/windmill/issues/9750)) ([250a05f](https://github.com/windmill-labs/windmill/commit/250a05f544ae397bb91af5fc83bf408cfe1c554d))
|
||||
* **apps:** realign legacy raw-app drafts to raw_app draft kind ([#9761](https://github.com/windmill-labs/windmill/issues/9761)) ([288318a](https://github.com/windmill-labs/windmill/commit/288318ac269714fc03b15622dbb86b1c28268a36))
|
||||
* **backend:** resolve folder_labels search_path on non-public (PG_SCHEMA) schemas ([#9758](https://github.com/windmill-labs/windmill/issues/9758)) ([f582878](https://github.com/windmill-labs/windmill/commit/f5828780fd6a8be070b2933ebd41ee6dff98a9e1))
|
||||
* forbid superadmin job tokens from global user and token management ([#9715](https://github.com/windmill-labs/windmill/issues/9715)) ([043c2c0](https://github.com/windmill-labs/windmill/commit/043c2c05b7678c49faca0ccb28e5f6393567ba4d))
|
||||
* **frontend:** highlight the runtime-chosen branch in flow graph viewer ([#9755](https://github.com/windmill-labs/windmill/issues/9755)) ([de6192b](https://github.com/windmill-labs/windmill/commit/de6192bec1695883a07452f7db2fb51c94dbfd43))
|
||||
* **frontend:** keep #content portal target present on AI-session route ([#9754](https://github.com/windmill-labs/windmill/issues/9754)) ([5e09c50](https://github.com/windmill-labs/windmill/commit/5e09c501713ebbe05b28ce0084eca641f0dbe95c))
|
||||
* **frontend:** show AI skills settings only when global mode enabled ([#9747](https://github.com/windmill-labs/windmill/issues/9747)) ([c017f7f](https://github.com/windmill-labs/windmill/commit/c017f7f8919a51292ddf01574961d1774bc1ba23))
|
||||
* **frontend:** stop flow step id generation from being poisoned by non-canonical keys ([#9766](https://github.com/windmill-labs/windmill/issues/9766)) ([4dbf873](https://github.com/windmill-labs/windmill/commit/4dbf8737238ccc4dc2c67365e6d43f04f46c75b5))
|
||||
* persist on-behalf-of user across app deploy paths ([#9773](https://github.com/windmill-labs/windmill/issues/9773)) ([f99781c](https://github.com/windmill-labs/windmill/commit/f99781ca5f77248206c951935cc44acfa5f072eb))
|
||||
* reject symlink traversal in job-dir path validation ([#9713](https://github.com/windmill-labs/windmill/issues/9713)) ([b5bd824](https://github.com/windmill-labs/windmill/commit/b5bd8245d81b84fc14d3ea955bf1e66ac576bf37))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* **audit:** adaptive timestamp floor for S3 audit-log export ([#9752](https://github.com/windmill-labs/windmill/issues/9752)) ([55bed4a](https://github.com/windmill-labs/windmill/commit/55bed4abcfce2a611b16054573980d2eb613ccb3))
|
||||
* **monitor:** vacuum job_perms/job_result_stream right after each orphan sweep ([#9753](https://github.com/windmill-labs/windmill/issues/9753)) ([8912e21](https://github.com/windmill-labs/windmill/commit/8912e21d1571e57b5cf21b7d4d9520e20a28e70d))
|
||||
|
||||
## [1.738.0](https://github.com/windmill-labs/windmill/compare/v1.737.0...v1.738.0) (2026-06-23)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* add resource and infrastructure telemetry ([#9737](https://github.com/windmill-labs/windmill/issues/9737)) ([9793d01](https://github.com/windmill-labs/windmill/commit/9793d01575415963a89609a1baf2cd64f0d050cc))
|
||||
* render mermaid diagrams in chat code blocks ([#9738](https://github.com/windmill-labs/windmill/issues/9738)) ([cfb9f1d](https://github.com/windmill-labs/windmill/commit/cfb9f1dbc23110ecf8f91bb3c8c81fc6e35dc09b))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **ai-chat:** Fix incorrect editor edits from ai chat [#1](https://github.com/windmill-labs/windmill/issues/1) ([#9741](https://github.com/windmill-labs/windmill/issues/9741)) ([fc797a3](https://github.com/windmill-labs/windmill/commit/fc797a35fe7885630c81453df0fc94769e73873a))
|
||||
* allow object storage test for non-super-admins, harden on cloud ([#9739](https://github.com/windmill-labs/windmill/issues/9739)) ([24446e8](https://github.com/windmill-labs/windmill/commit/24446e80093ade349f7fbf65063d2d1cb5551c1e))
|
||||
* **frontend:** debounce external code→Monaco sync in Editor ([#9743](https://github.com/windmill-labs/windmill/issues/9743)) ([29c67ce](https://github.com/windmill-labs/windmill/commit/29c67ced97bf2919584986f9d9eceb4337c34ad9))
|
||||
* **frontend:** preserve editor content when closing instance settings drawer ([#9740](https://github.com/windmill-labs/windmill/issues/9740)) ([11d0e65](https://github.com/windmill-labs/windmill/commit/11d0e65f3af9a048bc1921bbdd3d676a07483a57))
|
||||
* pipeline annotation false-positives from body comments ([#9736](https://github.com/windmill-labs/windmill/issues/9736)) ([984ea72](https://github.com/windmill-labs/windmill/commit/984ea728d98649b66b1cae899bdab9af3176caa7))
|
||||
* preserve fork parent linkage on workspace id change ([#9716](https://github.com/windmill-labs/windmill/issues/9716)) ([cbf54d4](https://github.com/windmill-labs/windmill/commit/cbf54d4eb432638e27f67c4c8b879cbcc0291da3))
|
||||
* prevent variable push from corrupting is_secret variables ([#9705](https://github.com/windmill-labs/windmill/issues/9705)) ([ba4b368](https://github.com/windmill-labs/windmill/commit/ba4b368706e95e22f346a10e5fe145b0795ac3f6))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* **monitor:** skip protected prefix in retention delete via cross-batch watermark (WIN-2088) ([#9744](https://github.com/windmill-labs/windmill/issues/9744)) ([e90b2be](https://github.com/windmill-labs/windmill/commit/e90b2be8fade1eb78cd685890291f5a4553a6a10))
|
||||
|
||||
## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911))
|
||||
* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6))
|
||||
* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a))
|
||||
* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39))
|
||||
* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011))
|
||||
|
||||
## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72))
|
||||
* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c))
|
||||
* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76))
|
||||
* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9))
|
||||
* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6))
|
||||
|
||||
## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* clarify session draft bar tracks all workspace draft changes ([#9714](https://github.com/windmill-labs/windmill/issues/9714)) ([ed016a5](https://github.com/windmill-labs/windmill/commit/ed016a5edb4527877bf7e6bf92feafc2710669c2))
|
||||
* **copilot:** improve global-mode path selection + add path-selection evals ([#9698](https://github.com/windmill-labs/windmill/issues/9698)) ([74a2329](https://github.com/windmill-labs/windmill/commit/74a2329d2e8395141807c43acef07ef132490039))
|
||||
* link files & folders to the global AI chat ([#9520](https://github.com/windmill-labs/windmill/issues/9520)) ([84cc043](https://github.com/windmill-labs/windmill/commit/84cc043406d63a1e1472165cf24ce8c09905fc5f))
|
||||
* scope default instance db name to workspace (dt_/dl_) ([#9699](https://github.com/windmill-labs/windmill/issues/9699)) ([4a8a724](https://github.com/windmill-labs/windmill/commit/4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* enforce job_dir containment when writing module files ([#9703](https://github.com/windmill-labs/windmill/issues/9703)) ([e403f92](https://github.com/windmill-labs/windmill/commit/e403f92d7e84cebc78709dce1a0928048ba2506d))
|
||||
* **frontend:** deploy full script/flow draft from AI chat via shared module ([#9642](https://github.com/windmill-labs/windmill/issues/9642)) ([23bf6bf](https://github.com/windmill-labs/windmill/commit/23bf6bf3da01d552d2dfab6dbcfd30f758ed34d2))
|
||||
* **frontend:** strip raw-app post-deploy diff noise (raw_app/lock/data) ([#9706](https://github.com/windmill-labs/windmill/issues/9706)) ([e20a277](https://github.com/windmill-labs/windmill/commit/e20a27745a08d552e6d2c5a8bbaf08ccfe89c68f))
|
||||
* ignore NotFound errors when deleting log files from object store ([#9707](https://github.com/windmill-labs/windmill/issues/9707)) ([8a0b0ab](https://github.com/windmill-labs/windmill/commit/8a0b0abead71320c4f69eb3007739a19f76d4126))
|
||||
* **oauth:** restore bring-your-own CC token URL override ([#9711](https://github.com/windmill-labs/windmill/issues/9711)) ([ef4962e](https://github.com/windmill-labs/windmill/commit/ef4962e52aba0bc79bf72523de9101853c660654))
|
||||
* sanitize git credentials from ansible executor errors and logs ([#9697](https://github.com/windmill-labs/windmill/issues/9697)) ([ace7b68](https://github.com/windmill-labs/windmill/commit/ace7b68a28b00d715298ffcb6ae907c1974a74b8))
|
||||
|
||||
## [1.734.0](https://github.com/windmill-labs/windmill/compare/v1.733.1...v1.734.0) (2026-06-20)
|
||||
|
||||
|
||||
|
||||
+5
-5
@@ -163,14 +163,14 @@ ENV PATH /usr/local/bin:/root/.local/bin:/tmp/.local/bin:$PATH
|
||||
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 software-properties-common tini gnupg lsb-release \
|
||||
&& apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 tini gnupg libargon2-1 \
|
||||
&& if echo "$features" | grep -q "ee"; then apt-get install -y --no-install-recommends libsasl2-modules-gssapi-mit krb5-user; fi \
|
||||
&& apt-get clean \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install latest PostgreSQL client (pg_dump) from official PostgreSQL apt repository
|
||||
RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /usr/share/keyrings/postgresql-archive-keyring.gpg \
|
||||
&& echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
|
||||
&& echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(. /etc/os-release; echo "$VERSION_CODENAME")-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y --no-install-recommends postgresql-client \
|
||||
&& apt-get clean \
|
||||
@@ -184,11 +184,11 @@ RUN if [ "$WITH_GIT" = "true" ]; then \
|
||||
else echo 'Building the image without git'; fi;
|
||||
|
||||
RUN if [ "$WITH_POWERSHELL" = "true" ]; then \
|
||||
if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \
|
||||
if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu72 -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \
|
||||
&& rm -rf /var/lib/apt/lists/* && \
|
||||
dpkg --install 'pwsh.deb' && \
|
||||
rm 'pwsh.deb'; \
|
||||
elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \
|
||||
elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu72 -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \
|
||||
&& rm -rf /var/lib/apt/lists/* && \
|
||||
mkdir -p /opt/microsoft/powershell/7 && \
|
||||
tar zxf powershell.tar.gz -C /opt/microsoft/powershell/7 && \
|
||||
@@ -233,7 +233,7 @@ ENV PATH="${PATH}:/usr/local/go/bin"
|
||||
ENV GO_PATH=/usr/local/go/bin/go
|
||||
|
||||
# Install UV
|
||||
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
|
||||
RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv
|
||||
|
||||
# Preinstall python runtimes to temp build location (will copy with world-writable perms later)
|
||||
# --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run
|
||||
|
||||
+10
-204
@@ -1,208 +1,14 @@
|
||||
# AI Evals Authoring Guide
|
||||
# AI Evals
|
||||
|
||||
This folder contains black-box benchmark cases for:
|
||||
Black-box benchmark cases for the Windmill AI generation modes (`flow`, `app`,
|
||||
`script`, `cli`, `global`).
|
||||
|
||||
- `flow`
|
||||
- `app`
|
||||
- `script`
|
||||
- `cli`
|
||||
- `global`
|
||||
**Authoring and running cases is documented in the `ai-evals` skill** — load it
|
||||
before adding/changing a case or running a benchmark. Claude Code reads
|
||||
`.claude/skills/ai-evals/SKILL.md`; Codex and Pi read
|
||||
`.agents/skills/ai-evals/SKILL.md` (same canonical file). Invoke with `/ai-evals` in
|
||||
Claude Code, `$ai-evals` in Codex, or `pi --skill ai-evals`.
|
||||
|
||||
The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape.
|
||||
For AI chat / copilot changes that these evals measure, see the `ai-chat` skill.
|
||||
|
||||
## Core rules
|
||||
|
||||
1. Write prompts like a real user request.
|
||||
2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details.
|
||||
3. Keep deterministic validation narrow and hard.
|
||||
4. Put semantic expectations in `judgeChecklist`.
|
||||
5. Use `expected` fixtures only when exact structure really matters.
|
||||
|
||||
## Prompt writing
|
||||
|
||||
Prompts should sound like something a user would naturally ask.
|
||||
|
||||
Good:
|
||||
|
||||
- "Create a flow that routes support requests based on customer tier."
|
||||
- "Add a reset button that sets the counter back to 0."
|
||||
- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
|
||||
|
||||
Bad:
|
||||
|
||||
- "Use `branchone` with 3 branches and a default branch."
|
||||
- "Create a `rawscript` step with this exact topology."
|
||||
- "This is a benchmark harness."
|
||||
|
||||
Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow.
|
||||
|
||||
## Flow-specific rules
|
||||
|
||||
This is the main principle you asked for:
|
||||
|
||||
- flow prompts should read like requests from a user who does not know the product internals
|
||||
- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs
|
||||
|
||||
That means:
|
||||
|
||||
- creation cases should describe the business behavior and expected result
|
||||
- modification cases may mention existing step names, because the user can see the current flow
|
||||
- only mention special Windmill constructs when the case is explicitly about those constructs
|
||||
|
||||
Examples:
|
||||
|
||||
- acceptable creation prompt:
|
||||
"Create a purchase approval flow that pauses for approval and asks the approver for a comment."
|
||||
- avoid:
|
||||
"Create a suspend step with one required event and a resume form."
|
||||
|
||||
For flow cases, do not fail a case just because the model chose a different valid topology.
|
||||
|
||||
## App-specific rules
|
||||
|
||||
App prompts should focus on user-visible behavior:
|
||||
|
||||
- what the UI should let the user do
|
||||
- what should persist
|
||||
- what backend behavior is needed
|
||||
|
||||
Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app.
|
||||
|
||||
## CLI-specific rules
|
||||
|
||||
CLI prompts can be more explicit about paths and file names because real CLI users often do specify them.
|
||||
|
||||
Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction.
|
||||
|
||||
When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior.
|
||||
|
||||
## Global-specific rules
|
||||
|
||||
Global prompts should exercise workspace-level drafting behavior:
|
||||
|
||||
- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant
|
||||
- writing AI drafts rather than saving or deploying by default
|
||||
- producing coherent multi-artifact changes when the request crosses artifact boundaries
|
||||
|
||||
Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.
|
||||
|
||||
Datatable cases should set `skipJudge: true` and validate through tool-use
|
||||
(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions
|
||||
(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`,
|
||||
`['update', 'insert into']`). Two reasons the judge is unreliable here:
|
||||
|
||||
- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql`
|
||||
produce no drafts, and the global judge only sees the drafts artifact — it
|
||||
scores a no-draft conversational answer as empty (same as the
|
||||
`askUserQuestion` cases).
|
||||
- Even a case that *does* produce a draft (a script reading the data table via
|
||||
`wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK
|
||||
reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the
|
||||
SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']`
|
||||
plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from
|
||||
runtime SDK use).
|
||||
|
||||
`stringIncludesAnyOf` is existential over calls (at least one matching call), so a
|
||||
mutation case still passes when the model mixes its UPDATE/INSERT with
|
||||
verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful
|
||||
within a case — writes persist, so a model that re-queries to verify its
|
||||
CREATE/UPDATE sees the change and does not loop. But the engine is best-effort
|
||||
(SELECT returns all rows of the referenced/first table with no WHERE/projection),
|
||||
so still never assert specific returned row values. Seed data via
|
||||
`workspace.datatables` in the `initial` fixture (see README).
|
||||
|
||||
## Deterministic validation
|
||||
|
||||
Use deterministic validation only for hard failures such as:
|
||||
|
||||
- missing required files
|
||||
- unexpected extra files when the prompt says not to create them
|
||||
- syntax errors
|
||||
- unresolved flow refs
|
||||
- missing required special modules or suspend config
|
||||
- obvious artifact corruption
|
||||
|
||||
Do not use deterministic validation to enforce one preferred implementation for broad creation tasks.
|
||||
|
||||
Examples of bad hard checks:
|
||||
|
||||
- exact step topology for a creation flow
|
||||
- exact branch structure when the prompt only asked for routing behavior
|
||||
- exact input shape when multiple reasonable shapes are acceptable
|
||||
|
||||
## Judge checklist
|
||||
|
||||
Every non-trivial case should have a `judgeChecklist`.
|
||||
|
||||
The checklist should capture:
|
||||
|
||||
- the user-visible behavior that must be present
|
||||
- important constraints
|
||||
- key completion criteria
|
||||
|
||||
The checklist should not duplicate low-level implementation details unless they are truly required by the task.
|
||||
|
||||
Good checklist items:
|
||||
|
||||
- "the flow calculates the order total with 8% tax"
|
||||
- "the app persists recipes appropriately for a raw Windmill app"
|
||||
- "the flow reuses the existing workspace script instead of rewriting the logic"
|
||||
|
||||
Bad checklist items:
|
||||
|
||||
- "uses `branchone`"
|
||||
- "contains a `rawscript` node"
|
||||
|
||||
## When to use `expected`
|
||||
|
||||
Use `expected` fixtures when the case is structure-sensitive, for example:
|
||||
|
||||
- exact file creation
|
||||
- exact script content
|
||||
- modification cases where a specific file must change in a specific way
|
||||
- cases where preserving an existing structure is part of the requirement
|
||||
|
||||
Do not use a full `expected` artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass.
|
||||
|
||||
## When to use `initial`
|
||||
|
||||
Use `initial` when the benchmark is about:
|
||||
|
||||
- editing an existing artifact
|
||||
- reusing existing workspace assets
|
||||
- preserving existing behavior while adding a change
|
||||
|
||||
If the case is greenfield, prefer no `initial`.
|
||||
|
||||
## Case design ladder
|
||||
|
||||
Prefer suites that get gradually harder:
|
||||
|
||||
1. trivial create case
|
||||
2. realistic create case
|
||||
3. reuse-existing-assets case
|
||||
4. modification case
|
||||
5. refactor case
|
||||
6. edge-case or niche product behavior
|
||||
|
||||
The last cases in a suite should cover unusual or product-specific behavior.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
Avoid these:
|
||||
|
||||
- benchmark framing in prompts
|
||||
- over-specified internal topology for creation tasks
|
||||
- judge checklists that just restate implementation details
|
||||
- deterministic validation that encodes one preferred solution
|
||||
- fixtures that are so minimal or brittle that they create false negatives
|
||||
|
||||
## Before adding a case
|
||||
|
||||
Ask:
|
||||
|
||||
1. Would a real user plausibly write this prompt?
|
||||
2. If the model solves it in a different valid way, would the case still pass?
|
||||
3. Are the hard deterministic checks only catching objectively broken output?
|
||||
4. Does the `judgeChecklist` describe the real success criteria?
|
||||
5. If this case fails, will the reason be understandable from the saved artifacts?
|
||||
The full case format, fields, and fixture details remain in `ai_evals/README.md`.
|
||||
|
||||
+3
-1
@@ -75,6 +75,8 @@ Public CLI surface:
|
||||
- `--model <alias>`: choose the model under test
|
||||
- `--models <a,b,c>`: run the same cases sequentially against several model aliases
|
||||
- `--verbose`: stream assistant output for frontend runs
|
||||
- `--skip-judge`: skip LLM judge scoring for the run
|
||||
- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring
|
||||
- `--record`: append a compact tracked summary line to `ai_evals/history/<mode>.jsonl` for full-suite runs only
|
||||
- `--backend-validation <mode>`: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals
|
||||
|
||||
@@ -99,7 +101,7 @@ Notes:
|
||||
- the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5`
|
||||
- frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases
|
||||
- `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there
|
||||
- the judge model is separate and currently defaults to `claude-sonnet-4-6`
|
||||
- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs
|
||||
|
||||
## Case Format
|
||||
|
||||
|
||||
@@ -25,6 +25,12 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
|
||||
);
|
||||
const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1";
|
||||
const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1";
|
||||
const executionOnly =
|
||||
process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1";
|
||||
const judgeModel =
|
||||
process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly
|
||||
? null
|
||||
: DEFAULT_JUDGE_MODEL;
|
||||
const model = resolveEvalModel(
|
||||
mode,
|
||||
process.env.WMILL_FRONTEND_AI_EVAL_MODEL,
|
||||
@@ -48,7 +54,8 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
|
||||
cases: selectedCases,
|
||||
runs,
|
||||
runModel,
|
||||
judgeModel: DEFAULT_JUDGE_MODEL,
|
||||
judgeModel,
|
||||
executionOnly,
|
||||
concurrency: verbose ? 1 : undefined,
|
||||
verbose,
|
||||
onProgress: emitProgress
|
||||
@@ -60,7 +67,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
|
||||
mode,
|
||||
runs,
|
||||
runModel,
|
||||
judgeModel: DEFAULT_JUDGE_MODEL,
|
||||
judgeModel,
|
||||
caseResults,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -50,6 +50,20 @@ export interface GlobalLiveEditorDraftFixture {
|
||||
value?: unknown;
|
||||
}
|
||||
|
||||
// Identity the global system prompt builds paths from. Production reads
|
||||
// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs
|
||||
// in, so without this the prompt sees an empty username (`u//...`) and no
|
||||
// path-selection case is meaningful. Seeded per-case via the initial fixture and
|
||||
// passed straight to `prepareGlobalSystemMessage` (no global-store mutation).
|
||||
export interface GlobalUserFixture {
|
||||
username: string;
|
||||
is_admin?: boolean;
|
||||
/** Folders the user can write to (the writable set whoami returns). */
|
||||
folders?: string[];
|
||||
/** Folders the user can read; read-only folders = folders_read \ folders. */
|
||||
folders_read?: string[];
|
||||
}
|
||||
|
||||
export interface GlobalEvalResult {
|
||||
success: boolean;
|
||||
state: GlobalDraftState;
|
||||
@@ -65,6 +79,7 @@ export interface GlobalEvalResult {
|
||||
export interface GlobalEvalOptions {
|
||||
workspaceFixtures?: BenchmarkWorkspaceRunnables;
|
||||
liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
|
||||
user?: GlobalUserFixture;
|
||||
model?: string;
|
||||
maxIterations?: number;
|
||||
provider?: AIProvider;
|
||||
@@ -90,9 +105,11 @@ export async function runGlobalEval(
|
||||
const model = options.model ?? "claude-haiku-4-5-20251001";
|
||||
const injectActiveEditorContext =
|
||||
process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1";
|
||||
// Pass the seeded identity straight to the prompt builder rather than mutating
|
||||
// the process-global `userStore`, so concurrent cases never race on it.
|
||||
const rawResult = await runEval({
|
||||
userPrompt,
|
||||
systemMessage: prepareGlobalSystemMessage(),
|
||||
systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }),
|
||||
userMessage: prepareGlobalUserMessage(
|
||||
userPrompt,
|
||||
[],
|
||||
|
||||
@@ -90,6 +90,13 @@ export function resetBenchmarkMockBackend(): void {
|
||||
benchmarkDrafts.clear()
|
||||
}
|
||||
|
||||
// Stand-in for FolderService.createFolder so the global create_folder tool runs in
|
||||
// memory instead of mutating the real backend. Folders aren't otherwise modelled
|
||||
// (no folder-listing in evals), so this just echoes the created name.
|
||||
export function createBenchmarkFolder(_workspace: string, name: string): string {
|
||||
return name
|
||||
}
|
||||
|
||||
export function registerBenchmarkWorkspace(workspace: string): void {
|
||||
benchmarkWorkspaces.add(workspace)
|
||||
}
|
||||
|
||||
@@ -24,6 +24,8 @@ export async function runFrontendBenchmarkAdapter(input: {
|
||||
runs: number;
|
||||
model?: string;
|
||||
verbose?: boolean;
|
||||
skipJudge?: boolean;
|
||||
executionOnly?: boolean;
|
||||
backendValidation?: string;
|
||||
}): Promise<BenchmarkRunResult> {
|
||||
const tempDir = await mkdtemp(
|
||||
@@ -40,6 +42,9 @@ export async function runFrontendBenchmarkAdapter(input: {
|
||||
WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "",
|
||||
WMILL_FRONTEND_AI_EVAL_PROGRESS: "1",
|
||||
WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0",
|
||||
WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE:
|
||||
input.skipJudge || input.executionOnly ? "1" : "0",
|
||||
WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY: input.executionOnly ? "1" : "0",
|
||||
WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "",
|
||||
};
|
||||
|
||||
|
||||
@@ -49,6 +49,7 @@ vi.mock('$lib/gen', async () => {
|
||||
listBenchmarkFlows,
|
||||
listBenchmarkJobs,
|
||||
listBenchmarkScripts,
|
||||
createBenchmarkFolder,
|
||||
createBenchmarkHttpTrigger,
|
||||
createBenchmarkSchedule,
|
||||
previewBenchmarkSchedule,
|
||||
@@ -90,6 +91,12 @@ vi.mock('$lib/gen', async () => {
|
||||
? listBenchmarkDrafts(data.workspace)
|
||||
: actual.DraftService.listDrafts(data)
|
||||
}),
|
||||
FolderService: wrapService(actual.FolderService, {
|
||||
createFolder: async (data: { workspace: string; requestBody: { name: string } }) =>
|
||||
hasBenchmarkWorkspace(data.workspace)
|
||||
? createBenchmarkFolder(data.workspace, data.requestBody.name)
|
||||
: actual.FolderService.createFolder(data)
|
||||
}),
|
||||
ScriptService: wrapService(actual.ScriptService, {
|
||||
listScripts: async (data: { workspace: string }) =>
|
||||
hasBenchmarkWorkspace(data.workspace)
|
||||
|
||||
+136
-1
@@ -3,8 +3,9 @@
|
||||
Create a draft Bun script at `f/evals/global/greet_user`.
|
||||
It should take a string `name` input and return `Hello, ${name}!`.
|
||||
Leave it as an AI draft only; do not deploy or save it.
|
||||
initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
|
||||
runtime:
|
||||
maxTurns: 8
|
||||
maxTurns: 10
|
||||
validate:
|
||||
draftCountExactly: 1
|
||||
requiredDrafts:
|
||||
@@ -1113,3 +1114,137 @@
|
||||
- renames the formatCurrency definition, imports, and all call sites to formatMoney
|
||||
- leaves the unrelated formatCurrencyPrecise helper unchanged
|
||||
- leaves the result as an AI draft only
|
||||
|
||||
# --- Path selection (u/<user> vs f/<folder>) ---
|
||||
# These cases assert how the assistant picks a workspace path when the user gives
|
||||
# none: a bare name defaults to the personal scope `u/<user>/`, an existing folder
|
||||
# whose purpose matches is used, a non-admin targets a writable folder and never a
|
||||
# read-only one, and shared intent with no matching folder asks rather than invents.
|
||||
# Each depends on the seeded `user` fixture (username / is_admin / folders /
|
||||
# folders_read) so the prompt's folder guidance and `u/{username}` are well-formed.
|
||||
|
||||
- id: global-path1-bare-name-defaults-to-personal
|
||||
prompt: |-
|
||||
Stage a quick draft helper that takes a string and returns it trimmed of
|
||||
leading and trailing whitespace. Just keep it as a draft.
|
||||
initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
|
||||
runtime:
|
||||
maxTurns: 8
|
||||
validate:
|
||||
draftCountExactly: 1
|
||||
requiredDrafts:
|
||||
- type: script
|
||||
pathStartsWith: u/admin/
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- write_script
|
||||
forbiddenToolsUsed:
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- stages a single script draft for a trim helper
|
||||
- defaults the path to the current user's personal scope (u/admin/...) since no path or folder was given
|
||||
- does not invent an f/<folder> path
|
||||
- leaves the result as a draft only
|
||||
|
||||
- id: global-path2-match-existing-folder
|
||||
prompt: |-
|
||||
Draft a flow for the marketing team's weekly campaign report.
|
||||
It should take a week number and return a short summary string.
|
||||
Keep it as a draft only.
|
||||
initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
|
||||
runtime:
|
||||
maxTurns: 10
|
||||
validate:
|
||||
draftCountExactly: 1
|
||||
requiredDrafts:
|
||||
- type: flow
|
||||
pathStartsWith: f/marketing/
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- write_flow
|
||||
forbiddenToolsUsed:
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- drafts a flow for the marketing campaign report
|
||||
- places it in the existing marketing folder (f/marketing/...) rather than the personal scope or an invented folder
|
||||
- leaves the result as a draft only
|
||||
|
||||
- id: global-path3-shared-intent-unknown-folder-asks
|
||||
prompt: |-
|
||||
Put together a draft onboarding checklist flow for the People Ops team to use
|
||||
when a new hire joins. Keep it as a draft.
|
||||
initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
|
||||
runtime:
|
||||
maxTurns: 8
|
||||
validate:
|
||||
draftCountExactly: 0
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- askUserQuestion
|
||||
forbiddenToolsUsed:
|
||||
- write_flow
|
||||
- write_script
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- recognizes the request implies shared/team work but names no existing folder (none of marketing/data_engineering/shared_utils fit People Ops)
|
||||
- asks which folder to use instead of guessing or inventing one
|
||||
- does not create a draft until the folder is known
|
||||
|
||||
- id: global-path4-nonadmin-avoids-readonly-folder
|
||||
prompt: |-
|
||||
Draft a small flow that returns today's date as an ISO string, and stage it in
|
||||
one of our shared team folders. Keep it as a draft.
|
||||
initial: ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json
|
||||
runtime:
|
||||
maxTurns: 10
|
||||
validate:
|
||||
draftCountExactly: 1
|
||||
requiredDrafts:
|
||||
- type: flow
|
||||
pathStartsWith: f/team_a/
|
||||
forbiddenDrafts:
|
||||
- type: flow
|
||||
pathStartsWith: f/team_b/
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- write_flow
|
||||
forbiddenToolsUsed:
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- drafts a flow that returns the current date as an ISO string
|
||||
- places it in team_a (writable by this non-admin user) and not team_b (read-only)
|
||||
- leaves the result as a draft only
|
||||
|
||||
- id: global-path5-create-folder-then-draft
|
||||
prompt: |-
|
||||
Create a new shared folder called "analytics" for our data work, then draft a
|
||||
script in it that returns the current timestamp as an ISO string. Keep the
|
||||
script as a draft.
|
||||
initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
|
||||
runtime:
|
||||
maxTurns: 10
|
||||
validate:
|
||||
draftCountExactly: 1
|
||||
requiredDrafts:
|
||||
- type: script
|
||||
pathStartsWith: f/analytics/
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- create_folder
|
||||
- write_script
|
||||
forbiddenToolsUsed:
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- creates a new shared folder named "analytics" via create_folder
|
||||
- drafts a script placed in that folder (f/analytics/...) returning an ISO timestamp
|
||||
- leaves the script as a draft only
|
||||
|
||||
+25
-3
@@ -25,7 +25,9 @@ import {
|
||||
import { runSuite } from "../core/runSuite";
|
||||
import { EVAL_MODES, type EvalMode } from "../core/types";
|
||||
import { DEFAULT_JUDGE_MODEL } from "../core/judge";
|
||||
import { createCliModeRunner } from "../modes/cli";
|
||||
// createCliModeRunner is imported lazily in runCliBenchmark so the non-cli modes
|
||||
// (global/flow/script/app) don't pull in the wmill CLI toolchain and its JSR deps
|
||||
// (e.g. @cliffy/*) just to load this entrypoint.
|
||||
import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime";
|
||||
import { resolveWindmillBackendSettings } from "../core/windmillBackendSettings";
|
||||
import { assertWindmillBackendReachable } from "../adapters/frontend/windmillBackend";
|
||||
@@ -97,6 +99,11 @@ async function main() {
|
||||
"comma-separated model aliases to run sequentially",
|
||||
)
|
||||
.option("--verbose", "stream assistant output during frontend runs")
|
||||
.option("--skip-judge", "skip LLM judge scoring for this run")
|
||||
.option(
|
||||
"--execution-only",
|
||||
"only require the model/proxy/frontend loop to complete",
|
||||
)
|
||||
.option(
|
||||
"--record",
|
||||
"append a compact summary line to ai_evals/history/<mode>.jsonl",
|
||||
@@ -115,6 +122,8 @@ async function main() {
|
||||
model?: string;
|
||||
models?: string;
|
||||
verbose?: boolean;
|
||||
skipJudge?: boolean;
|
||||
executionOnly?: boolean;
|
||||
record?: boolean;
|
||||
backendValidation?: string;
|
||||
},
|
||||
@@ -127,6 +136,8 @@ async function main() {
|
||||
model: options.model,
|
||||
models: options.models,
|
||||
verbose: options.verbose ?? false,
|
||||
skipJudge: options.skipJudge ?? false,
|
||||
executionOnly: options.executionOnly ?? false,
|
||||
record: options.record ?? false,
|
||||
backendValidation: options.backendValidation,
|
||||
});
|
||||
@@ -175,6 +186,8 @@ async function handleRun(input: {
|
||||
model?: string;
|
||||
models?: string;
|
||||
verbose: boolean;
|
||||
skipJudge: boolean;
|
||||
executionOnly: boolean;
|
||||
record: boolean;
|
||||
backendValidation?: string;
|
||||
}) {
|
||||
@@ -230,6 +243,8 @@ async function handleRun(input: {
|
||||
input.runs,
|
||||
getCliEvalModel(model),
|
||||
runModel,
|
||||
input.skipJudge,
|
||||
input.executionOnly,
|
||||
)
|
||||
: await runFrontendBenchmarkAdapter({
|
||||
mode: input.mode,
|
||||
@@ -237,6 +252,8 @@ async function handleRun(input: {
|
||||
runs: input.runs,
|
||||
model: model.id,
|
||||
verbose: input.verbose,
|
||||
skipJudge: input.skipJudge,
|
||||
executionOnly: input.executionOnly,
|
||||
backendValidation,
|
||||
});
|
||||
|
||||
@@ -278,20 +295,25 @@ async function runCliBenchmark(
|
||||
runs: number,
|
||||
model: ReturnType<typeof getCliEvalModel>,
|
||||
runModel: string,
|
||||
skipJudge: boolean,
|
||||
executionOnly: boolean,
|
||||
) {
|
||||
const { createCliModeRunner } = await import("../modes/cli");
|
||||
const judgeModel = skipJudge || executionOnly ? null : DEFAULT_JUDGE_MODEL;
|
||||
const caseResults = await runSuite({
|
||||
modeRunner: createCliModeRunner(model),
|
||||
cases,
|
||||
runs,
|
||||
runModel,
|
||||
judgeModel: DEFAULT_JUDGE_MODEL,
|
||||
judgeModel,
|
||||
executionOnly,
|
||||
});
|
||||
|
||||
return buildRunResult({
|
||||
mode: "cli",
|
||||
runs,
|
||||
runModel,
|
||||
judgeModel: DEFAULT_JUDGE_MODEL,
|
||||
judgeModel,
|
||||
caseResults,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -212,6 +212,9 @@ describe("loadCases", () => {
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(caseEntry?.initialPath).toContain(
|
||||
"ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json"
|
||||
);
|
||||
expect(caseEntry?.toolExpect).toMatchObject({
|
||||
requiredToolsUsed: ["write_script"],
|
||||
forbiddenToolsUsed: ["deploy_workspace_item", "delete_workspace_item"],
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
import { describe, expect, it } from "bun:test";
|
||||
import { runSuite } from "./runSuite";
|
||||
import type { ModeRunner } from "./types";
|
||||
|
||||
const modeRunner: ModeRunner<undefined, undefined, { ok: boolean }> = {
|
||||
mode: "global",
|
||||
concurrency: 1,
|
||||
loadInitial: async () => undefined,
|
||||
loadExpected: async () => undefined,
|
||||
run: async () => ({
|
||||
success: true,
|
||||
actual: { ok: true },
|
||||
assistantMessageCount: 1,
|
||||
toolCallCount: 0,
|
||||
toolsUsed: [],
|
||||
skillsInvoked: [],
|
||||
tokenUsage: null,
|
||||
}),
|
||||
validate: () => [],
|
||||
};
|
||||
|
||||
describe("runSuite", () => {
|
||||
it("skips judge checks when the run disables judge scoring", async () => {
|
||||
const [caseResult] = await runSuite({
|
||||
modeRunner,
|
||||
cases: [
|
||||
{
|
||||
id: "case-1",
|
||||
prompt: "Create a draft script",
|
||||
judgeChecklist: ["the output satisfies the prompt"],
|
||||
},
|
||||
],
|
||||
runs: 1,
|
||||
runModel: "model-under-test",
|
||||
judgeModel: null,
|
||||
});
|
||||
|
||||
const [attempt] = caseResult.attempts;
|
||||
expect(attempt.passed).toBe(true);
|
||||
expect(attempt.judgeScore).toBeNull();
|
||||
expect(attempt.judgeSummary).toBeNull();
|
||||
expect(attempt.checks.map((check) => check.name)).toEqual([
|
||||
"run succeeded",
|
||||
]);
|
||||
});
|
||||
|
||||
it("only requires run success when execution-only is enabled", async () => {
|
||||
let loadExpectedCalls = 0;
|
||||
let validateCalls = 0;
|
||||
let backendValidateCalls = 0;
|
||||
|
||||
const executionOnlyRunner: ModeRunner<
|
||||
undefined,
|
||||
undefined,
|
||||
{ ok: boolean }
|
||||
> = {
|
||||
...modeRunner,
|
||||
loadExpected: async () => {
|
||||
loadExpectedCalls++;
|
||||
return undefined;
|
||||
},
|
||||
validate: () => {
|
||||
validateCalls++;
|
||||
return [{ name: "validator failed", passed: false }];
|
||||
},
|
||||
backendValidate: async () => {
|
||||
backendValidateCalls++;
|
||||
return {
|
||||
checks: [{ name: "backend validation failed", passed: false }],
|
||||
};
|
||||
},
|
||||
};
|
||||
|
||||
const [caseResult] = await runSuite({
|
||||
modeRunner: executionOnlyRunner,
|
||||
cases: [
|
||||
{
|
||||
id: "case-1",
|
||||
prompt: "Create a draft script",
|
||||
expectedPath: "fixtures/expected.json",
|
||||
toolExpect: { requiredToolsUsed: ["write_script"] },
|
||||
judgeChecklist: ["the output satisfies the prompt"],
|
||||
},
|
||||
],
|
||||
runs: 1,
|
||||
runModel: "model-under-test",
|
||||
judgeModel: "judge-model",
|
||||
executionOnly: true,
|
||||
});
|
||||
|
||||
const [attempt] = caseResult.attempts;
|
||||
expect(attempt.passed).toBe(true);
|
||||
expect(attempt.judgeScore).toBeNull();
|
||||
expect(attempt.judgeSummary).toBeNull();
|
||||
expect(attempt.checks.map((check) => check.name)).toEqual([
|
||||
"run succeeded",
|
||||
]);
|
||||
expect(loadExpectedCalls).toBe(0);
|
||||
expect(validateCalls).toBe(0);
|
||||
expect(backendValidateCalls).toBe(0);
|
||||
});
|
||||
});
|
||||
+36
-17
@@ -15,11 +15,13 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
|
||||
runs: number;
|
||||
runModel: string | null;
|
||||
judgeModel?: string | null;
|
||||
executionOnly?: boolean;
|
||||
concurrency?: number;
|
||||
verbose?: boolean;
|
||||
onProgress?: (event: FrontendBenchmarkProgressEvent) => void;
|
||||
}): Promise<BenchmarkCaseResult[]> {
|
||||
const judgeModel = input.judgeModel ?? DEFAULT_JUDGE_MODEL;
|
||||
const judgeModel =
|
||||
input.judgeModel === undefined ? DEFAULT_JUDGE_MODEL : input.judgeModel;
|
||||
const concurrency = Math.max(1, input.concurrency ?? input.modeRunner.concurrency);
|
||||
const results = new Array<BenchmarkCaseResult>(input.cases.length);
|
||||
let cursor = 0;
|
||||
@@ -52,6 +54,7 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
|
||||
runs: input.runs,
|
||||
judgeModel,
|
||||
judgeThreshold: input.modeRunner.judgeThreshold ?? 80,
|
||||
executionOnly: input.executionOnly ?? false,
|
||||
modeRunner: input.modeRunner,
|
||||
totalCases: input.cases.length,
|
||||
verbose: input.verbose ?? false,
|
||||
@@ -72,8 +75,9 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
|
||||
caseIndex: number;
|
||||
evalCase: EvalCase;
|
||||
runs: number;
|
||||
judgeModel: string;
|
||||
judgeModel: string | null;
|
||||
judgeThreshold: number;
|
||||
executionOnly: boolean;
|
||||
modeRunner: ModeRunner<TInitial, TExpected, TActual>;
|
||||
totalCases: number;
|
||||
verbose: boolean;
|
||||
@@ -99,7 +103,9 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
|
||||
|
||||
try {
|
||||
const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath);
|
||||
const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath);
|
||||
const expected = input.executionOnly
|
||||
? undefined
|
||||
: await input.modeRunner.loadExpected(input.evalCase.expectedPath);
|
||||
const run = await input.modeRunner.run(input.evalCase.prompt, initial, {
|
||||
evalCase: input.evalCase,
|
||||
caseId: input.evalCase.id,
|
||||
@@ -162,22 +168,30 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
|
||||
});
|
||||
const checks: BenchmarkCheck[] = [
|
||||
buildCheck("run succeeded", run.success, run.error),
|
||||
...input.modeRunner.validate({
|
||||
evalCase: input.evalCase,
|
||||
prompt: input.evalCase.prompt,
|
||||
initial,
|
||||
expected,
|
||||
actual: run.actual,
|
||||
run,
|
||||
}),
|
||||
...validateToolExpectations({
|
||||
run,
|
||||
toolExpect: input.evalCase.toolExpect,
|
||||
}),
|
||||
];
|
||||
if (!input.executionOnly) {
|
||||
checks.push(
|
||||
...input.modeRunner.validate({
|
||||
evalCase: input.evalCase,
|
||||
prompt: input.evalCase.prompt,
|
||||
initial,
|
||||
expected,
|
||||
actual: run.actual,
|
||||
run,
|
||||
}),
|
||||
...validateToolExpectations({
|
||||
run,
|
||||
toolExpect: input.evalCase.toolExpect,
|
||||
})
|
||||
);
|
||||
}
|
||||
const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? [];
|
||||
|
||||
if (run.success && input.modeRunner.backendValidate) {
|
||||
if (
|
||||
run.success &&
|
||||
!input.executionOnly &&
|
||||
input.modeRunner.backendValidate
|
||||
) {
|
||||
try {
|
||||
const backendValidation = await input.modeRunner.backendValidate({
|
||||
evalCase: input.evalCase,
|
||||
@@ -218,7 +232,12 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
|
||||
let judgeScore: number | null = null;
|
||||
let judgeSummary: string | null = null;
|
||||
|
||||
if (run.success && !input.evalCase.skipJudge) {
|
||||
if (
|
||||
run.success &&
|
||||
!input.executionOnly &&
|
||||
input.judgeModel !== null &&
|
||||
!input.evalCase.skipJudge
|
||||
) {
|
||||
const judge = await judgeOutput({
|
||||
mode: input.modeRunner.mode,
|
||||
prompt: input.evalCase.prompt,
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"user": {
|
||||
"username": "admin",
|
||||
"is_admin": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"user": {
|
||||
"username": "admin",
|
||||
"is_admin": true,
|
||||
"folders": ["evals"],
|
||||
"folders_read": ["evals"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"user": {
|
||||
"username": "admin",
|
||||
"is_admin": true,
|
||||
"folders": ["marketing", "data_engineering", "shared_utils"],
|
||||
"folders_read": ["marketing", "data_engineering", "shared_utils"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"user": {
|
||||
"username": "bob",
|
||||
"is_admin": false,
|
||||
"folders": ["team_a"],
|
||||
"folders_read": ["team_a", "team_b"]
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import { loadAppFixtureForEval } from "../adapters/frontend/core/app/appFixtureL
|
||||
import {
|
||||
runGlobalEval,
|
||||
type GlobalLiveEditorDraftFixture,
|
||||
type GlobalUserFixture,
|
||||
} from "../adapters/frontend/core/global/globalEvalRunner";
|
||||
import type { BenchmarkWorkspaceRunnables } from "../adapters/frontend/mockBackend";
|
||||
import type { FrontendEvalModelConfig } from "../core/models";
|
||||
@@ -15,6 +16,7 @@ import { getFrontendApiKey } from "./frontendCommon";
|
||||
export interface GlobalInitialFixture {
|
||||
workspace?: BenchmarkWorkspaceRunnables;
|
||||
liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
|
||||
user?: GlobalUserFixture;
|
||||
}
|
||||
|
||||
export function createGlobalModeRunner(
|
||||
@@ -38,6 +40,7 @@ export function createGlobalModeRunner(
|
||||
{
|
||||
workspaceFixtures: initial?.workspace,
|
||||
liveEditorDrafts: initial?.liveEditorDrafts,
|
||||
user: initial?.user,
|
||||
maxIterations: context.evalCase?.runtime?.maxTurns,
|
||||
provider: modelConfig.provider,
|
||||
model: modelConfig.model,
|
||||
@@ -104,6 +107,7 @@ async function loadGlobalInitialFixture(path: string): Promise<GlobalInitialFixt
|
||||
return {
|
||||
workspace: parsed.workspace ?? {},
|
||||
liveEditorDrafts: parsed.liveEditorDrafts ?? [],
|
||||
user: parsed.user,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n COUNT(*)::bigint AS \"total!\",\n COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS \"replacing!\"\n FROM ai_skill\n WHERE workspace_id = $1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "total!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "replacing!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"TextArray"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL AND consumed_at < now() - interval '10 minutes'\n RETURNING 1\n ) SELECT count(*) FROM del",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "count",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "0035bf99ce6fc00c7338bebfeb7e79bb9e7bc3d216b84279dee0018603965941"
|
||||
}
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "UPDATE materialized_asset_schema\n SET snapshot_id = $5, job_id = $6, captured_at = now()\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n AND version = $4",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
{
|
||||
"Custom": {
|
||||
"name": "asset_kind",
|
||||
"kind": {
|
||||
"Enum": [
|
||||
"s3object",
|
||||
"resource",
|
||||
"variable",
|
||||
"ducklake",
|
||||
"datatable",
|
||||
"volume"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"Text",
|
||||
"Int8",
|
||||
"Int8",
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT SUM(pg_database_size(datname))::BIGINT AS \"v!\" FROM pg_database",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "v!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c"
|
||||
}
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT args->>$2 FROM v2_job WHERE id = ANY($1)",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "?column?",
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"UuidArray",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca"
|
||||
}
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n SELECT language AS \"language!: _\", COUNT(*)::BIGINT AS \"count!\"\n FROM script\n WHERE archived = false AND deleted = false AND kind = 'script'\n AND (auto_kind IS NULL OR auto_kind <> 'wac')\n GROUP BY language\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "language!: _",
|
||||
"type_info": {
|
||||
"Custom": {
|
||||
"name": "script_lang",
|
||||
"kind": {
|
||||
"Enum": [
|
||||
"python3",
|
||||
"deno",
|
||||
"go",
|
||||
"bash",
|
||||
"postgresql",
|
||||
"nativets",
|
||||
"bun",
|
||||
"mysql",
|
||||
"bigquery",
|
||||
"snowflake",
|
||||
"graphql",
|
||||
"powershell",
|
||||
"mssql",
|
||||
"php",
|
||||
"bunnative",
|
||||
"rust",
|
||||
"ansible",
|
||||
"csharp",
|
||||
"oracledb",
|
||||
"nu",
|
||||
"java",
|
||||
"duckdb",
|
||||
"ruby",
|
||||
"rlang"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "count!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8"
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT kind::text as \"kind!\", parent_job, runnable_path\n FROM v2_job WHERE id = $1 AND workspace_id = $2",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "kind!",
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "parent_job",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "runnable_path",
|
||||
"type_info": "Varchar"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4"
|
||||
}
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT version, columns AS \"columns: Json<Vec<SchemaColumn>>\"\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC\n LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "version",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "columns: Json<Vec<SchemaColumn>>",
|
||||
"type_info": "Jsonb"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
{
|
||||
"Custom": {
|
||||
"name": "asset_kind",
|
||||
"kind": {
|
||||
"Enum": [
|
||||
"s3object",
|
||||
"resource",
|
||||
"variable",
|
||||
"ducklake",
|
||||
"datatable",
|
||||
"volume"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7"
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM native_retry_attempt nra WHERE NOT EXISTS (SELECT 1 FROM v2_job WHERE id = nra.job_id)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd"
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM job_perms\n WHERE ctid IN (\n SELECT jp.ctid FROM job_perms jp\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)\n LIMIT 100000\n )",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab"
|
||||
}
|
||||
+3
-3
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT 1",
|
||||
"query": "SELECT NOT pg_is_in_recovery()",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "?column?",
|
||||
"type_info": "Int4"
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
@@ -16,5 +16,5 @@
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5"
|
||||
"hash": "282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL AND consumed_at < now() - interval '10 minutes'\n RETURNING 1\n ) SELECT count(*) as \"c!\" FROM del",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "c!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "299b94a7972443267dd664c178a1704d195a7fc0d4e66e1014a18398e3a294f4"
|
||||
}
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT version, columns AS \"columns: Json<Vec<SchemaColumn>>\",\n snapshot_id, job_id, captured_at\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "version",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "columns: Json<Vec<SchemaColumn>>",
|
||||
"type_info": "Jsonb"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "snapshot_id",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 3,
|
||||
"name": "job_id",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 4,
|
||||
"name": "captured_at",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
{
|
||||
"Custom": {
|
||||
"name": "asset_kind",
|
||||
"kind": {
|
||||
"Enum": [
|
||||
"s3object",
|
||||
"resource",
|
||||
"variable",
|
||||
"ducklake",
|
||||
"datatable",
|
||||
"volume"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb"
|
||||
}
|
||||
+2
-1
@@ -42,7 +42,8 @@
|
||||
"trigger_cli",
|
||||
"trigger_nextcloud",
|
||||
"trigger_google",
|
||||
"trigger_github"
|
||||
"trigger_github",
|
||||
"data_pipeline"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = $1) AS \"exists!\"",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "exists!",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a"
|
||||
}
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ORDER BY path, created_at DESC\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "path!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "content!",
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db"
|
||||
}
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, $2)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Int4"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91"
|
||||
}
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "UPDATE v2_job_queue SET running = true WHERE id = $1",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT count(*) as \"c!\" FROM v2_job_debounce_batch",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "c!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT COUNT(*)::INT AS \"v!\" FROM pg_stat_activity",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "v!",
|
||||
"type_info": "Int4"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM script WHERE archived = false AND deleted = false AND auto_kind = 'wac'",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "count!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79"
|
||||
}
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Varchar",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e"
|
||||
}
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\"\n FROM workspace f\n JOIN workspace p ON p.id = f.parent_workspace_id\n WHERE f.id = $1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "deleted!",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM job_result_stream_v2\n WHERE job_id NOT IN (SELECT id FROM v2_job_queue)\n AND job_id NOT IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at > NOW() - INTERVAL '60 seconds'\n )\n RETURNING job_id",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "job_id",
|
||||
"type_info": "Uuid"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51"
|
||||
}
|
||||
-24
@@ -1,24 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Uuid"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Int8",
|
||||
"UuidArray"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f"
|
||||
}
|
||||
-35
@@ -1,35 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "debounced_times",
|
||||
"type_info": "Int4"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "first_started_at",
|
||||
"type_info": "Timestamptz"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "job_id_to_debounce",
|
||||
"type_info": "Uuid"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Varchar"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618"
|
||||
}
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM flow_conversation_message m\n USING flow_conversation c\n WHERE m.conversation_id = c.id AND c.workspace_id = $1 AND m.job_id = ANY($2)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"UuidArray"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c"
|
||||
}
|
||||
+2
-1
@@ -41,7 +41,8 @@
|
||||
"trigger_cli",
|
||||
"trigger_nextcloud",
|
||||
"trigger_google",
|
||||
"trigger_github"
|
||||
"trigger_github",
|
||||
"data_pipeline"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT pg_database_size(current_database())::BIGINT AS \"v!\"",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "v!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT current_setting('max_connections')::INT AS \"v!\"",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "v!",
|
||||
"type_info": "Int4"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617"
|
||||
}
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT pg_advisory_xact_lock(hashtext($1))",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "pg_advisory_xact_lock",
|
||||
"type_info": "Void"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f"
|
||||
}
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM native_retry_attempt WHERE job_id = ANY($1)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"UuidArray"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM flow WHERE archived = false",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "count!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265"
|
||||
}
|
||||
+2
-1
@@ -35,7 +35,8 @@
|
||||
"trigger_cli",
|
||||
"trigger_nextcloud",
|
||||
"trigger_google",
|
||||
"trigger_github"
|
||||
"trigger_github",
|
||||
"data_pipeline"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT attempt FROM native_retry_attempt WHERE job_id = $1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "attempt",
|
||||
"type_info": "Int4"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c"
|
||||
}
|
||||
+2
-1
@@ -42,7 +42,8 @@
|
||||
"trigger_cli",
|
||||
"trigger_nextcloud",
|
||||
"trigger_google",
|
||||
"trigger_github"
|
||||
"trigger_github",
|
||||
"data_pipeline"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT current_setting('server_version_num')::INT AS \"v!\"",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "v!",
|
||||
"type_info": "Int4"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee"
|
||||
}
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH mine AS (\n SELECT debounce_batch, consumed_by FROM v2_job_debounce_batch WHERE id = $1\n ), claim_self AS (\n UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1\n WHERE id = $1 AND consumed_at IS NULL\n RETURNING debounce_batch\n ), claim_rest AS (\n UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1\n WHERE debounce_batch = (SELECT debounce_batch FROM claim_self)\n AND id <> $1 AND consumed_at IS NULL\n RETURNING id\n )\n SELECT\n EXISTS (SELECT 1 FROM mine) AS \"had_row!\",\n (SELECT debounce_batch FROM claim_self) AS claimed_batch,\n (SELECT consumed_by FROM mine) AS prev_consumed_by,\n ARRAY(SELECT id FROM claim_rest) AS \"claimed_ids!\"\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "had_row!",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "claimed_batch",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "prev_consumed_by",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 3,
|
||||
"name": "claimed_ids!",
|
||||
"type_info": "UuidArray"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "57f375e89d63ac118c5c6767487af401e6d800488ade04facb6af14578c30a89"
|
||||
}
|
||||
+2
-1
@@ -35,7 +35,8 @@
|
||||
"trigger_cli",
|
||||
"trigger_nextcloud",
|
||||
"trigger_google",
|
||||
"trigger_github"
|
||||
"trigger_github",
|
||||
"data_pipeline"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM dispatch_event WHERE workspace_id = $1 AND producer_job_id = ANY($2)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"UuidArray"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5"
|
||||
}
|
||||
+1
-1
@@ -15,7 +15,7 @@
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55"
|
||||
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT (status = 'success' OR EXISTS (\n SELECT 1 FROM native_retry_attempt nra\n JOIN v2_job jc ON jc.id = nra.job_id\n JOIN v2_job_completed cc ON cc.id = nra.job_id\n WHERE jc.parent_job = j.id AND cc.status = 'success'\n )) AS \"success!\",\n result AS \"result: Json<Box<RawValue>>\",\n started_at AS \"started_at!\"FROM v2_job j JOIN v2_job_completed USING (id)\n WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2\n AND parent_job IS NULL\n AND runnable_path = $3\n AND j.id != $4\n ORDER BY created_at DESC\n LIMIT $5",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "success!",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "result: Json<Box<RawValue>>",
|
||||
"type_info": "Jsonb"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "started_at!",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"Text",
|
||||
"Text",
|
||||
"Uuid",
|
||||
"Int8"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391"
|
||||
}
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n SELECT\n COUNT(*) FILTER (WHERE av.raw_app = false)::BIGINT AS \"low_code!\",\n COUNT(*) FILTER (WHERE av.raw_app = true)::BIGINT AS \"raw!\"\n FROM app a\n JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "low_code!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "raw!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174"
|
||||
}
|
||||
+2
-1
@@ -46,7 +46,8 @@
|
||||
"trigger_cli",
|
||||
"trigger_nextcloud",
|
||||
"trigger_google",
|
||||
"trigger_github"
|
||||
"trigger_github",
|
||||
"data_pipeline"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL AND consumed_at < now() - interval '1 hour'\n RETURNING 1\n ) SELECT count(*) FROM del",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "count",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "70f01b322765442de8888b6d9b79984da751f9ae10c8b15ada925bd371501e18"
|
||||
}
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO ai_skill (workspace_id, name, description, instructions, edited_at, edited_by)\n VALUES ($1, $2, $3, $4, now(), $5)\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET description = EXCLUDED.description,\n instructions = EXCLUDED.instructions,\n edited_at = now(),\n edited_by = EXCLUDED.edited_by",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Varchar",
|
||||
"Varchar",
|
||||
"Text",
|
||||
"Text",
|
||||
"Varchar"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8"
|
||||
}
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM debounce_key WHERE key = $1",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT audit_logs_s3_oldest_inflight_ts() AS \"x\"",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "x",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5"
|
||||
}
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = (\n SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1\n )",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684"
|
||||
}
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT concurrency_settings, debouncing_settings, retry_settings FROM runnable_settings WHERE hash = $1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "concurrency_settings",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "debouncing_settings",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "retry_settings",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496"
|
||||
}
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM dispatch_event WHERE producer_job_id = ANY($1)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"UuidArray"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab"
|
||||
}
|
||||
-35
@@ -1,35 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "debounced_times",
|
||||
"type_info": "Int4"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "first_started_at",
|
||||
"type_info": "Timestamptz"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "job_id_to_debounce",
|
||||
"type_info": "Uuid"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid",
|
||||
"Varchar"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b"
|
||||
}
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\",\n language AS \"language!: windmill_common::scripts::ScriptLang\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ORDER BY path, created_at DESC\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "path!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "content!",
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "language!: windmill_common::scripts::ScriptLang",
|
||||
"type_info": {
|
||||
"Custom": {
|
||||
"name": "script_lang",
|
||||
"kind": {
|
||||
"Enum": [
|
||||
"python3",
|
||||
"deno",
|
||||
"go",
|
||||
"bash",
|
||||
"postgresql",
|
||||
"nativets",
|
||||
"bun",
|
||||
"mysql",
|
||||
"bigquery",
|
||||
"snowflake",
|
||||
"graphql",
|
||||
"powershell",
|
||||
"mssql",
|
||||
"php",
|
||||
"bunnative",
|
||||
"rust",
|
||||
"ansible",
|
||||
"csharp",
|
||||
"oracledb",
|
||||
"nu",
|
||||
"java",
|
||||
"duckdb",
|
||||
"ruby",
|
||||
"rlang"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d"
|
||||
}
|
||||
+4
-3
@@ -1,16 +1,17 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, false, $3, now(), now(), now())\n ON CONFLICT (name) DO UPDATE SET\n value = $2, running = false, owner = $3,\n finished_at = now(), updated_at = now()",
|
||||
"query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, false, $3, now(), now(), now())\n ON CONFLICT (name) DO UPDATE SET\n value = $2, running = false, owner = $3,\n finished_at = now(), updated_at = now()\n WHERE (background_task_state.value->>'last_xmin')::bigint <= $4",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"Jsonb",
|
||||
"Text"
|
||||
"Text",
|
||||
"Int8"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d"
|
||||
"hash": "881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829"
|
||||
}
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO materialized_asset_schema\n (workspace_id, asset_kind, asset_path, version, columns,\n snapshot_id, job_id, captured_at)\n VALUES ($1, $2, $3, $4, $5, $6, $7, now())",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Varchar",
|
||||
{
|
||||
"Custom": {
|
||||
"name": "asset_kind",
|
||||
"kind": {
|
||||
"Enum": [
|
||||
"s3object",
|
||||
"resource",
|
||||
"variable",
|
||||
"ducklake",
|
||||
"datatable",
|
||||
"volume"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"Varchar",
|
||||
"Int8",
|
||||
"Jsonb",
|
||||
"Int8",
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0"
|
||||
}
|
||||
+2
-1
@@ -35,7 +35,8 @@
|
||||
"trigger_cli",
|
||||
"trigger_nextcloud",
|
||||
"trigger_google",
|
||||
"trigger_github"
|
||||
"trigger_github",
|
||||
"data_pipeline"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM workspace WHERE deleted = false AND id NOT LIKE 'wm-fork%'",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "count!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf"
|
||||
}
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM v2_job WHERE workspace_id = $1 AND id = ANY($2))",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"UuidArray"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL\n AND consumed_at < now() - interval '10 minutes'\n AND id NOT IN (SELECT id FROM v2_job_queue)\n RETURNING 1\n ) SELECT count(*) as \"c!\" FROM del",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "c!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d"
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM v2_job_debounce_batch WHERE consumed_at IS NOT NULL",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL\n AND consumed_at < now() - interval '10 minutes'\n AND id NOT IN (SELECT id FROM v2_job_queue)\n RETURNING 1\n ) SELECT count(*) FROM del",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "count",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77"
|
||||
}
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($3::timestamptz IS NULL OR completed_at >= $3)\n ORDER BY completed_at ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING id, completed_at",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "completed_at",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Int8",
|
||||
"Timestamptz"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614"
|
||||
}
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id)\n SELECT $1, $2, owner, false, premium,\n CASE WHEN $4 THEN parent_workspace_id ELSE NULL END\n FROM workspace WHERE id = $3",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Varchar",
|
||||
"Varchar",
|
||||
"Text",
|
||||
"Bool"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d"
|
||||
}
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "WITH mine AS (\n SELECT debounce_batch, consumed_by FROM v2_job_debounce_batch WHERE id = $1\n ), claimed AS (\n -- Claim the whole batch in ONE update so concurrent same-batch\n -- survivors lock rows in identical scan order (no lock-ordering\n -- deadlock); each re-evaluates `consumed_at IS NULL` under EvalPlanQual\n -- and skips rows the other already took. A claim therefore consumes\n -- every still-unclaimed row of the batch atomically.\n UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1\n WHERE debounce_batch = (SELECT debounce_batch FROM mine)\n AND consumed_at IS NULL\n RETURNING id\n )\n SELECT\n EXISTS (SELECT 1 FROM mine) AS \"had_row!\",\n (SELECT consumed_by FROM mine) AS prev_consumed_by,\n ARRAY(SELECT id FROM claimed) AS \"claimed_ids!\",\n EXISTS (SELECT 1 FROM claimed WHERE id = $1) AS \"claimed_self!\"\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "had_row!",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "prev_consumed_by",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "claimed_ids!",
|
||||
"type_info": "UuidArray"
|
||||
},
|
||||
{
|
||||
"ordinal": 3,
|
||||
"name": "claimed_self!",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be"
|
||||
}
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b"
|
||||
}
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n SELECT\n EXISTS(SELECT 1 FROM pg_proc WHERE proname = 'aurora_version') AS \"aurora!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'rds_superuser') AS \"rds!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'cloudsqlsuperuser') AS \"cloudsql!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname IN ('azure_pg_admin', 'azuresu')) AS \"azure!\"\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "aurora!",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "rds!",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "cloudsql!",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 3,
|
||||
"name": "azure!",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db"
|
||||
}
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT EXISTS(SELECT 1 FROM v2_job_debounce_batch WHERE id = $1) as \"e!\"",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "e!",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Uuid"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c"
|
||||
}
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings, retry_settings)\n VALUES ($1, $2, $3, $4)\n ON CONFLICT (hash)\n DO NOTHING",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Int8",
|
||||
"Int8",
|
||||
"Int8"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT audit_logs_s3_oldest_inflight_ts() AS \"cutoff?\"",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "cutoff?",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e"
|
||||
}
|
||||
+2
-1
@@ -46,7 +46,8 @@
|
||||
"trigger_cli",
|
||||
"trigger_nextcloud",
|
||||
"trigger_google",
|
||||
"trigger_github"
|
||||
"trigger_github",
|
||||
"data_pipeline"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user