mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-07 16:03:21 +00:00
fix: teach the AI the raw-app job bindings, the SDK reference and the draft/deployed split (#10754)
* feat: teach the AI the raw-app job bindings and the draft/deployed split Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: scope the raw-app deploy advice to the referenced item, and stop kind-conversion from stranding fields The draft/deployed guidance added in the previous commit was read as "deploy the app too": the agent asked for both the flow and the app and routed a one-item dependency through the review-and-deploy page. Only the referenced flow or script has to exist deployed — the preview runs the app's draft — so the prompts, the `write_app_runnable` warning and the testing rule now say to offer that one deploy and leave the app a draft. `buildPersistedRunnable` spread the existing runnable when rewriting it, so converting a path runnable to inline left `runType`/`path` behind (and the reverse left `inlineScript`). `isRunnableByName` matches the inline branch first, so an app "wired to a flow" silently ran stale inline code. `test_run_app_runnable` now fills ctx-bound inputs with `$ctx:<prop>` the way RawAppBackgroundRunner does, so a ctx argument no longer arrives missing. The SDK-reference rationale claimed WM_TOKEN may be unset, that a missing base URL falls back to localhost, and that a job token is scoped enough to 403 a hand-rolled REST call. None of the three is true, and it shipped to every write-script prompt; the text now only says the client configures itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address review round on the raw-app AI instructions The eval case could pass on the exact answer it exists to reject. Every `requiredMentionsAnyOf` alternative but one was flow-agnostic, so "the app must be deployed" satisfied "must be deployed". All alternatives now name the flow, and a unit test pins that the app-only phrasing fails. `instanceLine` asserted "self-hosted Community Edition" outside the browser, where `isCloudHosted()` reads false and the license store is unset — so every global eval was told that regardless of what it pointed at. It is now emitted only under BROWSER. `assistantExpect.forbiddenMentions` defaulted a missing `assistantText` to "", which passes every entry forever on a mode whose runner does not report it. It now fails with that as the reason. `buildPersistedRunnable` carried `schema` across a retarget, so a path runnable pointed at a new flow kept the previous item's schema and `genWmillTs` typed `backend.<key>(args)` from the wrong inputs. It survives only while kind and path both match. The SDK header claimed "a function that is not listed below does not exist". `windmill-client` also exports the generated services, and the Python client exposes `Windmill.get`/`.post`, so an endpoint without a helper had no legal move. Each language now names its own escape hatch. `getAppInstructions` said the attached reference carries the TypeScript SDK even when `language: "python3"` had swapped in the Python one — on the very sentence telling the model to make that call. The kind-conversion comment claimed a hybrid runnable "silently runs stale inline code". It does not: `isRunnableByName`, `isRunnableByPath`, `convertPersistedToBackendRunnable` and `rawAppPolicy.processRunnable` all dispatch on `type` alone. The leftovers contradict the runnable's kind rather than override it, which is what the comment now says. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address round-2 review nits on the raw-app AI instructions `flow is deployed` was satisfied both by "once the flow is deployed, the button works" and by a hallucinated "done — the flow is deployed", which eval mode makes impossible and the drafts-only judge cannot see. Every alternative now states an outstanding obligation, and two more real phrasings ("will need to be deployed") are accepted so a correct answer is not failed on wording. Condenses the three comment blocks that ran past the four-line limit in AGENTS.md, and drops two claims inside them that no longer hold: the `testRunAppRunnable` doc said it runs a runnable the way the app's own frontend does (it is the editor preview, which a deployed app's stored policy does not match), and `undeployedRunnableTargets` described its argument as the write tool's raw input when the call site passes the persisted runnable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: report the real cause when a test run fails, and label the app-runnable card Driving `test_run_app_runnable` in a live session surfaced two defects the API-level check could not see. `executeTestRun` built its failure message from `error.message`, which the generated client leaves as the bare status text while the server's message sits in `body`. A path runnable aimed at an undeployed flow reported "Not Found" instead of "Not found: flow not found at name u/admin/current_time" — dropping the one diagnostic the run exists to produce. `formatToolError`, in the same file and written for exactly this, now does it. This also applies to test_run_script and test_run_flow, which had the same loss. The completion card read "Flow test completed successfully" for an app runnable, because `contextName` doubles as the jobs-tray kind and a path runnable pointing at a flow really does queue a flow job. A `completionName` override now names what ran without changing the kind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: pin the deploy expectation against wrong answers, not just correct ones `deploying the flow` was satisfied by "done deploying the flow" — a deploy the agent only claims to have made, which eval mode makes impossible and the drafts-only judge cannot see. Replaced with the prospective forms, and dropped the same reading from the workflow variant. Three review rounds each found this same class of hole in the phrasing list, so the list is now exercised against the wrong answers themselves rather than eyeballed: naming the app as what needs deploying, claiming the deploy is already done, claiming to have deployed the flow, and saying nothing about deploying all have to fail, while four real correct phrasings have to pass. The test reads the case out of global.yaml, so a future edit to the alternatives is checked by it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: drop the tense-neutral deploy alternatives and cover completed claims A gerund after a preposition carries no tense, so `before`/`after`/`by deploying the flow` all match a deploy the agent only claims to have made ("after deploying the flow, I clicked the button and it returns the greeting") just as the bare gerund did. All three are gone rather than swapped for whichever reads least badly, and the two completed-deploy phrasings are now negative fixtures. The remaining alternatives are imperative or obligational, which a claim of having already deployed cannot satisfy. Condenses the two comments this list carries: the YAML block to four lines, and the test's rationale to the durable constraint about substring matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: encrypt sensitive inputs when test-running an app runnable `test_run_app_runnable` sent `force_viewer_static_fields` but not `force_viewer_sensitive_inputs`, which every other preview path derives from the runnable's `sensitive` user fields. That list is the only thing driving the encryption loop in apps.rs, so testing a runnable with a sensitive input wrote the real value into the job's args in plaintext, readable by anyone with run access to the workspace. Verified against a running EE instance. With the list, `api_key` is stored as `$encrypted:mvqtSRI9…` and the sentinel appears nowhere in the job record; without it, the sentinel is readable in run details. A non-sensitive field is left plaintext either way. The tool claims parity with the editor preview, so it uses that same filter (`type == 'user' && sensitive`) and omits the field entirely when empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
This commit is contained in:
Guilhem
co-authored by
Claude Opus 5
Ruben Fiszel
Claude Opus 5
Ruben Fiszel
parent
ac27d0200d
commit
574775d50c
@@ -1,6 +1,7 @@
|
||||
import { mkdtemp, rm } from "fs/promises";
|
||||
import { tmpdir } from "os";
|
||||
import { join } from "path";
|
||||
import type { ChatCompletionMessageParam } from "openai/resources/chat/completions.mjs";
|
||||
import type { AIProvider } from "$lib/gen/types.gen";
|
||||
import {
|
||||
globalToolsFor,
|
||||
@@ -75,11 +76,35 @@ export interface GlobalUserFixture {
|
||||
folders_read?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Flatten every assistant turn's text. Tool calls are excluded — only what the
|
||||
* user would actually read counts as having been said to them.
|
||||
*/
|
||||
function assistantTextOf(messages: ChatCompletionMessageParam[]): string {
|
||||
const parts: string[] = [];
|
||||
for (const message of messages) {
|
||||
if (message.role !== "assistant") continue;
|
||||
const content = message.content;
|
||||
if (typeof content === "string") {
|
||||
parts.push(content);
|
||||
} else if (Array.isArray(content)) {
|
||||
for (const part of content) {
|
||||
if (part && typeof part === "object" && "text" in part) {
|
||||
parts.push(String((part as { text?: unknown }).text ?? ""));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return parts.join("\n");
|
||||
}
|
||||
|
||||
export interface GlobalEvalResult {
|
||||
success: boolean;
|
||||
state: GlobalDraftState;
|
||||
error?: string;
|
||||
assistantMessageCount: number;
|
||||
/** Everything the assistant said to the user, for `assistantExpect` checks. */
|
||||
assistantText: string;
|
||||
toolCallCount: number;
|
||||
toolsUsed: string[];
|
||||
toolCallDetails: ToolCallDetail[];
|
||||
@@ -209,6 +234,7 @@ export async function runGlobalEval(
|
||||
success: rawResult.success,
|
||||
error: rawResult.error,
|
||||
assistantMessageCount: rawResult.iterations,
|
||||
assistantText: assistantTextOf(rawResult.messages),
|
||||
toolCallCount: rawResult.toolCallsCount,
|
||||
toolsUsed: rawResult.toolsCalled,
|
||||
toolCallDetails: rawResult.toolCallDetails,
|
||||
|
||||
@@ -324,3 +324,32 @@
|
||||
- reuses the existing datatable configuration rather than creating new tables
|
||||
- presents a read-only dashboard or summary of available analytics data
|
||||
- keeps the configured datatable references available in the app artifact
|
||||
|
||||
# GIT-967, app mode: asked to track a long-running job, the agent hand-wrote a
|
||||
# runnable that fetched the jobs REST API — guessing at WM_TOKEN and a base URL
|
||||
# until it fell back to localhost — instead of using backendAsync + getJob/waitJob,
|
||||
# which the generated ./wmill bindings already provide.
|
||||
- id: app-long-job-progress
|
||||
prompt: |-
|
||||
Add a "Generate report" button. Building the report takes a few minutes, so as soon
|
||||
as the user clicks it the app should show the run's job id and keep updating its
|
||||
status until it finishes, then display the result.
|
||||
runtime:
|
||||
maxTurns: 22
|
||||
validate:
|
||||
forbiddenAppContent:
|
||||
- BASE_INTERNAL_URL
|
||||
- WM_BASE_URL
|
||||
- WM_TOKEN
|
||||
- localhost:8000
|
||||
- getResultMaybe
|
||||
- jobs/list
|
||||
- getWorkspaceToken
|
||||
- getBaseUrl
|
||||
judgeChecklist:
|
||||
- adds a Generate report button that starts the report
|
||||
- shows the run's job id as soon as the run starts
|
||||
- keeps the status updating while the run is in flight and shows the result when it completes
|
||||
- starts the run with backendAsync and tracks it with getJob, waitJob or streamJob — all three are real exports of the generated ./wmill module, alongside backend and backendAsync
|
||||
- does not write a backend runnable that polls job status or lists jobs itself
|
||||
- does not call the Windmill API with fetch from the frontend, and does not read WM_TOKEN, BASE_INTERNAL_URL or WM_BASE_URL anywhere
|
||||
|
||||
@@ -2191,3 +2191,84 @@
|
||||
# one — the judge has to be told, or it reads the "" as the value having been cleared.
|
||||
- 'the draft''s empty value string is expected and correct: a secret variable draft stores "" when no new value is staged, which is exactly right for a description-only edit, so it does NOT mean the value was cleared or changed'
|
||||
- leaves the result as an AI draft and does not deploy it
|
||||
|
||||
# Reproduces GIT-967: a hello-world flow with a React app in front of it. The agent
|
||||
# built a backend runnable that fetched the Windmill REST API directly, guessing at
|
||||
# WM_TOKEN / BASE_INTERNAL_URL / WM_BASE_URL until it fell back to localhost:8000, and
|
||||
# wired the app to a flow that was never deployed. Both are pinned here.
|
||||
- id: global-app-triggers-flow
|
||||
prompt: |-
|
||||
Create a hello world workflow at `f/evals/global/hello_world_flow` that just returns a
|
||||
greeting, and a React app at `f/evals/global/hello_world_app` with a button that runs
|
||||
that workflow and shows the result on the page.
|
||||
initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json
|
||||
runtime:
|
||||
maxTurns: 24
|
||||
validate:
|
||||
draftCountAtLeast: 2
|
||||
requiredDrafts:
|
||||
- type: flow
|
||||
path: f/evals/global/hello_world_flow
|
||||
- type: app
|
||||
path: f/evals/global/hello_world_app
|
||||
valueIncludes:
|
||||
# The frontend must go through the generated bindings, which is the only
|
||||
# credentialed way it can reach anything server-side.
|
||||
- './wmill'
|
||||
valueExcludes:
|
||||
# Every artifact of hand-rolling HTTP against the Windmill API from a
|
||||
# runnable instead of using the client or a path runnable.
|
||||
- BASE_INTERNAL_URL
|
||||
- WM_BASE_URL
|
||||
- WM_TOKEN
|
||||
- localhost:8000
|
||||
- getResultMaybe
|
||||
- jobs/list
|
||||
- getWorkspaceToken
|
||||
- getBaseUrl
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- write_flow
|
||||
- init_app
|
||||
- write_app_runnable
|
||||
judgeChecklist:
|
||||
- creates a flow at f/evals/global/hello_world_flow that returns a greeting
|
||||
- creates a React raw app at f/evals/global/hello_world_app with a button that runs the flow
|
||||
- the app actually invokes the flow rather than reimplementing its logic in an inline runnable
|
||||
# Both items staying drafts is the CORRECT outcome — the chat must not deploy without
|
||||
# being asked. What is judged is that the flow is named as the one item needing a deploy.
|
||||
- 'leaving both the flow and the app as drafts is expected and correct: the chat deploys nothing unless asked. Judge only whether the flow is identified as the single item that will need deploying, and that the app is NOT presented as needing deployment to be tried'
|
||||
# The judge has repeatedly flagged a correct app as broken over this.
|
||||
- '`/wmill.d.ts` is generated by Windmill from the app''s runnables and is deliberately absent from the app''s files — its absence is correct and is NOT a missing-module bug'
|
||||
- the app reaches the flow through a backend runnable, not through hand-written HTTP calls to the Windmill API
|
||||
- the app's frontend calls the runnable via the generated ./wmill bindings — backend, or backendAsync together with waitJob/getJob/streamJob, all of which are real exports of that module
|
||||
- does not read WM_TOKEN, BASE_INTERNAL_URL or WM_BASE_URL, and does not construct a Windmill API URL anywhere
|
||||
- does not call windmill-client functions that do not exist, such as getBaseUrl or getWorkspaceToken
|
||||
# The point of the case: a path runnable (and wmill.runFlow*) resolves the deployed
|
||||
# item, so a flow left as a draft makes the app dead on arrival and the user has to be
|
||||
# told. This can't live in judgeChecklist — the global judge only ever sees the drafts,
|
||||
# never what the assistant said.
|
||||
assistantExpect:
|
||||
# Plain substring test: an alternative must name the FLOW and read as an outstanding
|
||||
# obligation. Flow-agnostic wording is satisfied by "the app must be deployed"; tense-neutral
|
||||
# wording by a deploy the agent only claims to have made. The mirror expectation "don't ask
|
||||
# to deploy the app" can't be a forbiddenMentions entry, since correct answers negate it.
|
||||
requiredMentionsAnyOf:
|
||||
- - deploy the flow
|
||||
- deploy that flow
|
||||
- deploy this flow
|
||||
- deploy just the flow
|
||||
- deploy the workflow
|
||||
- deploy that workflow
|
||||
- deploy hello_world_flow
|
||||
- flow must be deployed
|
||||
- flow needs to be deployed
|
||||
- flow has to be deployed
|
||||
- flow needs deploying
|
||||
- flow will need to be deployed
|
||||
- flow will have to be deployed
|
||||
- once the flow is deployed
|
||||
- until the flow is deployed
|
||||
- workflow must be deployed
|
||||
- workflow needs to be deployed
|
||||
- workflow has to be deployed
|
||||
|
||||
@@ -21,6 +21,7 @@ interface RawEvalCase {
|
||||
validate?: EvalValidationSpec;
|
||||
toolExpect?: EvalCase["toolExpect"];
|
||||
cliExpect?: CliValidationSpec;
|
||||
assistantExpect?: EvalCase["assistantExpect"];
|
||||
judgeChecklist?: string[];
|
||||
skipJudge?: boolean;
|
||||
runtime?: EvalCaseRuntimeSpec;
|
||||
@@ -50,6 +51,7 @@ export async function loadCases(mode: EvalMode): Promise<EvalCase[]> {
|
||||
validate: entry.validate,
|
||||
toolExpect: entry.toolExpect,
|
||||
cliExpect: entry.cliExpect,
|
||||
assistantExpect: entry.assistantExpect,
|
||||
judgeChecklist: entry.judgeChecklist,
|
||||
skipJudge: entry.skipJudge,
|
||||
runtime: entry.runtime,
|
||||
|
||||
@@ -7,7 +7,10 @@ import type {
|
||||
FrontendBenchmarkProgressEvent,
|
||||
ModeRunner,
|
||||
} from "./types";
|
||||
import { validateToolExpectations } from "./validators";
|
||||
import {
|
||||
validateAssistantExpectations,
|
||||
validateToolExpectations,
|
||||
} from "./validators";
|
||||
|
||||
export async function runSuite<TInitial, TExpected, TActual>(input: {
|
||||
modeRunner: ModeRunner<TInitial, TExpected, TActual>;
|
||||
@@ -182,6 +185,10 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
|
||||
...validateToolExpectations({
|
||||
run,
|
||||
toolExpect: input.evalCase.toolExpect,
|
||||
}),
|
||||
...validateAssistantExpectations({
|
||||
run,
|
||||
assistantExpect: input.evalCase.assistantExpect,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
@@ -213,6 +213,24 @@ export type EvalValidationSpec =
|
||||
| AppValidationSpec
|
||||
| GlobalValidationSpec;
|
||||
|
||||
/**
|
||||
* Expectations on what the assistant SAID, for cases where the deliverable is
|
||||
* partly a warning to the user. The `global` judge only ever sees the resulting
|
||||
* drafts, so "tells the user X" is invisible to it and has to be checked here.
|
||||
* Needs a mode whose runner reports `assistantText`.
|
||||
*/
|
||||
export interface AssistantValidationSpec {
|
||||
/** Each entry: at least one of its phrases appears somewhere in the assistant's text. */
|
||||
requiredMentionsAnyOf?: string[][];
|
||||
/**
|
||||
* Plain case-insensitive substring test, so it cannot see negation: a phrase the correct
|
||||
* answer might use in the negative ("you don't need to deploy the app") is not a valid
|
||||
* entry. Use it for tokens that never legitimately appear, and leave nuanced "did the
|
||||
* assistant say the right thing" expectations to the judge checklist.
|
||||
*/
|
||||
forbiddenMentions?: string[];
|
||||
}
|
||||
|
||||
export interface EvalCase {
|
||||
id: string;
|
||||
prompt: string;
|
||||
@@ -221,6 +239,7 @@ export interface EvalCase {
|
||||
validate?: EvalValidationSpec;
|
||||
toolExpect?: ToolValidationSpec;
|
||||
cliExpect?: CliValidationSpec;
|
||||
assistantExpect?: AssistantValidationSpec;
|
||||
judgeChecklist?: string[];
|
||||
skipJudge?: boolean;
|
||||
runtime?: EvalCaseRuntimeSpec;
|
||||
@@ -287,6 +306,8 @@ export interface ModeRunOutput<TActual> {
|
||||
toolsUsed: string[];
|
||||
toolCallDetails?: ToolCallDetail[];
|
||||
skillsInvoked: string[];
|
||||
/** Concatenated assistant-visible text of the run, when the mode reports it. */
|
||||
assistantText?: string;
|
||||
tokenUsage?: BenchmarkTokenUsage | null;
|
||||
/**
|
||||
* Total input tokens occupying the context window on the LAST model request
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
import { describe, expect, it } from "bun:test";
|
||||
import { loadCases } from "./cases";
|
||||
import {
|
||||
validateAppState,
|
||||
validateCliWorkspace,
|
||||
validateGlobalState,
|
||||
validateScriptState,
|
||||
validateAssistantExpectations,
|
||||
validateToolExpectations,
|
||||
} from "./validators";
|
||||
|
||||
@@ -41,6 +43,113 @@ describe("validateScriptState", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("validateAssistantExpectations", () => {
|
||||
it("checks assistant mentions across the whole run, not just the last turn", () => {
|
||||
const run = {
|
||||
success: true,
|
||||
actual: {},
|
||||
assistantMessageCount: 2,
|
||||
toolCallCount: 0,
|
||||
toolsUsed: [],
|
||||
skillsInvoked: [],
|
||||
assistantText: "Wired the app up.\nThe flow HAS TO BE DEPLOYED before the app works.",
|
||||
};
|
||||
|
||||
const checks = validateAssistantExpectations({
|
||||
run,
|
||||
assistantExpect: {
|
||||
requiredMentionsAnyOf: [["must be deployed", "has to be deployed"], ["never said"]],
|
||||
forbiddenMentions: ["WM_TOKEN"],
|
||||
},
|
||||
});
|
||||
|
||||
expect(checks.map((c) => c.passed)).toEqual([true, false, true]);
|
||||
});
|
||||
|
||||
it("rejects a deploy claim that names the app instead of the flow", () => {
|
||||
const checks = validateAssistantExpectations({
|
||||
run: {
|
||||
success: true,
|
||||
actual: {},
|
||||
assistantMessageCount: 1,
|
||||
toolCallCount: 0,
|
||||
toolsUsed: [],
|
||||
skillsInvoked: [],
|
||||
assistantText: "Built both. The app must be deployed before the button works.",
|
||||
},
|
||||
assistantExpect: {
|
||||
requiredMentionsAnyOf: [["deploy the flow", "flow must be deployed"]],
|
||||
},
|
||||
});
|
||||
|
||||
expect(checks.map((c) => c.passed)).toEqual([false]);
|
||||
});
|
||||
|
||||
it("fails instead of passing green when the mode reports no assistant text", () => {
|
||||
const checks = validateAssistantExpectations({
|
||||
run: {
|
||||
success: true,
|
||||
actual: {},
|
||||
assistantMessageCount: 1,
|
||||
toolCallCount: 0,
|
||||
toolsUsed: [],
|
||||
skillsInvoked: [],
|
||||
},
|
||||
assistantExpect: { forbiddenMentions: ["WM_TOKEN"] },
|
||||
});
|
||||
|
||||
expect(checks.map((c) => c.passed)).toEqual([false]);
|
||||
});
|
||||
});
|
||||
|
||||
// The matcher is a plain substring test, so its failure mode is accepting an answer it should
|
||||
// reject. The real alternatives are therefore exercised against wrong answers rather than
|
||||
// eyeballed, and read out of global.yaml so an edit there cannot silently loosen them.
|
||||
describe("global-app-triggers-flow deploy expectation", () => {
|
||||
const run = (assistantText: string) => ({
|
||||
success: true,
|
||||
actual: {},
|
||||
assistantMessageCount: 1,
|
||||
toolCallCount: 0,
|
||||
toolsUsed: [],
|
||||
skillsInvoked: [],
|
||||
assistantText,
|
||||
});
|
||||
|
||||
const passes = async (assistantText: string) => {
|
||||
const cases = await loadCases("global");
|
||||
const target = cases.find((c) => c.id === "global-app-triggers-flow");
|
||||
if (!target?.assistantExpect) throw new Error("case or its assistantExpect is missing");
|
||||
const checks = validateAssistantExpectations({
|
||||
run: run(assistantText),
|
||||
assistantExpect: target.assistantExpect,
|
||||
});
|
||||
return checks.every((c) => c.passed);
|
||||
};
|
||||
|
||||
// Deploying is impossible in eval mode and the judge only sees drafts, so a claim of
|
||||
// having deployed is a hallucination this case has to reject, not evidence of success.
|
||||
it.each([
|
||||
["names the app as what needs deploying", "Built both. The app must be deployed before the button works."],
|
||||
["claims the deploy is already done", "All set — done deploying the flow, everything works now."],
|
||||
["claims it deployed the flow itself", "I deployed the flow for you, so the button works."],
|
||||
["reports a completed deploy after the fact", "After deploying the flow, I clicked the button and it returns the greeting."],
|
||||
["reports a completed deploy instrumentally", "I fixed it by deploying the flow; everything works now."],
|
||||
["says nothing about deploying", "Built the flow and the app. The button calls the flow."],
|
||||
])("rejects an answer that %s", async (_label, text) => {
|
||||
expect(await passes(text)).toBe(false);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["you'll need to deploy the flow before the app's button will work"],
|
||||
["the flow has to be deployed first; the app can stay a draft"],
|
||||
["once the flow is deployed, the button will work in the preview"],
|
||||
["want me to deploy just the flow? the app stays a draft"],
|
||||
])("accepts a correct answer: %s", async (text) => {
|
||||
expect(await passes(text)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("validateToolExpectations", () => {
|
||||
it("accepts Windmill-prefixed schedule paths", () => {
|
||||
const checks = validateToolExpectations({
|
||||
|
||||
@@ -2,6 +2,7 @@ import path from "node:path";
|
||||
import ts from "typescript";
|
||||
import type {
|
||||
AppValidationSpec,
|
||||
AssistantValidationSpec,
|
||||
BenchmarkCheck,
|
||||
CliTrace,
|
||||
CliValidationSpec,
|
||||
@@ -160,6 +161,52 @@ function valueIncludesAnyOf(value: unknown, lowercaseNeedles: string[]): boolean
|
||||
);
|
||||
}
|
||||
|
||||
export function validateAssistantExpectations(input: {
|
||||
run: ModeRunOutput<unknown>;
|
||||
assistantExpect?: AssistantValidationSpec;
|
||||
}): BenchmarkCheck[] {
|
||||
const expect = input.assistantExpect;
|
||||
if (!expect) {
|
||||
return [];
|
||||
}
|
||||
// Only some mode runners report assistantText. Defaulting a missing one to "" would pass
|
||||
// every forbiddenMentions entry forever, so a case that expects to inspect what the
|
||||
// assistant said fails on the mode that cannot show it.
|
||||
if (input.run.assistantText === undefined) {
|
||||
return [
|
||||
check(
|
||||
"assistant text is available to check",
|
||||
false,
|
||||
"this mode's runner does not report assistantText, so assistantExpect cannot be evaluated"
|
||||
),
|
||||
];
|
||||
}
|
||||
const text = input.run.assistantText;
|
||||
const checks: BenchmarkCheck[] = [];
|
||||
|
||||
for (const phrases of expect.requiredMentionsAnyOf ?? []) {
|
||||
checks.push(
|
||||
check(
|
||||
`assistant mentions one of: ${phrases.join(" / ")}`,
|
||||
phrases.some((phrase) => assistantMentions(text, phrase)),
|
||||
truncateForDetails(text)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
for (const phrase of expect.forbiddenMentions ?? []) {
|
||||
checks.push(
|
||||
check(
|
||||
`assistant does not mention '${phrase}'`,
|
||||
!assistantMentions(text, phrase),
|
||||
truncateForDetails(text)
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
return checks;
|
||||
}
|
||||
|
||||
export function validateToolExpectations(input: {
|
||||
run: ModeRunOutput<unknown>;
|
||||
toolExpect?: ToolValidationSpec;
|
||||
|
||||
@@ -70,6 +70,7 @@ export function createGlobalModeRunner(
|
||||
toolsUsed: result.toolsUsed,
|
||||
toolCallDetails: result.toolCallDetails,
|
||||
skillsInvoked: [],
|
||||
assistantText: result.assistantText,
|
||||
tokenUsage: result.tokenUsage,
|
||||
finalContextTokens: result.finalContextTokens,
|
||||
};
|
||||
|
||||
Generated
+107
-4
@@ -547,6 +547,17 @@ const result: wmill.S3Object = await wmill.writeS3File(
|
||||
|
||||
Import: import * as wmill from 'windmill-client'
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
||||
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
||||
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
||||
neither listed below nor a service method, do not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
\`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL\`. WM_END_USER_EMAIL is the app viewer when
|
||||
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
|
||||
@@ -1319,6 +1330,17 @@ const result: wmill.S3Object = await wmill.writeS3File(
|
||||
|
||||
Import: import * as wmill from 'windmill-client'
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
||||
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
||||
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
||||
neither listed below nor a service method, do not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
\`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL\`. WM_END_USER_EMAIL is the app viewer when
|
||||
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
|
||||
@@ -2185,6 +2207,17 @@ const result: wmill.S3Object = await wmill.writeS3File(
|
||||
|
||||
Import: import * as wmill from 'windmill-client'
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
||||
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
||||
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
||||
neither listed below nor a service method, do not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
\`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL\`. WM_END_USER_EMAIL is the app viewer when
|
||||
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
|
||||
@@ -3881,6 +3914,17 @@ result: S3Object = wmill.write_s3_file(
|
||||
|
||||
Import: import wmill
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The functions below are the surface to prefer. For an endpoint none of them covers,
|
||||
wmill.Windmill().get(endpoint) and .post(endpoint) issue an authenticated request against this
|
||||
instance. What does not exist is a function name you guessed at: if it is not listed below, do
|
||||
not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
\`os.environ.get("WM_END_USER_EMAIL") or os.environ.get("WM_EMAIL")\`. WM_END_USER_EMAIL is the app
|
||||
viewer when the run was triggered from an app and empty otherwise (both variables are always
|
||||
@@ -5745,16 +5789,50 @@ The frontend imports a generated module that mirrors the backend runnables. **Ne
|
||||
|
||||
### Calling backend runnables
|
||||
|
||||
Import the generated bindings and call the runnable like a function:
|
||||
Import the generated bindings and call the runnable like a function. \`./wmill\` is the **only** way the frontend reaches anything server-side — datatables, workspace items, external services. Never \`fetch\` the Windmill API from frontend code: the bundle holds no token and builds no API URL.
|
||||
|
||||
\`\`\`typescript
|
||||
| Export | Resolves to | Use it for |
|
||||
|---|---|---|
|
||||
| \`backend.<key>(args)\` | the runnable's result | the default — run and wait |
|
||||
| \`backendAsync.<key>(args)\` | the **job id** (a string) | long-running work you want to track |
|
||||
| \`waitJob(jobId)\` | the job's **result** (rejects if the job failed) | awaiting a \`backendAsync\` job |
|
||||
| \`getJob(jobId)\` | a \`Job\` (\`{ type, success, result, duration_ms, ... }\`) | polling status without blocking |
|
||||
| \`streamJob(jobId, onUpdate?)\` | the final result, calling \`onUpdate\` per chunk | showing output as it is produced |
|
||||
|
||||
Run and wait — the common case:
|
||||
|
||||
\`\`\`tsx
|
||||
import { backend } from './wmill';
|
||||
|
||||
// Call a backend runnable
|
||||
const user = await backend.get_user({ user_id: '123' });
|
||||
\`\`\`
|
||||
|
||||
The frontend cannot reach datatables, workspace items, or external services on its own — it goes through \`backend.<key>(args)\` for everything server-side.
|
||||
Start a long job, then await it:
|
||||
|
||||
\`\`\`tsx
|
||||
import { backendAsync, waitJob } from './wmill';
|
||||
|
||||
const jobId = await backendAsync.run_report({ month: '2026-08' }); // a string
|
||||
const report = await waitJob(jobId); // the result itself
|
||||
\`\`\`
|
||||
|
||||
Or poll it without blocking, to render progress:
|
||||
|
||||
\`\`\`tsx
|
||||
import { getJob } from './wmill';
|
||||
|
||||
const job = await getJob(jobId);
|
||||
if (job.type === 'CompletedJob') setReport(job.result);
|
||||
\`\`\`
|
||||
|
||||
\`backendAsync\` resolves a job id and nothing else — guard on it before storing or polling. A poll loop started on an \`undefined\` id never completes and shows as a row stuck "running" forever:
|
||||
|
||||
\`\`\`tsx
|
||||
const jobId = await backendAsync.run_report(args);
|
||||
if (!jobId) throw new Error('run_report did not start a job');
|
||||
\`\`\`
|
||||
|
||||
**Never hand-write a job-polling runnable.** A backend runnable that calls \`jobs/list\`, or that returns \`getResultMaybe(...)\` for the frontend to poll, reimplements \`backendAsync\` + \`waitJob\` / \`getJob\` / \`streamJob\` — and it is what leads to guessing at base URLs and tokens.
|
||||
|
||||
### Keeping data out of recorded demos
|
||||
|
||||
@@ -5806,10 +5884,33 @@ def main(user_id: str):
|
||||
return user
|
||||
\`\`\`
|
||||
|
||||
#### The \`wmill\` client is already authenticated
|
||||
|
||||
An inline runnable runs as an ordinary Windmill job. \`import * as wmill from 'windmill-client'\` (TypeScript) and \`import wmill\` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
|
||||
|
||||
**Don't read \`WM_TOKEN\` or \`BASE_INTERNAL_URL\` and build an API URL to \`fetch\`.** The client's own \`setClient\` already reads exactly those, and it also sets the credentials mode a raw app needs (\`WM_RAW_APP\` suppresses credentials, because a sandboxed bundle calls the API from an opaque origin that can never pair with \`Access-Control-Allow-Origin: *\`). Rebuilding that by hand drops the parts you can't see. Use \`wmill.*\` for everything Windmill, and \`fetch\` only for third-party APIs.
|
||||
|
||||
Prefer the \`wmill\` functions that appear in the SDK reference; for an endpoint none of them covers, the generated service classes (\`JobService\`, \`ScriptService\`, ...) are importable from \`windmill-client\`. What is not available is a name you guessed at: \`getBaseUrl\` and \`getWorkspaceToken\` are inventions, not API.
|
||||
|
||||
### Path runnables (script / flow / hubscript)
|
||||
|
||||
When \`type\` is \`script\`, \`flow\`, or \`hubscript\`, the runnable just stores a \`path\` to an existing workspace or hub item — no inline code. The referenced item's input/output schema becomes the runnable's surface.
|
||||
|
||||
### Draft code vs deployed code
|
||||
|
||||
This decides whether an app works before anything is deployed:
|
||||
|
||||
- **Inline runnables run the app's current code.** The editor sends the runnable's source with each request, so an inline runnable works in the preview with nothing deployed.
|
||||
- **Path runnables (\`script\` / \`flow\` / \`hubscript\`) run the DEPLOYED item at that path.** So do \`wmill.runFlow\`, \`wmill.runFlowAsync\` and \`wmill.runScriptByPath\` called from inside a runnable. A draft — including a draft you just created — does not exist for them.
|
||||
|
||||
So an app wired to a flow you just wrote does nothing until **that flow is deployed**. The app itself does NOT have to be deployed for this: the preview runs the app's draft, so the referenced flow is the only thing that has to exist deployed.
|
||||
|
||||
That makes the fix a one-item deploy, not a release. Offer to deploy exactly the referenced flow or script and leave the app a draft the user keeps testing in the preview — do not push the whole change set through the review-and-deploy page, and do not ask the user to deploy the app, unless they said they want to ship it.
|
||||
|
||||
Do NOT quietly reimplement the flow inside an inline runnable to dodge the deployment: that leaves the user with two copies of the same logic and an app that ignores the flow they asked for. Inline the logic only when the user actually wants it inline.
|
||||
|
||||
Prefer a **path runnable of type \`flow\`** over an inline runnable that calls \`wmill.runFlowAsync\`. The path runnable gives the frontend the flow's real input schema and works with \`backend\` / \`backendAsync\` / \`waitJob\` like any other runnable; a hand-written wrapper gives up all of that.
|
||||
|
||||
### Static inputs
|
||||
|
||||
\`staticInputs\` is an optional \`Record<string, any>\` for arguments not overridable from the frontend. Useful with path runnables to pre-fill some args while leaving the rest to the frontend caller.
|
||||
@@ -5871,6 +5972,8 @@ def main(user_id: str):
|
||||
4. **Use descriptive keys** — \`get_user\`, not \`a\`.
|
||||
5. **Always whitelist tables** — adding a runnable that queries a new table requires the table to be in \`data.tables\` first.
|
||||
6. **Mark sensitive UI with \`data-wm-no-record\`** — it is what keeps that data out of a recorded demo; passwords are handled for you.
|
||||
7. **Reach for \`backendAsync\` + \`waitJob\`** for long work — never a hand-written job-polling runnable.
|
||||
8. **Deploy what a path runnable points at** — a path runnable aimed at a draft fails at runtime; tell the user what needs deploying.
|
||||
`,
|
||||
"triggers": `---
|
||||
name: triggers
|
||||
|
||||
@@ -1,7 +1,12 @@
|
||||
import { BROWSER } from 'esm-env'
|
||||
|
||||
export function isCloudHosted(): boolean {
|
||||
return BROWSER && window.location.hostname == 'app.windmill.dev'
|
||||
// `BROWSER` alone isn't enough: it resolves true wherever the browser export
|
||||
// condition is picked (SSR bundles, vitest's server project), where `window`
|
||||
// may be missing or a stub with no `location`. Same defensive shape as
|
||||
// `isChromiumBrowser`.
|
||||
if (!BROWSER) return false
|
||||
return globalThis.window?.location?.hostname == 'app.windmill.dev'
|
||||
}
|
||||
|
||||
// On the managed cloud, the public demo workspace is kept clean and consistent by
|
||||
|
||||
@@ -1113,6 +1113,12 @@ await backend.myFunction()
|
||||
|
||||
When you are using the windmill-client, do not forget that as id for variables or resources, those are path that are of the form \'u/<user>/<name>\' or \'f/<folder>/<name>\'.
|
||||
|
||||
Besides \`backend\`, the generated \`./wmill\` module exports \`backendAsync.<key>(args)\` (resolves the job id as a string), \`waitJob(jobId)\` (resolves that job's result, rejects if it failed), \`getJob(jobId)\` (the current job state, for rendering progress) and \`streamJob(jobId, onUpdate)\`. Use \`backendAsync\` + \`waitJob\`/\`getJob\` for long-running work — never hand-write a runnable that polls job status, and never \`fetch\` the Windmill API from frontend code, which holds no token.
|
||||
|
||||
A \`script\`/\`flow\` runnable runs the DEPLOYED item at that path, and so do \`wmill.runFlowAsync\`/\`wmill.runScriptByPath\` called inside a runnable — a draft is invisible to them, so an app pointed at an undeployed flow fails at runtime. The app itself does not need deploying — the preview runs its draft — so the fix is to deploy that one referenced item, not the whole change set. Say so instead of working around it, and never reimplement the flow inline to dodge the deployment. An \`inline\` runnable runs the app's own code and needs nothing deployed.
|
||||
|
||||
Inside an inline runnable the \`wmill\` client configures itself from the job environment: don't read \`WM_TOKEN\` or \`BASE_INTERNAL_URL\` and build an API URL by hand — the client already does that, plus the credentials mode a raw app needs. Only call \`wmill\` functions that actually exist; \`getBaseUrl\` and \`getWorkspaceToken\` are inventions.
|
||||
|
||||
## Instructions
|
||||
|
||||
1. Use the smallest context needed. If the target file or runnable is clear, inspect only that item with \`get_frontend_file(path)\` or \`get_backend_runnable(key)\`.
|
||||
|
||||
@@ -187,6 +187,7 @@ vi.mock('$lib/gen', async () => {
|
||||
listAzureTriggers: vi.fn(async () => [])
|
||||
}),
|
||||
AppService: wrapService(actual.AppService, {
|
||||
executeComponent: vi.fn(async () => 'job-app-component'),
|
||||
existsApp: vi.fn(async () => false),
|
||||
createAppRaw: vi.fn(async () => 'created'),
|
||||
updateAppRaw: vi.fn(async () => 'updated'),
|
||||
@@ -1803,6 +1804,261 @@ describe('global AI tools', () => {
|
||||
expect(getBackendDraft('raw_app', 'u/admin/live_app', { workspace: WORKSPACE })).toBeUndefined()
|
||||
})
|
||||
|
||||
// A path runnable executes the DEPLOYED item, so pointing one at a draft-only flow
|
||||
// produces an app that silently does nothing. The write still succeeds — flow and app
|
||||
// are normally built together — but the model has to be told what is missing.
|
||||
it('warns when a path runnable points at an item that is not deployed', async () => {
|
||||
seedBackendDraft(
|
||||
'raw_app',
|
||||
'u/admin/wired_app',
|
||||
{ summary: 'Wired app', files: {}, runnables: {}, data: { tables: [] } },
|
||||
{ workspace: WORKSPACE }
|
||||
)
|
||||
|
||||
vi.mocked(FlowService.existsFlowByPath).mockResolvedValueOnce(false)
|
||||
const undeployed = JSON.parse(
|
||||
await callGlobalTool('write_app_runnable', {
|
||||
path: 'u/admin/wired_app',
|
||||
key: 'run_flow',
|
||||
runnable: { name: 'Run the flow', type: 'flow', path: 'u/admin/hello_flow' }
|
||||
})
|
||||
)
|
||||
expect(undeployed.success).toBe(true)
|
||||
expect(undeployed.warning).toContain('u/admin/hello_flow')
|
||||
expect(undeployed.warning).toContain('NOT deployed')
|
||||
// The remedy is one item, not a release: the app runs its draft in the preview.
|
||||
expect(undeployed.warning).toContain('deploy_workspace_item')
|
||||
|
||||
vi.mocked(FlowService.existsFlowByPath).mockResolvedValueOnce(true)
|
||||
const deployed = JSON.parse(
|
||||
await callGlobalTool('write_app_runnable', {
|
||||
path: 'u/admin/wired_app',
|
||||
key: 'run_flow',
|
||||
runnable: { name: 'Run the flow', type: 'flow', path: 'u/admin/hello_flow' }
|
||||
})
|
||||
)
|
||||
expect(deployed.success).toBe(true)
|
||||
expect(deployed.warning).toBeUndefined()
|
||||
|
||||
// A script target resolves through existsScriptByPath, then the archived-inclusive
|
||||
// getScriptByPath fallback — a 404 there is the only thing that means "not deployed".
|
||||
vi.mocked(ScriptService.existsScriptByPath).mockResolvedValueOnce(false)
|
||||
vi.mocked(ScriptService.getScriptByPath).mockRejectedValueOnce(
|
||||
Object.assign(new Error('not found'), { status: 404 })
|
||||
)
|
||||
const scriptTarget = JSON.parse(
|
||||
await callGlobalTool('write_app_runnable', {
|
||||
path: 'u/admin/wired_app',
|
||||
key: 'run_script',
|
||||
runnable: { name: 'Run the script', type: 'script', path: 'u/admin/hello_script' }
|
||||
})
|
||||
)
|
||||
expect(scriptTarget.warning).toContain('u/admin/hello_script')
|
||||
|
||||
// A hub script lives outside the workspace and has no deployed/draft distinction,
|
||||
// so it must never be probed or warned about.
|
||||
const hub = JSON.parse(
|
||||
await callGlobalTool('write_app_runnable', {
|
||||
path: 'u/admin/wired_app',
|
||||
key: 'run_hub',
|
||||
runnable: { name: 'Hub', type: 'hubscript', path: 'hub/123/slack/send' }
|
||||
})
|
||||
)
|
||||
expect(hub.warning).toBeUndefined()
|
||||
})
|
||||
|
||||
// The execute_component payload is what makes this tool faithful to how the app
|
||||
// really runs: force_viewer_static_fields is what selects preview mode server-side
|
||||
// (apps.rs `is_preview`), and it must be sent even when there are no static fields.
|
||||
it('runs an inline app runnable as a preview job with its draft code', async () => {
|
||||
seedBackendDraft(
|
||||
'raw_app',
|
||||
'u/admin/tested_app',
|
||||
{
|
||||
summary: 'Tested app',
|
||||
files: {},
|
||||
runnables: {
|
||||
greet: {
|
||||
name: 'Greet',
|
||||
type: 'inline',
|
||||
inlineScript: { language: 'bun', content: 'export async function main() { return 1 }' },
|
||||
fields: {
|
||||
who: { type: 'ctx', ctx: 'email' },
|
||||
fixed: { type: 'static', value: 7 },
|
||||
api_key: { type: 'user', sensitive: true },
|
||||
plain: { type: 'user' }
|
||||
}
|
||||
}
|
||||
},
|
||||
data: { tables: [] }
|
||||
},
|
||||
{ workspace: WORKSPACE }
|
||||
)
|
||||
|
||||
await callGlobalTool('test_run_app_runnable', {
|
||||
path: 'u/admin/tested_app',
|
||||
key: 'greet',
|
||||
args: { name: 'ada' }
|
||||
})
|
||||
|
||||
const body = vi.mocked(AppService.executeComponent).mock.calls.at(-1)?.[0].requestBody as any
|
||||
expect(body.force_viewer_static_fields).toEqual({ fixed: 7 })
|
||||
// A ctx-bound input is resolved server-side; sending it absent would fail the run
|
||||
// for a reason unrelated to the runnable's code.
|
||||
expect(body.args).toEqual({ name: 'ada', who: '$ctx:email' })
|
||||
expect(body.raw_code).toMatchObject({ language: 'bun' })
|
||||
expect(body.path).toBeUndefined()
|
||||
// Only names listed here get encrypted before the args are queued, so a sensitive
|
||||
// field left out of it is stored in plaintext for anyone with run access to read.
|
||||
expect(body.force_viewer_sensitive_inputs).toEqual(['api_key'])
|
||||
})
|
||||
|
||||
// The undeployed-flow 404 is the whole reason this tool exists, and the generated client
|
||||
// leaves the server's message in `body` while `message` is the bare status text.
|
||||
it("surfaces the server's message when a path runnable's target is not deployed", async () => {
|
||||
seedBackendDraft(
|
||||
'raw_app',
|
||||
'u/admin/broken_app',
|
||||
{
|
||||
summary: 'Broken app',
|
||||
files: {},
|
||||
runnables: {
|
||||
go: { name: 'Go', type: 'path', runType: 'flow', path: 'u/admin/never_deployed' }
|
||||
},
|
||||
data: { tables: [] }
|
||||
},
|
||||
{ workspace: WORKSPACE }
|
||||
)
|
||||
vi.mocked(AppService.executeComponent).mockRejectedValueOnce({
|
||||
status: 404,
|
||||
message: 'Not Found',
|
||||
body: 'Not found: flow not found at name u/admin/never_deployed'
|
||||
})
|
||||
|
||||
await expect(
|
||||
callGlobalTool('test_run_app_runnable', { path: 'u/admin/broken_app', key: 'go' })
|
||||
).rejects.toThrow(/flow not found at name u\/admin\/never_deployed/)
|
||||
})
|
||||
|
||||
it('runs a path app runnable against the deployed item it names', async () => {
|
||||
seedBackendDraft(
|
||||
'raw_app',
|
||||
'u/admin/wired_app2',
|
||||
{
|
||||
summary: 'Wired app',
|
||||
files: {},
|
||||
runnables: {
|
||||
run_flow: { name: 'Run', type: 'path', runType: 'flow', path: 'u/admin/hello_flow' }
|
||||
},
|
||||
data: { tables: [] }
|
||||
},
|
||||
{ workspace: WORKSPACE }
|
||||
)
|
||||
|
||||
await callGlobalTool('test_run_app_runnable', { path: 'u/admin/wired_app2', key: 'run_flow' })
|
||||
|
||||
const body = vi.mocked(AppService.executeComponent).mock.calls.at(-1)?.[0].requestBody as any
|
||||
expect(body.path).toBe('flow/u/admin/hello_flow')
|
||||
expect(body.raw_code).toBeUndefined()
|
||||
// Absent rather than [], matching what the editor preview sends.
|
||||
expect(body.force_viewer_sensitive_inputs).toBeUndefined()
|
||||
})
|
||||
|
||||
// A hybrid runnable — inline code plus a leftover runType/path — contradicts its own
|
||||
// kind, and convertPersistedToBackendRunnable is what reports it back to the model.
|
||||
it('drops the other kind\'s fields when a runnable changes type', async () => {
|
||||
seedBackendDraft(
|
||||
'raw_app',
|
||||
'u/admin/converted_app',
|
||||
{ summary: 'Converted', files: {}, runnables: {}, data: { tables: [] } },
|
||||
{ workspace: WORKSPACE }
|
||||
)
|
||||
|
||||
await callGlobalTool('write_app_runnable', {
|
||||
path: 'u/admin/converted_app',
|
||||
key: 'go',
|
||||
runnable: { name: 'Run the flow', type: 'flow', path: 'u/admin/hello_flow' }
|
||||
})
|
||||
await callGlobalTool('write_app_runnable', {
|
||||
path: 'u/admin/converted_app',
|
||||
key: 'go',
|
||||
runnable: {
|
||||
name: 'Now inline',
|
||||
type: 'inline',
|
||||
inlineScript: { language: 'bun', content: 'export async function main() { return 1 }' }
|
||||
}
|
||||
})
|
||||
|
||||
const asInline = getBackendDraft<any>('raw_app', 'u/admin/converted_app', {
|
||||
workspace: WORKSPACE
|
||||
}).runnables.go
|
||||
expect(asInline.type).toBe('inline')
|
||||
expect(asInline.runType).toBeUndefined()
|
||||
expect(asInline.path).toBeUndefined()
|
||||
|
||||
await callGlobalTool('write_app_runnable', {
|
||||
path: 'u/admin/converted_app',
|
||||
key: 'go',
|
||||
runnable: { name: 'Back to flow', type: 'flow', path: 'u/admin/hello_flow' }
|
||||
})
|
||||
|
||||
const asPath = getBackendDraft<any>('raw_app', 'u/admin/converted_app', {
|
||||
workspace: WORKSPACE
|
||||
}).runnables.go
|
||||
expect(asPath.type).toBe('path')
|
||||
expect(asPath.runType).toBe('flow')
|
||||
expect(asPath.inlineScript).toBeUndefined()
|
||||
})
|
||||
|
||||
it("resets a path runnable's schema when it is retargeted, and keeps it when it is not", async () => {
|
||||
// The editor populates `schema` from the item the runnable points at, and
|
||||
// genWmillTs types `backend.<key>(args)` from it — so it must not outlive the target.
|
||||
const flowSchema = {
|
||||
type: 'object',
|
||||
properties: { old_arg: { type: 'string' } }
|
||||
}
|
||||
seedBackendDraft(
|
||||
'raw_app',
|
||||
'u/admin/retargeted_app',
|
||||
{
|
||||
summary: 'Retargeted',
|
||||
files: {},
|
||||
runnables: {
|
||||
go: {
|
||||
name: 'Run the flow',
|
||||
type: 'path',
|
||||
runType: 'flow',
|
||||
path: 'u/admin/first_flow',
|
||||
fields: {},
|
||||
schema: flowSchema
|
||||
}
|
||||
},
|
||||
data: { tables: [] }
|
||||
},
|
||||
{ workspace: WORKSPACE }
|
||||
)
|
||||
|
||||
await callGlobalTool('write_app_runnable', {
|
||||
path: 'u/admin/retargeted_app',
|
||||
key: 'go',
|
||||
runnable: { name: 'Run the flow', type: 'flow', path: 'u/admin/first_flow' }
|
||||
})
|
||||
expect(
|
||||
getBackendDraft<any>('raw_app', 'u/admin/retargeted_app', { workspace: WORKSPACE }).runnables
|
||||
.go.schema
|
||||
).toEqual(flowSchema)
|
||||
|
||||
await callGlobalTool('write_app_runnable', {
|
||||
path: 'u/admin/retargeted_app',
|
||||
key: 'go',
|
||||
runnable: { name: 'Run the other flow', type: 'flow', path: 'u/admin/second_flow' }
|
||||
})
|
||||
expect(
|
||||
getBackendDraft<any>('raw_app', 'u/admin/retargeted_app', { workspace: WORKSPACE }).runnables
|
||||
.go.schema
|
||||
).toEqual({})
|
||||
})
|
||||
|
||||
it('does not echo the app value back to the model on write', async () => {
|
||||
const sentinel = 'SENTINEL_DO_NOT_ECHO_DEADBEEF'
|
||||
seedBackendDraft(
|
||||
|
||||
@@ -56,6 +56,8 @@ import { sanitizeAttachmentName, textLineCount, type AttachedTextFile } from '..
|
||||
import { modelSupportsVision } from '../../modelConfig'
|
||||
import { tryGetCurrentModel } from '$lib/aiStore'
|
||||
import { isChromiumBrowser } from '$lib/utils'
|
||||
import { isCloudHosted } from '$lib/cloud'
|
||||
import { BROWSER } from 'esm-env'
|
||||
import {
|
||||
applyEditableFlowJsonToFlow,
|
||||
buildEditableFlowJson,
|
||||
@@ -114,6 +116,7 @@ import { fileTools } from '../files/fileTools'
|
||||
import type { AttachedFilesStore } from '../files/attachedFiles.svelte'
|
||||
import { artifactTools } from '../artifacts/artifactTools'
|
||||
import type { SessionArtifactsStore } from '../artifacts/artifactsState.svelte'
|
||||
import type { Runnable } from '$lib/components/apps/inputType'
|
||||
import { UserDraft } from '$lib/userDraft.svelte'
|
||||
import { emptySchema } from '$lib/utils'
|
||||
import { inferArgs } from '$lib/infer'
|
||||
@@ -295,7 +298,7 @@ const getInstructionsSchema = z.object({
|
||||
language: scriptLangSchema
|
||||
.optional()
|
||||
.describe(
|
||||
'The target language. Required when subject is script. For subject "datatable" it selects which SDK to return (e.g. "bun" for TypeScript, "python3" for Python) and defaults to TypeScript if omitted. Use the existing language when modifying, or the requested target language when creating. Other subjects ignore it.'
|
||||
'The target language. Required when subject is script. For subjects "datatable" and "app" it selects which SDK reference to return (e.g. "bun" for TypeScript, "python3" for Python) and defaults to TypeScript if omitted. Use the existing language when modifying, or the requested target language when creating. Other subjects ignore it.'
|
||||
)
|
||||
})
|
||||
|
||||
@@ -1029,6 +1032,21 @@ const deleteAppRunnableSchema = z.object({
|
||||
key: z.string().describe('Key of the backend runnable to remove.')
|
||||
})
|
||||
|
||||
const testRunAppRunnableSchema = z.object({
|
||||
path: z.string().describe('Workspace path of the app.'),
|
||||
key: z.string().describe('Key of the backend runnable to run.'),
|
||||
args: testRunArgsSchema,
|
||||
background: backgroundArgSchema,
|
||||
wait_seconds: waitSecondsArgSchema
|
||||
})
|
||||
|
||||
const testRunAppRunnableToolDef = createToolDef(
|
||||
testRunAppRunnableSchema,
|
||||
'test_run_app_runnable',
|
||||
"Run one of an app's backend runnables with test arguments, the way the editor preview runs it. An inline runnable executes the draft code; a path runnable executes the deployed script/flow it points at. A deployed app runs under its stored policy instead, so this does not prove the runnable is reachable once deployed.",
|
||||
{ strict: false }
|
||||
)
|
||||
|
||||
const openPreviewSchema = z.object({
|
||||
kind: z
|
||||
.enum(['script', 'flow', 'raw_app', 'pipeline'])
|
||||
@@ -1224,9 +1242,21 @@ const buildGlobalSystemPrompt = (
|
||||
? '\n- If the user message includes an ACTIVE PREVIEW section, that is the page the side panel is showing — resolve "this page", "here" and "it" against it, and against `open` (the row the page is anchored at, whose drawer the user opened) when there is one. It already tells you what get_preview_status would, so do not call that tool to learn what is on screen; call it only to check the panel\'s *other* tabs.'
|
||||
: ''
|
||||
const pipelineBullet = `- A "data pipeline" is NOT a flow: it is a DAG of independent scripts in one folder, wired by storage assets (DuckLake/data tables/S3) and triggers via top-of-file \`pipeline\` / \`on <ref>\` annotation comments written in each script's comment syntax (\`--\` for SQL, \`#\` for Python/Bash, \`//\` for TS — a \`//\` line in a SQL node is a syntax error). When the user asks for a data pipeline (or to ingest/transform/materialize data across steps), call get_instructions with subject "pipeline" and build annotated script drafts — do not build a flow.${pipelineAlphaNote}`
|
||||
// Hosting and edition come from the hostname and a store the app populates at init, so
|
||||
// they are knowable only in the browser: a non-browser caller reads false for both and
|
||||
// would be told "self-hosted Community Edition" whatever it targets. No base URL here
|
||||
// either — offering one is what invites a hardcoded URL into a script.
|
||||
const instanceLine = BROWSER
|
||||
? ` This is ${
|
||||
isCloudHosted() ? 'Windmill Cloud (app.windmill.dev)' : 'a self-hosted Windmill instance'
|
||||
}, running ${
|
||||
get(enterpriseLicense) ? 'Enterprise Edition' : 'Community Edition'
|
||||
}. Use that to judge whether a feature is available before promising it.`
|
||||
: ''
|
||||
|
||||
return `You are Windmill's global workspace assistant.
|
||||
|
||||
The current user's workspace username is "${username}".
|
||||
The current user's workspace username is "${username}".${instanceLine}
|
||||
|
||||
Use tools to inspect workspace items and create per-user drafts (saved server-side, visible only to this user — not deployed) for scripts, flows, schedules, triggers, resources, variables, and raw apps.
|
||||
|
||||
@@ -1252,6 +1282,7 @@ Rules:
|
||||
- Use get_instructions before writing scripts, flows, resources, or apps. For scripts, pass the target language.
|
||||
${pipelineBullet}
|
||||
- After creating or editing a script or flow draft, run test_run_script, test_run_flow, or test_run_step with representative args before reporting that it works. These tools prefer drafts, so testing does not require deployment.
|
||||
- Do the same for a raw app: run test_run_app_runnable on each backend runnable you wrote or changed before saying the app works. A bundle that compiles proves nothing about whether the runnables run. An inline runnable executes the app's draft code; a path runnable executes the DEPLOYED script/flow it names, so a path runnable aimed at something you have not deployed fails here — that failure is the point: report it and offer to deploy that one target. The app itself does not need deploying to be tested.
|
||||
- Use list_runs to find recent runs (optionally filtered by path, creator, label, or status), then get_job_logs with a returned id to inspect a specific run's logs — without starting a new test run.
|
||||
- To see what a flow run actually did per step — statuses and results across the whole execution tree, subflow steps and loop iterations included — use get_flow_run_details with the run id (it also works while the flow is still running). Pass step to read one step's result in full (capped at 12k chars). Prefer it over get_job_logs when you need step results rather than logs.
|
||||
- Use open_page to show a workspace page with filters applied — Runs, Schedules, Variables, Resources, Assets, Audit logs, or Workspace settings on a specific tab (e.g. "open the failed runs of f/foo/bar", "open the schedule for X", "open the git sync settings"). Carry over every filter the user described — Runs takes the page's whole filter set (time window, path, user, folder, label, tag, worker, trigger kind, args/result, ...), so don't drop a criterion just because it wasn't in the request's main clause. Only the pages listed for this user in the tool are available; don't offer pages that aren't listed. Don't use it as a substitute for list_runs when you just need the data yourself.
|
||||
@@ -1550,12 +1581,26 @@ function buildPersistedRunnable(
|
||||
)
|
||||
: (existing?.fields ?? {})
|
||||
|
||||
// Converting between kinds must drop the other kind's fields. Every consumer dispatches on
|
||||
// `type` (isRunnableByName / isRunnableByPath), so a leftover does not change which code
|
||||
// runs — it contradicts it: a runnable reported back as inline still names a flow in
|
||||
// `path`, and that is what the model reads on its next turn.
|
||||
const { runType: _runType, path: _path, inlineScript: _inlineScript, ...carried } = existing ?? {}
|
||||
|
||||
// `schema` describes the item a path runnable points at, so it only survives while that
|
||||
// target is unchanged. Carried across a retarget it types `backend.<key>(args)` from the
|
||||
// previous item's inputs, because hiddenRunnableToTsType reads it for the path branch.
|
||||
const sameTarget =
|
||||
(existing?.type === 'path' || existing?.type === 'runnableByPath') &&
|
||||
existing?.runType === input.type &&
|
||||
existing?.path === input.path
|
||||
|
||||
if (input.type === 'inline') {
|
||||
if (!input.inlineScript) {
|
||||
throw new Error('inlineScript is required when runnable type is "inline".')
|
||||
}
|
||||
return {
|
||||
...(existing ?? {}),
|
||||
...carried,
|
||||
name: input.name,
|
||||
type: 'inline',
|
||||
inlineScript: {
|
||||
@@ -1570,13 +1615,13 @@ function buildPersistedRunnable(
|
||||
throw new Error('path is required when runnable type is "script", "flow", or "hubscript".')
|
||||
}
|
||||
return {
|
||||
...(existing ?? {}),
|
||||
...carried,
|
||||
name: input.name,
|
||||
type: 'path',
|
||||
runType: input.type,
|
||||
path: input.path,
|
||||
fields,
|
||||
schema: existing?.schema ?? {}
|
||||
schema: sameTarget ? (existing?.schema ?? {}) : {}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2097,7 +2142,14 @@ ${getFlowPrompt()}`
|
||||
|
||||
type InstructionSubject = (typeof ALL_INSTRUCTION_SUBJECTS)[number]
|
||||
|
||||
function getAppInstructions(): string {
|
||||
function getAppInstructions(language?: ScriptLang): string {
|
||||
// getRawAppPrompt swaps in the Python SDK for python3, so the pointer has to swap with
|
||||
// it: telling the model to re-fetch a reference it is already holding wastes a turn, and
|
||||
// naming the wrong SDK contradicts the text right below the sentence.
|
||||
const sdkLine =
|
||||
language === 'python3'
|
||||
? '- The authoring reference below carries the Python SDK, for `python3` inline runnables. For a `bun` runnable, call `get_instructions` again with `subject: "app"` and no language.'
|
||||
: '- The authoring reference below carries the TypeScript SDK. For a `python3` runnable, call `get_instructions` again with `subject: "app"` and `language: "python3"`.'
|
||||
return `# Global draft app instructions
|
||||
|
||||
- Global mode edits raw app drafts only; it does not save or deploy unless the user explicitly asks to deploy.
|
||||
@@ -2108,6 +2160,8 @@ function getAppInstructions(): string {
|
||||
- Backend inline runnables are addressed as \`backend/<key>/main.{ts|py}\` from the file tools, but you create or update them via \`write_app_runnable\` / \`delete_app_runnable\` (which take the runnable shape directly: \`{ name, type, inlineScript?, path?, staticInputs? }\`).
|
||||
- \`/wmill.d.ts\` (or \`wmill.ts\`) is generated automatically from the backend runnables — never write it directly.
|
||||
- Inline runnables only support \`bun\` or \`python3\` in chat. Path runnables (\`script\`/\`flow\`/\`hubscript\`) reference an existing item.
|
||||
- Inline runnables run the app's DRAFT code, so they work in the preview with nothing deployed. Path runnables — and \`wmill.runFlow*\` / \`runScriptByPath\` called from inside any runnable — run the DEPLOYED item at that path, and a draft is invisible to them. An app wired to a flow you just drafted does nothing until that flow is deployed — but the APP does not have to be deployed for that: the preview runs its draft. So offer to deploy just the referenced flow/script with deploy_workspace_item and leave the app a draft the user keeps testing in the preview; don't route a one-item dependency deploy through the compare page, and don't ask them to deploy the app unless they want to ship it. Never dodge it by reimplementing the flow inside an inline runnable — that leaves two copies of the same logic and an app that ignores the flow they asked for.
|
||||
${sdkLine}
|
||||
- Use \`deploy_workspace_item\` after explicit user deploy intent. The deploy tool bundles JS/CSS before saving the raw app.
|
||||
- Use \`read_workspace_item\` with \`type: 'app'\` for a metadata summary (file paths and runnable list, no contents). Use \`read_app_file\` to read an individual file; large files are truncated to a head slice, so pass \`offset\`/\`limit\` to page through the rest rather than re-reading the whole file.
|
||||
- To find where a symbol or string lives across the app, call \`search_app\` (greps every frontend file and inline runnable, returns matching \`file:line\` rows) instead of reading files one by one — then \`read_app_file\` only the ranges you need. The loop is list (\`read_workspace_item\`) → locate (\`search_app\`) → inspect (\`read_app_file\` with \`offset\`/\`limit\`).
|
||||
@@ -2115,7 +2169,7 @@ function getAppInstructions(): string {
|
||||
|
||||
# Windmill raw app authoring reference
|
||||
|
||||
${getRawAppPrompt()}`
|
||||
${getRawAppPrompt(language)}`
|
||||
}
|
||||
|
||||
function getResourceInstructions(): string {
|
||||
@@ -2162,7 +2216,7 @@ function getInstructions(subject: InstructionSubject, language?: ScriptLang): st
|
||||
case 'resource':
|
||||
return getResourceInstructions()
|
||||
case 'app':
|
||||
return getAppInstructions()
|
||||
return getAppInstructions(language)
|
||||
case 'datatable':
|
||||
return getDatatableInstructions(language)
|
||||
case 'pipeline':
|
||||
@@ -3789,6 +3843,19 @@ export const globalTools: Tool<{}>[] = [
|
||||
return deleteAppRunnable(parsed, ctx)
|
||||
}
|
||||
},
|
||||
{
|
||||
def: testRunAppRunnableToolDef,
|
||||
fn: async (ctx) => {
|
||||
const parsed = testRunAppRunnableSchema.parse(ctx.args)
|
||||
return testRunAppRunnable(parsed, ctx)
|
||||
},
|
||||
requiresConfirmation: true,
|
||||
confirmationMessage: (args) =>
|
||||
`Run the backend runnable "${args?.key ?? ''}" of ${pathLeaf(args?.path, 'the app')}`,
|
||||
queuedLabel: (args) => `Test runnable "${args?.key ?? ''}" of ${args?.path ?? 'the app'}`,
|
||||
showDetails: true,
|
||||
autoCollapseDetails: false
|
||||
},
|
||||
...artifactTools,
|
||||
{
|
||||
def: createToolDef(
|
||||
@@ -4549,15 +4616,15 @@ function maybeAttachPreviewCard(
|
||||
function finishAppDraftWrite(
|
||||
result: DraftPersistResult,
|
||||
ctx: WriteDraftCtx,
|
||||
onSaved: () => { content: string; message: string }
|
||||
onSaved: () => { content: string; message: string; warning?: string }
|
||||
): string {
|
||||
const failure = draftWriteFailure(result, ctx)
|
||||
if (failure) return failure
|
||||
ctx.toolCallbacks.onItemModified?.(result.itemKind, result.storagePath)
|
||||
maybeAttachPreviewCard(ctx, result.itemKind, result.item.path)
|
||||
const { content, message } = onSaved()
|
||||
const { content, message, warning } = onSaved()
|
||||
ctx.toolCallbacks.setToolStatus(ctx.toolId, { content, result: 'Saved as draft' })
|
||||
return JSON.stringify({ success: true, message }, null, 2)
|
||||
return JSON.stringify({ success: true, message, warning }, null, 2)
|
||||
}
|
||||
|
||||
function finishDraftWrite(
|
||||
@@ -5662,13 +5729,102 @@ async function writeAppRunnable(
|
||||
const persisted = buildPersistedRunnable(input, existing)
|
||||
value.runnables = { ...value.runnables, [key]: persisted }
|
||||
await recomputeAppPolicy(value)
|
||||
const undeployed = await undeployedRunnableTargets(workspace, { [key]: persisted })
|
||||
const result = await saveAppDraft(workspace, path, value)
|
||||
return finishAppDraftWrite(result, ctx, () => ({
|
||||
content: `Updated runnable "${key}" in app "${path}"`,
|
||||
message: `Updated draft app "${path}" with runnable "${key}".`
|
||||
message: `Updated draft app "${path}" with runnable "${key}".`,
|
||||
warning: undeployed.length
|
||||
? `This runnable points at an item that is NOT deployed (${undeployed[0]}), so it fails at runtime — ` +
|
||||
`a path runnable runs the deployed item, never a draft. Offer to deploy just that item with ` +
|
||||
`deploy_workspace_item; the app itself does not need deploying, since the preview runs its draft.`
|
||||
: undefined
|
||||
}))
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs one backend runnable through `execute_component` in preview mode, the way the editor
|
||||
* preview does: inline executes draft code, a path runnable the deployed item it names.
|
||||
* Without it the chat can only wire an app up and hope.
|
||||
*/
|
||||
async function testRunAppRunnable(
|
||||
args: z.infer<typeof testRunAppRunnableSchema>,
|
||||
ctx: WriteDraftCtx
|
||||
): Promise<string> {
|
||||
const { workspace, toolId, toolCallbacks } = ctx
|
||||
const { path, key } = args
|
||||
|
||||
const { value } = await loadAppDraftValue(path, workspace)
|
||||
const runnable = value.runnables?.[key] as PersistedRunnable | undefined
|
||||
if (!runnable) {
|
||||
const known = Object.keys(value.runnables ?? {})
|
||||
throw new Error(
|
||||
`App "${path}" has no backend runnable "${key}".` +
|
||||
(known.length ? ` Available runnables: ${known.join(', ')}.` : '')
|
||||
)
|
||||
}
|
||||
|
||||
// Copied, not aliased: the ctx pass below writes into it.
|
||||
const testArgs = { ...normalizeTestRunArgs(args.args) }
|
||||
// Setting force_viewer_static_fields is what puts execute_component in preview
|
||||
// mode (apps.rs `is_preview`), which is what makes inline draft code run at all.
|
||||
// It must be sent even when the runnable has no static fields.
|
||||
const staticFields = Object.fromEntries(
|
||||
Object.entries(runnable.fields ?? {})
|
||||
.filter(([, field]) => field?.type === 'static')
|
||||
.map(([name, field]) => [name, field?.value])
|
||||
)
|
||||
// A ctx-bound input is filled by the server from `$ctx:<prop>`, exactly as
|
||||
// RawAppBackgroundRunner does before executing. Without this the argument arrives
|
||||
// missing and the runnable fails for a reason that has nothing to do with its code.
|
||||
for (const [name, field] of Object.entries(runnable.fields ?? {})) {
|
||||
if (field?.type === 'ctx' && field?.ctx) testArgs[name] = `$ctx:${field.ctx}`
|
||||
}
|
||||
// The server encrypts a queued argument only when its name is in this list
|
||||
// (apps.rs wraps it as `$encrypted:` with a job-scoped key). Omitting it writes a
|
||||
// `sensitive` field's real value into job args as plaintext, readable by anyone
|
||||
// with run access. Same filter the editor preview applies.
|
||||
const sensitiveInputs = Object.entries(runnable.fields ?? {})
|
||||
.filter(([, field]) => field?.type === 'user' && field?.sensitive)
|
||||
.map(([name]) => name)
|
||||
|
||||
// Imported lazily: statically pulling the apps module graph into the chat's
|
||||
// import chain drags the whole app-editor runtime in behind it.
|
||||
const { executeRunnable } = await import(
|
||||
'$lib/components/apps/components/helpers/executeRunnable'
|
||||
)
|
||||
|
||||
return executeTestRun({
|
||||
jobStarter: () =>
|
||||
executeRunnable(
|
||||
runnable as unknown as Runnable,
|
||||
workspace,
|
||||
undefined,
|
||||
get(userStore)?.username,
|
||||
path,
|
||||
key,
|
||||
{
|
||||
component: key,
|
||||
args: testArgs,
|
||||
force_viewer_static_fields: staticFields,
|
||||
force_viewer_sensitive_inputs: sensitiveInputs.length ? sensitiveInputs : undefined
|
||||
},
|
||||
undefined
|
||||
),
|
||||
workspace,
|
||||
toolCallbacks,
|
||||
toolId,
|
||||
startMessage: `Running backend runnable "${key}" of app "${path}"...`,
|
||||
// A path runnable pointing at a flow really does queue a flow job, so the
|
||||
// failure path can offer get_flow_run_details; everything else is a script job.
|
||||
contextName: runnable.runType === 'flow' ? 'flow' : 'script',
|
||||
completionName: 'backend runnable',
|
||||
background: args.background,
|
||||
detachAfterMs: waitSecondsToDetachMs(args.wait_seconds),
|
||||
label: `${path} / ${key}`
|
||||
})
|
||||
}
|
||||
|
||||
async function deleteAppRunnable(
|
||||
args: { path: string; key: string },
|
||||
ctx: WriteDraftCtx
|
||||
@@ -7034,6 +7190,16 @@ async function deployDraft(
|
||||
throw new Error(`Draft app "${path}" has no policy to deploy.`)
|
||||
}
|
||||
|
||||
// An app deployed on its own while a path runnable still points at a draft
|
||||
// is deployed and broken — the deploy has to say so, not just report success.
|
||||
const undeployedTargets = await undeployedRunnableTargets(workspace, appValue.runnables)
|
||||
if (undeployedTargets.length > 0) {
|
||||
deployNote =
|
||||
`These backend runnables point at items that are NOT deployed, so they fail at runtime: ` +
|
||||
`${undeployedTargets.join(', ')}. Deploy those items too, and tell the user the app is ` +
|
||||
`not working until they are.`
|
||||
}
|
||||
|
||||
toolCallbacks.setToolStatus(toolId, {
|
||||
content: `Bundling app "${path}"...`
|
||||
})
|
||||
@@ -7219,6 +7385,38 @@ async function validateDeleteWorkspaceItemTarget(args: {
|
||||
: `No ${type} at "${path}": neither a deployed item nor a draft. Nothing to delete.`
|
||||
}
|
||||
|
||||
/**
|
||||
* A path runnable executes the DEPLOYED item at its path, so an app wired to a draft-only
|
||||
* script or flow fails at runtime with no diagnostic. Writing one stays allowed — flow and
|
||||
* app are normally built together — but the missing deployment has to be visible. Hub
|
||||
* scripts have no deployed/draft distinction, so they are not probed.
|
||||
*/
|
||||
async function undeployedRunnableTargets(
|
||||
workspace: string,
|
||||
runnables: Record<string, { type?: string; runType?: string; path?: string } | undefined>
|
||||
): Promise<string[]> {
|
||||
const targets: string[] = []
|
||||
for (const [key, runnable] of Object.entries(runnables)) {
|
||||
// A persisted path runnable carries its kind in `runType`; any other shape names the
|
||||
// kind in `type` itself.
|
||||
const type =
|
||||
runnable?.type === 'path' || runnable?.type === 'runnableByPath'
|
||||
? runnable?.runType
|
||||
: runnable?.type
|
||||
const path = runnable?.path
|
||||
if ((type !== 'script' && type !== 'flow') || !path) continue
|
||||
try {
|
||||
if (await deployedItemExists(workspace, type, path, undefined)) continue
|
||||
} catch {
|
||||
// The probe is advisory: a lookup failure must never block the write or
|
||||
// the deploy it annotates.
|
||||
continue
|
||||
}
|
||||
targets.push(`"${key}" → ${type} "${path}"`)
|
||||
}
|
||||
return targets
|
||||
}
|
||||
|
||||
async function deployedItemExists(
|
||||
workspace: string,
|
||||
type: WorkspaceItemType,
|
||||
|
||||
@@ -1519,6 +1519,10 @@ export interface TestRunConfig {
|
||||
/** Overrides the default "…test started, waiting for completion" status while the
|
||||
* job runs inline (e.g. an SQL tool shows "SQL running…"). */
|
||||
runningMessage?: string
|
||||
/** Noun for the human-facing status strings ("<X> test completed successfully").
|
||||
* Defaults to `contextName`, which also carries the jobs-tray kind and so cannot
|
||||
* always name what ran: an app's path runnable queues a flow job. */
|
||||
completionName?: string
|
||||
/** Custom terminal formatting for the INLINE completion path (callers whose
|
||||
* result isn't a plain test-run summary, e.g. exec_datatable_sql shaping rows).
|
||||
* Returns the string handed to the model plus the tool-card patch. When omitted,
|
||||
@@ -1739,7 +1743,8 @@ export async function executeTestRun(config: TestRunConfig): Promise<string> {
|
||||
|
||||
const jobId = await config.jobStarter()
|
||||
|
||||
const contextName = config.contextName.charAt(0).toUpperCase() + config.contextName.slice(1)
|
||||
const shown = config.completionName ?? config.contextName
|
||||
const contextName = shown.charAt(0).toUpperCase() + shown.slice(1)
|
||||
|
||||
// Register the job so the tray shows it from the moment it is queued. Carry the
|
||||
// serializable resultFormat so a job that later detaches (and may outlive a
|
||||
@@ -1811,7 +1816,11 @@ export async function executeTestRun(config: TestRunConfig): Promise<string> {
|
||||
}
|
||||
return summary
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : 'Unknown error occurred'
|
||||
// formatToolError, not `error.message`: the generated client puts the server's
|
||||
// message in `body` and leaves `message` as the bare status text, so a path
|
||||
// runnable aimed at an undeployed flow reported "Not Found" instead of naming
|
||||
// the flow it could not find — losing the one diagnostic the run exists for.
|
||||
const errorMessage = formatToolError(error)
|
||||
config.toolCallbacks.setToolStatus(config.toolId, {
|
||||
content: `Test execution failed`,
|
||||
error: errorMessage
|
||||
|
||||
@@ -261,10 +261,10 @@ export type Job = {
|
||||
};
|
||||
|
||||
/**
|
||||
* Execute a job and wait for it to complete and return the completed job
|
||||
* Wait for a job to complete and return its result. Rejects if the job failed.
|
||||
* @param id
|
||||
*/
|
||||
export declare function waitJob(id: string): Promise<Job>;
|
||||
export declare function waitJob(id: string): Promise<any>;
|
||||
|
||||
/**
|
||||
* Get a job by id and return immediately with the current state of the job
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@ export * from './prompts';
|
||||
export declare function getScriptPrompt(language: string): string;
|
||||
export declare function getFlowPrompt(): string;
|
||||
export declare function getResourcePrompt(): string;
|
||||
export declare function getRawAppPrompt(): string;
|
||||
export declare function getRawAppPrompt(language?: string): string;
|
||||
export declare function getPipelinePrompt(): string;
|
||||
export declare function getDatatableSdkReference(language?: string): string;
|
||||
export declare function getWorkflowAsCodePrompt(language?: string): string;
|
||||
|
||||
@@ -49,9 +49,19 @@ export function getResourcePrompt(): string {
|
||||
return prompts.RESOURCES_BASE;
|
||||
}
|
||||
|
||||
// Helper for raw app authoring (chat consumers)
|
||||
export function getRawAppPrompt(): string {
|
||||
return prompts.RAW_APP_BASE;
|
||||
// Helper for raw app authoring (chat consumers). Inline backend runnables are
|
||||
// ordinary Windmill jobs, so the reference has to carry the SDK the runnable
|
||||
// calls — without it an agent invents client functions and hand-rolls HTTP.
|
||||
// Only one SDK is returned: both would double an already large tool result.
|
||||
export function getRawAppPrompt(language?: string): string {
|
||||
const sdkPrompt = PY_SDK_LANGUAGES.includes(language ?? '')
|
||||
? prompts.SDK_PYTHON
|
||||
: prompts.SDK_TYPESCRIPT;
|
||||
|
||||
return [
|
||||
prompts.RAW_APP_BASE,
|
||||
sdkPrompt
|
||||
].filter(Boolean).join('\n\n');
|
||||
}
|
||||
|
||||
// Helper for data pipeline authoring (chat consumers)
|
||||
|
||||
@@ -723,16 +723,50 @@ The frontend imports a generated module that mirrors the backend runnables. **Ne
|
||||
|
||||
### Calling backend runnables
|
||||
|
||||
Import the generated bindings and call the runnable like a function:
|
||||
Import the generated bindings and call the runnable like a function. \`./wmill\` is the **only** way the frontend reaches anything server-side — datatables, workspace items, external services. Never \`fetch\` the Windmill API from frontend code: the bundle holds no token and builds no API URL.
|
||||
|
||||
\`\`\`typescript
|
||||
| Export | Resolves to | Use it for |
|
||||
|---|---|---|
|
||||
| \`backend.<key>(args)\` | the runnable's result | the default — run and wait |
|
||||
| \`backendAsync.<key>(args)\` | the **job id** (a string) | long-running work you want to track |
|
||||
| \`waitJob(jobId)\` | the job's **result** (rejects if the job failed) | awaiting a \`backendAsync\` job |
|
||||
| \`getJob(jobId)\` | a \`Job\` (\`{ type, success, result, duration_ms, ... }\`) | polling status without blocking |
|
||||
| \`streamJob(jobId, onUpdate?)\` | the final result, calling \`onUpdate\` per chunk | showing output as it is produced |
|
||||
|
||||
Run and wait — the common case:
|
||||
|
||||
\`\`\`tsx
|
||||
import { backend } from './wmill';
|
||||
|
||||
// Call a backend runnable
|
||||
const user = await backend.get_user({ user_id: '123' });
|
||||
\`\`\`
|
||||
|
||||
The frontend cannot reach datatables, workspace items, or external services on its own — it goes through \`backend.<key>(args)\` for everything server-side.
|
||||
Start a long job, then await it:
|
||||
|
||||
\`\`\`tsx
|
||||
import { backendAsync, waitJob } from './wmill';
|
||||
|
||||
const jobId = await backendAsync.run_report({ month: '2026-08' }); // a string
|
||||
const report = await waitJob(jobId); // the result itself
|
||||
\`\`\`
|
||||
|
||||
Or poll it without blocking, to render progress:
|
||||
|
||||
\`\`\`tsx
|
||||
import { getJob } from './wmill';
|
||||
|
||||
const job = await getJob(jobId);
|
||||
if (job.type === 'CompletedJob') setReport(job.result);
|
||||
\`\`\`
|
||||
|
||||
\`backendAsync\` resolves a job id and nothing else — guard on it before storing or polling. A poll loop started on an \`undefined\` id never completes and shows as a row stuck "running" forever:
|
||||
|
||||
\`\`\`tsx
|
||||
const jobId = await backendAsync.run_report(args);
|
||||
if (!jobId) throw new Error('run_report did not start a job');
|
||||
\`\`\`
|
||||
|
||||
**Never hand-write a job-polling runnable.** A backend runnable that calls \`jobs/list\`, or that returns \`getResultMaybe(...)\` for the frontend to poll, reimplements \`backendAsync\` + \`waitJob\` / \`getJob\` / \`streamJob\` — and it is what leads to guessing at base URLs and tokens.
|
||||
|
||||
### Keeping data out of recorded demos
|
||||
|
||||
@@ -784,10 +818,33 @@ def main(user_id: str):
|
||||
return user
|
||||
\`\`\`
|
||||
|
||||
#### The \`wmill\` client is already authenticated
|
||||
|
||||
An inline runnable runs as an ordinary Windmill job. \`import * as wmill from 'windmill-client'\` (TypeScript) and \`import wmill\` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
|
||||
|
||||
**Don't read \`WM_TOKEN\` or \`BASE_INTERNAL_URL\` and build an API URL to \`fetch\`.** The client's own \`setClient\` already reads exactly those, and it also sets the credentials mode a raw app needs (\`WM_RAW_APP\` suppresses credentials, because a sandboxed bundle calls the API from an opaque origin that can never pair with \`Access-Control-Allow-Origin: *\`). Rebuilding that by hand drops the parts you can't see. Use \`wmill.*\` for everything Windmill, and \`fetch\` only for third-party APIs.
|
||||
|
||||
Prefer the \`wmill\` functions that appear in the SDK reference; for an endpoint none of them covers, the generated service classes (\`JobService\`, \`ScriptService\`, ...) are importable from \`windmill-client\`. What is not available is a name you guessed at: \`getBaseUrl\` and \`getWorkspaceToken\` are inventions, not API.
|
||||
|
||||
### Path runnables (script / flow / hubscript)
|
||||
|
||||
When \`type\` is \`script\`, \`flow\`, or \`hubscript\`, the runnable just stores a \`path\` to an existing workspace or hub item — no inline code. The referenced item's input/output schema becomes the runnable's surface.
|
||||
|
||||
### Draft code vs deployed code
|
||||
|
||||
This decides whether an app works before anything is deployed:
|
||||
|
||||
- **Inline runnables run the app's current code.** The editor sends the runnable's source with each request, so an inline runnable works in the preview with nothing deployed.
|
||||
- **Path runnables (\`script\` / \`flow\` / \`hubscript\`) run the DEPLOYED item at that path.** So do \`wmill.runFlow\`, \`wmill.runFlowAsync\` and \`wmill.runScriptByPath\` called from inside a runnable. A draft — including a draft you just created — does not exist for them.
|
||||
|
||||
So an app wired to a flow you just wrote does nothing until **that flow is deployed**. The app itself does NOT have to be deployed for this: the preview runs the app's draft, so the referenced flow is the only thing that has to exist deployed.
|
||||
|
||||
That makes the fix a one-item deploy, not a release. Offer to deploy exactly the referenced flow or script and leave the app a draft the user keeps testing in the preview — do not push the whole change set through the review-and-deploy page, and do not ask the user to deploy the app, unless they said they want to ship it.
|
||||
|
||||
Do NOT quietly reimplement the flow inside an inline runnable to dodge the deployment: that leaves the user with two copies of the same logic and an app that ignores the flow they asked for. Inline the logic only when the user actually wants it inline.
|
||||
|
||||
Prefer a **path runnable of type \`flow\`** over an inline runnable that calls \`wmill.runFlowAsync\`. The path runnable gives the frontend the flow's real input schema and works with \`backend\` / \`backendAsync\` / \`waitJob\` like any other runnable; a hand-written wrapper gives up all of that.
|
||||
|
||||
### Static inputs
|
||||
|
||||
\`staticInputs\` is an optional \`Record<string, any>\` for arguments not overridable from the frontend. Useful with path runnables to pre-fill some args while leaving the rest to the frontend caller.
|
||||
@@ -849,6 +906,8 @@ def main(user_id: str):
|
||||
4. **Use descriptive keys** — \`get_user\`, not \`a\`.
|
||||
5. **Always whitelist tables** — adding a runnable that queries a new table requires the table to be in \`data.tables\` first.
|
||||
6. **Mark sensitive UI with \`data-wm-no-record\`** — it is what keeps that data out of a recorded demo; passwords are handled for you.
|
||||
7. **Reach for \`backendAsync\` + \`waitJob\`** for long work — never a hand-written job-polling runnable.
|
||||
8. **Deploy what a path runnable points at** — a path runnable aimed at a draft fails at runtime; tell the user what needs deploying.
|
||||
`;
|
||||
|
||||
export const PIPELINE_BASE = `# Data pipeline authoring
|
||||
@@ -1186,6 +1245,17 @@ export const SDK_TYPESCRIPT = `# TypeScript SDK (windmill-client)
|
||||
|
||||
Import: import * as wmill from 'windmill-client'
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
||||
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
||||
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
||||
neither listed below nor a service method, do not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
\`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL\`. WM_END_USER_EMAIL is the app viewer when
|
||||
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
|
||||
@@ -1788,6 +1858,17 @@ export const SDK_PYTHON = `# Python SDK (wmill)
|
||||
|
||||
Import: import wmill
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The functions below are the surface to prefer. For an endpoint none of them covers,
|
||||
wmill.Windmill().get(endpoint) and .post(endpoint) issue an authenticated request against this
|
||||
instance. What does not exist is a function name you guessed at: if it is not listed below, do
|
||||
not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
\`os.environ.get("WM_END_USER_EMAIL") or os.environ.get("WM_EMAIL")\`. WM_END_USER_EMAIL is the app
|
||||
viewer when the run was triggered from an app and empty otherwise (both variables are always
|
||||
|
||||
@@ -1437,6 +1437,17 @@ being buffered, bypassing the 10000-row return cap.
|
||||
|
||||
Import: import * as wmill from 'windmill-client'
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
||||
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
||||
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
||||
neither listed below nor a service method, do not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL`. WM_END_USER_EMAIL is the app viewer when
|
||||
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
|
||||
@@ -2039,6 +2050,17 @@ appendPartition(opts: Omit<DucklakeMaterializeOptions, "uniqueKey">,): SqlStatem
|
||||
|
||||
Import: import wmill
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The functions below are the surface to prefer. For an endpoint none of them covers,
|
||||
wmill.Windmill().get(endpoint) and .post(endpoint) issue an authenticated request against this
|
||||
instance. What does not exist is a function name you guessed at: if it is not listed below, do
|
||||
not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
`os.environ.get("WM_END_USER_EMAIL") or os.environ.get("WM_EMAIL")`. WM_END_USER_EMAIL is the app
|
||||
viewer when the run was triggered from an app and empty otherwise (both variables are always
|
||||
|
||||
@@ -2,6 +2,17 @@
|
||||
|
||||
Import: import wmill
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The functions below are the surface to prefer. For an endpoint none of them covers,
|
||||
wmill.Windmill().get(endpoint) and .post(endpoint) issue an authenticated request against this
|
||||
instance. What does not exist is a function name you guessed at: if it is not listed below, do
|
||||
not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
`os.environ.get("WM_END_USER_EMAIL") or os.environ.get("WM_EMAIL")`. WM_END_USER_EMAIL is the app
|
||||
viewer when the run was triggered from an app and empty otherwise (both variables are always
|
||||
|
||||
@@ -2,6 +2,17 @@
|
||||
|
||||
Import: import * as wmill from 'windmill-client'
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
||||
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
||||
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
||||
neither listed below nor a service method, do not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL`. WM_END_USER_EMAIL is the app viewer when
|
||||
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
|
||||
|
||||
@@ -273,16 +273,50 @@ The frontend imports a generated module that mirrors the backend runnables. **Ne
|
||||
|
||||
### Calling backend runnables
|
||||
|
||||
Import the generated bindings and call the runnable like a function:
|
||||
Import the generated bindings and call the runnable like a function. `./wmill` is the **only** way the frontend reaches anything server-side — datatables, workspace items, external services. Never `fetch` the Windmill API from frontend code: the bundle holds no token and builds no API URL.
|
||||
|
||||
```typescript
|
||||
| Export | Resolves to | Use it for |
|
||||
|---|---|---|
|
||||
| `backend.<key>(args)` | the runnable's result | the default — run and wait |
|
||||
| `backendAsync.<key>(args)` | the **job id** (a string) | long-running work you want to track |
|
||||
| `waitJob(jobId)` | the job's **result** (rejects if the job failed) | awaiting a `backendAsync` job |
|
||||
| `getJob(jobId)` | a `Job` (`{ type, success, result, duration_ms, ... }`) | polling status without blocking |
|
||||
| `streamJob(jobId, onUpdate?)` | the final result, calling `onUpdate` per chunk | showing output as it is produced |
|
||||
|
||||
Run and wait — the common case:
|
||||
|
||||
```tsx
|
||||
import { backend } from './wmill';
|
||||
|
||||
// Call a backend runnable
|
||||
const user = await backend.get_user({ user_id: '123' });
|
||||
```
|
||||
|
||||
The frontend cannot reach datatables, workspace items, or external services on its own — it goes through `backend.<key>(args)` for everything server-side.
|
||||
Start a long job, then await it:
|
||||
|
||||
```tsx
|
||||
import { backendAsync, waitJob } from './wmill';
|
||||
|
||||
const jobId = await backendAsync.run_report({ month: '2026-08' }); // a string
|
||||
const report = await waitJob(jobId); // the result itself
|
||||
```
|
||||
|
||||
Or poll it without blocking, to render progress:
|
||||
|
||||
```tsx
|
||||
import { getJob } from './wmill';
|
||||
|
||||
const job = await getJob(jobId);
|
||||
if (job.type === 'CompletedJob') setReport(job.result);
|
||||
```
|
||||
|
||||
`backendAsync` resolves a job id and nothing else — guard on it before storing or polling. A poll loop started on an `undefined` id never completes and shows as a row stuck "running" forever:
|
||||
|
||||
```tsx
|
||||
const jobId = await backendAsync.run_report(args);
|
||||
if (!jobId) throw new Error('run_report did not start a job');
|
||||
```
|
||||
|
||||
**Never hand-write a job-polling runnable.** A backend runnable that calls `jobs/list`, or that returns `getResultMaybe(...)` for the frontend to poll, reimplements `backendAsync` + `waitJob` / `getJob` / `streamJob` — and it is what leads to guessing at base URLs and tokens.
|
||||
|
||||
### Keeping data out of recorded demos
|
||||
|
||||
@@ -334,10 +368,33 @@ def main(user_id: str):
|
||||
return user
|
||||
```
|
||||
|
||||
#### The `wmill` client is already authenticated
|
||||
|
||||
An inline runnable runs as an ordinary Windmill job. `import * as wmill from 'windmill-client'` (TypeScript) and `import wmill` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
|
||||
|
||||
**Don't read `WM_TOKEN` or `BASE_INTERNAL_URL` and build an API URL to `fetch`.** The client's own `setClient` already reads exactly those, and it also sets the credentials mode a raw app needs (`WM_RAW_APP` suppresses credentials, because a sandboxed bundle calls the API from an opaque origin that can never pair with `Access-Control-Allow-Origin: *`). Rebuilding that by hand drops the parts you can't see. Use `wmill.*` for everything Windmill, and `fetch` only for third-party APIs.
|
||||
|
||||
Prefer the `wmill` functions that appear in the SDK reference; for an endpoint none of them covers, the generated service classes (`JobService`, `ScriptService`, ...) are importable from `windmill-client`. What is not available is a name you guessed at: `getBaseUrl` and `getWorkspaceToken` are inventions, not API.
|
||||
|
||||
### Path runnables (script / flow / hubscript)
|
||||
|
||||
When `type` is `script`, `flow`, or `hubscript`, the runnable just stores a `path` to an existing workspace or hub item — no inline code. The referenced item's input/output schema becomes the runnable's surface.
|
||||
|
||||
### Draft code vs deployed code
|
||||
|
||||
This decides whether an app works before anything is deployed:
|
||||
|
||||
- **Inline runnables run the app's current code.** The editor sends the runnable's source with each request, so an inline runnable works in the preview with nothing deployed.
|
||||
- **Path runnables (`script` / `flow` / `hubscript`) run the DEPLOYED item at that path.** So do `wmill.runFlow`, `wmill.runFlowAsync` and `wmill.runScriptByPath` called from inside a runnable. A draft — including a draft you just created — does not exist for them.
|
||||
|
||||
So an app wired to a flow you just wrote does nothing until **that flow is deployed**. The app itself does NOT have to be deployed for this: the preview runs the app's draft, so the referenced flow is the only thing that has to exist deployed.
|
||||
|
||||
That makes the fix a one-item deploy, not a release. Offer to deploy exactly the referenced flow or script and leave the app a draft the user keeps testing in the preview — do not push the whole change set through the review-and-deploy page, and do not ask the user to deploy the app, unless they said they want to ship it.
|
||||
|
||||
Do NOT quietly reimplement the flow inside an inline runnable to dodge the deployment: that leaves the user with two copies of the same logic and an app that ignores the flow they asked for. Inline the logic only when the user actually wants it inline.
|
||||
|
||||
Prefer a **path runnable of type `flow`** over an inline runnable that calls `wmill.runFlowAsync`. The path runnable gives the frontend the flow's real input schema and works with `backend` / `backendAsync` / `waitJob` like any other runnable; a hand-written wrapper gives up all of that.
|
||||
|
||||
### Static inputs
|
||||
|
||||
`staticInputs` is an optional `Record<string, any>` for arguments not overridable from the frontend. Useful with path runnables to pre-fill some args while leaving the rest to the frontend caller.
|
||||
@@ -399,3 +456,5 @@ def main(user_id: str):
|
||||
4. **Use descriptive keys** — `get_user`, not `a`.
|
||||
5. **Always whitelist tables** — adding a runnable that queries a new table requires the table to be in `data.tables` first.
|
||||
6. **Mark sensitive UI with `data-wm-no-record`** — it is what keeps that data out of a recorded demo; passwords are handled for you.
|
||||
7. **Reach for `backendAsync` + `waitJob`** for long work — never a hand-written job-polling runnable.
|
||||
8. **Deploy what a path runnable points at** — a path runnable aimed at a draft fails at runtime; tell the user what needs deploying.
|
||||
|
||||
@@ -173,6 +173,17 @@ const result: wmill.S3Object = await wmill.writeS3File(
|
||||
|
||||
Import: import * as wmill from 'windmill-client'
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
||||
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
||||
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
||||
neither listed below nor a service method, do not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL`. WM_END_USER_EMAIL is the app viewer when
|
||||
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
|
||||
|
||||
@@ -173,6 +173,17 @@ const result: wmill.S3Object = await wmill.writeS3File(
|
||||
|
||||
Import: import * as wmill from 'windmill-client'
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
||||
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
||||
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
||||
neither listed below nor a service method, do not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL`. WM_END_USER_EMAIL is the app viewer when
|
||||
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
|
||||
|
||||
@@ -175,6 +175,17 @@ const result: wmill.S3Object = await wmill.writeS3File(
|
||||
|
||||
Import: import * as wmill from 'windmill-client'
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
||||
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
||||
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
||||
neither listed below nor a service method, do not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL`. WM_END_USER_EMAIL is the app viewer when
|
||||
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
|
||||
|
||||
@@ -187,6 +187,17 @@ result: S3Object = wmill.write_s3_file(
|
||||
|
||||
Import: import wmill
|
||||
|
||||
The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs.
|
||||
|
||||
The functions below are the surface to prefer. For an endpoint none of them covers,
|
||||
wmill.Windmill().get(endpoint) and .post(endpoint) issue an authenticated request against this
|
||||
instance. What does not exist is a function name you guessed at: if it is not listed below, do
|
||||
not call it.
|
||||
|
||||
To know who is running the script, read the contextual variables rather than calling the API:
|
||||
`os.environ.get("WM_END_USER_EMAIL") or os.environ.get("WM_EMAIL")`. WM_END_USER_EMAIL is the app
|
||||
viewer when the run was triggered from an app and empty otherwise (both variables are always
|
||||
|
||||
@@ -38,16 +38,50 @@ The frontend imports a generated module that mirrors the backend runnables. **Ne
|
||||
|
||||
### Calling backend runnables
|
||||
|
||||
Import the generated bindings and call the runnable like a function:
|
||||
Import the generated bindings and call the runnable like a function. `./wmill` is the **only** way the frontend reaches anything server-side — datatables, workspace items, external services. Never `fetch` the Windmill API from frontend code: the bundle holds no token and builds no API URL.
|
||||
|
||||
```typescript
|
||||
| Export | Resolves to | Use it for |
|
||||
|---|---|---|
|
||||
| `backend.<key>(args)` | the runnable's result | the default — run and wait |
|
||||
| `backendAsync.<key>(args)` | the **job id** (a string) | long-running work you want to track |
|
||||
| `waitJob(jobId)` | the job's **result** (rejects if the job failed) | awaiting a `backendAsync` job |
|
||||
| `getJob(jobId)` | a `Job` (`{ type, success, result, duration_ms, ... }`) | polling status without blocking |
|
||||
| `streamJob(jobId, onUpdate?)` | the final result, calling `onUpdate` per chunk | showing output as it is produced |
|
||||
|
||||
Run and wait — the common case:
|
||||
|
||||
```tsx
|
||||
import { backend } from './wmill';
|
||||
|
||||
// Call a backend runnable
|
||||
const user = await backend.get_user({ user_id: '123' });
|
||||
```
|
||||
|
||||
The frontend cannot reach datatables, workspace items, or external services on its own — it goes through `backend.<key>(args)` for everything server-side.
|
||||
Start a long job, then await it:
|
||||
|
||||
```tsx
|
||||
import { backendAsync, waitJob } from './wmill';
|
||||
|
||||
const jobId = await backendAsync.run_report({ month: '2026-08' }); // a string
|
||||
const report = await waitJob(jobId); // the result itself
|
||||
```
|
||||
|
||||
Or poll it without blocking, to render progress:
|
||||
|
||||
```tsx
|
||||
import { getJob } from './wmill';
|
||||
|
||||
const job = await getJob(jobId);
|
||||
if (job.type === 'CompletedJob') setReport(job.result);
|
||||
```
|
||||
|
||||
`backendAsync` resolves a job id and nothing else — guard on it before storing or polling. A poll loop started on an `undefined` id never completes and shows as a row stuck "running" forever:
|
||||
|
||||
```tsx
|
||||
const jobId = await backendAsync.run_report(args);
|
||||
if (!jobId) throw new Error('run_report did not start a job');
|
||||
```
|
||||
|
||||
**Never hand-write a job-polling runnable.** A backend runnable that calls `jobs/list`, or that returns `getResultMaybe(...)` for the frontend to poll, reimplements `backendAsync` + `waitJob` / `getJob` / `streamJob` — and it is what leads to guessing at base URLs and tokens.
|
||||
|
||||
### Keeping data out of recorded demos
|
||||
|
||||
@@ -99,10 +133,33 @@ def main(user_id: str):
|
||||
return user
|
||||
```
|
||||
|
||||
#### The `wmill` client is already authenticated
|
||||
|
||||
An inline runnable runs as an ordinary Windmill job. `import * as wmill from 'windmill-client'` (TypeScript) and `import wmill` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
|
||||
|
||||
**Don't read `WM_TOKEN` or `BASE_INTERNAL_URL` and build an API URL to `fetch`.** The client's own `setClient` already reads exactly those, and it also sets the credentials mode a raw app needs (`WM_RAW_APP` suppresses credentials, because a sandboxed bundle calls the API from an opaque origin that can never pair with `Access-Control-Allow-Origin: *`). Rebuilding that by hand drops the parts you can't see. Use `wmill.*` for everything Windmill, and `fetch` only for third-party APIs.
|
||||
|
||||
Prefer the `wmill` functions that appear in the SDK reference; for an endpoint none of them covers, the generated service classes (`JobService`, `ScriptService`, ...) are importable from `windmill-client`. What is not available is a name you guessed at: `getBaseUrl` and `getWorkspaceToken` are inventions, not API.
|
||||
|
||||
### Path runnables (script / flow / hubscript)
|
||||
|
||||
When `type` is `script`, `flow`, or `hubscript`, the runnable just stores a `path` to an existing workspace or hub item — no inline code. The referenced item's input/output schema becomes the runnable's surface.
|
||||
|
||||
### Draft code vs deployed code
|
||||
|
||||
This decides whether an app works before anything is deployed:
|
||||
|
||||
- **Inline runnables run the app's current code.** The editor sends the runnable's source with each request, so an inline runnable works in the preview with nothing deployed.
|
||||
- **Path runnables (`script` / `flow` / `hubscript`) run the DEPLOYED item at that path.** So do `wmill.runFlow`, `wmill.runFlowAsync` and `wmill.runScriptByPath` called from inside a runnable. A draft — including a draft you just created — does not exist for them.
|
||||
|
||||
So an app wired to a flow you just wrote does nothing until **that flow is deployed**. The app itself does NOT have to be deployed for this: the preview runs the app's draft, so the referenced flow is the only thing that has to exist deployed.
|
||||
|
||||
That makes the fix a one-item deploy, not a release. Offer to deploy exactly the referenced flow or script and leave the app a draft the user keeps testing in the preview — do not push the whole change set through the review-and-deploy page, and do not ask the user to deploy the app, unless they said they want to ship it.
|
||||
|
||||
Do NOT quietly reimplement the flow inside an inline runnable to dodge the deployment: that leaves the user with two copies of the same logic and an app that ignores the flow they asked for. Inline the logic only when the user actually wants it inline.
|
||||
|
||||
Prefer a **path runnable of type `flow`** over an inline runnable that calls `wmill.runFlowAsync`. The path runnable gives the frontend the flow's real input schema and works with `backend` / `backendAsync` / `waitJob` like any other runnable; a hand-written wrapper gives up all of that.
|
||||
|
||||
### Static inputs
|
||||
|
||||
`staticInputs` is an optional `Record<string, any>` for arguments not overridable from the frontend. Useful with path runnables to pre-fill some args while leaving the rest to the frontend caller.
|
||||
@@ -164,3 +221,5 @@ def main(user_id: str):
|
||||
4. **Use descriptive keys** — `get_user`, not `a`.
|
||||
5. **Always whitelist tables** — adding a runnable that queries a new table requires the table to be in `data.tables` first.
|
||||
6. **Mark sensitive UI with `data-wm-no-record`** — it is what keeps that data out of a recorded demo; passwords are handled for you.
|
||||
7. **Reach for `backendAsync` + `waitJob`** for long work — never a hand-written job-polling runnable.
|
||||
8. **Deploy what a path runnable points at** — a path runnable aimed at a draft fails at runtime; tell the user what needs deploying.
|
||||
|
||||
@@ -726,10 +726,34 @@ viewer when the run was triggered from an app and empty otherwise (both variable
|
||||
defined), WM_EMAIL is the user the job is permissioned as. WM_USERNAME is the matching username."""
|
||||
|
||||
|
||||
# `setClient` reads BASE_INTERNAL_URL/BASE_URL and WM_TOKEN itself; agents that miss this rebuild
|
||||
# that logic by hand and get it wrong.
|
||||
PRECONFIGURED_CLIENT = """The client configures itself from the job's environment — base URL, token and credentials mode
|
||||
are all set before your code runs, so there is nothing to initialize and no reason to read
|
||||
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
||||
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
||||
HTTP for third-party APIs."""
|
||||
|
||||
# The listing is the helper surface, not the whole API, so each language needs its own escape
|
||||
# hatch: a flat "not listed means it does not exist" is false and leaves an uncovered endpoint
|
||||
# with no legal move.
|
||||
UNLISTED_ENDPOINTS_TS = """The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
||||
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
||||
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
||||
neither listed below nor a service method, do not call it."""
|
||||
|
||||
UNLISTED_ENDPOINTS_PY = """The functions below are the surface to prefer. For an endpoint none of them covers,
|
||||
wmill.Windmill().get(endpoint) and .post(endpoint) issue an authenticated request against this
|
||||
instance. What does not exist is a function name you guessed at: if it is not listed below, do
|
||||
not call it."""
|
||||
|
||||
|
||||
def generate_ts_sdk_markdown(functions: list[dict], _types: list[dict]) -> str:
|
||||
"""Generate compact documentation for TypeScript SDK."""
|
||||
md = "# TypeScript SDK (windmill-client)\n\n"
|
||||
md += "Import: import * as wmill from 'windmill-client'\n\n"
|
||||
md += PRECONFIGURED_CLIENT + "\n\n"
|
||||
md += UNLISTED_ENDPOINTS_TS + "\n\n"
|
||||
md += IDENTITY_OF_THE_RUN_TS + "\n\n"
|
||||
|
||||
for i, func in enumerate(functions):
|
||||
@@ -753,6 +777,8 @@ def generate_py_sdk_markdown(functions: list[dict], _classes: list[dict]) -> str
|
||||
"""Generate compact documentation for Python SDK."""
|
||||
md = "# Python SDK (wmill)\n\n"
|
||||
md += "Import: import wmill\n\n"
|
||||
md += PRECONFIGURED_CLIENT + "\n\n"
|
||||
md += UNLISTED_ENDPOINTS_PY + "\n\n"
|
||||
md += IDENTITY_OF_THE_RUN_PY + "\n\n"
|
||||
|
||||
for func in functions:
|
||||
@@ -2644,9 +2670,19 @@ export function getResourcePrompt(): string {
|
||||
return prompts.RESOURCES_BASE;
|
||||
}
|
||||
|
||||
// Helper for raw app authoring (chat consumers)
|
||||
export function getRawAppPrompt(): string {
|
||||
return prompts.RAW_APP_BASE;
|
||||
// Helper for raw app authoring (chat consumers). Inline backend runnables are
|
||||
// ordinary Windmill jobs, so the reference has to carry the SDK the runnable
|
||||
// calls — without it an agent invents client functions and hand-rolls HTTP.
|
||||
// Only one SDK is returned: both would double an already large tool result.
|
||||
export function getRawAppPrompt(language?: string): string {
|
||||
const sdkPrompt = PY_SDK_LANGUAGES.includes(language ?? '')
|
||||
? prompts.SDK_PYTHON
|
||||
: prompts.SDK_TYPESCRIPT;
|
||||
|
||||
return [
|
||||
prompts.RAW_APP_BASE,
|
||||
sdkPrompt
|
||||
].filter(Boolean).join('\\n\\n');
|
||||
}
|
||||
|
||||
// Helper for data pipeline authoring (chat consumers)
|
||||
@@ -2705,7 +2741,7 @@ export function getWorkflowAsCodePrompt(language?: string): string {
|
||||
export declare function getScriptPrompt(language: string): string;
|
||||
export declare function getFlowPrompt(): string;
|
||||
export declare function getResourcePrompt(): string;
|
||||
export declare function getRawAppPrompt(): string;
|
||||
export declare function getRawAppPrompt(language?: string): string;
|
||||
export declare function getPipelinePrompt(): string;
|
||||
export declare function getDatatableSdkReference(language?: string): string;
|
||||
export declare function getWorkflowAsCodePrompt(language?: string): string;
|
||||
|
||||
Reference in New Issue
Block a user