mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-05 08:02:28 +00:00
* feat: lint AI agent tool names and flow groups in wmill lint * refactor: assemble chat and CLI AI guidance from one topic table * feat: share flow groups, reuse and pipeline guidance with the CLI skills * feat: share raw app, data table and secret guidance between chat and CLI * docs: document the shared AI guidance source for contributors * fix: keep wmill lint running on flows with malformed collections * fix: reject skill descriptions that are not plain YAML text * fix: tighten fence typos, script base scope and app prompt order * fix: skip tool name checks on agent steps linked to a saved agent * fix: catch any misspelled prompt fence and soften the tool name claim * fix: align cli eval harness with the files and steps wmill init adds * fix: drop cli eval checks that expect unrequested deploy commands * fix: list ansible as mainless and c# Main in script base guidance
2854 lines
92 KiB
Markdown
2854 lines
92 KiB
Markdown
# Windmill Script Writing Guide
|
|
|
|
## General Principles
|
|
|
|
- A script's inputs are its parameters. Credentials and configuration come in as resource-typed parameters, never hard-coded or read from the environment; the language section below shows how that language declares parameters
|
|
- Libraries are installed automatically - do not show installation instructions
|
|
- In a language with an entrypoint function (TypeScript, Python, Go, Rust, PHP, R, …), name it `main` (`Main` in C#) and do not call it; in TypeScript it must be async. SQL, GraphQL, Bash, PowerShell and Ansible scripts have no `main`: their language section shows how they take arguments
|
|
- Where the language has a Windmill client (`wmill`), use it to interact with the platform
|
|
|
|
## Return Values
|
|
|
|
- A script can return any JSON-serializable value; a SQL script returns the rows its query produces
|
|
- Return values become available to subsequent flow steps via `results.step_id`
|
|
|
|
## Preprocessor Scripts
|
|
|
|
Preprocessor scripts process raw trigger data from various sources (webhook, custom HTTP route, SQS, WebSocket, Kafka, NATS, MQTT, AMQP, Postgres, GCP Pub/Sub, Azure, or email) before passing it to the flow. This separates the trigger logic from the flow logic and keeps the auto-generated UI clean.
|
|
|
|
A preprocessor is written in TypeScript or Python: its function is named `preprocessor` instead of `main`, and it receives a single parameter called `event` (the language section gives its type).
|
|
|
|
The returned object determines the parameter values passed to the flow.
|
|
e.g., `{ b: 1, a: 2 }` calls the flow with `a = 2` and `b = 1`, assuming the flow has two inputs called `a` and `b`.
|
|
|
|
|
|
# Ansible
|
|
|
|
Windmill runs Ansible playbooks with `ansible-playbook`. A script is a single YAML
|
|
document made of two parts separated by a `---` line: a Windmill **header** and one or
|
|
more standard Ansible **plays**.
|
|
|
|
## Structure
|
|
|
|
```yaml
|
|
---
|
|
# Windmill header: configures inventories, file resources, arguments and dependencies
|
|
extra_vars:
|
|
world_qualifier:
|
|
type: string
|
|
dependencies:
|
|
galaxy:
|
|
collections:
|
|
- name: community.general
|
|
python:
|
|
- jmespath
|
|
---
|
|
# Standard Ansible plays
|
|
- name: Echo
|
|
hosts: 127.0.0.1
|
|
connection: local
|
|
tasks:
|
|
- name: Print debug message
|
|
debug:
|
|
msg: "Hello, {{ world_qualifier }} world!"
|
|
```
|
|
|
|
## Header
|
|
|
|
The header is **not** standard Ansible — it is parsed by Windmill to build the script's
|
|
inputs and runtime environment. Supported keys:
|
|
|
|
- `extra_vars`: defines the script arguments. Each entry is passed to the playbook via
|
|
`--extra-vars` and becomes a Jinja variable usable as `{{ name }}` in the plays. Give
|
|
each argument a `type` (`string`, `number`, `boolean`, `object`, ...) so Windmill can
|
|
generate the input form.
|
|
- `inventory`: lists inventories. Use `resource_type: ansible_inventory` (optionally
|
|
pinned with `resource: u/user/your_resource`) or `resource_type: dynamic_inventory`.
|
|
- `files`: writes Windmill resources/variables to files before the run, e.g.
|
|
`- resource: u/user/template` with `target: ./config.j2`, or
|
|
`- variable: u/user/ssh_key` with `target: ./ssh_key` and `mode: '0600'`.
|
|
- `dependencies`: `galaxy` collections/roles (installed with `ansible-galaxy`) and
|
|
`python` pip packages available to the playbook.
|
|
- `options`: extra `ansible-playbook` flags such as `- verbosity: vvv`.
|
|
- `vault_password`: a Windmill variable path to use as the Ansible Vault password.
|
|
|
|
## Arguments
|
|
|
|
Reference header `extra_vars` directly as Jinja variables in the plays:
|
|
|
|
```yaml
|
|
extra_vars:
|
|
name:
|
|
type: string
|
|
count:
|
|
type: number
|
|
---
|
|
- hosts: localhost
|
|
tasks:
|
|
- debug:
|
|
msg: "{{ name }} x {{ count }}"
|
|
```
|
|
|
|
## Environment variables
|
|
|
|
Windmill contextual variables are available as environment variables and read with the
|
|
`env` lookup:
|
|
|
|
```yaml
|
|
- debug:
|
|
msg: "Running in workspace {{ lookup('env', 'WM_WORKSPACE') }}"
|
|
```
|
|
|
|
## Output
|
|
|
|
To return a result, write JSON to a `result.json` file in the job directory:
|
|
|
|
```yaml
|
|
- hosts: localhost
|
|
tasks:
|
|
- name: Write result
|
|
copy:
|
|
content: "{{ { 'ok': true, 'value': 42 } | to_json }}"
|
|
dest: result.json
|
|
```
|
|
|
|
|
|
# Bash
|
|
|
|
## Structure
|
|
|
|
Do not include `#!/bin/bash`. Arguments are obtained as positional parameters:
|
|
|
|
```bash
|
|
# Get arguments
|
|
var1="$1"
|
|
var2="$2"
|
|
|
|
echo "Processing $var1 and $var2"
|
|
|
|
# Return JSON by echoing to stdout
|
|
echo "{\"result\": \"$var1\", \"count\": $var2}"
|
|
```
|
|
|
|
**Important:**
|
|
- Do not include shebang (`#!/bin/bash`)
|
|
- Arguments are always strings
|
|
- Access with `$1`, `$2`, etc.
|
|
|
|
## Output
|
|
|
|
The script output is captured as the result. For structured data, output valid JSON:
|
|
|
|
```bash
|
|
name="$1"
|
|
count="$2"
|
|
|
|
# Output JSON result
|
|
cat << EOF
|
|
{
|
|
"name": "$name",
|
|
"count": $count,
|
|
"timestamp": "$(date -Iseconds)"
|
|
}
|
|
EOF
|
|
```
|
|
|
|
## Environment Variables
|
|
|
|
Environment variables set in Windmill are available:
|
|
|
|
```bash
|
|
# Access environment variable
|
|
echo "Workspace: $WM_WORKSPACE"
|
|
echo "Job ID: $WM_JOB_ID"
|
|
```
|
|
|
|
|
|
# BigQuery
|
|
|
|
Arguments use `@name` syntax.
|
|
|
|
Name the parameters by adding comments before the statement:
|
|
|
|
```sql
|
|
-- @name1 (string)
|
|
-- @name2 (int64) = 0
|
|
SELECT * FROM users WHERE name = @name1 AND age > @name2;
|
|
```
|
|
|
|
## Receiving an S3Object as a script parameter
|
|
|
|
Declare the arg with type `(s3object)`. Windmill renders an S3 file picker for
|
|
it, downloads the file, and binds it as a `STRING` JSON parameter — Parquet/CSV
|
|
files are decoded server-side into a JSON array of records, JSON/JSONL pass
|
|
through. Consume with `JSON_EXTRACT_ARRAY` / `JSON_VALUE`:
|
|
|
|
```sql
|
|
-- @file (s3object)
|
|
SELECT
|
|
CAST(JSON_VALUE(row, '$.id') AS INT64) AS id,
|
|
JSON_VALUE(row, '$.name') AS name
|
|
FROM UNNEST(JSON_EXTRACT_ARRAY(@file)) AS row;
|
|
```
|
|
|
|
## Streaming query results to S3
|
|
|
|
Add a `-- s3` directive at the top of the script to stream the result set to S3
|
|
instead of returning rows. Windmill writes the file and returns its `S3Object`
|
|
as the script result.
|
|
|
|
```sql
|
|
-- s3 prefix=exports/users format=parquet
|
|
SELECT id, name FROM users;
|
|
```
|
|
|
|
All keys are optional: `prefix` (object key prefix), `storage` (named storage —
|
|
omit to use the workspace default), `format` (`json` (default), `parquet`, or
|
|
`csv`). Use this for large result sets — rows stream directly to S3 instead of
|
|
being buffered, bypassing the 10000-row return cap.
|
|
|
|
|
|
# TypeScript (Bun)
|
|
|
|
Bun runtime with full npm ecosystem and fastest execution. **Bun is the default and preferred TypeScript runtime** — choose it for any TypeScript script unless there is a major reason to use Deno for that specific use-case.
|
|
|
|
## Structure
|
|
|
|
Export a single **async** function called `main`:
|
|
|
|
```typescript
|
|
export async function main(param1: string, param2: number) {
|
|
// Your code here
|
|
return { result: param1, count: param2 };
|
|
}
|
|
```
|
|
|
|
Do not call the main function. Libraries are installed automatically.
|
|
|
|
## Resource Types
|
|
|
|
On Windmill, credentials and configuration are stored in resources and passed as parameters to main.
|
|
|
|
Use the `RT` namespace for resource types:
|
|
|
|
```typescript
|
|
export async function main(stripe: RT.Stripe) {
|
|
// stripe contains API key and config from the resource
|
|
}
|
|
```
|
|
|
|
Only use resource types if you need them to satisfy the instructions. Always use the RT namespace.
|
|
|
|
## Imports
|
|
|
|
```typescript
|
|
import Stripe from "stripe";
|
|
import { someFunction } from "some-package";
|
|
```
|
|
|
|
## Prefer `//native` when the runtime allows it
|
|
|
|
If a script only needs `fetch` and the JavaScript standard library — including when it uses `windmill-client` — prefer making it a **native** script: add `//native` as the first line and write it with the `write-script-bunnative` skill. Native scripts run on a lightweight V8 isolate, start faster, and parallelize heavily. `windmill-client` works on the native worker (its calls go over `fetch`), so needing the Windmill client is **not** a reason to avoid `//native`. Use the regular `bun` language only when the code (or a dependency) needs Node/Bun runtime APIs — `node:*` modules, the filesystem, child processes, or native addons.
|
|
|
|
## Windmill Client
|
|
|
|
Import the windmill client for platform interactions:
|
|
|
|
```typescript
|
|
import * as wmill from "windmill-client";
|
|
```
|
|
|
|
**Prefer `windmill-client` over raw `fetch` for anything that talks to Windmill** — reading resources/variables/states, running scripts and flows, S3 object operations, etc. It handles auth, the workspace, and the base URL for you, so you don't hand-roll URLs or tokens. Reserve `fetch` for calling *external* HTTP APIs that aren't Windmill.
|
|
|
|
The full `windmill-client` API reference (every exported function and its signature) is included below — consult it for the exact method to use instead of guessing or falling back to `fetch`.
|
|
|
|
## Preprocessor Scripts
|
|
|
|
For preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:
|
|
|
|
```typescript
|
|
type Event = {
|
|
kind:
|
|
| "webhook"
|
|
| "http"
|
|
| "websocket"
|
|
| "kafka"
|
|
| "email"
|
|
| "nats"
|
|
| "postgres"
|
|
| "sqs"
|
|
| "mqtt"
|
|
| "amqp"
|
|
| "gcp"
|
|
| "azure";
|
|
body: any;
|
|
headers: Record<string, string>;
|
|
query: Record<string, string>;
|
|
};
|
|
|
|
export async function preprocessor(event: Event) {
|
|
return {
|
|
param1: event.body.field1,
|
|
param2: event.query.id,
|
|
};
|
|
}
|
|
```
|
|
|
|
## S3 Object Operations
|
|
|
|
Windmill provides built-in support for S3-compatible storage operations. The `wmill.S3Object` type covers both the `s3://storage/key` URI form (`s3:///key` for the workspace default storage) and the `{ s3, storage? }` record form — always use it instead of redefining your own.
|
|
|
|
### Receiving an S3Object as a script parameter
|
|
|
|
```typescript
|
|
import * as wmill from "windmill-client";
|
|
|
|
export async function main(file: wmill.S3Object) {
|
|
const content = await wmill.loadS3File(file);
|
|
// ...
|
|
}
|
|
```
|
|
|
|
### S3 operations
|
|
|
|
```typescript
|
|
import * as wmill from "windmill-client";
|
|
|
|
// Load file content from S3
|
|
const content: Uint8Array = await wmill.loadS3File(s3object);
|
|
|
|
// Load file as stream
|
|
const blob: Blob = await wmill.loadS3FileStream(s3object);
|
|
|
|
// Write file to S3
|
|
const result: wmill.S3Object = await wmill.writeS3File(
|
|
s3object, // Target path (or undefined to auto-generate)
|
|
fileContent, // string or Blob
|
|
s3ResourcePath // Optional: specific S3 resource to use
|
|
);
|
|
```
|
|
|
|
|
|
# TypeScript (Bun Native)
|
|
|
|
Native TypeScript execution. Native scripts are Bun scripts that run on the native worker — a lightweight V8 isolate that exposes `fetch` and the JavaScript standard library — and can be heavily parallelized. Every script MUST start with `//native` on its first line so Windmill routes it to the native worker; without it the exact same script runs on the regular Bun worker. You may import npm packages and other Windmill scripts (e.g. `./helper.ts`) — imports are resolved and bundled just like a regular Bun script — as long as everything (your code and its dependencies) relies only on `fetch` and the standard library. Libraries that need Node/Bun runtime APIs (filesystem, `node:*` modules, child processes, native addons) will not work on the native worker; use the regular `bun` language for those.
|
|
|
|
## Structure
|
|
|
|
Export a single **async** function called `main`:
|
|
|
|
```typescript
|
|
//native
|
|
export async function main(param1: string, param2: number) {
|
|
// Your code here
|
|
return { result: param1, count: param2 };
|
|
}
|
|
```
|
|
|
|
Do not call the main function.
|
|
|
|
## Resource Types
|
|
|
|
On Windmill, credentials and configuration are stored in resources and passed as parameters to main.
|
|
|
|
Use the `RT` namespace for resource types:
|
|
|
|
```typescript
|
|
//native
|
|
export async function main(stripe: RT.Stripe) {
|
|
// stripe contains API key and config from the resource
|
|
}
|
|
```
|
|
|
|
Only use resource types if you need them to satisfy the instructions. Always use the RT namespace.
|
|
|
|
## Imports
|
|
|
|
**The constraint is the runtime, not the import list.** You may import npm packages and relative Windmill scripts; they are resolved and bundled exactly like a regular Bun script. But the native worker only provides `fetch` and the JavaScript standard library, so any imported code must work using only those. Anything requiring Node/Bun built-ins (`node:fs`, `child_process`, the `Bun` API, native modules) belongs in a regular `bun` script instead. Use the globally available `fetch` for HTTP:
|
|
|
|
```typescript
|
|
//native
|
|
export async function main(url: string) {
|
|
const response = await fetch(url);
|
|
return await response.json();
|
|
}
|
|
```
|
|
|
|
## Windmill Client
|
|
|
|
`windmill-client` works on the native worker (its calls go over `fetch`), so use it as the **preferred way to talk to Windmill** — reading resources/variables/states, running scripts and flows, and the S3 helpers below (`loadS3File`, `loadS3FileStream`, `writeS3File`, `S3Object`). It handles auth, the workspace, and the base URL for you. Reserve raw `fetch` for calling *external* HTTP APIs that aren't Windmill.
|
|
|
|
The full `windmill-client` API reference (every exported function and its signature) is included below — consult it for the exact method instead of hand-rolling a `fetch` against the Windmill API.
|
|
|
|
## Preprocessor Scripts
|
|
|
|
For preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:
|
|
|
|
```typescript
|
|
//native
|
|
type Event = {
|
|
kind:
|
|
| "webhook"
|
|
| "http"
|
|
| "websocket"
|
|
| "kafka"
|
|
| "email"
|
|
| "nats"
|
|
| "postgres"
|
|
| "sqs"
|
|
| "mqtt"
|
|
| "amqp"
|
|
| "gcp"
|
|
| "azure";
|
|
body: any;
|
|
headers: Record<string, string>;
|
|
query: Record<string, string>;
|
|
};
|
|
|
|
export async function preprocessor(event: Event) {
|
|
return {
|
|
param1: event.body.field1,
|
|
param2: event.query.id,
|
|
};
|
|
}
|
|
```
|
|
|
|
## S3 Object Operations
|
|
|
|
Windmill provides built-in support for S3-compatible storage operations. The `wmill.S3Object` type covers both the `s3://storage/key` URI form (`s3:///key` for the workspace default storage) and the `{ s3, storage? }` record form — always use it instead of redefining your own.
|
|
|
|
### Receiving an S3Object as a script parameter
|
|
|
|
```typescript
|
|
//native
|
|
import * as wmill from "windmill-client";
|
|
|
|
export async function main(file: wmill.S3Object) {
|
|
const content = await wmill.loadS3File(file);
|
|
// ...
|
|
}
|
|
```
|
|
|
|
### S3 operations
|
|
|
|
```typescript
|
|
//native
|
|
import * as wmill from "windmill-client";
|
|
|
|
// Load file content from S3
|
|
const content: Uint8Array = await wmill.loadS3File(s3object);
|
|
|
|
// Load file as stream
|
|
const blob: Blob = await wmill.loadS3FileStream(s3object);
|
|
|
|
// Write file to S3
|
|
const result: wmill.S3Object = await wmill.writeS3File(
|
|
s3object, // Target path (or undefined to auto-generate)
|
|
fileContent, // string or Blob
|
|
s3ResourcePath // Optional: specific S3 resource to use
|
|
);
|
|
```
|
|
|
|
|
|
# C#
|
|
|
|
The script must contain a public static `Main` method inside a class:
|
|
|
|
```csharp
|
|
public class Script
|
|
{
|
|
public static object Main(string name, int count)
|
|
{
|
|
return new { Name = name, Count = count };
|
|
}
|
|
}
|
|
```
|
|
|
|
**Important:**
|
|
- Class name is irrelevant
|
|
- Method must be `public static`
|
|
- Return type can be `object` or specific type
|
|
|
|
## NuGet Packages
|
|
|
|
Add packages using the `#r` directive at the top:
|
|
|
|
```csharp
|
|
#r "nuget: Newtonsoft.Json, 13.0.3"
|
|
#r "nuget: RestSharp, 110.2.0"
|
|
|
|
using Newtonsoft.Json;
|
|
using RestSharp;
|
|
|
|
public class Script
|
|
{
|
|
public static object Main(string url)
|
|
{
|
|
var client = new RestClient(url);
|
|
var request = new RestRequest();
|
|
var response = client.Get(request);
|
|
return JsonConvert.DeserializeObject(response.Content);
|
|
}
|
|
}
|
|
```
|
|
|
|
|
|
# TypeScript (Deno)
|
|
|
|
Deno runtime with npm support via `npm:` prefix and native Deno libraries.
|
|
|
|
**Prefer Bun (`write-script-bun`) for TypeScript.** Only use Deno when the script specifically requires the Deno runtime — Deno's standard library or `deno.land` URL imports that have no npm equivalent. For all other TypeScript, use Bun instead.
|
|
|
|
## Structure
|
|
|
|
Export a single **async** function called `main`:
|
|
|
|
```typescript
|
|
export async function main(param1: string, param2: number) {
|
|
// Your code here
|
|
return { result: param1, count: param2 };
|
|
}
|
|
```
|
|
|
|
Do not call the main function. Libraries are installed automatically.
|
|
|
|
## Resource Types
|
|
|
|
On Windmill, credentials and configuration are stored in resources and passed as parameters to main.
|
|
|
|
Use the `RT` namespace for resource types:
|
|
|
|
```typescript
|
|
export async function main(stripe: RT.Stripe) {
|
|
// stripe contains API key and config from the resource
|
|
}
|
|
```
|
|
|
|
Only use resource types if you need them to satisfy the instructions. Always use the RT namespace.
|
|
|
|
## Imports
|
|
|
|
```typescript
|
|
// npm packages use npm: prefix
|
|
import Stripe from "npm:stripe";
|
|
import { someFunction } from "npm:some-package";
|
|
|
|
// Deno standard library
|
|
import { serve } from "https://deno.land/std/http/server.ts";
|
|
```
|
|
|
|
## Windmill Client
|
|
|
|
Import the windmill client for platform interactions:
|
|
|
|
```typescript
|
|
import * as wmill from "windmill-client";
|
|
```
|
|
|
|
**Prefer `windmill-client` over raw `fetch` for anything that talks to Windmill** — reading resources/variables/states, running scripts and flows, S3 object operations, etc. It handles auth, the workspace, and the base URL for you. Reserve `fetch` for calling *external* HTTP APIs that aren't Windmill.
|
|
|
|
The full `windmill-client` API reference (every exported function and its signature) is included below — consult it for the exact method instead of guessing or falling back to `fetch`.
|
|
|
|
## Preprocessor Scripts
|
|
|
|
For preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:
|
|
|
|
```typescript
|
|
type Event = {
|
|
kind:
|
|
| "webhook"
|
|
| "http"
|
|
| "websocket"
|
|
| "kafka"
|
|
| "email"
|
|
| "nats"
|
|
| "postgres"
|
|
| "sqs"
|
|
| "mqtt"
|
|
| "amqp"
|
|
| "gcp"
|
|
| "azure";
|
|
body: any;
|
|
headers: Record<string, string>;
|
|
query: Record<string, string>;
|
|
};
|
|
|
|
export async function preprocessor(event: Event) {
|
|
return {
|
|
param1: event.body.field1,
|
|
param2: event.query.id,
|
|
};
|
|
}
|
|
```
|
|
|
|
## S3 Object Operations
|
|
|
|
Windmill provides built-in support for S3-compatible storage operations. The `wmill.S3Object` type covers both the `s3://storage/key` URI form (`s3:///key` for the workspace default storage) and the `{ s3, storage? }` record form — always use it instead of redefining your own.
|
|
|
|
### Receiving an S3Object as a script parameter
|
|
|
|
```typescript
|
|
import * as wmill from "windmill-client";
|
|
|
|
export async function main(file: wmill.S3Object) {
|
|
const content = await wmill.loadS3File(file);
|
|
// ...
|
|
}
|
|
```
|
|
|
|
### S3 operations
|
|
|
|
```typescript
|
|
import * as wmill from "windmill-client";
|
|
|
|
// Load file content from S3
|
|
const content: Uint8Array = await wmill.loadS3File(s3object);
|
|
|
|
// Load file as stream
|
|
const blob: Blob = await wmill.loadS3FileStream(s3object);
|
|
|
|
// Write file to S3
|
|
const result: wmill.S3Object = await wmill.writeS3File(
|
|
s3object, // Target path (or undefined to auto-generate)
|
|
fileContent, // string or Blob
|
|
s3ResourcePath // Optional: specific S3 resource to use
|
|
);
|
|
```
|
|
|
|
|
|
# DuckDB
|
|
|
|
Arguments are defined with comments and used with `$name` syntax:
|
|
|
|
```sql
|
|
-- $name (text) = default
|
|
-- $age (integer)
|
|
SELECT * FROM users WHERE name = $name AND age > $age;
|
|
```
|
|
|
|
## Ducklake Integration
|
|
|
|
Attach Ducklake for data lake operations:
|
|
|
|
```sql
|
|
-- Main ducklake
|
|
ATTACH 'ducklake' AS dl;
|
|
|
|
-- Named ducklake
|
|
ATTACH 'ducklake://my_lake' AS dl;
|
|
|
|
-- Then query
|
|
SELECT * FROM dl.schema.table;
|
|
```
|
|
|
|
## External Database Connections
|
|
|
|
Connect to external databases using resources:
|
|
|
|
```sql
|
|
ATTACH '$res:path/to/resource' AS db (TYPE postgres);
|
|
SELECT * FROM db.schema.table;
|
|
```
|
|
|
|
## S3 File Operations
|
|
|
|
Read files from S3 storage:
|
|
|
|
```sql
|
|
-- Default storage
|
|
SELECT * FROM read_csv('s3:///path/to/file.csv');
|
|
|
|
-- Named storage
|
|
SELECT * FROM read_csv('s3://storage_name/path/to/file.csv');
|
|
|
|
-- Parquet files
|
|
SELECT * FROM read_parquet('s3:///path/to/file.parquet');
|
|
|
|
-- JSON files
|
|
SELECT * FROM read_json('s3:///path/to/file.json');
|
|
```
|
|
|
|
### Receiving an S3Object as a script parameter
|
|
|
|
Declare the arg with type `(s3object)`. Windmill renders an S3 file picker for it
|
|
and binds the arg as the bare `s3://storage/key` URI, which DuckDB's reader
|
|
functions consume directly:
|
|
|
|
```sql
|
|
-- $file (s3object)
|
|
SELECT * FROM read_parquet($file);
|
|
```
|
|
|
|
Works with any DuckDB reader: `read_csv($file)`, `read_json($file)`, etc.
|
|
|
|
### Writing query results to S3
|
|
|
|
DuckDB writes to S3 natively via `COPY ... TO`:
|
|
|
|
```sql
|
|
COPY (SELECT * FROM users) TO 's3:///exports/users.parquet' (FORMAT PARQUET);
|
|
```
|
|
|
|
Use this instead of the `-- s3` streaming directive supported by the other SQL
|
|
dialects — that directive is not available in DuckDB.
|
|
|
|
|
|
# Go
|
|
|
|
## Structure
|
|
|
|
The file package must be `inner` and export a function called `main`:
|
|
|
|
```go
|
|
package inner
|
|
|
|
func main(param1 string, param2 int) (map[string]interface{}, error) {
|
|
return map[string]interface{}{
|
|
"result": param1,
|
|
"count": param2,
|
|
}, nil
|
|
}
|
|
```
|
|
|
|
**Important:**
|
|
- Package must be `inner`
|
|
- Return type must be `({return_type}, error)`
|
|
- Function name is `main` (lowercase)
|
|
|
|
## Return Types
|
|
|
|
The return type can be any Go type that can be serialized to JSON:
|
|
|
|
```go
|
|
package inner
|
|
|
|
type Result struct {
|
|
Name string `json:"name"`
|
|
Count int `json:"count"`
|
|
}
|
|
|
|
func main(name string, count int) (Result, error) {
|
|
return Result{
|
|
Name: name,
|
|
Count: count,
|
|
}, nil
|
|
}
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
Return errors as the second return value:
|
|
|
|
```go
|
|
package inner
|
|
|
|
import "errors"
|
|
|
|
func main(value int) (string, error) {
|
|
if value < 0 {
|
|
return "", errors.New("value must be positive")
|
|
}
|
|
return "success", nil
|
|
}
|
|
```
|
|
|
|
|
|
# GraphQL
|
|
|
|
## Structure
|
|
|
|
Write GraphQL queries or mutations. Arguments can be added as query parameters:
|
|
|
|
```graphql
|
|
query GetUser($id: ID!) {
|
|
user(id: $id) {
|
|
id
|
|
name
|
|
email
|
|
}
|
|
}
|
|
```
|
|
|
|
## Variables
|
|
|
|
Variables are passed as script arguments and automatically bound to the query:
|
|
|
|
```graphql
|
|
query SearchProducts($query: String!, $limit: Int = 10) {
|
|
products(search: $query, first: $limit) {
|
|
edges {
|
|
node {
|
|
id
|
|
name
|
|
price
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Mutations
|
|
|
|
```graphql
|
|
mutation CreateUser($input: CreateUserInput!) {
|
|
createUser(input: $input) {
|
|
id
|
|
name
|
|
createdAt
|
|
}
|
|
}
|
|
```
|
|
|
|
|
|
# Java
|
|
|
|
The script must contain a Main public class with a `public static main()` method:
|
|
|
|
```java
|
|
public class Main {
|
|
public static Object main(String name, int count) {
|
|
java.util.Map<String, Object> result = new java.util.HashMap<>();
|
|
result.put("name", name);
|
|
result.put("count", count);
|
|
return result;
|
|
}
|
|
}
|
|
```
|
|
|
|
**Important:**
|
|
- Class must be named `Main`
|
|
- Method must be `public static Object main(...)`
|
|
- Return type is `Object` or `void`
|
|
|
|
## Maven Dependencies
|
|
|
|
Add dependencies using comments at the top:
|
|
|
|
```java
|
|
//requirements:
|
|
//com.google.code.gson:gson:2.10.1
|
|
//org.apache.httpcomponents:httpclient:4.5.14
|
|
|
|
import com.google.gson.Gson;
|
|
|
|
public class Main {
|
|
public static Object main(String input) {
|
|
Gson gson = new Gson();
|
|
return gson.fromJson(input, Object.class);
|
|
}
|
|
}
|
|
```
|
|
|
|
|
|
# Microsoft SQL Server (MSSQL)
|
|
|
|
Arguments use `@P1`, `@P2`, etc.
|
|
|
|
Name the parameters by adding comments before the statement:
|
|
|
|
```sql
|
|
-- @P1 name1 (varchar)
|
|
-- @P2 name2 (int) = 0
|
|
SELECT * FROM users WHERE name = @P1 AND age > @P2;
|
|
```
|
|
|
|
## Receiving an S3Object as a script parameter
|
|
|
|
Declare the arg with type `(s3object)`. Windmill renders an S3 file picker for
|
|
it, downloads the file, and binds it as `nvarchar(max)` JSON text — Parquet/CSV
|
|
files are decoded server-side into a JSON array of records, JSON/JSONL pass
|
|
through. Consume with `OPENJSON`:
|
|
|
|
```sql
|
|
-- @P1 file (s3object)
|
|
SELECT id, name
|
|
FROM OPENJSON(@P1)
|
|
WITH (id INT, name NVARCHAR(200));
|
|
```
|
|
|
|
## Streaming query results to S3
|
|
|
|
Add a `-- s3` directive at the top of the script to stream the result set to S3
|
|
instead of returning rows. Windmill writes the file and returns its `S3Object`
|
|
as the script result.
|
|
|
|
```sql
|
|
-- s3 prefix=exports/users format=parquet
|
|
SELECT id, name FROM users;
|
|
```
|
|
|
|
All keys are optional: `prefix` (object key prefix), `storage` (named storage —
|
|
omit to use the workspace default), `format` (`json` (default), `parquet`, or
|
|
`csv`). Use this for large result sets — rows stream directly to S3 instead of
|
|
being buffered as the script return value.
|
|
|
|
|
|
# MySQL
|
|
|
|
Arguments use `?` placeholders.
|
|
|
|
Name the parameters by adding comments before the statement:
|
|
|
|
```sql
|
|
-- ? name1 (text)
|
|
-- ? name2 (int) = 0
|
|
SELECT * FROM users WHERE name = ? AND age > ?;
|
|
```
|
|
|
|
## Receiving an S3Object as a script parameter
|
|
|
|
Declare the arg with type `(s3object)`. Windmill renders an S3 file picker for
|
|
it, downloads the file, and binds it as JSON text — Parquet/CSV files are
|
|
decoded server-side into a JSON array of records, JSON/JSONL pass through.
|
|
Consume with `JSON_TABLE`:
|
|
|
|
```sql
|
|
-- ? file (s3object)
|
|
SELECT id, name
|
|
FROM JSON_TABLE(?, '$[*]'
|
|
COLUMNS (id INT PATH '$.id', name VARCHAR(200) PATH '$.name')
|
|
) AS r;
|
|
```
|
|
|
|
## Streaming query results to S3
|
|
|
|
Add a `-- s3` directive at the top of the script to stream the result set to S3
|
|
instead of returning rows. Windmill writes the file and returns its `S3Object`
|
|
as the script result.
|
|
|
|
```sql
|
|
-- s3 prefix=exports/users format=parquet
|
|
SELECT id, name FROM users;
|
|
```
|
|
|
|
All keys are optional: `prefix` (object key prefix), `storage` (named storage —
|
|
omit to use the workspace default), `format` (`json` (default), `parquet`, or
|
|
`csv`). Use this for large result sets — rows stream directly to S3 instead of
|
|
being buffered as the script return value.
|
|
|
|
|
|
# PHP
|
|
|
|
## Structure
|
|
|
|
The script must start with `<?php` and contain at least one function called `main`:
|
|
|
|
```php
|
|
<?php
|
|
|
|
function main(string $param1, int $param2) {
|
|
return ["result" => $param1, "count" => $param2];
|
|
}
|
|
```
|
|
|
|
## Resource Types
|
|
|
|
On Windmill, credentials and configuration are stored in resources and passed as parameters to main.
|
|
|
|
You need to **redefine** the type of the resources that are needed before the main function. Always check if the class already exists using `class_exists`:
|
|
|
|
```php
|
|
<?php
|
|
|
|
if (!class_exists('Postgresql')) {
|
|
class Postgresql {
|
|
public string $host;
|
|
public int $port;
|
|
public string $user;
|
|
public string $password;
|
|
public string $dbname;
|
|
}
|
|
}
|
|
|
|
function main(Postgresql $db) {
|
|
// $db contains the database connection details
|
|
}
|
|
```
|
|
|
|
The resource type name has to be exactly as specified.
|
|
|
|
## Library Dependencies
|
|
|
|
Specify library dependencies as comments before the main function:
|
|
|
|
```php
|
|
<?php
|
|
|
|
// require:
|
|
// guzzlehttp/guzzle
|
|
// stripe/stripe-php@^10.0
|
|
|
|
function main() {
|
|
// Libraries are available
|
|
}
|
|
```
|
|
|
|
One dependency per line. No need to require autoload, it is already done.
|
|
|
|
|
|
# PostgreSQL
|
|
|
|
Arguments are obtained directly in the statement with `$1::{type}`, `$2::{type}`, etc.
|
|
|
|
Name the parameters by adding comments at the beginning of the script (without specifying the type):
|
|
|
|
```sql
|
|
-- $1 name1
|
|
-- $2 name2 = default_value
|
|
SELECT * FROM users WHERE name = $1::TEXT AND age > $2::INT;
|
|
```
|
|
|
|
## Receiving an S3Object as a script parameter
|
|
|
|
Declare the arg with type `(s3object)`. Windmill renders an S3 file picker for
|
|
it, downloads the file, and binds it as a `jsonb` parameter — Parquet/CSV files
|
|
are decoded server-side into a JSON array of records, JSON/JSONL pass through.
|
|
Consume with `jsonb_to_recordset` (or any `jsonb` API):
|
|
|
|
```sql
|
|
-- $1 file (s3object)
|
|
SELECT *
|
|
FROM jsonb_to_recordset($1::jsonb) AS r(id INT, name TEXT);
|
|
```
|
|
|
|
## Streaming query results to S3
|
|
|
|
Add a `-- s3` directive at the top of the script to stream the result set to S3
|
|
instead of returning rows. Windmill writes the file and returns its `S3Object`
|
|
as the script result.
|
|
|
|
```sql
|
|
-- s3 prefix=exports/users format=parquet
|
|
SELECT id, name FROM users;
|
|
```
|
|
|
|
All keys are optional: `prefix` (object key prefix), `storage` (named storage —
|
|
omit to use the workspace default), `format` (`json` (default), `parquet`, or
|
|
`csv`). Use this for large result sets — rows stream directly to S3 instead of
|
|
being buffered as the script return value.
|
|
|
|
|
|
# PowerShell
|
|
|
|
## Structure
|
|
|
|
Arguments are obtained by calling the `param` function on the first line:
|
|
|
|
```powershell
|
|
param($Name, $Count = 0, [int]$Age)
|
|
|
|
# Your code here
|
|
Write-Output "Processing $Name, count: $Count, age: $Age"
|
|
|
|
# Return object
|
|
@{
|
|
name = $Name
|
|
count = $Count
|
|
age = $Age
|
|
}
|
|
```
|
|
|
|
## Parameter Types
|
|
|
|
You can specify types for parameters:
|
|
|
|
```powershell
|
|
param(
|
|
[string]$Name,
|
|
[int]$Count = 0,
|
|
[bool]$Enabled = $true,
|
|
[array]$Items
|
|
)
|
|
|
|
@{
|
|
name = $Name
|
|
count = $Count
|
|
enabled = $Enabled
|
|
items = $Items
|
|
}
|
|
```
|
|
|
|
## Return Values
|
|
|
|
Return values by outputting them at the end of the script:
|
|
|
|
```powershell
|
|
param($Input)
|
|
|
|
$result = @{
|
|
processed = $true
|
|
data = $Input
|
|
timestamp = Get-Date -Format "o"
|
|
}
|
|
|
|
$result
|
|
```
|
|
|
|
|
|
# Python
|
|
|
|
## Structure
|
|
|
|
The script must contain at least one function called `main`:
|
|
|
|
```python
|
|
def main(param1: str, param2: int):
|
|
# Your code here
|
|
return {"result": param1, "count": param2}
|
|
```
|
|
|
|
Do not call the main function. Libraries are installed automatically.
|
|
|
|
## Resource Types
|
|
|
|
On Windmill, credentials and configuration are stored in resources and passed as parameters to main.
|
|
|
|
You need to **redefine** the type of the resources that are needed before the main function as TypedDict:
|
|
|
|
```python
|
|
from typing import TypedDict
|
|
|
|
class postgresql(TypedDict):
|
|
host: str
|
|
port: int
|
|
user: str
|
|
password: str
|
|
dbname: str
|
|
|
|
def main(db: postgresql):
|
|
# db contains the database connection details
|
|
pass
|
|
```
|
|
|
|
**Important rules:**
|
|
|
|
- The resource type name must be **IN LOWERCASE**
|
|
- Only include resource types if they are actually needed
|
|
- If an import conflicts with a resource type name, **rename the imported object, not the type name**
|
|
- Make sure to import TypedDict from typing **if you're using it**
|
|
|
|
## Imports
|
|
|
|
Libraries are installed automatically. Do not show installation instructions.
|
|
|
|
```python
|
|
import requests
|
|
import pandas as pd
|
|
from datetime import datetime
|
|
```
|
|
|
|
If an import name conflicts with a resource type:
|
|
|
|
```python
|
|
# Wrong - don't rename the type
|
|
import stripe as stripe_lib
|
|
class stripe_type(TypedDict): ...
|
|
|
|
# Correct - rename the import
|
|
import stripe as stripe_sdk
|
|
class stripe(TypedDict):
|
|
api_key: str
|
|
```
|
|
|
|
## Windmill Client
|
|
|
|
Import the windmill client for platform interactions:
|
|
|
|
```python
|
|
import wmill
|
|
```
|
|
|
|
See the SDK documentation for available methods.
|
|
|
|
## Preprocessor Scripts
|
|
|
|
For preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:
|
|
|
|
```python
|
|
from typing import TypedDict, Literal, Any
|
|
|
|
class Event(TypedDict):
|
|
kind: Literal["webhook", "http", "websocket", "kafka", "email", "nats", "postgres", "sqs", "mqtt", "amqp", "gcp", "azure"]
|
|
body: Any
|
|
headers: dict[str, str]
|
|
query: dict[str, str]
|
|
|
|
def preprocessor(event: Event):
|
|
# Transform the event into flow input parameters
|
|
return {
|
|
"param1": event["body"]["field1"],
|
|
"param2": event["query"]["id"]
|
|
}
|
|
```
|
|
|
|
## S3 Object Operations
|
|
|
|
Windmill provides built-in support for S3-compatible storage operations.
|
|
|
|
### Receiving an S3Object as a script parameter
|
|
|
|
To accept a file from S3 as input to a script, type the parameter with `S3Object` (imported from `wmill`):
|
|
|
|
```python
|
|
import wmill
|
|
from wmill import S3Object
|
|
|
|
def main(file: S3Object):
|
|
content = wmill.load_s3_file(file)
|
|
# ...
|
|
```
|
|
|
|
### S3 operations
|
|
|
|
```python
|
|
import wmill
|
|
|
|
# Load file content from S3
|
|
content: bytes = wmill.load_s3_file(s3object)
|
|
|
|
# Load file as stream reader
|
|
reader: BufferedReader = wmill.load_s3_file_reader(s3object)
|
|
|
|
# Write file to S3
|
|
result: S3Object = wmill.write_s3_file(
|
|
s3object, # Target path (or None to auto-generate)
|
|
file_content, # bytes or BufferedReader
|
|
s3_resource_path, # Optional: specific S3 resource
|
|
content_type, # Optional: MIME type
|
|
content_disposition # Optional: Content-Disposition header
|
|
)
|
|
```
|
|
|
|
|
|
# R
|
|
|
|
## Structure
|
|
|
|
Define a `main` function using `<-` or `=` assignment. Parameters become the script inputs:
|
|
|
|
```r
|
|
library(dplyr)
|
|
library(jsonlite)
|
|
|
|
main <- function(x, name = "default", flag = TRUE) {
|
|
df <- tibble(x = x, name = name)
|
|
result <- df %>% mutate(greeting = paste("Hello", name))
|
|
return(toJSON(result, auto_unbox = TRUE))
|
|
}
|
|
```
|
|
|
|
**Important:**
|
|
- The `main` function is required
|
|
- Use `library()` to load packages — they are resolved and installed automatically
|
|
- `jsonlite` is always available (used internally for argument parsing)
|
|
- Return values must be JSON-serializable
|
|
|
|
## Parameters
|
|
|
|
R types map to Windmill types:
|
|
- `numeric` → float/int
|
|
- `character` → string
|
|
- `logical` → bool (use `TRUE`/`FALSE`)
|
|
- `list` → object/dict
|
|
- `NULL` → null
|
|
|
|
Default values are inferred from the function signature:
|
|
|
|
```r
|
|
main <- function(
|
|
name, # required string
|
|
count = 10, # optional int, default 10
|
|
verbose = FALSE # optional bool, default FALSE
|
|
) {
|
|
# ...
|
|
}
|
|
```
|
|
|
|
## Resources and Variables
|
|
|
|
Use the built-in Windmill helpers (no import needed):
|
|
|
|
```r
|
|
main <- function() {
|
|
# Get a variable
|
|
api_key <- get_variable("f/my_folder/api_key")
|
|
|
|
# Get a resource (returns a list)
|
|
db <- get_resource("f/my_folder/postgres_config")
|
|
host <- db$host
|
|
port <- db$port
|
|
|
|
return(list(host = host, port = port))
|
|
}
|
|
```
|
|
|
|
## Output
|
|
|
|
Return any JSON-serializable value from `main`. The return value becomes the step result:
|
|
|
|
```r
|
|
main <- function(x) {
|
|
# Return a scalar
|
|
return(x + 1)
|
|
|
|
# Or a list (becomes JSON object)
|
|
return(list(result = x + 1, status = "ok"))
|
|
}
|
|
```
|
|
|
|
## Annotations
|
|
|
|
Control execution behavior with comment annotations:
|
|
|
|
```r
|
|
#renv_verbose = true # Show verbose renv output during resolution
|
|
#renv_install_verbose = true # Show verbose output during package installation
|
|
#sandbox = true # Run in nsjail sandbox (requires nsjail)
|
|
```
|
|
|
|
|
|
# Rust
|
|
|
|
## Structure
|
|
|
|
The script must contain a function called `main` with proper return type:
|
|
|
|
```rust
|
|
use anyhow::anyhow;
|
|
use serde::Serialize;
|
|
|
|
#[derive(Serialize, Debug)]
|
|
struct ReturnType {
|
|
result: String,
|
|
count: i32,
|
|
}
|
|
|
|
fn main(param1: String, param2: i32) -> anyhow::Result<ReturnType> {
|
|
Ok(ReturnType {
|
|
result: param1,
|
|
count: param2,
|
|
})
|
|
}
|
|
```
|
|
|
|
**Important:**
|
|
- Arguments should be owned types
|
|
- Return type must be serializable (`#[derive(Serialize)]`)
|
|
- Return type is `anyhow::Result<T>`
|
|
|
|
## Dependencies
|
|
|
|
Packages must be specified with a partial cargo.toml at the beginning of the script:
|
|
|
|
```rust
|
|
//! ```cargo
|
|
//! [dependencies]
|
|
//! anyhow = "1.0.86"
|
|
//! reqwest = { version = "0.11", features = ["json"] }
|
|
//! tokio = { version = "1", features = ["full"] }
|
|
//! ```
|
|
|
|
use anyhow::anyhow;
|
|
// ... rest of the code
|
|
```
|
|
|
|
**Note:** Serde is already included, no need to add it again.
|
|
|
|
## Async Functions
|
|
|
|
If you need to handle async functions (e.g., using tokio), keep the main function sync and create the runtime inside:
|
|
|
|
```rust
|
|
//! ```cargo
|
|
//! [dependencies]
|
|
//! anyhow = "1.0.86"
|
|
//! tokio = { version = "1", features = ["full"] }
|
|
//! reqwest = { version = "0.11", features = ["json"] }
|
|
//! ```
|
|
|
|
use anyhow::anyhow;
|
|
use serde::Serialize;
|
|
|
|
#[derive(Serialize, Debug)]
|
|
struct Response {
|
|
data: String,
|
|
}
|
|
|
|
fn main(url: String) -> anyhow::Result<Response> {
|
|
let rt = tokio::runtime::Runtime::new()?;
|
|
rt.block_on(async {
|
|
let resp = reqwest::get(&url).await?.text().await?;
|
|
Ok(Response { data: resp })
|
|
})
|
|
}
|
|
```
|
|
|
|
|
|
# Snowflake
|
|
|
|
Arguments use `?` placeholders.
|
|
|
|
Name the parameters by adding comments before the statement:
|
|
|
|
```sql
|
|
-- ? name1 (text)
|
|
-- ? name2 (number) = 0
|
|
SELECT * FROM users WHERE name = ? AND age > ?;
|
|
```
|
|
|
|
## Receiving an S3Object as a script parameter
|
|
|
|
Declare the arg with type `(s3object)`. Windmill renders an S3 file picker for
|
|
it, downloads the file, and binds it as JSON text — Parquet/CSV files are
|
|
decoded server-side into a JSON array of records, JSON/JSONL pass through.
|
|
Wrap the bind with `PARSE_JSON(?)` and walk it with `LATERAL FLATTEN`:
|
|
|
|
```sql
|
|
-- ? file (s3object)
|
|
SELECT
|
|
v.value:id::NUMBER AS id,
|
|
v.value:name::STRING AS name
|
|
FROM LATERAL FLATTEN(input => PARSE_JSON(?)) v;
|
|
```
|
|
|
|
## Streaming query results to S3
|
|
|
|
Add a `-- s3` directive at the top of the script to stream the result set to S3
|
|
instead of returning rows. Windmill writes the file and returns its `S3Object`
|
|
as the script result.
|
|
|
|
```sql
|
|
-- s3 prefix=exports/users format=parquet
|
|
SELECT id, name FROM users;
|
|
```
|
|
|
|
All keys are optional: `prefix` (object key prefix), `storage` (named storage —
|
|
omit to use the workspace default), `format` (`json` (default), `parquet`, or
|
|
`csv`). Use this for large result sets — rows stream directly to S3 instead of
|
|
being buffered, bypassing the 10000-row return cap.
|
|
|
|
|
|
# TypeScript SDK (windmill-client)
|
|
|
|
Import: import * as wmill from 'windmill-client'
|
|
|
|
The client configures itself from the job's environment — base URL, token and credentials mode
|
|
are all set before your code runs, so there is nothing to initialize and no reason to read
|
|
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
|
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
|
HTTP for third-party APIs.
|
|
|
|
The helpers below are the surface to prefer. For an endpoint none of them covers, import the
|
|
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
|
|
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
|
|
neither listed below nor a service method, do not call it.
|
|
|
|
To know who is running the script, read the contextual variables rather than calling the API:
|
|
`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL`. WM_END_USER_EMAIL is the app viewer when
|
|
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
|
|
is the user the job is permissioned as. WM_USERNAME is the matching username.
|
|
|
|
workerHasInternalServer(): boolean
|
|
|
|
/**
|
|
* Initialize the Windmill client with authentication token and base URL
|
|
* @param token - Authentication token (defaults to WM_TOKEN env variable)
|
|
* @param baseUrl - API base URL (defaults to BASE_INTERNAL_URL or BASE_URL env variable)
|
|
*/
|
|
setClient(token?: string, baseUrl?: string): void
|
|
|
|
/**
|
|
* Create a client configuration from env variables
|
|
* @returns client configuration
|
|
*/
|
|
getWorkspace(): string
|
|
|
|
/**
|
|
* Get a resource value by path
|
|
* @param path path of the resource, default to internal state path
|
|
* @param undefinedIfEmpty if the resource does not exist, return undefined instead of throwing an error
|
|
* @returns resource value
|
|
*/
|
|
async getResource(path?: string, undefinedIfEmpty?: boolean): Promise<any>
|
|
|
|
/**
|
|
* Get the true root job id
|
|
* @param jobId job id to get the root job id from (default to current job)
|
|
* @returns root job id
|
|
*/
|
|
async getRootJobId(jobId?: string): Promise<string>
|
|
|
|
/**
|
|
* Run a script synchronously by its path and wait for the result
|
|
* @param path - Script path in Windmill
|
|
* @param args - Arguments to pass to the script
|
|
* @param verbose - Enable verbose logging
|
|
* @param tag - Override the worker tag the job runs on
|
|
* @returns Script execution result
|
|
*/
|
|
async runScriptByPath(path: string, args: Record<string, any> | null = null, verbose: boolean = false, tag: string | null = null): Promise<any>
|
|
|
|
/**
|
|
* Run a script synchronously by its hash and wait for the result
|
|
* @param hash_ - Script hash in Windmill
|
|
* @param args - Arguments to pass to the script
|
|
* @param verbose - Enable verbose logging
|
|
* @param tag - Override the worker tag the job runs on
|
|
* @returns Script execution result
|
|
*/
|
|
async runScriptByHash(hash_: string, args: Record<string, any> | null = null, verbose: boolean = false, tag: string | null = null): Promise<any>
|
|
|
|
/**
|
|
* Append a text to the result stream
|
|
* @param text text to append to the result stream
|
|
*/
|
|
appendToResultStream(text: string): void
|
|
|
|
/**
|
|
* Stream to the result stream
|
|
* @param stream stream to stream to the result stream
|
|
*/
|
|
async streamResult(stream: AsyncIterable<string>): Promise<void>
|
|
|
|
/**
|
|
* Run a flow synchronously by its path and wait for the result
|
|
* @param path - Flow path in Windmill
|
|
* @param args - Arguments to pass to the flow
|
|
* @param verbose - Enable verbose logging
|
|
* @param tag - Override the worker tag the job runs on
|
|
* @returns Flow execution result
|
|
*/
|
|
async runFlow(path: string | null = null, args: Record<string, any> | null = null, verbose: boolean = false, tag: string | null = null): Promise<any>
|
|
|
|
/**
|
|
* Wait for a job to complete and return its result
|
|
* @param jobId - ID of the job to wait for
|
|
* @param verbose - Enable verbose logging
|
|
* @returns Job result when completed
|
|
*/
|
|
async waitJob(jobId: string, verbose: boolean = false): Promise<any>
|
|
|
|
/**
|
|
* Get the result of a completed job
|
|
* @param jobId - ID of the completed job
|
|
* @returns Job result
|
|
*/
|
|
async getResult(jobId: string): Promise<any>
|
|
|
|
/**
|
|
* Get the result of a job if completed, or its current status
|
|
* @param jobId - ID of the job
|
|
* @returns Object with started, completed, success, and result properties
|
|
*/
|
|
async getResultMaybe(jobId: string): Promise<any>
|
|
|
|
/**
|
|
* Cancel a queued or running job by ID.
|
|
* @param jobId - UUID of the job to cancel
|
|
* @param reason - Optional reason for cancellation
|
|
* @returns Response message from the cancel endpoint
|
|
*/
|
|
async cancelJob(jobId: string, reason: string | undefined = undefined): Promise<string>
|
|
|
|
/**
|
|
* Run a script asynchronously by its path
|
|
* @param path - Script path in Windmill
|
|
* @param args - Arguments to pass to the script
|
|
* @param scheduledInSeconds - Schedule execution for a future time (in seconds)
|
|
* @param tag - Override the worker tag the job runs on
|
|
* @returns Job ID of the created job
|
|
*/
|
|
async runScriptByPathAsync(path: string, args: Record<string, any> | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise<string>
|
|
|
|
/**
|
|
* Run a script asynchronously by its hash
|
|
* @param hash_ - Script hash in Windmill
|
|
* @param args - Arguments to pass to the script
|
|
* @param scheduledInSeconds - Schedule execution for a future time (in seconds)
|
|
* @param tag - Override the worker tag the job runs on
|
|
* @returns Job ID of the created job
|
|
*/
|
|
async runScriptByHashAsync(hash_: string, args: Record<string, any> | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise<string>
|
|
|
|
/**
|
|
* Run a flow asynchronously by its path
|
|
* @param path - Flow path in Windmill
|
|
* @param args - Arguments to pass to the flow
|
|
* @param scheduledInSeconds - Schedule execution for a future time (in seconds)
|
|
* @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job)
|
|
* @param tag - Override the worker tag the job runs on
|
|
* @returns Job ID of the created job
|
|
*/
|
|
async runFlowAsync(path: string | null, args: Record<string, any> | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise<string>
|
|
|
|
/**
|
|
* Resolve a resource value in case the default value was picked because the input payload was undefined
|
|
* @param obj resource value or path of the resource under the format `$res:path`
|
|
* @returns resource value
|
|
*/
|
|
async resolveDefaultResource(obj: any): Promise<any>
|
|
|
|
/**
|
|
* Get the state file path from environment variables
|
|
* @returns State path string
|
|
*/
|
|
getStatePath(): string
|
|
|
|
/**
|
|
* Set a resource value by path
|
|
* @param path path of the resource to set, default to state path
|
|
* @param value new value of the resource to set
|
|
* @param initializeToTypeIfNotExist if the resource does not exist, initialize it with this type
|
|
*/
|
|
async setResource(value: any, path?: string, initializeToTypeIfNotExist?: string): Promise<void>
|
|
|
|
/**
|
|
* Set the state
|
|
* @param state state to set
|
|
* @param path Optional state resource path override. Defaults to `getStatePath()`.
|
|
*/
|
|
async setState(state: any, path?: string): Promise<void>
|
|
|
|
/**
|
|
* Set the progress
|
|
* Progress cannot go back and limited to 0% to 99% range
|
|
* @param percent Progress to set in %
|
|
* @param jobId? Job to set progress for
|
|
*/
|
|
async setProgress(percent: number, jobId?: any): Promise<void>
|
|
|
|
/**
|
|
* Get the progress
|
|
* @param jobId? Job to get progress from
|
|
* @returns Optional clamped between 0 and 100 progress value
|
|
*/
|
|
async getProgress(jobId?: any): Promise<number | null>
|
|
|
|
/**
|
|
* Set a flow user state
|
|
* @param key key of the state
|
|
* @param value value of the state
|
|
*/
|
|
async setFlowUserState(key: string, value: any, errorIfNotPossible?: boolean): Promise<void>
|
|
|
|
/**
|
|
* Get a flow user state
|
|
* @param path path of the variable
|
|
*/
|
|
async getFlowUserState(key: string, errorIfNotPossible?: boolean): Promise<any>
|
|
|
|
/**
|
|
* Get the state shared across executions
|
|
* @param path Optional state resource path override. Defaults to `getStatePath()`.
|
|
*/
|
|
async getState(path?: string): Promise<any>
|
|
|
|
/**
|
|
* Get a variable by path
|
|
* @param path path of the variable
|
|
* @returns variable value
|
|
*/
|
|
async getVariable(path: string): Promise<string>
|
|
|
|
/**
|
|
* Set a variable by path, create if not exist
|
|
* @param path path of the variable
|
|
* @param value value of the variable
|
|
* @param isSecretIfNotExist if the variable does not exist, create it as secret or not (default: false)
|
|
* @param descriptionIfNotExist if the variable does not exist, create it with this description (default: "")
|
|
*/
|
|
async setVariable(path: string, value: string, isSecretIfNotExist?: boolean, descriptionIfNotExist?: string): Promise<void>
|
|
|
|
/**
|
|
* Build a PostgreSQL connection URL from a database resource
|
|
* @param path - Path to the database resource
|
|
* @returns PostgreSQL connection URL string
|
|
*/
|
|
async databaseUrlFromResource(path: string): Promise<string>
|
|
|
|
async polarsConnectionSettings(s3_resource_path: string | undefined): Promise<any>
|
|
|
|
async duckdbConnectionSettings(s3_resource_path: string | undefined): Promise<any>
|
|
|
|
/**
|
|
* Get S3 client settings from a resource or workspace default
|
|
* @param s3_resource_path - Path to S3 resource (uses workspace default if undefined)
|
|
* @param workspace - Workspace to read from (defaults to the `WM_WORKSPACE` env var)
|
|
* @returns S3 client configuration settings
|
|
*/
|
|
async denoS3LightClientSettings(s3_resource_path: string | undefined, workspace: string | undefined = undefined): Promise<DenoS3LightClientSettings>
|
|
|
|
/**
|
|
* Load the content of a file stored in S3. If the s3ResourcePath is undefined, it will default to the workspace S3 resource.
|
|
*
|
|
* ```typescript
|
|
* let fileContent = await wmill.loadS3FileContent(inputFile)
|
|
* // if the file is a raw text file, it can be decoded and printed directly:
|
|
* const text = new TextDecoder().decode(fileContentStream)
|
|
* console.log(text);
|
|
* ```
|
|
*
|
|
* @param workspace - Workspace to read from (defaults to the `WM_WORKSPACE` env var)
|
|
*/
|
|
async loadS3File(s3object: S3Object, s3ResourcePath: string | undefined = undefined, workspace: string | undefined = undefined): Promise<Uint8Array | undefined>
|
|
|
|
/**
|
|
* Load the content of a file stored in S3 as a stream. If the s3ResourcePath is undefined, it will default to the workspace S3 resource.
|
|
*
|
|
* ```typescript
|
|
* let fileContentBlob = await wmill.loadS3FileStream(inputFile)
|
|
* // if the content is plain text, the blob can be read directly:
|
|
* console.log(await fileContentBlob.text());
|
|
* ```
|
|
*
|
|
* @param workspace - Workspace to read from (defaults to the `WM_WORKSPACE` env var)
|
|
*/
|
|
async loadS3FileStream(s3object: S3Object, s3ResourcePath: string | undefined = undefined, workspace: string | undefined = undefined): Promise<Blob | undefined>
|
|
|
|
/**
|
|
* Persist a file to the S3 bucket. If the s3ResourcePath is undefined, it will default to the workspace S3 resource.
|
|
*
|
|
* ```typescript
|
|
* const s3object = await writeS3File(s3Object, "Hello Windmill!")
|
|
* const fileContentAsUtf8Str = (await s3object.toArray()).toString('utf-8')
|
|
* console.log(fileContentAsUtf8Str)
|
|
* ```
|
|
*
|
|
* @param workspace - Workspace to write to (defaults to the `WM_WORKSPACE` env var)
|
|
*/
|
|
async writeS3File(s3object: S3Object | undefined, fileContent: string | Blob, s3ResourcePath: string | undefined = undefined, contentType: string | undefined = undefined, contentDisposition: string | undefined = undefined, workspace: string | undefined = undefined): Promise<S3Object>
|
|
|
|
/**
|
|
* Permanently delete a file from S3 by key.
|
|
*
|
|
* ```typescript
|
|
* await wmill.deleteS3File({ s3: "path/to/file.txt" })
|
|
* ```
|
|
*
|
|
* @param s3object - S3 object identifying the file to delete (must have `s3` set)
|
|
* @param workspace - Workspace to delete from (defaults to the `WM_WORKSPACE` env var)
|
|
*/
|
|
async deleteS3File(s3object: S3Object, workspace: string | undefined = undefined): Promise<void>
|
|
|
|
/**
|
|
* Sign S3 objects to be used by anonymous users in public apps
|
|
* @param s3objects s3 objects to sign
|
|
* @param expirySecs how long the signature stays valid, in seconds (default 43200 = 12h, clamped to [60, 604800])
|
|
* @returns signed s3 objects
|
|
*/
|
|
async signS3Objects(s3objects: S3Object[], { expirySecs }: { expirySecs?: number } = {}): Promise<S3Object[]>
|
|
|
|
/**
|
|
* Sign S3 object to be used by anonymous users in public apps
|
|
* @param s3object s3 object to sign
|
|
* @param expirySecs how long the signature stays valid, in seconds (default 43200 = 12h, clamped to [60, 604800])
|
|
* @returns signed s3 object
|
|
*/
|
|
async signS3Object(s3object: S3Object, { expirySecs }: { expirySecs?: number } = {}): Promise<S3Object>
|
|
|
|
/**
|
|
* Generate a presigned public URL for an array of S3 objects.
|
|
* If an S3 object is not signed yet, it will be signed first.
|
|
* @param s3Objects s3 objects to sign
|
|
* @param expirySecs how long the signature stays valid, in seconds (default 43200 = 12h, clamped to [60, 604800])
|
|
* @returns list of signed public URLs
|
|
*/
|
|
async getPresignedS3PublicUrls(s3Objects: S3Object[], { baseUrl, expirySecs }: { baseUrl?: string; expirySecs?: number } = {}): Promise<string[]>
|
|
|
|
/**
|
|
* Generate a presigned public URL for an S3 object. If the S3 object is not signed yet, it will be signed first.
|
|
* @param s3Object s3 object to sign
|
|
* @param expirySecs how long the signature stays valid, in seconds (default 43200 = 12h, clamped to [60, 604800])
|
|
* @returns signed public URL
|
|
*/
|
|
async getPresignedS3PublicUrl(s3Objects: S3Object, { baseUrl, expirySecs }: { baseUrl?: string; expirySecs?: number } = {}): Promise<string>
|
|
|
|
/**
|
|
* Get URLs needed for resuming a flow after this step
|
|
* @param approver approver name
|
|
* @param flowLevel if true, generate resume URLs for the parent flow instead of the specific step.
|
|
* This allows pre-approvals that can be consumed by any later suspend step in the same flow.
|
|
* @returns approval page UI URL, resume and cancel API URLs for resuming the flow
|
|
*/
|
|
async getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{
|
|
approvalPage: string;
|
|
resume: string;
|
|
cancel: string;
|
|
}>
|
|
|
|
/**
|
|
* Get an OIDC jwt token for auth to external services (e.g: Vault, AWS) (ee only)
|
|
* @param audience audience of the token
|
|
* @param expiresIn Optional number of seconds until the token expires
|
|
* @returns jwt token
|
|
*/
|
|
async getIdToken(audience: string, expiresIn?: number): Promise<string>
|
|
|
|
/**
|
|
* Convert a base64-encoded string to Uint8Array
|
|
* @param data - Base64-encoded string
|
|
* @returns Decoded Uint8Array
|
|
*/
|
|
base64ToUint8Array(data: string): Uint8Array
|
|
|
|
/**
|
|
* Convert a Uint8Array to base64-encoded string
|
|
* @param arrayBuffer - Uint8Array to encode
|
|
* @returns Base64-encoded string
|
|
*/
|
|
uint8ArrayToBase64(arrayBuffer: Uint8Array): string
|
|
|
|
/**
|
|
* Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields.
|
|
*
|
|
* **[Enterprise Edition Only]** To include form fields in the Slack approval request, go to **Advanced -> Suspend -> Form**
|
|
* and define a form. Learn more at [Windmill Documentation](https://www.windmill.dev/docs/flows/flow_approval#form).
|
|
*
|
|
* @param {Object} options - The configuration options for the Slack approval request.
|
|
* @param {string} options.slackResourcePath - The path to the Slack resource in Windmill.
|
|
* @param {string} options.channelId - The Slack channel ID where the approval request will be sent.
|
|
* @param {string} [options.message] - Optional custom message to include in the Slack approval request.
|
|
* @param {string} [options.approver] - Optional user ID or name of the approver for the request.
|
|
* @param {DefaultArgs} [options.defaultArgsJson] - Optional object defining or overriding the default arguments to a form field.
|
|
* @param {Enums} [options.dynamicEnumsJson] - Optional object overriding the enum default values of an enum form field.
|
|
* @param {string} [options.resumeButtonText] - Optional text for the resume button.
|
|
* @param {string} [options.cancelButtonText] - Optional text for the cancel button.
|
|
*
|
|
* @returns {Promise<void>} Resolves when the Slack approval request is successfully sent.
|
|
*
|
|
* @throws {Error} If the function is not called within a flow or flow preview.
|
|
* @throws {Error} If the `JobService.getSlackApprovalPayload` call fails.
|
|
*
|
|
* **Usage Example:**
|
|
* ```typescript
|
|
* await requestInteractiveSlackApproval({
|
|
* slackResourcePath: "/u/alex/my_slack_resource",
|
|
* channelId: "admins-slack-channel",
|
|
* message: "Please approve this request",
|
|
* approver: "approver123",
|
|
* defaultArgsJson: { key1: "value1", key2: 42 },
|
|
* dynamicEnumsJson: { foo: ["choice1", "choice2"], bar: ["optionA", "optionB"] },
|
|
* resumeButtonText: "Resume",
|
|
* cancelButtonText: "Cancel",
|
|
* });
|
|
* ```
|
|
*
|
|
* **Note:** This function requires execution within a Windmill flow or flow preview.
|
|
*/
|
|
async requestInteractiveSlackApproval({ slackResourcePath, channelId, message, approver, defaultArgsJson, dynamicEnumsJson, resumeButtonText, cancelButtonText, }: SlackApprovalOptions): Promise<void>
|
|
|
|
/**
|
|
* Sends an interactive approval request via Teams, allowing optional customization of the message, approver, and form fields.
|
|
*
|
|
* **[Enterprise Edition Only]** To include form fields in the Teams approval request, go to **Advanced -> Suspend -> Form**
|
|
* and define a form. Learn more at [Windmill Documentation](https://www.windmill.dev/docs/flows/flow_approval#form).
|
|
*
|
|
* @param {Object} options - The configuration options for the Teams approval request.
|
|
* @param {string} options.teamName - The Teams team name where the approval request will be sent.
|
|
* @param {string} options.channelName - The Teams channel name where the approval request will be sent.
|
|
* @param {string} [options.message] - Optional custom message to include in the Teams approval request.
|
|
* @param {string} [options.approver] - Optional user ID or name of the approver for the request.
|
|
* @param {DefaultArgs} [options.defaultArgsJson] - Optional object defining or overriding the default arguments to a form field.
|
|
* @param {Enums} [options.dynamicEnumsJson] - Optional object overriding the enum default values of an enum form field.
|
|
*
|
|
* @returns {Promise<void>} Resolves when the Teams approval request is successfully sent.
|
|
*
|
|
* @throws {Error} If the function is not called within a flow or flow preview.
|
|
* @throws {Error} If the `JobService.getTeamsApprovalPayload` call fails.
|
|
*
|
|
* **Usage Example:**
|
|
* ```typescript
|
|
* await requestInteractiveTeamsApproval({
|
|
* teamName: "admins-teams",
|
|
* channelName: "admins-teams-channel",
|
|
* message: "Please approve this request",
|
|
* approver: "approver123",
|
|
* defaultArgsJson: { key1: "value1", key2: 42 },
|
|
* dynamicEnumsJson: { foo: ["choice1", "choice2"], bar: ["optionA", "optionB"] },
|
|
* });
|
|
* ```
|
|
*
|
|
* **Note:** This function requires execution within a Windmill flow or flow preview.
|
|
*/
|
|
async requestInteractiveTeamsApproval({ teamName, channelName, message, approver, defaultArgsJson, dynamicEnumsJson, }: TeamsApprovalOptions): Promise<void>
|
|
|
|
setWorkflowCtx(ctx: WorkflowCtx | null): void
|
|
|
|
async sleep(seconds: number): Promise<void>
|
|
|
|
/**
|
|
* Execute `fn` inline and checkpoint the result. On replay the cached value is
|
|
* returned without re-executing `fn`.
|
|
*
|
|
* `fn`'s result is encoded as JSON and decoded back before it is returned, so
|
|
* the round that runs the body sees the same types every replay sees: a `Date`
|
|
* comes back as a string, a `Map` as `{}`. {@link Jsonified} is that shape.
|
|
*/
|
|
async step<T>(name: string, fn: () => T | Promise<T>,): Promise<Jsonified<Awaited<T>>>
|
|
|
|
/**
|
|
* Create a task that dispatches to a separate Windmill script.
|
|
*
|
|
* @example
|
|
* const extract = taskScript("f/data/extract");
|
|
* // inside workflow: await extract({ url: "https://..." })
|
|
*/
|
|
taskScript(path: string, options?: TaskOptions): (...args: any[]) => PromiseLike<any>
|
|
|
|
/**
|
|
* Create a task that dispatches to a separate Windmill flow.
|
|
*
|
|
* @example
|
|
* const pipeline = taskFlow("f/etl/pipeline");
|
|
* // inside workflow: await pipeline({ input: data })
|
|
*/
|
|
taskFlow(path: string, options?: TaskOptions): (...args: any[]) => PromiseLike<any>
|
|
|
|
/**
|
|
* Mark an async function as a workflow-as-code entry point.
|
|
*
|
|
* The function must be **deterministic**: given the same inputs it must call
|
|
* tasks in the same order on every replay. Branching on task results is fine
|
|
* (results are replayed from checkpoint), but branching on external state
|
|
* (current time, random values, external API calls) must use `step()` to
|
|
* checkpoint the value so replays see the same result.
|
|
*/
|
|
workflow<T>(fn: (...args: any[]) => Promise<T>): void
|
|
|
|
/**
|
|
* Suspend the workflow and wait for an external approval.
|
|
*
|
|
* Pass `key` to name the step, then `getApprovalUrls(key)` yields the URLs that
|
|
* resume exactly this approval — route them through your own channel. Without a
|
|
* key the steps are named `approval`, `approval_2`, ...
|
|
*
|
|
* `skin: "minimal"` shows approvers only the request (form and approve/reject)
|
|
* instead of the detailed page with the workflow's details. `description` is
|
|
* shown above the form: a string, or a rich value such as `{ markdown: "..." }`.
|
|
*
|
|
* @example
|
|
* const urls = await step("urls", () => getApprovalUrls("manager"));
|
|
* await step("notify", () => sendEmail(urls.resume, urls.cancel));
|
|
* const { value, approver } = await waitForApproval({ key: "manager", timeout: 3600 });
|
|
*/
|
|
waitForApproval(options?: { timeout?: number; form?: object; selfApproval?: boolean; key?: string; skin?: "detailed" | "minimal"; description?: string | object; }): PromiseLike<{ value: any; approver: string; approved: boolean }>
|
|
|
|
/**
|
|
* Resume/cancel/approval-page URLs bound to one `waitForApproval` step.
|
|
*
|
|
* Unlike `getResumeUrls()`, which signs a random nonce, these address the very
|
|
* `resume_job` record the step's built-in approval buttons use, so they are
|
|
* stable across replays and safe to embed in a custom notification.
|
|
*
|
|
* `stepKey` must match the `key` given to `waitForApproval`. Keys must be unique
|
|
* within a workflow; reusing one throws rather than silently renaming it. The URL
|
|
* only resumes while that step is awaiting approval; used at any other moment it is
|
|
* rejected rather than banking a row a different approval would consume. Send it
|
|
* ahead of time — approvers just cannot act before the workflow reaches the step.
|
|
*
|
|
* `resume` and `cancel` are step-bound; `approvalPage` is not — it opens the job's
|
|
* approval page, which acts on whichever approval is pending when it is used.
|
|
*
|
|
* @example
|
|
* const urls = await step("urls", () => getApprovalUrls("manager"));
|
|
* await step("notify", () => sendEmail(urls.resume, urls.cancel));
|
|
* await waitForApproval({ key: "manager" });
|
|
*/
|
|
async getApprovalUrls(stepKey: string = "approval", approver?: string): Promise<{
|
|
approvalPage: string;
|
|
resume: string;
|
|
cancel: string;
|
|
}>
|
|
|
|
/**
|
|
* Process items in parallel with optional concurrency control.
|
|
*
|
|
* Each item is processed by calling `fn(item)`, which should be a task().
|
|
* Items are dispatched in batches of `concurrency` (default: all at once).
|
|
*
|
|
* @example
|
|
* const process = task(async (item: string) => { ... });
|
|
* const results = await parallel(items, process, { concurrency: 5 });
|
|
*/
|
|
async parallel<T, R>(items: T[], fn: (item: T) => PromiseLike<R> | R, options?: { concurrency?: number },): Promise<R[]>
|
|
|
|
/**
|
|
* Commit Kafka offsets for a trigger with auto_commit disabled.
|
|
* @param triggerPath - Path to the Kafka trigger (from event.wm_trigger.trigger_path)
|
|
* @param topic - Kafka topic name (from event.topic)
|
|
* @param partition - Partition number (from event.partition)
|
|
* @param offset - Message offset to commit (from event.offset)
|
|
*/
|
|
async commitKafkaOffsets(triggerPath: string, topic: string, partition: number, offset: number,): Promise<void>
|
|
|
|
/**
|
|
* Parse an S3 object from URI string or record format
|
|
* @param s3Object - S3 object as URI string (`s3://storage/key`, `s3:///key`
|
|
* for the default storage) or record. Any other string throws rather than
|
|
* falling back to an auto-generated key: an auto key is requested by
|
|
* omitting the object, and a fallback would silently misplace the upload
|
|
* on any typo.
|
|
* @returns S3 object record with storage and s3 key
|
|
*/
|
|
parseS3Object(s3Object: S3Object): S3ObjectRecord
|
|
|
|
/**
|
|
* Create a SQL template function for PostgreSQL/datatable queries
|
|
* @param name - Database/datatable name (default: "main")
|
|
* @param opts.role - Connect as this data table role instead of the data table's default one.
|
|
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
|
|
* @returns SQL template function for building parameterized queries
|
|
* @example
|
|
* let sql = wmill.datatable()
|
|
* let name = 'Robin'
|
|
* let age = 21
|
|
* await sql`
|
|
* SELECT * FROM friends
|
|
* WHERE name = ${name} AND age = ${age}::int
|
|
* `.fetch()
|
|
* @example
|
|
* // Read through a restricted role
|
|
* let sql = wmill.datatable("main", { role: "analytics" })
|
|
*/
|
|
datatable(name: string = "main", opts?: DatatableOptions): DatatableSqlTemplateFunction
|
|
|
|
/**
|
|
* Create a SQL template function for DuckDB/ducklake queries
|
|
* @param name - DuckDB database name, optionally with a schema as `name:schema` (default: "main")
|
|
* @returns SQL template function for building parameterized queries
|
|
* @example
|
|
* let sql = wmill.ducklake()
|
|
* let name = 'Robin'
|
|
* let age = 21
|
|
* await sql`
|
|
* SELECT * FROM friends
|
|
* WHERE name = ${name} AND age = ${age}
|
|
* `.fetch()
|
|
* @example
|
|
* // Target a specific schema within the ducklake
|
|
* let sql = wmill.ducklake("my_lake:analytics")
|
|
*/
|
|
ducklake(name: string = "main"): SqlTemplateFunction
|
|
|
|
/**
|
|
* Idempotently materialize `selectSql` into a ducklake table for one
|
|
* partition (or the whole table when `partition` is omitted) — the client-side
|
|
* equivalent of the `// materialize` engine.
|
|
* With `uniqueKey` it upserts the slice (delete-by-key + insert); otherwise it
|
|
* replaces it (whole table → `CREATE OR REPLACE`; partition → delete + insert).
|
|
* Safe to re-run for the same partition (backfill / failure-recovery).
|
|
*
|
|
* Returns a lazy statement — call `.execute()` to run it:
|
|
* `await wmill.upsertPartition({ table, selectSql, partition }).execute()`.
|
|
*/
|
|
upsertPartition(opts: DucklakeMaterializeOptions): SqlStatement<any>
|
|
|
|
/**
|
|
* INSERT-only materialization (no dedup/replace) for append-only tables.
|
|
* Re-running the same partition duplicates rows — use only for immutable
|
|
* event-log sources.
|
|
*
|
|
* Returns a lazy statement — call `.execute()` to run it:
|
|
* `await wmill.appendPartition({ table, selectSql, partition }).execute()`.
|
|
*/
|
|
appendPartition(opts: Omit<DucklakeMaterializeOptions, "uniqueKey">,): SqlStatement<any>
|
|
|
|
|
|
# Python SDK (wmill)
|
|
|
|
Import: import wmill
|
|
|
|
The client configures itself from the job's environment — base URL, token and credentials mode
|
|
are all set before your code runs, so there is nothing to initialize and no reason to read
|
|
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
|
|
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
|
|
HTTP for third-party APIs.
|
|
|
|
The functions below are the surface to prefer. For an endpoint none of them covers,
|
|
wmill.Windmill().get(endpoint) and .post(endpoint) issue an authenticated request against this
|
|
instance. What does not exist is a function name you guessed at: if it is not listed below, do
|
|
not call it.
|
|
|
|
To know who is running the script, read the contextual variables rather than calling the API:
|
|
`os.environ.get("WM_END_USER_EMAIL") or os.environ.get("WM_EMAIL")`. WM_END_USER_EMAIL is the app
|
|
viewer when the run was triggered from an app and empty otherwise (both variables are always
|
|
defined), WM_EMAIL is the user the job is permissioned as. WM_USERNAME is the matching username.
|
|
|
|
def worker_has_internal_server() -> bool
|
|
|
|
def get_mocked_api() -> Optional[dict]
|
|
|
|
# Get the HTTP client instance.
|
|
#
|
|
# Returns:
|
|
# Configured httpx.Client for API requests
|
|
def get_client() -> httpx.Client
|
|
|
|
# Make an HTTP GET request to the Windmill API.
|
|
#
|
|
# Args:
|
|
# endpoint: API endpoint path
|
|
# raise_for_status: Whether to raise an exception on HTTP errors
|
|
# **kwargs: Additional arguments passed to httpx.get
|
|
#
|
|
# Returns:
|
|
# HTTP response object
|
|
def get(endpoint, raise_for_status = True, **kwargs) -> httpx.Response
|
|
|
|
# Make an HTTP POST request to the Windmill API.
|
|
#
|
|
# Args:
|
|
# endpoint: API endpoint path
|
|
# raise_for_status: Whether to raise an exception on HTTP errors
|
|
# **kwargs: Additional arguments passed to httpx.post
|
|
#
|
|
# Returns:
|
|
# HTTP response object
|
|
def post(endpoint, raise_for_status = True, **kwargs) -> httpx.Response
|
|
|
|
# Create a new authentication token.
|
|
#
|
|
# Args:
|
|
# duration: Token validity duration (default: 1 day)
|
|
#
|
|
# Returns:
|
|
# New authentication token string
|
|
def create_token(duration = dt.timedelta(days=1)) -> str
|
|
|
|
# Create a script job by path and return its job id.
|
|
def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str
|
|
|
|
# Create a script job by hash and return its job id.
|
|
def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str
|
|
|
|
# Create a flow job and return its job id.
|
|
def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str
|
|
|
|
# Run script by path synchronously and return its result.
|
|
def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any
|
|
|
|
# Run script by hash synchronously and return its result.
|
|
def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any
|
|
|
|
# Run a script on the current worker without creating a job.
|
|
#
|
|
# On agent workers (no internal server), falls back to running a normal
|
|
# preview job and waiting for the result.
|
|
def run_inline_script_preview(content: str, language: str, args: dict = None) -> Any
|
|
|
|
# Wait for a job to complete and return its result.
|
|
#
|
|
# Args:
|
|
# job_id: ID of the job to wait for
|
|
# timeout: Maximum time to wait (seconds or timedelta)
|
|
# verbose: Enable verbose logging
|
|
# cleanup: Register cleanup handler to cancel job on exit
|
|
# assert_result_is_not_none: Raise exception if result is None
|
|
#
|
|
# Returns:
|
|
# Job result when completed
|
|
#
|
|
# Raises:
|
|
# TimeoutError: If timeout is reached
|
|
# Exception: If job fails
|
|
def wait_job(job_id, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False)
|
|
|
|
# Cancel a specific job by ID.
|
|
#
|
|
# Args:
|
|
# job_id: UUID of the job to cancel
|
|
# reason: Optional reason for cancellation
|
|
#
|
|
# Returns:
|
|
# Response message from the cancel endpoint
|
|
def cancel_job(job_id: str, reason: str = None) -> str
|
|
|
|
# Cancel currently running executions of the same script.
|
|
def cancel_running() -> dict
|
|
|
|
# Get job details by ID.
|
|
#
|
|
# Args:
|
|
# job_id: UUID of the job
|
|
#
|
|
# Returns:
|
|
# Job details dictionary
|
|
def get_job(job_id: str) -> dict
|
|
|
|
# Get the root job ID for a flow hierarchy.
|
|
#
|
|
# Args:
|
|
# job_id: Job ID (defaults to current WM_JOB_ID)
|
|
#
|
|
# Returns:
|
|
# Root job ID
|
|
def get_root_job_id(job_id: str | None = None) -> dict
|
|
|
|
# Get an OIDC JWT token for authentication to external services.
|
|
#
|
|
# Args:
|
|
# audience: Token audience (e.g., "vault", "aws")
|
|
# expires_in: Optional expiration time in seconds
|
|
#
|
|
# Returns:
|
|
# JWT token string
|
|
def get_id_token(audience: str, expires_in: int | None = None) -> str
|
|
|
|
# Get the status of a job.
|
|
#
|
|
# Args:
|
|
# job_id: UUID of the job
|
|
#
|
|
# Returns:
|
|
# Job status: "RUNNING", "WAITING", or "COMPLETED"
|
|
def get_job_status(job_id: str) -> JobStatus
|
|
|
|
# Get the result of a completed job.
|
|
#
|
|
# Args:
|
|
# job_id: UUID of the completed job
|
|
# assert_result_is_not_none: Raise exception if result is None
|
|
#
|
|
# Returns:
|
|
# Job result
|
|
def get_result(job_id: str, assert_result_is_not_none: bool = True) -> Any
|
|
|
|
# Get a variable value by path.
|
|
#
|
|
# Args:
|
|
# path: Variable path in Windmill
|
|
#
|
|
# Returns:
|
|
# Variable value as string
|
|
def get_variable(path: str) -> str
|
|
|
|
# Set a variable value by path, creating it if it doesn't exist.
|
|
#
|
|
# Args:
|
|
# path: Variable path in Windmill
|
|
# value: Variable value to set
|
|
# is_secret: Whether the variable should be secret (default: False)
|
|
def set_variable(path: str, value: str, is_secret: bool = False) -> None
|
|
|
|
# Get a resource value by path.
|
|
#
|
|
# Args:
|
|
# path: Resource path in Windmill
|
|
# none_if_undefined: Return None instead of raising if not found
|
|
# interpolated: if variables and resources are fully unrolled
|
|
#
|
|
# Returns:
|
|
# Resource value dictionary or None
|
|
def get_resource(path: str, none_if_undefined: bool = False, interpolated: bool = True) -> dict | None
|
|
|
|
# Set a resource value by path, creating it if it doesn't exist.
|
|
#
|
|
# Args:
|
|
# value: Resource value to set
|
|
# path: Resource path in Windmill
|
|
# resource_type: Resource type for creation
|
|
def set_resource(value: Any, path: str, resource_type: str)
|
|
|
|
# List resources from Windmill workspace.
|
|
#
|
|
# Args:
|
|
# resource_type: Optional resource type to filter by (e.g., "postgresql", "mysql", "s3")
|
|
# page: Optional page number for pagination
|
|
# per_page: Optional number of results per page
|
|
#
|
|
# Returns:
|
|
# List of resource dictionaries
|
|
def list_resources(resource_type: str = None, page: int = None, per_page: int = None) -> list[dict]
|
|
|
|
# Set the workflow state.
|
|
#
|
|
# Args:
|
|
# value: State value to set
|
|
# path: Optional state resource path override.
|
|
def set_state(value: Any, path: str | None = None) -> None
|
|
|
|
# Get the workflow state.
|
|
#
|
|
# Args:
|
|
# path: Optional state resource path override.
|
|
#
|
|
# Returns:
|
|
# State value or None if not set
|
|
def get_state(path: str | None = None) -> Any
|
|
|
|
# Set job progress percentage (0-99).
|
|
#
|
|
# Args:
|
|
# value: Progress percentage
|
|
# job_id: Job ID (defaults to current WM_JOB_ID)
|
|
def set_progress(value: int, job_id: Optional[str] = None)
|
|
|
|
# Get job progress percentage.
|
|
#
|
|
# Args:
|
|
# job_id: Job ID (defaults to current WM_JOB_ID)
|
|
#
|
|
# Returns:
|
|
# Progress value (0-100) or None if not set
|
|
def get_progress(job_id: Optional[str] = None) -> Any
|
|
|
|
# Set the user state of a flow at a given key
|
|
def set_flow_user_state(key: str, value: Any) -> None
|
|
|
|
# Get the user state of a flow at a given key
|
|
def get_flow_user_state(key: str) -> Any
|
|
|
|
# Get the Windmill server version.
|
|
#
|
|
# Returns:
|
|
# Version string
|
|
def version()
|
|
|
|
# Convenient helpers that takes an S3 resource as input and returns the settings necessary to
|
|
# initiate an S3 connection from DuckDB
|
|
def get_duckdb_connection_settings(s3_resource_path: str = '') -> DuckDbConnectionSettings | None
|
|
|
|
# Convenient helpers that takes an S3 resource as input and returns the settings necessary to
|
|
# initiate an S3 connection from Polars
|
|
def get_polars_connection_settings(s3_resource_path: str = '') -> PolarsConnectionSettings
|
|
|
|
# Convenient helpers that takes an S3 resource as input and returns the settings necessary to
|
|
# initiate an S3 connection using boto3
|
|
def get_boto3_connection_settings(s3_resource_path: str = '') -> Boto3ConnectionSettings
|
|
|
|
# Load a file from the workspace s3 bucket and returns its content as bytes.
|
|
#
|
|
# '''python
|
|
# from wmill import S3Object
|
|
#
|
|
# s3_obj = S3Object(s3="/path/to/my_file.txt")
|
|
# my_obj_content = client.load_s3_file(s3_obj)
|
|
# file_content = my_obj_content.decode("utf-8")
|
|
# '''
|
|
def load_s3_file(s3object: S3Object | str, s3_resource_path: str | None) -> bytes
|
|
|
|
# Load a file from the workspace s3 bucket and returns the bytes stream.
|
|
#
|
|
# '''python
|
|
# from wmill import S3Object
|
|
#
|
|
# s3_obj = S3Object(s3="/path/to/my_file.txt")
|
|
# with wmill.load_s3_file_reader(s3object, s3_resource_path) as file_reader:
|
|
# print(file_reader.read())
|
|
# '''
|
|
def load_s3_file_reader(s3object: S3Object | str, s3_resource_path: str | None) -> BufferedReader
|
|
|
|
# Write a file to the workspace S3 bucket
|
|
#
|
|
# '''python
|
|
# from wmill import S3Object
|
|
#
|
|
# s3_obj = S3Object(s3="/path/to/my_file.txt")
|
|
#
|
|
# # for an in memory bytes array:
|
|
# file_content = b'Hello Windmill!'
|
|
# client.write_s3_file(s3_obj, file_content)
|
|
#
|
|
# # for a file:
|
|
# with open("my_file.txt", "rb") as my_file:
|
|
# client.write_s3_file(s3_obj, my_file)
|
|
# '''
|
|
def write_s3_file(s3object: S3Object | str | None, file_content: BufferedReader | bytes, s3_resource_path: str | None, content_type: str | None = None, content_disposition: str | None = None) -> S3Object
|
|
|
|
# Permanently delete a file from the workspace S3 bucket.
|
|
#
|
|
# '''python
|
|
# from wmill import S3Object
|
|
#
|
|
# s3_obj = S3Object(s3="/path/to/my_file.txt")
|
|
# client.delete_s3_object(s3_obj)
|
|
# '''
|
|
def delete_s3_object(s3object: S3Object | str, s3_resource_path: str | None = None) -> None
|
|
|
|
# Sign S3 objects for use by anonymous users in public apps.
|
|
#
|
|
# Args:
|
|
# s3_objects: List of S3 objects to sign
|
|
# expiry_secs: How long the signature stays valid, in seconds
|
|
# (defaults to 43200 = 12h, clamped to [60, 604800])
|
|
#
|
|
# Returns:
|
|
# List of signed S3 objects
|
|
def sign_s3_objects(s3_objects: list[S3Object | str], expiry_secs: int | None = None) -> list[S3Object]
|
|
|
|
# Sign a single S3 object for use by anonymous users in public apps.
|
|
#
|
|
# Args:
|
|
# s3_object: S3 object to sign
|
|
# expiry_secs: How long the signature stays valid, in seconds
|
|
# (defaults to 43200 = 12h, clamped to [60, 604800])
|
|
#
|
|
# Returns:
|
|
# Signed S3 object
|
|
def sign_s3_object(s3_object: S3Object | str, expiry_secs: int | None = None) -> S3Object
|
|
|
|
# Generate presigned public URLs for an array of S3 objects.
|
|
# If an S3 object is not signed yet, it will be signed first.
|
|
#
|
|
# Args:
|
|
# s3_objects: List of S3 objects to sign
|
|
# base_url: Optional base URL for the presigned URLs (defaults to WM_BASE_URL)
|
|
# expiry_secs: How long the signatures stay valid, in seconds
|
|
# (defaults to 43200 = 12h, clamped to [60, 604800])
|
|
#
|
|
# Returns:
|
|
# List of signed public URLs
|
|
#
|
|
# Example:
|
|
# >>> s3_objs = [S3Object(s3="/path/to/file1.txt"), S3Object(s3="/path/to/file2.txt")]
|
|
# >>> urls = client.get_presigned_s3_public_urls(s3_objs)
|
|
def get_presigned_s3_public_urls(s3_objects: list[S3Object | str], base_url: str | None = None, expiry_secs: int | None = None) -> list[str]
|
|
|
|
# Generate a presigned public URL for an S3 object.
|
|
# If the S3 object is not signed yet, it will be signed first.
|
|
#
|
|
# Args:
|
|
# s3_object: S3 object to sign
|
|
# base_url: Optional base URL for the presigned URL (defaults to WM_BASE_URL)
|
|
# expiry_secs: How long the signature stays valid, in seconds
|
|
# (defaults to 43200 = 12h, clamped to [60, 604800])
|
|
#
|
|
# Returns:
|
|
# Signed public URL
|
|
#
|
|
# Example:
|
|
# >>> s3_obj = S3Object(s3="/path/to/file.txt")
|
|
# >>> url = client.get_presigned_s3_public_url(s3_obj)
|
|
def get_presigned_s3_public_url(s3_object: S3Object | str, base_url: str | None = None, expiry_secs: int | None = None) -> str
|
|
|
|
# Get the current user information.
|
|
#
|
|
# Returns:
|
|
# User details dictionary
|
|
def whoami() -> dict
|
|
|
|
# Get the current user information (alias for whoami).
|
|
#
|
|
# Returns:
|
|
# User details dictionary
|
|
def user() -> dict
|
|
|
|
# Get the state resource path from environment.
|
|
#
|
|
# Returns:
|
|
# State path string
|
|
def state_path() -> str
|
|
|
|
# Get the workflow state.
|
|
#
|
|
# Returns:
|
|
# State value or None if not set
|
|
def state() -> Any
|
|
|
|
# Set the state in the shared folder using pickle
|
|
def set_shared_state_pickle(value: Any, path: str = 'state.pickle') -> None
|
|
|
|
# Get the state in the shared folder using pickle
|
|
def get_shared_state_pickle(path: str = 'state.pickle') -> Any
|
|
|
|
# Set the state in the shared folder using pickle
|
|
def set_shared_state(value: Any, path: str = 'state.json') -> None
|
|
|
|
# Get the state in the shared folder using pickle
|
|
def get_shared_state(path: str = 'state.json') -> None
|
|
|
|
# Get URLs needed for resuming a flow after suspension.
|
|
#
|
|
# Args:
|
|
# approver: Optional approver name
|
|
# flow_level: If True, generate resume URLs for the parent flow instead of the
|
|
# specific step. This allows pre-approvals that can be consumed by any later
|
|
# suspend step in the same flow.
|
|
#
|
|
# Returns:
|
|
# Dictionary with approvalPage, resume, and cancel URLs
|
|
def get_resume_urls(approver: str = None, flow_level: bool = None) -> dict
|
|
|
|
# Get the resume URLs bound to one ``wait_for_approval`` step of this workflow.
|
|
#
|
|
# Args:
|
|
# step_key: Checkpoint key of the approval step, as passed to
|
|
# ``wait_for_approval(key=...)``
|
|
# approver: Optional approver name
|
|
#
|
|
# Returns:
|
|
# Dictionary with approvalPage, resume, and cancel URLs
|
|
def get_approval_urls(step_key: str = 'approval', approver: str = None) -> dict
|
|
|
|
# Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields.
|
|
#
|
|
# **[Enterprise Edition Only]** To include form fields in the Slack approval request, use the "Advanced -> Suspend -> Form" functionality.
|
|
# Learn more at: https://www.windmill.dev/docs/flows/flow_approval#form
|
|
#
|
|
# :param slack_resource_path: The path to the Slack resource in Windmill.
|
|
# :type slack_resource_path: str
|
|
# :param channel_id: The Slack channel ID where the approval request will be sent.
|
|
# :type channel_id: str
|
|
# :param message: Optional custom message to include in the Slack approval request.
|
|
# :type message: str, optional
|
|
# :param approver: Optional user ID or name of the approver for the request.
|
|
# :type approver: str, optional
|
|
# :param default_args_json: Optional dictionary defining or overriding the default arguments for form fields.
|
|
# :type default_args_json: dict, optional
|
|
# :param dynamic_enums_json: Optional dictionary overriding the enum default values of enum form fields.
|
|
# :type dynamic_enums_json: dict, optional
|
|
#
|
|
# :raises Exception: If the function is not called within a flow or flow preview.
|
|
# :raises Exception: If the required flow job or flow step environment variables are not set.
|
|
#
|
|
# :return: None
|
|
#
|
|
# **Usage Example:**
|
|
# >>> client.request_interactive_slack_approval(
|
|
# ... slack_resource_path="/u/alex/my_slack_resource",
|
|
# ... channel_id="admins-slack-channel",
|
|
# ... message="Please approve this request",
|
|
# ... approver="approver123",
|
|
# ... default_args_json={"key1": "value1", "key2": 42},
|
|
# ... dynamic_enums_json={"foo": ["choice1", "choice2"], "bar": ["optionA", "optionB"]},
|
|
# ... )
|
|
#
|
|
# **Notes:**
|
|
# - This function must be executed within a Windmill flow or flow preview.
|
|
# - The function checks for required environment variables (`WM_FLOW_JOB_ID`, `WM_FLOW_STEP_ID`) to ensure it is run in the appropriate context.
|
|
def request_interactive_slack_approval(slack_resource_path: str, channel_id: str, message: str = None, approver: str = None, default_args_json: dict = None, dynamic_enums_json: dict = None) -> None
|
|
|
|
# Send a message to a Microsoft Teams conversation with conversation_id, where success is used to style the message
|
|
def send_teams_message(conversation_id: str, text: str, success: bool = True, card_block: dict = None)
|
|
|
|
# Get a DataTable client for SQL queries.
|
|
#
|
|
# Args:
|
|
# name: Database name (default: "main")
|
|
# role: Connect as this data table role instead of the data table's default one.
|
|
# Only meaningful on a data table under roles, and only for a role you are a
|
|
# tenant of.
|
|
#
|
|
# Returns:
|
|
# DataTableClient instance
|
|
def datatable(name: str = 'main', *, role: Optional[str] = None)
|
|
|
|
# Get a DuckLake client for DuckDB queries.
|
|
#
|
|
# Args:
|
|
# name: Database name (default: "main")
|
|
#
|
|
# Returns:
|
|
# DucklakeClient instance
|
|
def ducklake(name: str = 'main')
|
|
|
|
def init_global_client(f)
|
|
|
|
def deprecate(in_favor_of: str)
|
|
|
|
# Get the current workspace ID.
|
|
#
|
|
# Returns:
|
|
# Workspace ID string
|
|
def get_workspace() -> str
|
|
|
|
def get_version() -> str
|
|
|
|
# Create a script job and return its job ID.
|
|
#
|
|
# Args:
|
|
# hash_or_path: Script hash or path (determined by presence of '/')
|
|
# args: Script arguments
|
|
# scheduled_in_secs: Delay before execution in seconds
|
|
# tag: Override the worker tag the job runs on
|
|
#
|
|
# Returns:
|
|
# Job ID string
|
|
def run_script_async(hash_or_path: str, args: Dict[str, Any] = None, scheduled_in_secs: int = None, tag: str = None) -> str
|
|
|
|
# Run a script synchronously by hash and return its result.
|
|
#
|
|
# Args:
|
|
# hash: Script hash
|
|
# args: Script arguments
|
|
# verbose: Enable verbose logging
|
|
# assert_result_is_not_none: Raise exception if result is None
|
|
# cleanup: Register cleanup handler to cancel job on exit
|
|
# timeout: Maximum time to wait
|
|
# tag: Override the worker tag the job runs on
|
|
#
|
|
# Returns:
|
|
# Script result
|
|
def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any
|
|
|
|
# Run a script synchronously by path and return its result.
|
|
#
|
|
# Args:
|
|
# path: Script path
|
|
# args: Script arguments
|
|
# verbose: Enable verbose logging
|
|
# assert_result_is_not_none: Raise exception if result is None
|
|
# cleanup: Register cleanup handler to cancel job on exit
|
|
# timeout: Maximum time to wait
|
|
# tag: Override the worker tag the job runs on
|
|
#
|
|
# Returns:
|
|
# Script result
|
|
def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any
|
|
|
|
# Convenient helpers that takes an S3 resource as input and returns the settings necessary to
|
|
# initiate an S3 connection from DuckDB
|
|
def duckdb_connection_settings(s3_resource_path: str = '') -> DuckDbConnectionSettings
|
|
|
|
# Convenient helpers that takes an S3 resource as input and returns the settings necessary to
|
|
# initiate an S3 connection from Polars
|
|
def polars_connection_settings(s3_resource_path: str = '') -> PolarsConnectionSettings
|
|
|
|
# Convenient helpers that takes an S3 resource as input and returns the settings necessary to
|
|
# initiate an S3 connection using boto3
|
|
def boto3_connection_settings(s3_resource_path: str = '') -> Boto3ConnectionSettings
|
|
|
|
# Get the state resource path from environment.
|
|
#
|
|
# Returns:
|
|
# State path string
|
|
def get_state_path() -> str
|
|
|
|
# Parse resource syntax from string.
|
|
def parse_resource_syntax(s: str) -> Optional[str]
|
|
|
|
# Parse S3 object from a `s3://<storage>/<key>` URI string (`s3:///<key>`
|
|
# for the default storage) or S3Object format. Any other string raises
|
|
# rather than falling back to an auto-generated key: an auto key is
|
|
# requested by omitting the object, and a fallback would silently misplace
|
|
# the upload on any typo.
|
|
def parse_s3_object(s3_object: S3Object | str) -> S3Object
|
|
|
|
# Parse variable syntax from string.
|
|
def parse_variable_syntax(s: str) -> Optional[str]
|
|
|
|
# Append a text to the result stream.
|
|
#
|
|
# Args:
|
|
# text: text to append to the result stream
|
|
def append_to_result_stream(text: str) -> None
|
|
|
|
# Stream to the result stream.
|
|
#
|
|
# Args:
|
|
# stream: stream to stream to the result stream
|
|
def stream_result(stream) -> None
|
|
|
|
# Execute a SQL query against the DataTable.
|
|
#
|
|
# Args:
|
|
# sql: SQL query string with $1, $2, etc. placeholders
|
|
# *args: Positional arguments to bind to query placeholders
|
|
#
|
|
# Returns:
|
|
# SqlQuery instance for fetching results
|
|
def query(sql: str, *args) -> SqlQuery
|
|
|
|
# Idempotently materialize the rows of `select_sql` into ducklake
|
|
# `table` for one `partition` (or the whole table when `partition` is
|
|
# None). Client-side equivalent of the `// materialize` engine: with
|
|
# `unique_key` it upserts within the slice (delete-by-key + insert);
|
|
# without it, it replaces (whole table → CREATE OR REPLACE; partition →
|
|
# delete the partition + insert). Re-running the same slice is safe — the
|
|
# backfill / failure-recovery contract.
|
|
#
|
|
# The partition value is bound as a DuckDB arg (never string-interpolated)
|
|
# so it cannot inject SQL. `select_sql` is trusted (your own query).
|
|
def upsert_partition(table: str, select_sql: str, partition: str = None, unique_key: str = None, partition_col: str = '_wm_partition', schema: str = None)
|
|
|
|
# INSERT-only materialization (no dedup / no replace) for an immutable
|
|
# event-log table — for one `partition`, or the whole table when
|
|
# `partition` is None. NOTE: unlike `upsert_partition`, re-running the same
|
|
# slice duplicates rows — use only for append-only sources.
|
|
def append_partition(table: str, select_sql: str, partition: str = None, partition_col: str = '_wm_partition', schema: str = None)
|
|
|
|
# Read a materialized ducklake table, optionally a single partition.
|
|
def read(table: str, partition: str = None, partition_col: str = '_wm_partition', schema: str = None)
|
|
|
|
# Execute query and fetch results.
|
|
#
|
|
# Args:
|
|
# result_collection: Optional result collection mode
|
|
#
|
|
# Returns:
|
|
# Query results
|
|
def fetch(result_collection: str | None = None)
|
|
|
|
# Execute query and fetch first row of results.
|
|
#
|
|
# Returns:
|
|
# First row of query results
|
|
def fetch_one()
|
|
|
|
# Execute query and fetch first row of results. Return result as a scalar value.
|
|
#
|
|
# Returns:
|
|
# First row of query result as a scalar value
|
|
def fetch_one_scalar()
|
|
|
|
# Execute query and don't return any results.
|
|
#
|
|
def execute()
|
|
|
|
# DuckDB executor requires explicit argument types at declaration
|
|
# These types exist in both DuckDB and Postgres
|
|
# Check that the types exist if you plan to extend this function for other SQL engines.
|
|
def infer_sql_type(value) -> str
|
|
|
|
def parse_sql_client_name(name: str) -> tuple[str, Optional[str]]
|
|
|
|
# Decorator that marks a function as a workflow task.
|
|
#
|
|
# Works in both WAC v1 (sync, HTTP-based dispatch) and WAC v2
|
|
# (async, checkpoint/replay) modes:
|
|
#
|
|
# - **v2 (inside @workflow)**: dispatches as a checkpoint step.
|
|
# - **v1 (WM_JOB_ID set, no @workflow)**: dispatches via HTTP API.
|
|
# - **Standalone**: executes the function body directly.
|
|
#
|
|
# A task runs as its own job, so its result is always encoded as JSON and
|
|
# decoded back before the caller sees it: a ``datetime`` comes back as a
|
|
# string, a tuple as a list.
|
|
#
|
|
# ``retry`` re-dispatches the task after a failure, inside ``@workflow`` only.
|
|
# Every attempt is a step of its own (``call_api``, ``call_api#2``, ...) and
|
|
# the wait between two of them is a durable sleep, so a retrying task holds no
|
|
# worker while it backs off. Keys: ``attempts`` (retries after the first
|
|
# failure, a whole number from 0 to 100), ``delay`` (seconds before the first
|
|
# retry, sub-second delays dropped), ``multiplier`` (applied to the delay
|
|
# after each attempt, 1 keeps it constant), ``max_delay`` (ceiling in
|
|
# seconds). ``attempts`` is required, and an out-of-range or unknown key is
|
|
# rejected where the policy is written.
|
|
#
|
|
# A workflow sleeps once per round, so tasks backing off in the same fan-out
|
|
# wait one after another rather than together: the delay before a fan-out
|
|
# retries is the sum of every backoff pending in it, not the longest one, and
|
|
# it grows with both the width of the fan-out and ``attempts``. Retries with
|
|
# no ``delay`` all go out in a single round.
|
|
#
|
|
# ``cache_ttl`` serves a previous result of the task for that many seconds
|
|
# instead of running it again. A task is keyed on its step key (its name and
|
|
# call order) and the workflow's input, not on the arguments it is called
|
|
# with, so cache one only when whether it runs, and what it receives, follow
|
|
# from the workflow's input alone. A ``task_script`` target is keyed on the
|
|
# arguments it is called with. It has no effect on a ``task_flow`` target,
|
|
# which keeps its flow's own cache policy.
|
|
#
|
|
# Usage::
|
|
#
|
|
# @task
|
|
# async def extract_data(url: str): ...
|
|
#
|
|
# @task(path="f/external_script", timeout=600, tag="gpu")
|
|
# async def run_external(x: int): ...
|
|
#
|
|
# @task(retry={"attempts": 3, "delay": 30, "multiplier": 2})
|
|
# async def call_api(payload: dict): ...
|
|
def task(_func = None, *, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
|
|
|
|
# Create a task that dispatches to a separate Windmill script.
|
|
#
|
|
# ``retry`` takes the same policy as :func:`task`.
|
|
#
|
|
# Usage::
|
|
#
|
|
# extract = task_script("f/data/extract", timeout=600)
|
|
#
|
|
# @workflow
|
|
# async def main():
|
|
# data = await extract(url="https://...")
|
|
def task_script(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
|
|
|
|
# Create a task that dispatches to a separate Windmill flow.
|
|
#
|
|
# ``retry`` takes the same policy as :func:`task`.
|
|
#
|
|
# Usage::
|
|
#
|
|
# pipeline = task_flow("f/etl/pipeline", priority=10)
|
|
#
|
|
# @workflow
|
|
# async def main():
|
|
# result = await pipeline(input=data)
|
|
def task_flow(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
|
|
|
|
# Decorator marking an async function as a workflow-as-code entry point.
|
|
#
|
|
# The function must be **deterministic**: given the same inputs it must call
|
|
# tasks in the same order on every replay. Branching on task results is fine
|
|
# (results are replayed from checkpoint), but branching on external state
|
|
# (current time, random values, external API calls) must use ``step()`` to
|
|
# checkpoint the value so replays see the same result.
|
|
def workflow(func)
|
|
|
|
# Execute ``fn`` inline and checkpoint the result.
|
|
#
|
|
# On replay the cached value is returned without re-executing ``fn``.
|
|
# Use for lightweight deterministic operations (timestamps, random IDs,
|
|
# config reads) that should not incur the overhead of a child job.
|
|
#
|
|
# ``fn``'s result is encoded as JSON and decoded back before it is returned,
|
|
# so the round that runs the body sees the same types every replay sees:
|
|
# a ``datetime`` comes back as a string, a tuple as a list.
|
|
async def step(name: str, fn)
|
|
|
|
# Server-side sleep — suspend the workflow for the given duration without holding a worker.
|
|
#
|
|
# Inside a @workflow, the parent job suspends and auto-resumes after ``seconds``.
|
|
# Outside a workflow, falls back to ``asyncio.sleep``.
|
|
async def sleep(seconds: int)
|
|
|
|
# Suspend the workflow and wait for an external approval.
|
|
#
|
|
# Pass ``key`` to name the step, then ``get_approval_urls(key)`` yields the URLs
|
|
# that resume exactly this approval — route them through your own channel.
|
|
# Without a key the steps are named ``approval``, ``approval_2``, ...
|
|
#
|
|
# Returns a dict with ``value`` (form data), ``approver``, and ``approved``.
|
|
#
|
|
# Args:
|
|
# timeout: Approval timeout in seconds (default 1800).
|
|
# form: Optional form schema for the approval page.
|
|
# self_approval: Whether the user who triggered the flow can approve it (default True).
|
|
# key: Optional checkpoint key naming this approval step.
|
|
# skin: ``"minimal"`` shows approvers only the request (form and approve/reject)
|
|
# instead of the detailed page with the workflow's details.
|
|
# description: Shown to approvers above the form: a string, or a rich value such as
|
|
# ``{"markdown": "..."}``.
|
|
#
|
|
# Example::
|
|
#
|
|
# urls = await step("urls", lambda: get_approval_urls("manager"))
|
|
# await step("notify", lambda: send_email(urls["resume"], urls["cancel"]))
|
|
# result = await wait_for_approval(key="manager", timeout=3600)
|
|
async def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_approval: bool = True, key: str | None = None, skin: Literal['detailed', 'minimal'] | None = None, description: str | dict | None = None) -> dict
|
|
|
|
# Process items in parallel with optional concurrency control.
|
|
#
|
|
# Each item is processed by calling ``fn(item)``, which should be a @task.
|
|
# Items are dispatched in batches of ``concurrency`` (default: all at once).
|
|
#
|
|
# Example::
|
|
#
|
|
# @task
|
|
# async def process(item: str):
|
|
# ...
|
|
#
|
|
# results = await parallel(items, process, concurrency=5)
|
|
async def parallel(items, fn, *, concurrency: Optional[int] = None)
|
|
|
|
# Commit Kafka offsets for a trigger with auto_commit disabled.
|
|
#
|
|
# Args:
|
|
# trigger_path: Path to the Kafka trigger (from event['wm_trigger']['trigger_path'])
|
|
# topic: Kafka topic name (from event['topic'])
|
|
# partition: Partition number (from event['partition'])
|
|
# offset: Message offset to commit (from event['offset'])
|
|
def commit_kafka_offsets(trigger_path: str, topic: str, partition: int, offset: int) -> None
|
|
|