diff --git a/backend/windmill-api/src/ai_session_tools/gen/instructions.json b/backend/windmill-api/src/ai_session_tools/gen/instructions.json index af8b45c68a..018490080f 100644 --- a/backend/windmill-api/src/ai_session_tools/gen/instructions.json +++ b/backend/windmill-api/src/ai_session_tools/gen/instructions.json @@ -289,7 +289,7 @@ "datatable.python3": "# Data tables: schema changes, migrations and permissions\n\nA data table is a PostgreSQL database managed by Windmill and shared by the whole workspace: scripts, flows and apps reach it by name, `main` being the default. It is configured in the workspace settings (Data Tables), never created by SQL.\n\n## Changing the schema\n\nA data table with migrations enabled keeps its schema as an ordered list of versioned SQL migrations, each an `up` and an optional `down`. That list is what git sync and a workspace merge carry, and what rebuilds the schema in another workspace. So first find out whether migrations are enabled on the data table:\n\n- **Enabled**: make every schema change (CREATE, ALTER, DROP, anything but row data) as a new migration. A change made with plain SQL exists in that one database only, and the history no longer describes it.\n- **Not enabled**: schema changes are plain SQL. Enabling migrations is a workspace admin's decision, made in the Data Tables settings, not a step to take on the way to something else.\n\nWriting a migration:\n\n- Wrap the statements in `BEGIN; ... END;` so it applies atomically. Leave the frame out only for a statement that cannot run in a transaction, such as `CREATE INDEX CONCURRENTLY`: such a migration can fail half applied.\n- Give it a `down` that undoes the `up` whenever the change is reversible. Without one it cannot be rolled back.\n- Declare the schemas it changes on a leading line of its `up`, `-- schemas sales, billing`, and qualify every name with its schema (`sales.orders`). A migration waits only on earlier migrations whose schemas overlap its own. One that declares none applies to the whole database and every later migration waits for it, so leave the line out only for a change that really spans the database, such as an extension.\n- Migrations apply oldest first. One that is applied is not edited: correct it with a new migration.\n- Row data is not schema. Insert, update and delete rows with plain SQL, migrations or not.\n\n## A migration is not a draft\n\nA migration is saved to the workspace and changes the database as soon as it is applied, for everything that reads those tables. To try a schema out without touching tables already in use:\n\n- put the new tables in a schema of their own, created by the same migration (`CREATE SCHEMA IF NOT EXISTS`), and point what you are building at it. A raw app names its schema in its `data` config, which is why a new app gets a new schema by default;\n- or work in a workspace fork whose data table was cloned when the fork was created (\"Clone schema only\" or \"Clone schema and data\"): migrations applied there change that copy alone, and reach the parent workspace when the fork is merged. A fork that kept the original data table points at the parent's database, so a migration applied in it changes the parent's tables at once.\n\nAltering or dropping a table other scripts or apps use is neither of those: say what depends on it before doing it.\n\n## Roles and permissions\n\nData table roles are a self-hosted Enterprise Edition feature, for data tables on managed Postgres (the instance database or the external managed instance). Without them, and always on a data table backed by a Postgres resource, everyone who can use the data table reaches all of it: there is no per-user, per-schema or per-row access to promise.\n\nA data table under roles is reached as a Postgres role, and Postgres enforces what that role was granted:\n\n- roles are assigned, per data table, to users, groups or folders. `admin` is the connection the data table had before roles: it owns the existing objects and ignores grants;\n- a connection either names a role (`wmill.datatable('main', { role: 'analyst' })`, `role='analyst'` in Python, `datatable://main?role=analyst`) and holds that role alone, or names none and is resolved from the data table's default role;\n- `permission denied` means the role in use lacks a grant. Say which role and which object: the grant is a data table admin's to give, in the roles drawer of the Database Manager. Do not switch to `admin` to get past it;\n- a migration runs as the role its SQL declares on its first line, `-- role `, the same one in its `down`. One that declares none runs as `admin`, which only those who may use `admin` can do.\n\nRead the Permissions section of https://www.windmill.dev/docs/core_concepts/persistent_storage/data_tables before designing who reaches what: it covers tenants, grants, the default role and the roles a connection that names none ends up holding, rows that differ per viewer through row-level security, and how forks and raw apps behave under roles.\n\n\n# Datatable SQL SDK reference\n\nDatatables are workspace-scoped managed PostgreSQL databases. In chat, explore and shape them with the datatable tools you were given. The reference below is for code you author inside runnables (inline app runnables, scripts, or flow rawscript modules) that reads or writes datatable data at runtime.\n\n- A runnable accesses a datatable via `wmill.datatable()` (the default \"main\") or `wmill.datatable('')`, referencing tables as `schema.table`.\n- Use parameterized queries (the tagged template in TypeScript, `$1`/`$2` placeholders in Python) — never interpolate untrusted values into SQL strings.\n\n## Python Datatable API (wmill)\n\nImport: `import wmill`\n\n# Get a DataTable client for SQL queries.\n# \n# Args:\n# name: Database name (default: \"main\")\n# role: Connect as this data table role instead of the data table's default one.\n# \n# Returns:\n# DataTableClient instance\ndef datatable(name: str = 'main', *, role: Optional[str] = None) -> DataTableClient\n\n# Client for executing SQL queries against Windmill DataTables.\nclass DataTableClient:\n # Initialize DataTableClient.\n # \n # Args:\n # client: Windmill client instance\n # name: DataTable name\n # role: Data table role to connect as, or None for the data table's default\n def __init__(client: Windmill, name: str, role: Optional[str] = None)\n\n # Execute a SQL query against the DataTable.\n # \n # Args:\n # sql: SQL query string with $1, $2, etc. placeholders\n # *args: Positional arguments to bind to query placeholders\n # \n # Returns:\n # SqlQuery instance for fetching results\n def query(sql: str, *args) -> SqlQuery\n\n\n# Query result handler for DataTable and DuckLake queries.\nclass SqlQuery:\n # Initialize SqlQuery.\n # \n # Args:\n # sql: SQL query string\n # fetch_fn: Function to execute the query\n def __init__(sql: str, fetch_fn)\n\n # Execute query and fetch results.\n # \n # Args:\n # result_collection: Optional result collection mode\n # \n # Returns:\n # Query results\n def fetch(result_collection: str | None = None)\n\n # Execute query and fetch first row of results.\n # \n # Returns:\n # First row of query results\n def fetch_one()\n\n # Execute query and fetch first row of results. Return result as a scalar value.\n # \n # Returns:\n # First row of query result as a scalar value\n def fetch_one_scalar()\n\n # Execute query and don't return any results.\n # \n def execute()\n\n\n", "datatable.go": "# Data tables: schema changes, migrations and permissions\n\nA data table is a PostgreSQL database managed by Windmill and shared by the whole workspace: scripts, flows and apps reach it by name, `main` being the default. It is configured in the workspace settings (Data Tables), never created by SQL.\n\n## Changing the schema\n\nA data table with migrations enabled keeps its schema as an ordered list of versioned SQL migrations, each an `up` and an optional `down`. That list is what git sync and a workspace merge carry, and what rebuilds the schema in another workspace. So first find out whether migrations are enabled on the data table:\n\n- **Enabled**: make every schema change (CREATE, ALTER, DROP, anything but row data) as a new migration. A change made with plain SQL exists in that one database only, and the history no longer describes it.\n- **Not enabled**: schema changes are plain SQL. Enabling migrations is a workspace admin's decision, made in the Data Tables settings, not a step to take on the way to something else.\n\nWriting a migration:\n\n- Wrap the statements in `BEGIN; ... END;` so it applies atomically. Leave the frame out only for a statement that cannot run in a transaction, such as `CREATE INDEX CONCURRENTLY`: such a migration can fail half applied.\n- Give it a `down` that undoes the `up` whenever the change is reversible. Without one it cannot be rolled back.\n- Declare the schemas it changes on a leading line of its `up`, `-- schemas sales, billing`, and qualify every name with its schema (`sales.orders`). A migration waits only on earlier migrations whose schemas overlap its own. One that declares none applies to the whole database and every later migration waits for it, so leave the line out only for a change that really spans the database, such as an extension.\n- Migrations apply oldest first. One that is applied is not edited: correct it with a new migration.\n- Row data is not schema. Insert, update and delete rows with plain SQL, migrations or not.\n\n## A migration is not a draft\n\nA migration is saved to the workspace and changes the database as soon as it is applied, for everything that reads those tables. To try a schema out without touching tables already in use:\n\n- put the new tables in a schema of their own, created by the same migration (`CREATE SCHEMA IF NOT EXISTS`), and point what you are building at it. A raw app names its schema in its `data` config, which is why a new app gets a new schema by default;\n- or work in a workspace fork whose data table was cloned when the fork was created (\"Clone schema only\" or \"Clone schema and data\"): migrations applied there change that copy alone, and reach the parent workspace when the fork is merged. A fork that kept the original data table points at the parent's database, so a migration applied in it changes the parent's tables at once.\n\nAltering or dropping a table other scripts or apps use is neither of those: say what depends on it before doing it.\n\n## Roles and permissions\n\nData table roles are a self-hosted Enterprise Edition feature, for data tables on managed Postgres (the instance database or the external managed instance). Without them, and always on a data table backed by a Postgres resource, everyone who can use the data table reaches all of it: there is no per-user, per-schema or per-row access to promise.\n\nA data table under roles is reached as a Postgres role, and Postgres enforces what that role was granted:\n\n- roles are assigned, per data table, to users, groups or folders. `admin` is the connection the data table had before roles: it owns the existing objects and ignores grants;\n- a connection either names a role (`wmill.datatable('main', { role: 'analyst' })`, `role='analyst'` in Python, `datatable://main?role=analyst`) and holds that role alone, or names none and is resolved from the data table's default role;\n- `permission denied` means the role in use lacks a grant. Say which role and which object: the grant is a data table admin's to give, in the roles drawer of the Database Manager. Do not switch to `admin` to get past it;\n- a migration runs as the role its SQL declares on its first line, `-- role `, the same one in its `down`. One that declares none runs as `admin`, which only those who may use `admin` can do.\n\nRead the Permissions section of https://www.windmill.dev/docs/core_concepts/persistent_storage/data_tables before designing who reaches what: it covers tenants, grants, the default role and the roles a connection that names none ends up holding, rows that differ per viewer through row-level security, and how forks and raw apps behave under roles.\n\n\n# Datatable SQL SDK reference\n\nDatatables are workspace-scoped managed PostgreSQL databases. In chat, explore and shape them with the datatable tools you were given. The reference below is for code you author inside runnables (inline app runnables, scripts, or flow rawscript modules) that reads or writes datatable data at runtime.\n\n- A runnable accesses a datatable via `wmill.datatable()` (the default \"main\") or `wmill.datatable('')`, referencing tables as `schema.table`.\n- Use parameterized queries (the tagged template in TypeScript, `$1`/`$2` placeholders in Python) — never interpolate untrusted values into SQL strings.\n\n## TypeScript Datatable API (windmill-client)\n\nImport: `import * as wmill from 'windmill-client'`\n\nSQL statement object with query content, arguments, and execution methods\n```typescript\ntype SqlStatement = {\n /** Raw SQL content with formatted arguments */\n content: string;\n\n /** Argument values keyed by parameter name */\n args: Record;\n\n /**\n * Execute the SQL query and return results\n * @param params - Optional parameters including result collection mode\n * @returns Query results based on the result collection mode\n */\n fetch(\n params?: FetchParams // The union is for auto-completion\n ): Promise>;\n\n /**\n * Execute the SQL query and return only the first row\n * @param params - Optional parameters\n * @returns First row of the query result\n */\n fetchOne(\n params?: Omit, \"resultCollection\">\n ): Promise>;\n\n /**\n * Execute the SQL query and return only the first row as a scalar value\n * @param params - Optional parameters\n * @returns First row of the query result\n */\n fetchOneScalar(\n params?: Omit<\n FetchParams<\"last_statement_first_row_scalar\">,\n \"resultCollection\"\n >\n ): Promise>;\n\n /**\n * Execute the SQL query without fetching rows\n * @param params - Optional parameters\n */\n execute(\n params?: Omit, \"resultCollection\">\n ): Promise;\n};\n```\n\n```typescript\n// Template tag function: sql`SELECT * FROM table WHERE id = ${id}`.fetch()\ninterface DatatableSqlTemplateFunction {\n // Tagged template usage:\n (strings: TemplateStringsArray, ...values: any[]): SqlStatement;\n query(sql: string, ...params: any[]): SqlStatement;\n};\n```\n\nCreate a SQL template function for PostgreSQL/datatable queries\n@param name - Database/datatable name (default: \"main\")\n@param opts.role - Connect as this data table role instead of the data table's default one.\n Only meaningful on a data table under roles, and only for a role you are a tenant of.\n@returns SQL template function for building parameterized queries\n@example\nlet sql = wmill.datatable()\nlet name = 'Robin'\nlet age = 21\nawait sql`\n SELECT * FROM friends\n WHERE name = ${name} AND age = ${age}::int\n`.fetch()\n@example\n// Read through a restricted role\nlet sql = wmill.datatable(\"main\", { role: \"analytics\" })\n```typescript\nfunction datatable(name: string = \"main\",\n opts?: DatatableOptions): DatatableSqlTemplateFunction\n```\n\n\n## Python Datatable API (wmill)\n\nImport: `import wmill`\n\n# Get a DataTable client for SQL queries.\n# \n# Args:\n# name: Database name (default: \"main\")\n# role: Connect as this data table role instead of the data table's default one.\n# \n# Returns:\n# DataTableClient instance\ndef datatable(name: str = 'main', *, role: Optional[str] = None) -> DataTableClient\n\n# Client for executing SQL queries against Windmill DataTables.\nclass DataTableClient:\n # Initialize DataTableClient.\n # \n # Args:\n # client: Windmill client instance\n # name: DataTable name\n # role: Data table role to connect as, or None for the data table's default\n def __init__(client: Windmill, name: str, role: Optional[str] = None)\n\n # Execute a SQL query against the DataTable.\n # \n # Args:\n # sql: SQL query string with $1, $2, etc. placeholders\n # *args: Positional arguments to bind to query placeholders\n # \n # Returns:\n # SqlQuery instance for fetching results\n def query(sql: str, *args) -> SqlQuery\n\n\n# Query result handler for DataTable and DuckLake queries.\nclass SqlQuery:\n # Initialize SqlQuery.\n # \n # Args:\n # sql: SQL query string\n # fetch_fn: Function to execute the query\n def __init__(sql: str, fetch_fn)\n\n # Execute query and fetch results.\n # \n # Args:\n # result_collection: Optional result collection mode\n # \n # Returns:\n # Query results\n def fetch(result_collection: str | None = None)\n\n # Execute query and fetch first row of results.\n # \n # Returns:\n # First row of query results\n def fetch_one()\n\n # Execute query and fetch first row of results. Return result as a scalar value.\n # \n # Returns:\n # First row of query result as a scalar value\n def fetch_one_scalar()\n\n # Execute query and don't return any results.\n # \n def execute()\n\n\n", "pipeline": "# Data pipeline authoring\n\nA **data pipeline** is NOT a flow. A flow is one runnable that orchestrates steps internally. A data pipeline is a set of **independent scripts**, each deployed on its own, that form a DAG by reading and writing shared **storage assets** (DuckLake tables, data tables, S3 objects, volumes, resources) and by declaring execution **triggers**. The pipeline is visualized and edited at `/pipeline/`; every node is a normal workspace script that happens to carry pipeline annotations. When the user asks for a \"data pipeline\" (or to \"ingest / transform / materialize\" data across steps), build pipeline-annotated scripts — do NOT build a flow.\n\n## Default to DuckDB + DuckLake\n\nA pipeline node that produces a table should almost always be a **`duckdb`** node that materializes its output into a **DuckLake** table with `-- materialize ducklake:///` (in a DuckDB node the annotation uses SQL `--` comment syntax; write the body as a bare `SELECT` and let the runtime do the write). DuckLake is the default lakehouse store for pipelines and is the shape the pipeline editor is built around, so prefer it unless the work specifically calls for something else:\n\n- `postgresql` / data tables — only for row-level, OLTP-style mutations against an existing Postgres data table (frequent single-row upserts/updates, transactional reads that an app queries live).\n- `bun` / `python3` — only for work that doesn't map to SQL: calling an external API, wrangling files, arbitrary glue. When such a node produces tabular data for downstream steps, write it **straight into DuckLake** (or a data table) from the same script — see \"Ingesting from Python / TypeScript\" below — rather than inventing a parallel store.\n\nDo not spread a pipeline across postgres, S3, and DuckLake when one DuckLake lake would do; a consistent DuckLake lakehouse is the goal.\n\n## Ingesting from Python / TypeScript\n\nA node that fetches data in code (an API, a SaaS export, a scraper) writes the rows **directly** into their DuckLake table with `wmill.ducklake()`, or into a data table with `wmill.datatable()`. DuckLake already stores every table as parquet files in the workspace object storage, so **never stage the raw payload first** — no intermediate JSON / CSV / parquet file in S3 and no second \"load the raw file\" node. One ingestion node, one table.\n\nPass the rows as a query argument (a list of dicts / array of objects is sent as JSON) and let SQL unnest them; declare the column types so the table gets a real schema:\n\n```python\n# pipeline\n# on schedule\nimport wmill\n\ndef main():\n rows = fetch_charges() # list[dict]\n lake = wmill.ducklake()\n lake.query(\"\"\"\n CREATE OR REPLACE TABLE stripe_charges AS\n SELECT unnest(from_json($rows, '[{\"id\":\"VARCHAR\",\"amount\":\"BIGINT\",\"created\":\"BIGINT\"}]'), recursive := true)\n \"\"\", rows=rows).fetch()\n```\n\n```ts\n// pipeline\n// on schedule\nimport * as wmill from \"windmill-client\"\n\nexport async function main() {\n const rows = await fetchCharges()\n const lake = wmill.ducklake()\n await lake`CREATE OR REPLACE TABLE stripe_charges AS\n SELECT unnest(from_json(${rows}, '[{\"id\":\"VARCHAR\",\"amount\":\"BIGINT\",\"created\":\"BIGINT\"}]'), recursive := true)`.fetch()\n}\n```\n\n- Use `INSERT INTO
SELECT ...` instead of `CREATE OR REPLACE` to append (an incremental pull), with a one-time `CREATE TABLE IF NOT EXISTS` first.\n- Assign the client to a variable (`lake = wmill.ducklake()`) and keep the SQL a **string literal** (Python) or a template without `sql.raw` (TS): that is what lets the graph record the node's write to `ducklake://main/
`. A chained `wmill.ducklake().query(...)` or SQL built in a variable records no output, so the downstream edge never forms.\n- `wmill.ducklake(\"\")` targets a non-default catalog; `wmill.datatable(\"\")` writes a Postgres data table the same way (Python `db.query(sql, *args)` with `$1` placeholders, TS the same template form) — use it only when the data belongs in Postgres (see above).\n- The downstream node triggers on the table itself: `-- on ducklake://main/stripe_charges`.\n\n## Storage prerequisites\n\nA DuckLake pipeline only runs once the workspace has **object storage** (S3 / Azure Blob / GCS) **and a DuckLake catalog** configured — DuckLake tables and `s3://` assets can't be materialized or read without it. Check which DuckLake catalogs the workspace has before you build.\nDrafting the annotated scripts does not require storage, but the pipeline can't ingest, materialize, or read its assets until it exists. So if there is none (or the user hits \"storage not configured\" errors), say so and give the right next step **by role**:\n\n- a workspace **admin** sets it up in Workspace settings → Object Storage (add an S3/Azure/GCS storage), then adds a DuckLake catalog on top of it;\n- anyone **without admin rights** should ask a workspace admin to configure object storage + a DuckLake catalog.\n\nNever hand back a DuckLake pipeline that cannot run without flagging the missing storage and pointing to who sets it up.\n\n## What makes a script a pipeline node\n\nA script joins the pipeline when its source begins with the `pipeline` annotation as a top-of-file comment, **written in the script's own comment syntax** — `//` for TS/JS (bun), `--` for SQL (DuckDB/Postgres), `#` for Python/Bash. So it's `-- pipeline` in a DuckDB node, `# pipeline` in a Python node, `// pipeline` in a bun node. Every annotation below uses that same prefix (the `//` shown is the TS form). All other wiring is expressed as annotation comments near the top of the file:\n\n- `// on ` — declares an execution-DAG **input** (what triggers/feeds this node). `` is either:\n - an **asset URI** (the node runs when that asset is produced upstream): `ducklake://main/orders`, `datatable://main/users`, `$res:f/folder/my_resource`, `volume://name/path`, or an S3 object (see the S3 storage-form rule below).\n - a **native trigger kind**: `schedule`, `webhook`, `email`, `kafka`, `mqtt`, `amqp`, `nats`, `postgres`, `sqs`, `gcp`, or `data_upload` (a user-uploaded S3 file). For these the actual trigger row (cron, topic, …) is created separately; the annotation only declares the binding. **`data_upload` is special**: there is no trigger row — the node instead declares an **`S3Object` input parameter** fed by the auto-generated upload picker; it never hard-codes a key. Any language can be the `data_upload` node:\n - Python (has `import wmill`): `def main(file: wmill.S3Object):` then `wmill.load_s3_file(file)`; TS (has `import * as wmill from \"windmill-client\"`): `export async function main(file: wmill.S3Object)`. Qualify the type as `wmill.S3Object` (or add `from wmill import S3Object` / `import { S3Object } from \"windmill-client\"`) — a bare `S3Object` is undefined.\n - **DuckDB** takes the s3object arg via a `-- $ (s3object)` declaration and reads it directly, so a single DuckDB node can ingest **and** materialize:\n ```\n -- pipeline\n -- on data_upload\n -- materialize ducklake://main/raw_uploads\n -- $file (s3object)\n SELECT * FROM read_csv($file)\n ```\n- **Outputs** are inferred from what the body writes — a `CREATE TABLE`, a `wmill.writeS3File(...)`, a DuckLake/datatable write. To declare a managed output explicitly, use `// materialize `.\n- Optional badges: `// partitioned `, `// freshness ` (e.g. `1h`), `// tag `, `// retry [delay]`, `// data_test ...` (managed DuckLake targets only — deploy rejects it beside a `dbt://` one), `// measure = [where ]` and `// dimension = ` (see \"Declared metrics\" below).\n\n## S3 object wiring (storage form matters)\n\nAn `s3://` URI's first slashes select the **storage**, not part of the key — get this wrong and the producer/consumer edge silently won't connect:\n\n- `s3:///` (**triple** slash, empty first segment) = the **default** workspace storage. A downstream node reading or triggering on that object uses `s3:///` — e.g. DuckDB `-- on s3:///orders/2024.parquet` and `read_parquet('s3:///orders/2024.parquet')`.\n- `s3:///` (**double** slash, non-empty first segment) = a **named secondary** storage called `` — so `s3://ingest/x` means storage `ingest`, key `x`, NOT key `ingest/x`. Only use this when the object genuinely lives in a configured secondary storage; never invent a bucket/storage name for a default-storage object (it breaks the edge).\n\nTo make the producer side visible to lineage, a Python/TS node MUST pass the **`S3Object` form**, not a bare key string: Python `wmill.write_s3_file(wmill.S3Object(s3=\"\"), data)` (or the import-free dict `{\"s3\": \"\"}`), TS `wmill.writeS3File({ s3: \"\" }, data)`. That records the default-storage asset `/`, which a downstream `s3:///` reader connects to (same key both sides). A bare `write_s3_file(\"\", ...)` records **no** asset and produces **no** edge. Add `storage=\"\"` only for a named secondary storage.\n\nThe key must be a **string literal** — the graph parser is static and cannot follow a variable, f-string, or computed path, so `write_s3_file(wmill.S3Object(s3=key_var), ...)` records no edge. Inline the literal (`s3=\"events/user_events.parquet\"`) on both the writing and reading node. The same rule applies to every asset URI in an annotation or SDK call (`ducklake://`, `datatable://`, `s3://`): write them literally, not via a variable.\n\n## Materialize (the managed output)\n\n> **A MANAGED `// materialize` is DuckDB-only**, and its target must be a DuckLake table (`ducklake:///
`). Deploy **rejects** a `ducklake://` `// materialize` on any other language (`python3`, `bun`, `postgresql`). For a non-DuckDB node writing the lake, do **not** use `// materialize` — write the table itself with `wmill.ducklake()` (see \"Ingesting from Python / TypeScript\") and let the output be inferred. Use `duckdb` when a node should materialize a DuckLake table.\n>\n> The one target ANY language may declare (except a dbt script, whose writes come from its manifest) is a **warehouse relation**: `// materialize manual dbt:////`, where `` is a warehouse the workspace configures under Settings → dbt. `manual` is the only mode it has — nothing generates warehouse DDL, so the node issues its own write (a postgresql `CREATE TABLE` / `INSERT`, an SDK load, …) and the annotation records the outcome. Use it on an ingestion node whose output a dbt project reads as a `source`: the declared relation and the dbt model land on ONE graph node, and a downstream `// on dbt:////` fires when the ingestion node completes.\n\nA managed `// materialize ducklake:///
` tells the runtime to write the node's output table **for you**: write the body as a single `SELECT` and the runtime wraps it in the create/replace — do **not** also write your own `CREATE TABLE` / `INSERT`. (The opposite holds for the `dbt://` target above: there the node writes its own DDL and the strategies below do not apply.)\n\nA target that names no table — `// materialize`, `// materialize ducklake://`, or options alone such as `// materialize key=id` — writes the table named after the script: node `f/shop/orders_daily` materializes `ducklake://main/orders_daily` (characters other than letters, digits and `_` become `_`, and a name starting with a digit gets a leading `_`). Name the node after the table it produces and the bare form is enough; write the full target when the table's name differs from the node's, and use that same name in every downstream `// on` and read.\n\nWrite strategy:\n\n- no option → **replace** the whole table each run (full refresh; the only mode whose output columns may change);\n- `// materialize append` → INSERT-append rows (incremental);\n- `// materialize key=` → merge/upsert on ``.\n\n`// materialize manual ` opts **out** of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage.\n\n`materialize` pairs with partitioning for incremental pipelines: a `// partitioned ` node runs **once per partition** (append/merge into a fixed-schema table). The `{partition}` token, usable in any asset URI **and** in the body SQL, is replaced at run time by the current partition's **identity string**:\n\n- To filter the source to the active slice on a time grain, use the runtime-injected macro: `WHERE wm_partition() = {partition}`. `wm_partition(ts)` buckets a timestamp in exactly the identity format the runtime uses for daily/hourly/weekly/monthly, so it always matches; never hand-write a `strftime` format.\n- Do NOT write `= TIMESTAMP {partition}`: the identity string is not a valid timestamp literal for hourly/weekly/monthly and errors at run time.\n- For `dynamic` partitioning the identity is the caller-supplied key (not a timestamp, no macro), so filter on it directly: `WHERE = {partition}`.\n\n`materialize` is an output **declaration** on a node — not a command. There is no \"materialize run\".\n\n## Declared metrics (`measure` / `dimension`)\n\nOn a node that materializes a DuckLake table, `// measure = [where ]` names the canonical way to aggregate that table (e.g. `// measure revenue = sum(amount) where not is_refund`), and `// dimension = ` names a way to slice it (e.g. `// dimension region = region`, `// dimension month = date_trunc('month', ordered_at)`). They execute nothing: they are catalogued at deploy so the editor and other agents reuse the definition instead of re-deriving it and silently disagreeing.\n\n- Keep the predicate in the `where` clause rather than folding it into the aggregate: it is rendered as ` FILTER (WHERE )`, which is what lets two measures with different predicates share one GROUP BY.\n- DuckLake-only, and only meaningful next to `// materialize`.\n- Declare one when a number carries a judgement call someone else would get wrong (refunds excluded, test rows dropped, which column is the amount); do NOT blanket every table with measures — an obvious `count(*)` earns nothing.\n- To use a metric another node declares, read that node and reuse its exact expression rather than guessing it.\n\n## How to build one\n\n1. Put every node in the **same folder**: `f//`. The folder is the pipeline.\n2. Write each node as its own script. Default to `duckdb` materializing into DuckLake (see \"Default to DuckDB + DuckLake\" above); pick `postgresql`, `bun`, or `python3` only when that section says the work calls for it.\n3. Start each body with `// pipeline`, then the `// on` input declarations, then the transform that writes the output.\n4. **Chain nodes by asset URI**: read an upstream node's output asset, then `// on ` in the downstream node so the edge forms. Reuse exact asset paths from existing nodes rather than inventing parallel ones.\n5. Never guess the columns of a file or table you have not seen: read them first with a query that writes nothing, such as `DESCRIBE SELECT * FROM read_csv('s3:///raw/orders/*.csv')` or `ATTACH 'ducklake://main' AS lake; DESCRIBE lake.orders`, and use those exact column names. A test run of a node that materializes a table writes that table into the workspace's lake: test-run such a node only while its table does not exist yet, and ask before running one that would replace a table already there.\n6. Don't deploy nodes unless the user asks to. A pipeline only \"runs\" once its scripts are deployed and their triggers exist.\n\n## Example (DuckDB → DuckLake, scheduled ingest + downstream transform)\n\nNode `f/sales/orders_ingest` (runs on a schedule, materializes a DuckLake table):\n\n```sql\n-- pipeline\n-- on schedule\n-- materialize ducklake://main/orders\nSELECT * FROM read_csv('s3:///raw/orders/*.csv')\n```\n\nNode `f/sales/orders_daily` (runs when `orders` is produced, writes a rollup):\n\n```sql\n-- pipeline\n-- on ducklake://main/orders\n-- materialize ducklake://main/orders_daily\nATTACH 'ducklake://main' AS lake;\nSELECT date_trunc('day', ts) AS day, count(*) AS n\nFROM lake.orders GROUP BY 1\n```\n\nA node reads a DuckLake table only through a lake it attaches itself (`ATTACH 'ducklake://' AS lake;`, then `lake.
`); the materialize target is attached by the runtime under a name of its own, so `ducklake.main.
` does not resolve.\n", - "workspace_settings": "# Workspace settings\n\nEverything an admin sets on the Workspace settings page is one document: a mapping of top-level\nkeys, the same layout as the `settings.yaml` file `wmill sync` keeps. Only a workspace admin can\nread or change it.\n\n## Applying a document\n\n- A partial document changes only the keys it states; every key it omits is left as it is.\n- Each key is replaced whole, so read its current value and send it back edited: a nested object\n given with one field replaces the stored object with that one field.\n- A key set to `null` clears that setting; an on/off switch cleared this way reads as `false`.\n- Each setting is written through the same path the settings page uses, with the same validation\n and audit log. The writes are not one transaction: when one fails the apply stops, and the\n error says what was already written.\n- A dry run reports the changes, as `before` / `after` per key, without writing anything.\n- `name` is never applied: rename a workspace on the settings page. `color` is applied only when\n set to a value, so `null` does not clear it.\n- A key the server does not know is ignored and named in the warnings: check them, since a typo\n otherwise reads as \"nothing to change\".\n- A setting this edition does not offer (Enterprise only) is refused, by the dry run too.\n\n## Secrets\n\n`slack_oauth_client_secret` is the one secret. A read shows `__SENSITIVE_AND_UNCHANGED__` in place\nof a set secret; sending that placeholder back keeps the stored value. A value starting with\n`$encrypted:` is decrypted with the workspace key, and one encrypted with another workspace's key\nis left unchanged with a warning; any other value is taken as plaintext. Never\nwrite a secret you were not given: keep the placeholder.\n\n## Keys\n\n| Key | Value |\n|---|---|\n| `auto_invite` | `{ enabled, mode: invite \\| add, operator, instance_groups?, instance_groups_roles? }`. The instance groups are managed only when `instance_groups` is stated; clearing a group removes the members it added. |\n| `error_handler` / `success_handler` | `{ path, extra_args?, muted_on_cancel?, muted_on_user_path? }`, `path` being `script/` or `flow/`. |\n| `error_handler_fallback_to_instance_alerts` | Report failures to the instance critical alert channels when no error handler is set. |\n| `webhook` | URL POSTed a JSON event on every change to the workspace's items. |\n| `ai_config` | Providers, default and completion models, custom prompts, `copilot_disabled`. |\n| `large_file_storage` | Workspace object storage: `{ type, s3_resource_path \\| azure_blob_resource_path \\| gcs_resource_path, public_resource?, secondary_storage?, advanced_permissions? }`. |\n| `git_sync` | `{ repositories: [...] }`, each with `git_repo_resource_path`, `script_path` and `settings` (include/exclude paths and types). Server-owned fields (webhook secrets, sync state, credentials) are never read nor written. |\n| `datatable` | `{ datatables: { : { database: { resource_type, resource_path } } } }`, both required: `resource_type: postgresql` with `resource_path` a Postgres resource's path, or `instance`, or `external_instance` (Enterprise only), with `resource_path` the name of a database a superadmin set up. A fork's entry can instead be `{ reference: { workspace_id, datatable } }`, pointing at its parent's data table; keep it as is. `main` is the one scripts use by default. Role permissions are `datatable_permissions`. |\n| `datatable_permissions` | `{ : { default_role, roles: { : [tenants] } } }`: who may connect to each data table as which role. Tenants are `u/`, `g/`, `f/` or `*`; `admin` is always a role, and `default_role: null` means no default. A data table the value omits is no longer under roles. Only a data table on a cluster Windmill manages can be. |\n| `protection_rules` | `{ : { rules: [...], bypass_groups: [...], bypass_users: [...], fork_role_mapping?: { : } } }`, rules among `DisableDirectDeployment`, `DisableWorkspaceForking`, `RestrictDeployToDeployers`, `RestrictAnonymousAppDeployment`, `RestrictPublicRunSharing`, `RestrictGuestAppDeployment`, `RequireIsolatedForks`. A rule the value omits is deleted. The dev workspace lock is not listed and cannot be set here. |\n| `ducklake` | `{ ducklakes: { : { catalog, storage, ... } } }`. |\n| `dbt_warehouses` | dbt warehouse connections. |\n| `default_app` | Path of the app shown as the workspace home. |\n| `default_scripts` | `{ order?, hidden?, default_script_content? }` for the script language picker. |\n| `deploy_ui` | `{ include_path?, include_type? }`: what the deploy-to-workspace UI offers. |\n| `role_overrides` | `{ : { label?, capabilities, workspace_only? } }`: the workspace's overrides of the instance's roles, and with `workspace_only: true` a role of this workspace only (Enterprise). A role the value omits loses its override. Overriding a role the instance does not define is refused. Edit operator rights in `role_overrides.operator`. |\n| `operator_settings` | Legacy view of the `operator` role, kept for older servers: page visibility flags (`runs`, `schedules`, `resources`, `variables`, `assets`, `audit_logs`, `groups`, `folders`, `workers`, `triggers`) and write rights (`manage_schedules`, `manage_triggers`, `builder_flows`, `builder_apps`). Not applied when the document also states `role_overrides`. |\n| `mute_critical_alerts` | Mute the workspace's critical alerts. |\n| `slack_team_id`, `slack_name` | Read only: set by connecting Slack on the settings page. |\n| `slack_command_script` / `teams_command_script` | Script or flow handling the Slack / Teams command. |\n| `slack_oauth_client_id` / `slack_oauth_client_secret` | The workspace's own Slack OAuth app. Clearing either removes both. |\n| `public_app_execution_limit_per_minute` | Rate limit for public apps, `null` for none. |\n| `guest_jwt_public_key`, `guest_jwt_jwks_url` | The key that signs guest JWTs for apps set to Guests. |\n| `add_admins_and_developers_to_forks` | Add the workspace's admins and developers to each fork. |\n| `color` | Workspace color, `#rrggbb`. |\n", + "workspace_settings": "# Workspace settings\n\nEverything an admin sets on the Workspace settings page is one document: a mapping of top-level\nkeys, the same layout as the `settings.yaml` file `wmill sync` keeps. Only a workspace admin can\nread or change it.\n\n## Applying a document\n\n- A partial document changes only the keys it states; every key it omits is left as it is.\n- Each key is replaced whole, so read its current value and send it back edited: a nested object\n given with one field replaces the stored object with that one field.\n- A key set to `null` clears that setting; an on/off switch cleared this way reads as `false`.\n- Each setting is written through the same path the settings page uses, with the same validation\n and audit log. The writes are not one transaction: when one fails the apply stops, and the\n error says what was already written.\n- A dry run reports the changes, as `before` / `after` per key, without writing anything.\n- `name` is never applied: rename a workspace on the settings page. `color` is applied only when\n set to a value, so `null` does not clear it.\n- A key the server does not know is ignored and named in the warnings: check them, since a typo\n otherwise reads as \"nothing to change\".\n- A setting this edition does not offer (Enterprise only) is refused, by the dry run too.\n\n## Secrets\n\n`webhook` (its URL usually carries the receiver's token) and `slack_oauth_client_secret` are the\nsecrets. A read shows `__SENSITIVE_AND_UNCHANGED__` in place\nof a set secret; sending that placeholder back keeps the stored value. A value starting with\n`$encrypted:` is decrypted with the workspace key, and one encrypted with another workspace's key\nis left unchanged with a warning; any other value is taken as plaintext. Never\nwrite a secret you were not given: keep the placeholder.\n\n## Keys\n\n| Key | Value |\n|---|---|\n| `auto_invite` | `{ enabled, mode: invite \\| add, operator, instance_groups?, instance_groups_roles? }`. The instance groups are managed only when `instance_groups` is stated; clearing a group removes the members it added. |\n| `error_handler` / `success_handler` | `{ path, extra_args?, muted_on_cancel?, muted_on_user_path? }`, `path` being `script/` or `flow/`. |\n| `error_handler_fallback_to_instance_alerts` | Report failures to the instance critical alert channels when no error handler is set. |\n| `webhook` | URL POSTed a JSON event on every change to the workspace's items. |\n| `ai_config` | Providers, default and completion models, custom prompts, `copilot_disabled`. |\n| `large_file_storage` | Workspace object storage: `{ type, s3_resource_path \\| azure_blob_resource_path \\| gcs_resource_path, public_resource?, secondary_storage?, advanced_permissions? }`. |\n| `git_sync` | `{ repositories: [...] }`, each with `git_repo_resource_path`, `script_path` and `settings` (include/exclude paths and types). Server-owned fields (webhook secrets, sync state, credentials) are never read nor written. |\n| `datatable` | `{ datatables: { : { database: { resource_type, resource_path } } } }`, both required: `resource_type: postgresql` with `resource_path` a Postgres resource's path, or `instance`, or `external_instance` (Enterprise only), with `resource_path` the name of a database a superadmin set up. A fork's entry can instead be `{ reference: { workspace_id, datatable } }`, pointing at its parent's data table; keep it as is. `main` is the one scripts use by default. Role permissions are `datatable_permissions`. |\n| `datatable_permissions` | `{ : { default_role, roles: { : [tenants] } } }`: who may connect to each data table as which role. Tenants are `u/`, `g/`, `f/` or `*`; `admin` is always a role, and `default_role: null` means no default. A data table the value omits is no longer under roles. Only a data table on a cluster Windmill manages can be. |\n| `protection_rules` | `{ : { rules: [...], bypass_groups: [...], bypass_users: [...], fork_role_mapping?: { : } } }`, rules among `DisableDirectDeployment`, `DisableWorkspaceForking`, `RestrictDeployToDeployers`, `RestrictAnonymousAppDeployment`, `RestrictPublicRunSharing`, `RestrictGuestAppDeployment`, `RequireIsolatedForks`. A rule the value omits is deleted. The dev workspace lock is not listed and cannot be set here. |\n| `ducklake` | `{ ducklakes: { : { catalog, storage, ... } } }`. |\n| `dbt_warehouses` | dbt warehouse connections. |\n| `default_app` | Path of the app shown as the workspace home. |\n| `default_scripts` | `{ order?, hidden?, default_script_content? }` for the script language picker. |\n| `deploy_ui` | `{ include_path?, include_type? }`: what the deploy-to-workspace UI offers. |\n| `role_overrides` | `{ : { label?, capabilities, workspace_only? } }`: the workspace's overrides of the instance's roles, and with `workspace_only: true` a role of this workspace only (Enterprise). A role the value omits loses its override. Overriding a role the instance does not define is refused. Edit operator rights in `role_overrides.operator`. |\n| `operator_settings` | Legacy view of the `operator` role, kept for older servers: page visibility flags (`runs`, `schedules`, `resources`, `variables`, `assets`, `audit_logs`, `groups`, `folders`, `workers`, `triggers`) and write rights (`manage_schedules`, `manage_triggers`, `builder_flows`, `builder_apps`). Not applied when the document also states `role_overrides`. |\n| `mute_critical_alerts` | Mute the workspace's critical alerts. |\n| `slack_team_id`, `slack_name` | Read only: set by connecting Slack on the settings page. |\n| `slack_command_script` / `teams_command_script` | Script or flow handling the Slack / Teams command. |\n| `slack_oauth_client_id` / `slack_oauth_client_secret` | The workspace's own Slack OAuth app. Clearing either removes both. |\n| `public_app_execution_limit_per_minute` | Rate limit for public apps, `null` for none. |\n| `guest_jwt_public_key`, `guest_jwt_jwks_url` | The key that signs guest JWTs for apps set to Guests. |\n| `add_admins_and_developers_to_forks` | Add the workspace's admins and developers to each fork. |\n| `color` | Workspace color, `#rrggbb`. |\n", "display": "# Rich results\n\nWindmill renders a script's or flow's return value according to its shape: on run pages, in the\nscript and flow editors, in app result displays and in AI chat run cards. Anything that matches none\nof the shapes below is shown as JSON.\n\n## Shapes\n\n- `{ \"table-row-object\": [{...}, ...] }`, or a bare array of objects: a table whose columns are the\n union of the objects' keys.\n- `{ \"table-col\": { \"col\": [...], ... } }`, or a bare object of equal-length arrays: a table built\n column by column.\n- `{ \"table-row\": [[\"col\", ...], [v, ...], ...] }`: a table whose first row holds the column names.\n- `{ \"chart\": {...} }`: a bar, line, pie or scatter chart drawn from rows (schema below). A `chart`\n value the schema rejects is shown as JSON, not as a broken chart.\n- `{ \"markdown\": \"...\" }` (or `md`): rendered markdown.\n- `{ \"html\": \"...\" }`: sanitized HTML.\n- `{ \"png\": \"\" }`, likewise `jpeg`, `gif`, `svg`: an image. `{ \"pdf\": \"\" }`: a PDF.\n- `{ \"file\": \"\", \"filename\": \"report.csv\" }`: a download.\n- `{ \"map\": { \"lat\": 0, \"lon\": 0, \"zoom\": 3, \"markers\": [{ \"lat\": 0, \"lon\": 0, \"title\": \"...\" }] } }`:\n a map.\n- `{ \"s3\": \"path/to/key\" }`: an object in the workspace storage, previewed by type.\n- `{ \"render_all\": [ ... ] }`: each item rendered in turn with the rules above.\n\nEach shape is an object with that one key (plus `filename` for a file, and an optional `storage` for\n`s3`). A result carrying any other key beside it is shown as JSON.\n\n## Choosing one\n\nA chart answers a question at a glance: a trend over time (line), a comparison across categories\n(bar), parts of a whole with few parts (pie), or a relation between two numbers (scatter). A table\nanswers lookups and holds the exact values. When both help, return both in a `render_all`: the chart\nfirst, then the table of the same rows.\n\nKeep a chart to what it shows well: at most 8 series. A pie with more than 8 slices keeps the 7\nlargest and folds the rest into \"Other\", so prefer a bar chart for many categories. Aggregate before\ncharting: a chart of thousands of raw points says less than one of the totals per day.\n\n\n### Chart and table schema\n\nA chart or table value, as JSON Schema. Beyond it: `x` and every `y` must be columns of `data`, `y` columns hold numbers, a scatter chart's `x` column holds numbers, and a pie chart takes exactly one `y`.\n\n```json\n{\"$schema\":\"https://json-schema.org/draft/2020-12/schema\",\"anyOf\":[{\"anyOf\":[{\"type\":\"object\",\"properties\":{\"chart\":{\"type\":\"object\",\"properties\":{\"type\":{\"description\":\"bar or line for categories or time, pie for parts of a whole, scatter for two numeric columns\",\"type\":\"string\",\"enum\":[\"bar\",\"line\",\"pie\",\"scatter\"]},\"title\":{\"type\":\"string\"},\"data\":{\"description\":\"The rows to plot, the same shape as a table-row-object result\",\"minItems\":1,\"maxItems\":5000,\"type\":\"array\",\"items\":{\"description\":\"One row: column name to value (string, number, boolean or null)\",\"type\":\"object\",\"propertyNames\":{\"type\":\"string\"},\"additionalProperties\":{\"anyOf\":[{\"type\":\"string\"},{\"type\":\"number\"},{\"type\":\"boolean\"},{\"type\":\"null\"}]}}},\"x\":{\"description\":\"Column holding the category, label or date of each row (bar, line, pie), or the numeric x value (scatter)\",\"type\":\"string\"},\"y\":{\"description\":\"Numeric column, or columns, to plot: one series each. A pie takes exactly one\",\"anyOf\":[{\"type\":\"string\"},{\"minItems\":1,\"maxItems\":8,\"type\":\"array\",\"items\":{\"type\":\"string\"}}]},\"stacked\":{\"description\":\"bar and line: stack the series\",\"type\":\"boolean\"},\"horizontal\":{\"description\":\"bar: draw the bars horizontally\",\"type\":\"boolean\"},\"x_label\":{\"type\":\"string\"},\"y_label\":{\"type\":\"string\"}},\"required\":[\"type\",\"data\",\"x\",\"y\"],\"additionalProperties\":false}},\"required\":[\"chart\"],\"additionalProperties\":false},{\"type\":\"object\",\"properties\":{\"table-row-object\":{\"description\":\"Rows as objects; the columns are the union of their keys\",\"minItems\":1,\"maxItems\":5000,\"type\":\"array\",\"items\":{\"description\":\"One row: column name to value (string, number, boolean or null)\",\"type\":\"object\",\"propertyNames\":{\"type\":\"string\"},\"additionalProperties\":{\"anyOf\":[{\"type\":\"string\"},{\"type\":\"number\"},{\"type\":\"boolean\"},{\"type\":\"null\"}]}}}},\"required\":[\"table-row-object\"],\"additionalProperties\":false},{\"type\":\"object\",\"properties\":{\"table-col\":{\"description\":\"Columns: column name to its values, all the same length\",\"type\":\"object\",\"propertyNames\":{\"type\":\"string\"},\"additionalProperties\":{\"maxItems\":5000,\"type\":\"array\",\"items\":{\"anyOf\":[{\"type\":\"string\"},{\"type\":\"number\"},{\"type\":\"boolean\"},{\"type\":\"null\"}]}}}},\"required\":[\"table-col\"],\"additionalProperties\":false},{\"type\":\"object\",\"properties\":{\"table-row\":{\"description\":\"Rows as arrays; the first row holds the column names\",\"minItems\":2,\"maxItems\":5000,\"type\":\"array\",\"items\":{\"type\":\"array\",\"items\":{\"anyOf\":[{\"type\":\"string\"},{\"type\":\"number\"},{\"type\":\"boolean\"},{\"type\":\"null\"}]}}}},\"required\":[\"table-row\"],\"additionalProperties\":false}]},{\"type\":\"object\",\"properties\":{\"render_all\":{\"description\":\"Several items shown one after another\",\"minItems\":1,\"maxItems\":10,\"type\":\"array\",\"items\":{\"anyOf\":[{\"type\":\"object\",\"properties\":{\"chart\":{\"type\":\"object\",\"properties\":{\"type\":{\"description\":\"bar or line for categories or time, pie for parts of a whole, scatter for two numeric columns\",\"type\":\"string\",\"enum\":[\"bar\",\"line\",\"pie\",\"scatter\"]},\"title\":{\"type\":\"string\"},\"data\":{\"description\":\"The rows to plot, the same shape as a table-row-object result\",\"minItems\":1,\"maxItems\":5000,\"type\":\"array\",\"items\":{\"description\":\"One row: column name to value (string, number, boolean or null)\",\"type\":\"object\",\"propertyNames\":{\"type\":\"string\"},\"additionalProperties\":{\"anyOf\":[{\"type\":\"string\"},{\"type\":\"number\"},{\"type\":\"boolean\"},{\"type\":\"null\"}]}}},\"x\":{\"description\":\"Column holding the category, label or date of each row (bar, line, pie), or the numeric x value (scatter)\",\"type\":\"string\"},\"y\":{\"description\":\"Numeric column, or columns, to plot: one series each. A pie takes exactly one\",\"anyOf\":[{\"type\":\"string\"},{\"minItems\":1,\"maxItems\":8,\"type\":\"array\",\"items\":{\"type\":\"string\"}}]},\"stacked\":{\"description\":\"bar and line: stack the series\",\"type\":\"boolean\"},\"horizontal\":{\"description\":\"bar: draw the bars horizontally\",\"type\":\"boolean\"},\"x_label\":{\"type\":\"string\"},\"y_label\":{\"type\":\"string\"}},\"required\":[\"type\",\"data\",\"x\",\"y\"],\"additionalProperties\":false}},\"required\":[\"chart\"],\"additionalProperties\":false},{\"type\":\"object\",\"properties\":{\"table-row-object\":{\"description\":\"Rows as objects; the columns are the union of their keys\",\"minItems\":1,\"maxItems\":5000,\"type\":\"array\",\"items\":{\"description\":\"One row: column name to value (string, number, boolean or null)\",\"type\":\"object\",\"propertyNames\":{\"type\":\"string\"},\"additionalProperties\":{\"anyOf\":[{\"type\":\"string\"},{\"type\":\"number\"},{\"type\":\"boolean\"},{\"type\":\"null\"}]}}}},\"required\":[\"table-row-object\"],\"additionalProperties\":false},{\"type\":\"object\",\"properties\":{\"table-col\":{\"description\":\"Columns: column name to its values, all the same length\",\"type\":\"object\",\"propertyNames\":{\"type\":\"string\"},\"additionalProperties\":{\"maxItems\":5000,\"type\":\"array\",\"items\":{\"anyOf\":[{\"type\":\"string\"},{\"type\":\"number\"},{\"type\":\"boolean\"},{\"type\":\"null\"}]}}}},\"required\":[\"table-col\"],\"additionalProperties\":false},{\"type\":\"object\",\"properties\":{\"table-row\":{\"description\":\"Rows as arrays; the first row holds the column names\",\"minItems\":2,\"maxItems\":5000,\"type\":\"array\",\"items\":{\"type\":\"array\",\"items\":{\"anyOf\":[{\"type\":\"string\"},{\"type\":\"number\"},{\"type\":\"boolean\"},{\"type\":\"null\"}]}}}},\"required\":[\"table-row\"],\"additionalProperties\":false}]}}},\"required\":[\"render_all\"],\"additionalProperties\":false}]}\n```\n\nBar chart, one series per y column:\n\n```json\n{\n \"chart\": {\n \"type\": \"bar\",\n \"title\": \"Runs per day\",\n \"data\": [\n {\n \"day\": \"2026-11-10\",\n \"success\": 42,\n \"failure\": 3\n },\n {\n \"day\": \"2026-11-11\",\n \"success\": 51,\n \"failure\": 1\n }\n ],\n \"x\": \"day\",\n \"y\": [\n \"success\",\n \"failure\"\n ],\n \"stacked\": true\n }\n}\n```\n\nPie chart (exactly one y column):\n\n```json\n{\n \"chart\": {\n \"type\": \"pie\",\n \"data\": [\n {\n \"language\": \"python3\",\n \"scripts\": 18\n },\n {\n \"language\": \"bun\",\n \"scripts\": 25\n }\n ],\n \"x\": \"language\",\n \"y\": \"scripts\"\n }\n}\n```\n\nA chart and a table together:\n\n```json\n{\n \"render_all\": [\n {\n \"chart\": {\n \"type\": \"line\",\n \"data\": [\n {\n \"hour\": \"10:00\",\n \"p95_ms\": 120\n },\n {\n \"hour\": \"11:00\",\n \"p95_ms\": 180\n }\n ],\n \"x\": \"hour\",\n \"y\": \"p95_ms\",\n \"y_label\": \"ms\"\n }\n },\n {\n \"table-row-object\": [\n {\n \"hour\": \"10:00\",\n \"p95_ms\": 120,\n \"jobs\": 310\n },\n {\n \"hour\": \"11:00\",\n \"p95_ms\": 180,\n \"jobs\": 402\n }\n ]\n }\n ]\n}\n```\n" } } diff --git a/backend/windmill-api/src/workspace_settings_document.rs b/backend/windmill-api/src/workspace_settings_document.rs index bcf7022184..a670340706 100644 --- a/backend/windmill-api/src/workspace_settings_document.rs +++ b/backend/windmill-api/src/workspace_settings_document.rs @@ -41,7 +41,8 @@ use crate::db::ApiAuthed; pub const SENSITIVE_UNCHANGED: &str = "__SENSITIVE_AND_UNCHANGED__"; use windmill_common::variables::SETTINGS_ENCRYPTED_PREFIX as ENCRYPTED_PREFIX; -const SECRET_KEYS: &[&str] = &["slack_oauth_client_secret"]; +/// The webhook URL is a secret because it usually carries the receiver's token. +const SECRET_KEYS: &[&str] = &["webhook", "slack_oauth_client_secret"]; #[derive(Clone, Copy, Debug, Deserialize, PartialEq, Eq, Default)] #[serde(rename_all = "snake_case")] @@ -366,17 +367,21 @@ pub async fn load_settings_document( .fetch_one(&mut *conn) .await?; - let slack_oauth_client_secret = match (row.slack_oauth_client_secret, secrets) { - (Some(s), SecretMode::Redacted) if !s.is_empty() => Some(SENSITIVE_UNCHANGED.to_string()), - (Some(s), SecretMode::Encrypted) if !s.is_empty() => { - let mc = build_crypt(db, w_id).await?; - Some(format!( - "{ENCRYPTED_PREFIX}{}", - mc.encrypt_str_deterministic(&s) - )) - } + let mc = match secrets { + SecretMode::Encrypted => Some(build_crypt(db, w_id).await?), + _ => None, + }; + let secret = |v: Option| match (v, &mc) { + (Some(s), _) if s.is_empty() => Some(s), + (Some(s), Some(mc)) => Some(format!( + "{ENCRYPTED_PREFIX}{}", + mc.encrypt_str_deterministic(&s) + )), + (Some(_), None) if secrets == SecretMode::Redacted => Some(SENSITIVE_UNCHANGED.to_string()), (s, _) => s, }; + let webhook = secret(row.webhook); + let slack_oauth_client_secret = secret(row.slack_oauth_client_secret); let has_permissions = row .datatable @@ -413,7 +418,7 @@ pub async fn load_settings_document( // added since is always written, so a pulled file states it and from then on manages it. let entries: [(&str, Option, bool); 33] = [ ("auto_invite", row.auto_invite, false), - ("webhook", row.webhook.map(Value::String), false), + ("webhook", webhook.map(Value::String), false), ("webhook_events", row.webhook_events.map(|e| json!(e)), true), ("error_handler", row.error_handler, true), ("success_handler", row.success_handler, true), @@ -2252,25 +2257,27 @@ mod tests { #[test] fn secrets_keep_their_value_for_the_placeholder_and_a_foreign_key() { let mc = WorkspaceCrypt::new("k"); - let ours = format!( - "{ENCRYPTED_PREFIX}{}", - mc.encrypt_str_deterministic("sekret") - ); let theirs = format!( "{ENCRYPTED_PREFIX}{}", WorkspaceCrypt::new("other").encrypt_str_deterministic("x") ); - for secret in [SENSITIVE_UNCHANGED.to_string(), ours, theirs.clone()] { - let p = plan_document( - current(), - doc(json!({ "slack_oauth_client_secret": secret })), - false, - &mc, - &DatatableNames::default(), - ) - .unwrap(); - assert!(p.changes.is_empty(), "{secret}: {:?}", keys(&p)); - assert_eq!(p.warnings.len(), usize::from(secret == theirs)); + for (key, stored) in [ + ("slack_oauth_client_secret", "sekret"), + ("webhook", "https://a"), + ] { + let ours = format!("{ENCRYPTED_PREFIX}{}", mc.encrypt_str_deterministic(stored)); + for secret in [SENSITIVE_UNCHANGED.to_string(), ours, theirs.clone()] { + let p = plan_document( + current(), + doc(json!({ key: secret })), + false, + &mc, + &DatatableNames::default(), + ) + .unwrap(); + assert!(p.changes.is_empty(), "{key} {secret}: {:?}", keys(&p)); + assert_eq!(p.warnings.len(), usize::from(secret == theirs)); + } } } diff --git a/cli/src/core/settings.ts b/cli/src/core/settings.ts index ec796de5d3..ec0035de79 100644 --- a/cli/src/core/settings.ts +++ b/cli/src/core/settings.ts @@ -232,6 +232,20 @@ export async function pushWorkspaceSettings( delete localSettings.datatable_permissions; } + if ( + unreadableSecret(localSettings.webhook) && + !(await documentKeepsRedactedWebhook(workspace)) + ) { + log.warn( + "Leaving the webhook unchanged: this server stores the webhook URL in plain text and cannot read an encrypted or redacted one. Upgrade it, or write the URL in plain text." + ); + const remote = await wmill.getWorkspaceSettingsDocument({ workspace }); + if (typeof remote.webhook === "string") { + localSettings.webhook = remote.webhook; + } else { + delete localSettings.webhook; + } + } if (await applySettingsDocument(workspace, localSettings)) { return; } @@ -240,10 +254,13 @@ export async function pushWorkspaceSettings( for (const key of SETTINGS_MANAGED_ONLY_WHEN_PRESENT) { delete (localSettings as any)[key]; } - const secret = localSettings.slack_oauth_client_secret; - const slackSecretUnreadable = - typeof secret === "string" && - (secret.startsWith("$encrypted:") || secret === "__SENSITIVE_AND_UNCHANGED__"); + const webhookUnreadable = unreadableSecret(localSettings.webhook); + if (webhookUnreadable) { + log.warn( + "Leaving the webhook unchanged: this server cannot read an encrypted or redacted webhook URL. Upgrade it, or write the URL in plain text." + ); + } + const slackSecretUnreadable = unreadableSecret(localSettings.slack_oauth_client_secret); if (slackSecretUnreadable) { log.warn( "Leaving the Slack OAuth client and secret unchanged: this server cannot read an encrypted or redacted secret. Upgrade it, or write the secret in plain text." @@ -301,7 +318,7 @@ export async function pushWorkspaceSettings( } log.debug(`Workspace settings are not up-to-date, updating...`); - if (localSettings.webhook !== settings.webhook) { + if (!webhookUnreadable && localSettings.webhook !== settings.webhook) { log.debug(`Updating webhook...`); await wmill.editWebhook({ workspace, @@ -550,6 +567,36 @@ export async function pushWorkspaceSettings( } } +function unreadableSecret(secret: unknown): boolean { + return ( + typeof secret === "string" && + (secret.startsWith("$encrypted:") || secret === "__SENSITIVE_AND_UNCHANGED__") + ); +} + +/** + * Whether the settings document treats the webhook URL as a secret: a server whose document + * predates that plans the placeholder as a new URL, which its webhook endpoint then refuses. + * True for a server without the document, whose fallback path handles the secret itself. + */ +async function documentKeepsRedactedWebhook(workspace: string): Promise { + try { + const result = await wmill.applyWorkspaceSettingsDocument({ + workspace, + requestBody: { + document: { webhook: "__SENSITIVE_AND_UNCHANGED__" }, + dry_run: true, + }, + }); + return !result.changes.some((c) => c.key === "webhook"); + } catch (err: any) { + if (err?.status === 404 && !err?.body) { + return true; + } + throw new Error(`Failed to read workspace settings: ${err?.body ?? err}`); + } +} + /** Applies settings.yaml server-side, which owns the rules for what an omitted or null setting * means. Returns false on a server that predates the settings document. */ async function applySettingsDocument( diff --git a/cli/src/guidance/skills.gen.ts b/cli/src/guidance/skills.gen.ts index 8acc13b621..bf2230cfd0 100644 --- a/cli/src/guidance/skills.gen.ts +++ b/cli/src/guidance/skills.gen.ts @@ -8504,7 +8504,8 @@ read or change it. ## Secrets -\`slack_oauth_client_secret\` is the one secret. A read shows \`__SENSITIVE_AND_UNCHANGED__\` in place +\`webhook\` (its URL usually carries the receiver's token) and \`slack_oauth_client_secret\` are the +secrets. A read shows \`__SENSITIVE_AND_UNCHANGED__\` in place of a set secret; sending that placeholder back keeps the stored value. A value starting with \`$encrypted:\` is decrypted with the workspace key, and one encrypted with another workspace's key is left unchanged with a warning; any other value is taken as plaintext. Never diff --git a/system_prompts/auto-generated/prompts.ts b/system_prompts/auto-generated/prompts.ts index 9351bace0e..8421e3c8d0 100644 --- a/system_prompts/auto-generated/prompts.ts +++ b/system_prompts/auto-generated/prompts.ts @@ -1611,7 +1611,8 @@ read or change it. ## Secrets -\`slack_oauth_client_secret\` is the one secret. A read shows \`__SENSITIVE_AND_UNCHANGED__\` in place +\`webhook\` (its URL usually carries the receiver's token) and \`slack_oauth_client_secret\` are the +secrets. A read shows \`__SENSITIVE_AND_UNCHANGED__\` in place of a set secret; sending that placeholder back keeps the stored value. A value starting with \`$encrypted:\` is decrypted with the workspace key, and one encrypted with another workspace's key is left unchanged with a warning; any other value is taken as plaintext. Never diff --git a/system_prompts/auto-generated/skills/workspace-settings/SKILL.md b/system_prompts/auto-generated/skills/workspace-settings/SKILL.md index 336910c27e..6bc8cdccbe 100644 --- a/system_prompts/auto-generated/skills/workspace-settings/SKILL.md +++ b/system_prompts/auto-generated/skills/workspace-settings/SKILL.md @@ -27,7 +27,8 @@ read or change it. ## Secrets -`slack_oauth_client_secret` is the one secret. A read shows `__SENSITIVE_AND_UNCHANGED__` in place +`webhook` (its URL usually carries the receiver's token) and `slack_oauth_client_secret` are the +secrets. A read shows `__SENSITIVE_AND_UNCHANGED__` in place of a set secret; sending that placeholder back keeps the stored value. A value starting with `$encrypted:` is decrypted with the workspace key, and one encrypted with another workspace's key is left unchanged with a warning; any other value is taken as plaintext. Never diff --git a/system_prompts/base/workspace-settings.md b/system_prompts/base/workspace-settings.md index 29b2f20859..ae2277e1b9 100644 --- a/system_prompts/base/workspace-settings.md +++ b/system_prompts/base/workspace-settings.md @@ -22,7 +22,8 @@ read or change it. ## Secrets -`slack_oauth_client_secret` is the one secret. A read shows `__SENSITIVE_AND_UNCHANGED__` in place +`webhook` (its URL usually carries the receiver's token) and `slack_oauth_client_secret` are the +secrets. A read shows `__SENSITIVE_AND_UNCHANGED__` in place of a set secret; sending that placeholder back keeps the stored value. A value starting with `$encrypted:` is decrypted with the workspace key, and one encrypted with another workspace's key is left unchanged with a warning; any other value is taken as plaintext. Never