Files
windmill/system_prompts/generate.py
T
0e807fb1dd feat: put a data table's connection under Postgres roles (#11020)
* feat(datatables): put a data table's connection under Postgres roles

A data table backed by the instance database resolved to exactly one Postgres connection,
`custom_instance_user`, for everyone who could reach it at all. There was no way to say
this job reads, that one writes, this one never sees the salaries table.

A data table role is now a real Postgres login on the cluster, defined once for the
instance by a superadmin and named exactly as they named it. A script that declares
`-- role analytics` connects as `analytics`, and Postgres decides what it may touch —
grants are ordinary SQL. Windmill answers only "may this caller ask for this role", from
the tenant lists on the data table entry: `u/alice`, `g/analysts`, `f/finance` or `*`.
A data table with no `permissions` block behaves exactly as before.

Everything that opens a connection on someone's behalf goes through one chokepoint,
`get_datatable_resource_from_db`, which takes the identity explicitly and fails closed when
there is none. The role logs in as itself — never `SET ROLE`, which a script could
`RESET ROLE` its way out of.

A fork's data table entry becomes a pointer at the workspace that governs it rather than a
copy of it. The settings clone used to hand a fork a byte-identical entry naming the
parent's database, which a fork admin could edit to grant themselves `admin` there; a
pointer has nothing local to edit, and its tenants are evaluated as a member of the
governing workspace, by email. `permissions` is stripped from the workspace export and
ignored on import: tenants name principals of one workspace, and a settings push is not
where an access decision should be made.

Operations that see the whole database whatever the roles grant stay with the governing
workspace's admins: editing the roles, a migration that declares none, and opening a
replication stream for a Postgres trigger or capture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): gate the paths that reach a whole database as admin

Auditing what still resolved through the unchecked resolver turned up three that act for a
caller and hand back the admin connection: `resolve_pg_source_checked` (behind schema
export, the full-schema read, database creation, import and the forked-database drop), the
connection test, and the schema snapshot a fork clone takes of its parent. On a data table
under roles each let any workspace member — or a fork admin who is nobody in the governing
workspace — read or copy the whole database whatever its roles grant.

All three now require admin reach on the governing workspace. A dump taken under a
restricted role would be a silently truncated copy rather than an error, so refusing is the
only right answer for the copy paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): confine roles to the instance database, and stop a fork reaching the parent's bookkeeping

A data table role is a login on Windmill's own Postgres. Nothing stopped a workspace admin
putting a *resource-backed* data table under roles, at which point the executor dialled the
host that resource names — one the admin chose — with the role's real cluster password, and
`CONNECT` is granted to every registered instance database. Both ends now refuse: the
permissions endpoint rejects the save, and the chokepoint refuses to substitute credentials
on a non-instance entry rather than trusting the record it read.

Two more places reached the governing database without answering to it. The initial-migration
generator returned a `pg_dump` of the whole schema to any member. And the migration
rename/delete cascade followed a fork's pointer into the parent, so a fork admin renaming or
removing their own local entry relabelled or wiped the parent's `_wm_migrations` — after
which the parent re-runs every migration from zero. The remote half is now skipped when the
entry resolves into another workspace, which is also just correct: a fork renaming what it
calls a data table changes nothing about the data table.

Also: revoking a tenant now bounces the replication streams of every workspace holding an
entry that resolves here, not only the governing one, so a fork's trigger stops rather than
living on inside its open connection; the instance role catalog and the governing workspace's
tenant lists are no longer returned to someone who cannot edit them; and the tenant rename
dedup collapses non-adjacent duplicates, per role rather than once any role changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): fail loudly where a role or a pointer can be left half-recorded

Three ways the feature could end up in a state nobody could see or undo.

Creating a role writes the cluster first and the catalog second, but the catalog write was an
`UPDATE` that matched nothing when the instance Postgres settings row was absent — leaving a
live login with a password nobody recorded: invisible to the catalog, un-recreatable because
the name is taken, and un-deletable because there is no entry to delete. It now errors, so
the operation is retryable once the row is restored.

Deleting a workspace only nulls the fork lineage; the data table entries pointing at it are
left resolving to nothing. Sweeping them is not an option — turning a pointer back into a copy
would hand each fork the database outright — so the delete now names the data tables it
stranded, and resolving one says which workspace is missing rather than reporting a data table
this workspace never had.

`InstanceDatatableRole` derived `Debug` while holding a Postgres password; it is now
hand-written so `{:?}` on the catalog cannot put a live credential in a log line.

Adds the two branches the reviews found unpinned: a caller who is not a member of the
governing workspace at all, and `NoIdentity` — the compatibility path for an agent worker that
predates this and sends no job id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): unbreak two operator messages and two comments that described other code

The two strings this branch added for states an operator hits once — the catalog write that
matched nothing, and the delete that stranded a pointer — were collapsed from their multi-line
form with the indentation left in, so both rendered with a fourteen-space gap mid-sentence.

`list_datatables` claimed to report a chain it cannot follow and then dropped it; it does drop
it, and the comment now says why that is the right place to stay quiet. The non-superadmin
check in `edit_datatable_config` was introduced as also covering references, which it does not
and need not: `reference` is overwritten from the stored entry for every caller before the
check runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): serialize role catalog mutations, and state each helper's authorization contract

The catalog is one JSON document, so create, rename, enable and delete are all
read-modify-write. Two concurrent creates read the same snapshot, both succeed in the
cluster, and the second write drops the first — leaving a live Postgres login with a password
nobody recorded, which is the exact state the delete path exists to prevent. Every mutation
now runs in one transaction holding an advisory lock across the read, the cluster DDL and the
write, so a lost update cannot happen and a failure rolls the whole thing back. The DDL
helpers take that transaction rather than the pool, which is what makes the lock cover them.

Their statements moved off `sqlx::raw_sql`: the simple protocol is only needed for genuinely
multi-statement SQL, and its future is not `Send`, which an axum handler holding the
transaction requires. Each of these is one statement anyway.

The new cross-crate surface now says what callers must do. `read_role_catalog` returns
plaintext credentials; `create`/`rename`/`set_login`/`drop_instance_role` and
`converge_connect_grants` mutate cluster-wide state; `read_datatable_entry` reads a workspace's
raw config. All of them are superadmin-gated by their current handlers, but nothing said so at
the definition, which is where the next caller looks.

Also: the roles table reloads after a failed login toggle instead of leaving it claiming a flip
that did not land; the rename affordance is the design-system `Button`, not a raw one; and
`resolve_datatable_pg_as_caller` drops a `role` parameter no caller ever filled — browsing
resolves as the data table's default until the database manager grows a picker.

Why role passwords stay a plain `String` while the instance user's password beside them is a
`StringOrSecretRef`, asked three times across reviews: that one is a secret ref because an
operator supplies it and may want it from their own backend, while these are minted here and
never entered by anyone, so there is nothing for a ref to point at. Encrypting generated
secrets at rest is a separate change that would take the replication password with it. Now
said at the field.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): give the role catalog its own row, out of reach of the config machinery

Putting it inside `custom_instance_pg_databases` was the wrong call, and it cost two ways.
The catalog serializes a generated Postgres password per role, and that row is the
operator-facing instance config, so the passwords reached `get_instance_config` and its YAML
editor — a live cluster credential in a response body, a UI field and any log of either.
Worse in the other direction: `to_settings_map` strips the catalog, so a full-row upsert of
that key writes the row back without it and the catalog is gone, while the cluster keeps every
login it described.

`custom_instance_replication_pwd` is the precedent and says exactly why — a generated secret,
written only by the server, never operator-authored, hidden so the config machinery cannot
read, rewrite or drop it. The catalog is the same thing, so it now has the same shape:
`datatable_roles`, in `HIDDEN_SETTINGS`, `PROTECTED_SETTINGS` and the agent-worker denylist.
No redaction to keep in step with three code paths, and no way for a neighbouring write to
take it out.

Two races on the same shared documents. `edit_datatable_config` read the stored data tables
outside its transaction and then wrote the whole `datatable` document, so a permissions save
committing in between was silently rolled back; it now reads under `FOR UPDATE`. And
`set_datatable_permissions` validated role ids against the catalog before opening its
transaction, so a deletion in between let it write a deleted role back — including as the
default, which every later job then fails on; it now holds the catalog lock and the settings
row across validation and write.

Completes the authorization contracts the previous commit claimed but did not finish:
`read_datatable_entry` (which it named and missed), `resolve_governing_datatable`, whose whole
job is to answer for a workspace the caller may not belong to, and
`converge_connect_grants_with`, which had not inherited its wrapper's.

Also the generic Python SDK reference: `_format_py_params` learned the bare `*` last time, but
`extract_py_functions` is a second formatter and still rendered `datatable(name, role)`, so
code written from that page passed a keyword-only argument positionally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): make the concurrency test pin the handlers, and the contracts describe what is enforced

The concurrency test reimplemented the read-modify-write inline, so deleting the lock from all
three handlers left it green — it pinned Postgres, not the code it was written for. It now
drives `create_datatable_role` twice concurrently and asserts the catalog kept both names.
Checked the way the last one should have been: removing the lock from the handler makes it
fail with "wmtest_a_… is a live cluster login the catalog forgot".

The contracts added last commit were stricter than this PR's own callers, which is worse than
none — the next reader sees a rule already broken and learns to ignore it.
`read_role_catalog` said superadmin-only while two of its four callers are open to any
workspace member, and `converge_connect_grants` said superadmin while
`set_datatable_permissions` reaches it as a workspace admin. Both were fine on substance: the
rule that actually holds is about the credential never reaching a response, log, audit record
or export, not about who may call. They now say that. `read_datatable_entry` gets the same
treatment rather than the one the earlier message claimed for it: it is the primitive every
resolution goes through, so it is deliberately open, and what must not escape is `permissions`
— it names the governing workspace's users, groups and folders.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): close the last ways a role or a pointer can be left pointing at nothing

The raw settings readers hand back whatever is in the row, so moving the catalog into its own
`global_settings` key protected the config machinery and left `GET /settings/global/datatable_roles`
and the settings listing returning every live password. Both now filter that one key. The
neighbouring `custom_instance_replication_pwd` has the same shape and is not touched here: it
predates this and widening the fix to it is a decision about an operator workflow, not a
consequence of this change.

Three ways a save could leave something resolving to nothing:

A permissioned data table could be moved to a PostgreSQL resource. The block was carried across
as a server-owned field, the runtime refuses roles on a resource-backed table, so the save
succeeded and every job afterwards failed. Refused instead — turning roles off first is one step,
and it keeps discarding an access decision something somebody chose.

Renaming a governing data table left every fork pointing at the old name: the data table
disappears from their pickers and their jobs stop, with nothing in the renaming workspace to
suggest why. The rename now follows into the pointers in the same transaction.

Deleting one cannot be followed the same way, so it is reported instead — the response names what
it stranded, the way deleting a workspace does, and the fork's own error already says which
workspace is gone.

Also: `ensure_instance_db_grant_options_unchecked` claimed superadmin while the permissions
handler reaches it as a workspace admin (the same class fixed last commit, one instance missed);
the role entry kept an `instance_config_schema` derive it no longer needs; `write_role_catalog`
was the one writer of that table not stamping `updated_at`; and the concurrency test dropped its
roles only on success — a failing run is exactly the one that creates them without recording them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* refactor(datatables): put the role catalog in its own table, not in global_settings

Five findings across three rounds were all the same choice. A set of live Postgres credentials
was living in `global_settings`, which has generic read, list, write, config-export and CLI
round-trip paths that know nothing about what they carry: the passwords reached the instance
config and its YAML editor, a full-row upsert of a neighbouring key erased the catalog,
`GET /settings/global/{key}` and the settings listing returned them raw, and this round the
redaction that fixed the last two turned `wmill instance push` into something that wipes every
password — a fix breaking the assumption the previous fix made. `POST /settings/global/datatable_roles`
could also empty it outside the lock.

The approved plan offered a table or `global_settings`, so this is the other option it already
allowed rather than a new design. `datatable_role` is a table: no generic settings path can read
it, list it, export it, write it or round-trip it, so none of the five needs a guard. The
redaction, the hidden/protected/agent-denylist entries and the JSON document all go with it.

One row per role also removes the read-modify-write the concurrency work was about: two
concurrent creates are two inserts, and the unique index on `name` is what settles a collision.
The advisory lock stays for the one window rows do not cover — `CREATE ROLE` is invisible to
another transaction until commit, so without it both creates pass their `pg_roles` check.

Also from this round: rename mappings are checked against the configuration they claim to
describe, since fork pointers are rewritten from them — a caller could otherwise submit
`main -> missing` against an unchanged config and repoint every fork of `main` at a name nothing
has, and `A -> B` plus `B -> C` moved what pointed at `A` all the way to `C`. And the warning
naming forks a delete stranded reached the response but not the screen: both the data table
settings save and the workspace delete now show it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): validate a rename against the save it describes, and re-check under the locks

Three from the round, all about deciding on state that could already have moved.

A permission save resolved the data table and checked it was instance-backed before taking any
lock, then wrote under one. A config save committing in between could move the table onto a
PostgreSQL resource — recreating exactly what the transition guard refuses — or rename it, in
which case the write targeted a key that no longer existed and reported success having changed
nothing. It now re-resolves and re-checks on the locked state.

Rename validation checked that the source existed before and the target existed after, which
still accepts `main -> decoy` against a save that keeps both: every fork of `main` then follows
onto a different data table, silently, because it keeps resolving. The rule is now the actual
old-to-new key transition — a source may only survive if another rename took its name, and a
target may only pre-exist if another rename freed it. That also stops two sources sharing one
target, and it admits a swap, which the previous guard refused: `datatables` is keyed by name, so
a swap cannot be done one save at a time, and refusing it was a regression against main. The
pointer cascade now runs in two passes through a temporary name, the way the migration cascade
one layer down already handles the same shape, so `A -> B` with `B -> C` moves each pointer once
from what it named before the save.

The tenant mutators say what they are for: they write an access decision for any workspace named,
with an arbitrary mutation, and exist for the transaction that frees or renames a principal.
Editing a decision on purpose belongs in the permissions endpoint.

Carried in the same change: the stranded-fork list is a field rather than a phrase to grep out of
a success string; the pointer cascade matches with `EXISTS` instead of a `LIKE` over the whole
document, so a workspace whose pointers name something else is not rewritten to a byte-identical
value under an exclusive lock; and `InstanceDatatableRole` drops the serde derives left over from
the JSON document, one of which would emit `pwd`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): cascade on the leave route that is used, gate migrations before the admin connection, and drop a role atomically

The tenant cascade on leaving went onto `/users/leave`. The UI and the generated client call
`/workspaces/leave` — a different handler in a different crate with the same name — which
deleted the membership and left `u/<username>` in the tenant lists. Leaving and rejoining
therefore restored the access the leave was supposed to end, and a later account taking the
username would have inherited it. The regression test drives the route the client actually
calls; without the fix it fails with "leaving kept the tenant".

The migration endpoints authorized too late. `run_datatable_migrations` opened the data table's
admin connection, created `_wm_migrations` and read it before reaching the per-migration role
check — so with nothing pending, nothing was checked at all. Rollback returned before its check
when nothing was applied, and the status endpoint had none. All three now ask, before any
connection is opened, whether the caller can reach the data table as any role at all; which role
a given migration runs as is still decided per migration, and by the executor after that.

Deleting a role committed the cluster drop and the catalog row, then swept the tenant lists in
separate transactions. A sweep failing part-way left workspaces naming a role nothing can connect
as, while the retry answered `NotFound` because the catalog entry was already gone. The sweep now
runs in the same transaction, so the drop, the row and every tenant list commit together.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse to copy a data table that is under roles

pg_dump carries no roles and the import runs with --no-privileges, so a copied
data table arrives owned by the admin connection with no GRANT for any role.
The settings clone brings `permissions` across, so the fork's tenants pass
Windmill's check, connect as the role they were given, and are denied by
Postgres on everything: an entry that reads as configured and answers nothing.

Refuse the copy — in the import endpoint before any data moves, and in the fork
path the CLI takes. Replaying the source's owners and ACLs into the clone is
what lifts this, and is a change of its own. Dropping `permissions` from the
copy instead would be the unsafe half, since the copy holds the parent's rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse the clone's database too, not only its data

A clone is two endpoints: `create_pg_database` then `import_pg_database`. Only
the second refused a data table under roles, so a fork asking to clone one
created and registered an empty `wm_fork_…` instance database and then failed —
and nothing collects it, since `drop_forked_datatable_databases` only drops
entries carrying `forked_from` and no entry names this one.

Refuse in both, so the clone stops before a database exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* nit worker error msg

* fix pg_dump stuck on version 17 on nix

* fix(datatables): refuse a malformed role annotation instead of ignoring it

`-- Role operator`, `-- role operator;` and `-- role operator -- why` all failed
the annotation parser's exact-match rule, so the query fell through to the data
table's default role and ran, silently, under a login the author did not choose.
Naming a role exists precisely to not do that.

A leading comment whose first word is `role` is now an annotation attempt: the
keyword matches case-insensitively, one trailing `;` is tolerated, and anything
else is an error naming the line. Only callers that already know the target is a
`datatable://` reference ever run this, so ordinary SQL keeps its comments.

Also bumps the dev shell's postgres client to 18 — it trailed the server the dev
database runs, which takes out every data table export, clone and fork-with-data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse a malformed role query string instead of ignoring it

`?Role=analytics`, `?role=` and `?x=1&role=…` all fell through the reference
parser's exact-match rule, so the connection resolved to the data table's default
role and ran under a login the caller never asked for — the URI half of the same
trap as a malformed `-- role` annotation.

The key now matches case-insensitively, and anything else in the query string is
an error naming it; `role` is the only parameter a reference takes. Callers that
only need the entry keep a lenient `datatable_ref_name`, since they never act on
the role. The DuckDB `ATTACH` parser propagates it rather than attaching under
the default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): carry the role annotation into the row_to_json retry

The retry rebuilds its SQL from `pruneComments(code)`, so the leading comment
block never reached the second attempt — and with it the `-- role <name>` line
that decides which login the query runs as. The retry connected as the data
table's default role instead, so a query the first attempt was denied could
succeed on the second, reported as "recovered with the row_to_json fix".

Carry the leading comment block over. The retry itself is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* chore(datatables): don't mount the roles UI until the ACL editor lands

Enforcement ships first. The permissions drawer is what turns roles on, and the
catalog section is what creates them — both are only useful once there is a way
to grant a role the privileges it needs, which arrives with the ACL editor. Left
mounted they would offer a feature whose other half does not exist.

The two components are complete and reviewed; only their call sites here are
commented out, with a note pointing the follow-up PRs at them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): honour `-- role: x`, and fix the DuckDB attach test

Two review findings, both real.

`attach_datatable_parses_name_and_role` never compiled: `parse_attach_datatable`
returns `Result<Option<_>>` now and one call site kept a single `unwrap`. Its
`?Role=analytics` case also asserted a refusal, contradicting the parser in the
same commit, which matches the key case-insensitively. Replaced with the cases
that are genuinely malformed, and a positive one for the cased key.

`-- role: analytics` fell through to the default role — the silent fallback the
strict parser exists to remove, for the spelling most likely to be typed. The
keyword now accepts an optional colon, attached or spaced, while a word that
merely starts with it (`rolebased`) is still not an attempt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): clone a fork's pointer instead of failing after the copy

Forking a fork with cloning left an orphan database. The preflight resolves the
pointer and sees the governing entry, so both endpoints ran and filled the new
database; `apply_forked_datatable` then refused the inherited pointer and rolled
the fork back, stranding a registered `wm_fork_*` that no entry names and whose
name blocks the retry.

Refusing earlier would have been the smaller change, but forking a fork and
cloning worked before pointers existed, so it would trade an orphan for a
regression. Resolve what the pointer names and write the terminal entry the
clone needs: the whole `database` object rather than a patch of its
`resource_path`, since a pointer has none, and `reference` removed with it.

Also accepts `-- role=x` and `-- Role = x`, two more spellings that fell through
to the default role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): refuse to roll back the catalog while roles exist

The down migration dropped the table and left every role behind: live Postgres
logins whose passwords only that table carried, so after a revert Windmill could
neither use, disable nor delete them, and re-applying could not recreate them
because the names were taken. Cleaning up here is not possible either — dropping
a role means reassigning what it owns in every instance database, and a
migration runs in one — so it now refuses while the catalog is non-empty and
says to delete the roles through instance settings, which does the cluster work.

Also enforces the instance-only invariant the resolved-pointer clone relies on
rather than only asserting it in a comment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* refactor(datatables): settle clonability in one place, before anything is created

A clone is three stages a workspace apart — `create_pg_database`, then
`import_pg_database`, then `apply_forked_datatable` inside the fork transaction.
Only the third can roll back, and `CREATE DATABASE` is not transactional, so any
refusal that lives there strands a registered `wm_fork_*` that no entry names
and whose name blocks the retry.

That orphan has now been fixed three times, most recently reintroduced by a
guard added one commit ago. Patching each new refusal into the first endpoint is
not the fix; having two places that can refuse is. `ensure_datatable_is_clonable`
now answers every reason a copy can be refused and returns what it resolved, and
the stage that writes the entry only does the work.

Also takes an ACCESS EXCLUSIVE lock before the rollback guard counts, so a role
created concurrently cannot slip between the check and the drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): let a retried clone reclaim its own leftover database

A clone creates its target database one request before it copies into it, and
the fork that would name it is written a request after that. Any failure in
between — a pg_dump error, a bad restore, a dropped connection, the source's
roles changing mid-flow — left a registered `wm_fork_*` that no entry names,
and every retry then failed on its name. This predates data table roles.

`create_pg_database` now reclaims such a leftover before creating: only a
`wm_fork_*` database Windmill registered as a data table database and that no
data table or ducklake entry names, in any workspace, archived ones included.
The drop never terminates connections, so a clone still copying into it makes
the reclaim fail instead of being cut off. It is limited to callers who
administer the source — reaching it is not enough, since on a data table
without roles every member reaches it — and anyone else gets the refusal an
existing database always got.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Revert "fix(datatables): let a retried clone reclaim its own leftover database"

This reverts commit 7dd3275a10.

The reclaim tied the caller to the source they administer, but not to the
database it dropped. Between another workspace's import and its final fork
request, that workspace's target is full, registered, unnamed and has no open
connection, so an admin of any instance data table could name it and have it
dropped and recreated empty. The victim's fork would then commit pointing at
the empty copy. Safe reclaim needs durable clone ownership and serialization
with the request that names the database; until then the leftover stays, as it
did before this PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(datatables): record the stale clone database as a known limitation

A clone is three requests and `CREATE DATABASE` is not transactional, so a
failure after the first leaves a registered `wm_fork_*` behind, as it did
before data table roles. Accepted for this PR: it is harmless to data and goes
away once the clone is a single server-side operation.

The comment also records why the obvious fix is wrong: reclaiming the leftover
on retry, without durable clone ownership, can drop another workspace's fully
copied database between its import and its final fork request.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): bounce the streams reading a data table when it is deleted

Deleting a governing data table, or the workspace that holds it, only collected
the fork pointers it stranded, for the warning. A Postgres trigger or capture
already streaming through one of those pointers kept the replication connection
it opened while the pointer still resolved, so it went on dispatching the
governing database's rows after the fork lost access — until its connection
happened to restart. The governing workspace's own streams on a deleted entry
did the same.

Both deletion paths now bounce the affected listeners inside their own
transaction, through the helper a permission change already uses, so a
listener that reconnects re-resolves the entry and finds it gone. The helper is
split so a caller can pass the (workspace, local name) pairs it already holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): keep the fork schema baseline, and bounce streams on every removal

Three fixes from review.

`edit_datatable_config` took `forked_from` wholesale from the stored entry, so
the fork schema diff's save of an advanced baseline was silently discarded and
an applied change was offered again. Whether an entry carries a clone stamp is
still carried from the store, since that is what marks its database droppable,
but the baseline inside it is now taken from the request.

The stranded-pointer warning and the stream bounce ran over the optional
`deleted_datatables` hint, which the settings-sync CLI never sends, so removing
a governing data table through `wmill` bounced nothing. Removals are now derived
from the stored configuration against the saved one.

`delete_workspace` read the pointers to bounce before its transaction, so a fork
committing a pointer during the deletion was missed. The read now happens inside
the transaction, after the workspace row is deleted: a fork's insert key-share
locks that row through its parent foreign key, so it is either seen or fails on
the missing parent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(datatables): keep Postgres triggers and data table roles apart

A replication stream reads every row of every table whatever the data table's
roles grant, and its listener checks access only when it connects. Rather than
chase every way access can change and bounce the streams each one affects, a
data table now carries one or the other:

- a Postgres trigger or capture cannot be created on, or connect to, a data
  table under roles;
- roles cannot be turned on while an enabled trigger or a live capture reads
  the data table, its own or a fork's through its pointer. The refusal names
  each one to disable.

This removes the stream bounces on roles edits and on data table and workspace
deletion, and the trigger gate that admitted admins. The fork schema baseline
fix from the same review round is kept.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): refuse a Postgres trigger on a data table under roles when it is saved

Creating or editing a trigger that points at a data table under roles was
accepted, and its listener then retried the refused connection every 30
seconds forever. The save is now refused, and a trigger that reaches such a
data table anyway (re-enabled, or cloned into a fork) is disabled by its
listener with the reason, as a missing replication slot is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): disable a data table role before deleting it

Deleting a role reassigns and drops what it owns in each registered
database on its own connection, and each of those passes commits as it
goes. A database failing part-way left the role enabled in the catalog and
able to log in, but already stripped in the databases reached before it.
The role is now disabled in its own commit first, so a failed delete
leaves a disabled role to retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): serialize roles going on with a stream starting

Turning roles on looked for enabled triggers and live captures once,
without a lock anything starting a stream also took. A trigger enabled in
that window could have its listener connect before roles committed, and a
healthy listener never checks again. Both transitions now serialize on one
advisory lock: roles going on hold it exclusive while they look, and
trigger create, edit and enable, and capture setup and ping hold it shared
while they commit. Either the look sees the stream, or the listener
connects after roles are committed and refuses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): wait out live listeners, and resolve stored names containing `?`

Turning roles on counted a trigger as gone once disabled, and a capture
once its client stopped pinging, but the listener keeps its replication
connection until its next heartbeat notices. A trigger or capture whose
listener pinged in the last 15 seconds, the window a server holds a
listener for, now still counts as streaming.

Data table names could contain `?` before they were restricted, and such
entries are still stored. Splitting `?role=` off a reference misread them:
`a?b` became `a` with an unknown parameter, and the clone checks looked at
a different entry than the one copied. An entry stored under the whole
reference is now looked up first, in the Postgres executor, DuckDB ATTACH
and the clone checks. Agent workers cannot read the workspace and keep
the strict parse, which refuses such a name rather than misreading it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): warn when a settings sync strands fork pointers

A settings save reported the fork pointers left resolving to nothing only
for the names in `deleted_datatables`, which `wmill sync push` never sends.
The save now works out what it removed from the locked entries, and the
CLI prints the stranded pointers it returns.

Also correct the replication helper's contract: no role or admin check
makes a replication connection safe, so a data table under roles is
refused outright rather than gated as an admin operation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): refuse a save that drops a data table's roles through an undeclared rename

A data table's roles follow its entry only through a declared rename. A
settings sync sends the whole map and never declares one, so renaming a
data table under roles there read as a delete and a new entry on the same
database: the new entry carried no roles, and every caller connected as
admin. Such a save is now refused, naming both entries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): no entry without roles may newly reach a database under roles

The previous guard only caught a new name replacing an entry under roles.
A whole-map save could also repoint an existing entry without roles at
that database, or another workspace could point one there, and every
caller of that entry would connect as admin. The rule is now stated on
the saved entries: one that carries no roles and newly points at an
instance database any entry under roles uses, in this workspace or
another, is refused. A declared rename carries its roles and passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move data table role catalog and resolution to the enterprise edition

Roles are an Enterprise Edition feature. The catalog, the Postgres logins,
CONNECT convergence, tenant evaluation and the role half of connection
resolution move to windmill-ee-private. Every public function keeps its path
and signature and forwards through datatable_roles_oss, which re-exports the
enterprise implementation or, without it, refuses.

Without the enterprise edition a data table under roles, or a caller naming a
role, is refused a connection rather than resolved as admin, and the reach and
admin-access checks refuse one under roles. A data table not under roles
resolves as before in every edition, and an instance database keeps the
CONNECT grants it was created with. The catalog lock, the stream lock, the
tenant cascades and the permissions stripping stay in OSS: they only restrict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move the data table permissions endpoints to the enterprise edition

The permissions read, save and usable-roles handlers move to
windmill-ee-private; the routes stay registered and, without the enterprise
edition, answer that data table roles are an Enterprise Edition feature.
ensure_governs_datatable and ensure_reaches_datatable keep their paths: the
first refuses, the second passes a data table not under roles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move the data table role catalog endpoints to the enterprise edition

The superadmin list, create, update and delete handlers move to
windmill-ee-private. The routes stay registered and, without the enterprise
edition, refuse after authentication.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* test(datatables): run the roles tests on the enterprise edition, refusals without it

Each test that exercises roles runs with private and enterprise. Two tests run
without them: every roles route answers the Enterprise refusal, and a data
table saved under roles, or a named role, is refused a connection while one
not under roles resolves as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): gate the roles UI mount sites on an enterprise license

Both mount sites are still commented out; the gate travels with them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* test(datatables): run the tenant matcher test on the enterprise edition

The matcher it covers is enterprise code now, so without the enterprise
edition the test hit the stub and failed the default windmill-common run. It
runs with private and enterprise, and a counterpart without them asserts that
no tenant list covers anyone, the wildcard and a workspace admin included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* chore: update ee-repo-ref to a1873dbb67f2302b85ff5362f8387b48eccdb607

This commit updates the EE repository reference after PR #783 was merged in windmill-ee-private.

Previous ee-repo-ref: 5c853e2c20eca6b748415fc0d6862a6ebfb5fec4

New ee-repo-ref: a1873dbb67f2302b85ff5362f8387b48eccdb607

Automated by sync-ee-ref workflow.

* fix(datatables): refuse roles while a same-workspace alias reaches the database

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): let CE migrations connect as an explicitly named admin

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): serialize roles going on with aliases saved from other workspaces

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(datatables): note that legacy names with ? cannot be migrated

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): drop a DuckDB data table secret once its ATTACH has used it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* perf(datatables): resolve a workspace's data tables per pointer hop, not per entry

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): hold the parent's settings while a fork points at its data tables

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to 7e338e4dabf91689bfd7fb0333c6534040b17b59

This commit updates the EE repository reference after PR #787 was merged in windmill-ee-private.

Previous ee-repo-ref: 38d6fcf2aeb39cfdac21814bbdbbcc02911e566a

New ee-repo-ref: 7e338e4dabf91689bfd7fb0333c6534040b17b59

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-09-18 13:56:41 +02:00

2825 lines
104 KiB
Python

#!/usr/bin/env python3
"""
Generate system prompts documentation from SDKs and OpenFlow schema.
This script:
1. Parses TypeScript SDK to extract function signatures
2. Parses Python SDK using Python's ast module
3. Parses OpenFlow YAML schema
4. Generates markdown files in sdks/ and schemas/
5. Assembles complete prompts and generates TypeScript exports in generated/
Usage:
python generate.py
python generate.py --plugin-dir /path/to/windmill-claude-plugin
python generate.py --context7-dir /path/to/windmill-cli-docs
"""
import argparse
import ast
import copy
import json
import re
import shutil
from pathlib import Path
import yaml
from utils import (
# Path constants
SCRIPT_DIR,
TS_SDK_DIR,
PY_SDK_PATH,
OPENFLOW_SCHEMA_PATH,
BACKEND_OPENAPI_PATH,
OUTPUT_SDKS_DIR,
OUTPUT_GENERATED_DIR,
OUTPUT_CLI_DIR,
OUTPUT_SKILLS_DIR,
OUTPUT_SCHEMAS_DIR,
CLI_GUIDANCE_DIR,
CLI_MAIN,
CLI_COMMANDS_DIR,
# Language metadata
LANGUAGE_METADATA,
TS_SDK_LANGUAGES,
PY_SDK_LANGUAGES,
# Schema mappings
SCHEMA_MAPPINGS,
# String/file utilities
clean_jsdoc,
clean_params,
escape_for_ts,
read_markdown_file,
# Parsing utilities
extract_balanced,
extract_return_type,
parse_default_imports,
extract_options,
# Schema utilities
extract_cli_schema,
format_schema_for_markdown,
format_schema_as_json,
)
# =============================================================================
# TypeScript SDK Parsing
# =============================================================================
def extract_ts_functions(content: str) -> list[dict]:
"""Extract exported function signatures from TypeScript SDK."""
functions = []
seen_names = set()
# Pattern to find export function declarations (with or without JSDoc)
# Captures JSDoc if present, then the function declaration
pattern = re.compile(
r'(?:(/\*\*(?:[^*]|\*(?!/))*\*/)\s*)?' # Optional JSDoc comment
r'export\s+(async\s+)?function\s+(\w+)\s*' # export [async] function name
r'(<[^>]+>)?\s*', # optional generic
re.MULTILINE
)
for match in pattern.finditer(content):
jsdoc_raw, is_async, name, generic = match.groups()
if name in seen_names:
continue
# Find the opening parenthesis for parameters
pos = match.end()
while pos < len(content) and content[pos] in ' \t\n':
pos += 1
if pos >= len(content) or content[pos] != '(':
continue
# Extract balanced parameters
params, paren_end = extract_balanced(content, pos, '(', ')')
if paren_end == -1:
continue
# Extract return type (handles multi-line types like Promise<{...}>)
return_type, _ = extract_return_type(content, paren_end + 1)
if not return_type:
return_type = 'Promise<void>' if is_async else 'void'
# `@internal` marks an export that exists for another module or for a
# test to reach, not for a user to call. The SDK reference these prompts
# become is a user-facing API list, so it must not advertise them.
# `@deprecated` exports stay callable for existing scripts but must not be
# suggested for new ones.
if jsdoc_raw and ('@internal' in jsdoc_raw or '@deprecated' in jsdoc_raw):
continue
docstring = clean_jsdoc(jsdoc_raw) if jsdoc_raw else ''
seen_names.add(name)
functions.append({
'name': name,
'generic': generic or '',
'params': clean_params(params),
'return_type': return_type,
'async': bool(is_async),
'docstring': docstring
})
return functions
def extract_ts_types(content: str) -> list[dict]:
"""Extract exported type definitions from TypeScript SDK."""
types = []
# Pattern for exported type aliases
type_pattern = re.compile(
r'export\s+type\s+(\w+)\s*=\s*([^;]+);',
re.MULTILINE
)
# Pattern for exported interfaces
interface_pattern = re.compile(
r'export\s+interface\s+(\w+)\s*\{([^}]+)\}',
re.MULTILINE | re.DOTALL
)
for match in type_pattern.finditer(content):
name, definition = match.groups()
types.append({
'name': name,
'kind': 'type',
'definition': definition.strip()
})
for match in interface_pattern.finditer(content):
name, body = match.groups()
types.append({
'name': name,
'kind': 'interface',
'definition': body.strip()
})
return types
# =============================================================================
# Python SDK Parsing
# =============================================================================
def extract_py_functions(content: str) -> list[dict]:
"""Extract function signatures from Python SDK using AST."""
functions = []
seen_names = set()
try:
tree = ast.parse(content)
except SyntaxError as e:
print(f"Warning: Could not parse Python SDK: {e}")
return functions
def process_function(node):
"""Process a function node and add to functions list if not duplicate."""
# Skip private functions
if node.name.startswith('_') and not node.name.startswith('__'):
return
# Skip duplicates
if node.name in seen_names:
return
# Get docstring
docstring = ast.get_docstring(node) or ''
# Same rule as the TypeScript SDK: a deprecated member stays callable for existing
# scripts but must not be suggested for new ones. The Python SDK marks them with the
# Sphinx `.. deprecated::` directive.
if '.. deprecated::' in docstring:
return
# Build parameter list
params = []
args = node.args
# Handle regular args
num_defaults = len(args.defaults)
num_args = len(args.args)
for i, arg in enumerate(args.args):
if arg.arg == 'self':
continue
param_str = arg.arg
if arg.annotation:
param_str += f": {ast.unparse(arg.annotation)}"
# Check if has default
default_idx = i - (num_args - num_defaults)
if default_idx >= 0:
default = args.defaults[default_idx]
param_str += f" = {ast.unparse(default)}"
params.append(param_str)
# Handle *args
if args.vararg:
params.append(f"*{args.vararg.arg}")
# Handle keyword-only args. Same bare `*` as `_format_py_params`: without it the rendered
# signature reads as all-positional, and code written against this reference passes a
# keyword-only argument positionally and gets a TypeError.
if args.kwonlyargs and not args.vararg:
params.append('*')
for i, arg in enumerate(args.kwonlyargs):
param_str = arg.arg
if arg.annotation:
param_str += f": {ast.unparse(arg.annotation)}"
if args.kw_defaults[i]:
param_str += f" = {ast.unparse(args.kw_defaults[i])}"
params.append(param_str)
# Handle **kwargs
if args.kwarg:
params.append(f"**{args.kwarg.arg}")
# Get return type
return_type = ''
if node.returns:
return_type = ast.unparse(node.returns)
seen_names.add(node.name)
functions.append({
'name': node.name,
'params': ', '.join(params),
'return_type': return_type,
'docstring': docstring,
'async': isinstance(node, ast.AsyncFunctionDef)
})
# Process top-level functions and class methods (but not nested functions)
for node in tree.body:
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
process_function(node)
elif isinstance(node, ast.ClassDef):
for item in node.body:
if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)):
process_function(item)
return functions
def extract_py_classes(content: str) -> list[dict]:
"""Extract class definitions from Python SDK."""
classes = []
try:
tree = ast.parse(content)
except SyntaxError:
return classes
for node in ast.walk(tree):
if isinstance(node, ast.ClassDef):
methods = []
for item in node.body:
if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)):
if not item.name.startswith('_') or item.name == '__init__':
docstring = ast.get_docstring(item) or ''
methods.append({
'name': item.name,
'docstring': docstring
})
classes.append({
'name': node.name,
'docstring': ast.get_docstring(node) or '',
'methods': methods
})
return classes
# =============================================================================
# CLI Command Parsing
# =============================================================================
# Reusable option pattern for CLI parsing. Matches both `.option(...)` and
# `.globalOption(...)` so subcommand-level global options surface in the docs.
OPTION_PATTERN = re.compile(
r'\.(?:option|globalOption)\(\s*"([^"]+)"\s*,\s*"([^"]+)"' # double-quoted
r'|'
r"\.(?:option|globalOption)\(\s*'([^']+)'\s*,\s*'([^']+)'", # single-quoted
re.MULTILINE | re.DOTALL
)
# A single JS string literal: double/single quoted or backtick. The other
# quote chars may appear inside (apostrophes inside a "..." string, etc.) and
# backslash escapes are consumed so a `\"` doesn't end the match early.
_STRING_LITERAL = (
r'"(?:[^"\\]|\\.)*"'
r"|'(?:[^'\\]|\\.)*'"
r'|`(?:[^`\\]|\\.)*`'
)
def _unquote_js_string(literal: str) -> str:
"""Drop the surrounding quotes of a JS string literal and unescape the
escapes that show up in command descriptions."""
body = literal[1:-1]
return (
body.replace('\\\\', '\x00')
.replace('\\n', '\n')
.replace('\\t', '\t')
.replace('\\"', '"')
.replace("\\'", "'")
.replace('\\`', '`')
.replace('\x00', '\\')
)
def extract_description(section: str) -> str | None:
"""Extract the text of the first chained `.description(...)` call.
Handles double/single-quoted and backtick strings (a quote of one kind may
appear inside a string delimited by another — e.g. an apostrophe inside a
"..." description), backslash escapes, and `"a" + "b"` concatenation across
lines. Returns None when `.description(` is absent or its argument is not a
string literal (e.g. a variable), matching the previous empty-description
behavior. Using `[^"\\']+` here instead would silently drop any description
containing an apostrophe.
"""
m = re.search(
r'\.description\(\s*((?:' + _STRING_LITERAL + r')(?:\s*\+\s*(?:' + _STRING_LITERAL + r'))*)',
section,
re.DOTALL,
)
if not m:
return None
parts = re.findall(_STRING_LITERAL, m.group(1), re.DOTALL)
return ''.join(_unquote_js_string(p) for p in parts).strip() or None
def extract_named_command_block(content: str, var_name: str) -> str | None:
"""Return the chained-call body of `const <var_name> = new Command() ...`,
from just after `new Command()` up to the next top-level statement.
Returns None when the var isn't a *direct* `new Command()` (e.g. it's wrapped
in a helper call like `auditListOptions(new Command()...)`), so callers can
fall back to a looser match.
"""
m = re.search(
r'const\s+' + re.escape(var_name) + r'\s*=\s*new\s+Command\(\)'
r'([\s\S]*?)(?=\n(?:const|let|var|async|function|export)\b)',
content,
)
return m.group(1) if m else None
def extract_exported_command_block(content: str) -> str | None:
"""Return the chained-call body of the command that is `export default`ed.
A command file may define helper `new Command()` groups (assigned to local
consts and mounted as nested subcommands via `.command("x", localCmd)`)
*before* the exported command. Anchoring on the first `new Command()` in the
file would merge those helpers into the top-level command, so resolve the
exported variable first and only then fall back to the first `new Command()`
(which covers inline/wrapped exports).
"""
export_match = re.search(r'export\s+default\s+(\w+)\s*;', content)
if export_match:
block = extract_named_command_block(content, export_match.group(1))
if block is not None:
return block
command_match = re.search(
r'(?:const\s+command\s*=\s*)?new\s+Command\(\)([\s\S]*?)(?=export\s+default)',
content,
)
return command_match.group(1) if command_match else None
def parse_command_block(
content: str, file_path: Path | None = None, block: str | None = None
) -> dict:
"""
Parse a Cliffy Command() definition block and extract metadata.
Returns a dict with: description, options, subcommands, arguments, alias
If file_path is provided, imported subcommands will be resolved by parsing
the imported files. `block` may be passed to parse a specific pre-extracted
command body (used to recurse into locally-defined nested command groups).
"""
result = {
'description': '',
'options': [],
'subcommands': [],
'arguments': '',
'alias': ''
}
# Find the command block
if block is None:
block = extract_exported_command_block(content)
if block is None:
return result
# Find where subcommands start
first_subcommand_pos = block.find('.command(')
if first_subcommand_pos == -1:
first_subcommand_pos = len(block)
top_section = block[:first_subcommand_pos]
# Extract main description
main_desc = extract_description(top_section)
if main_desc:
result['description'] = main_desc
# Extract alias
alias_match = re.search(r'\.alias\(\s*["\']([^"\']+)["\']\s*\)', top_section)
if alias_match:
result['alias'] = alias_match.group(1)
# Extract top-level options (before any .command() or .action())
top_section_until_action = re.split(r'\.action\(', top_section)[0]
result['options'] = extract_options(top_section_until_action, OPTION_PATTERN)
# Extract top-level arguments
args_match = re.search(r'\.arguments\(\s*["\']([^"\']+)["\']\s*\)', top_section)
if args_match:
result['arguments'] = args_match.group(1)
# Parse imports if we have a file path (for resolving imported subcommands)
imports = parse_default_imports(content) if file_path else {}
# Extract subcommands
subcommand_sections = re.split(r'(?=\.command\()', block)
for section in subcommand_sections:
# Second arg is either a quoted description, a bare identifier (imported
# command, e.g. `.command("app", app)`), or a more complex expression
# like `someWrapper(new Command()...)` — the `[^)]+` fallback covers
# the last case by matching up to the next `)`.
#
# Two subtleties:
# - The quoted-string alts are tried first so a description containing
# `(` like "(psql, DBeaver)" isn't truncated by the `[^)]+` fallback.
# - The trailing `,?` accommodates the prettier-style `\n )` close
# paren that follows a comma. Without it, the quoted alt would
# succeed but the outer `\s*\)` would fail, forcing a backtrack to
# `[^)]+` and producing a truncated description with a trailing `",`.
cmd_match = re.match(
r'\.command\(\s*["\']([^"\']+)["\']\s*'
r'(?:,\s*("(?:[^"\\]|\\.)*"|\'(?:[^\'\\]|\\.)*\'|[^)]+))?'
r'\s*,?\s*\)',
section,
)
if not cmd_match:
continue
# Explicit source marker for backwards-compatible CLI commands that
# should not be suggested in generated system prompts.
if '@deprecated' in section:
continue
# Hidden commands (Cliffy .hidden()) are internal — invoked by other
# Windmill components, not users — and must not surface in the
# generated agent system prompts or help.
if '.hidden()' in section:
continue
cmd_name = cmd_match.group(1)
second_arg = cmd_match.group(2).strip() if cmd_match.group(2) else ''
# Check if second arg is a string (description) or a variable (imported command)
is_string_desc = second_arg.startswith('"') or second_arg.startswith("'")
if is_string_desc:
cmd_desc = second_arg.strip('"\'')
elif second_arg and second_arg in imports and file_path:
# Imported command - resolve and parse the imported file
import_path = imports[second_arg]
if import_path.startswith('./') or import_path.startswith('../'):
imported_file = (file_path.parent / import_path).resolve()
if imported_file.exists():
try:
imported_content = imported_file.read_text()
imported_cmd = parse_command_block(imported_content, imported_file)
result['subcommands'].append({
'name': cmd_name,
'description': imported_cmd.get('description', ''),
'arguments': imported_cmd.get('arguments', ''),
'options': imported_cmd.get('options', []),
'subcommands': imported_cmd.get('subcommands', []),
})
continue
except Exception as e:
print(f" Warning: Could not parse imported command {second_arg}: {e}")
cmd_desc = ''
elif second_arg and re.search(
r'const\s+' + re.escape(second_arg) + r'\s*=\s*new\s+Command\(\)', content
):
# Locally-defined command group mounted as a subcommand
# (e.g. `.command("migrate", migrateCommand)`): recurse into its
# definition so its own subcommands/options are captured.
nested_block = extract_named_command_block(content, second_arg)
if nested_block is not None:
nested = parse_command_block(content, file_path, block=nested_block)
result['subcommands'].append({
'name': cmd_name,
'description': nested.get('description', ''),
'arguments': nested.get('arguments', ''),
'options': nested.get('options', []),
'subcommands': nested.get('subcommands', []),
})
continue
cmd_desc = ''
else:
cmd_desc = ''
# Check for description in chained .description() call
chained_desc = extract_description(section)
if chained_desc:
cmd_desc = chained_desc
# Check for arguments
args_match = re.search(r'\.arguments\(\s*["\']([^"\']+)["\']\s*\)', section)
cmd_args = args_match.group(1) if args_match else ''
# Extract options specific to this subcommand (before .action())
section_until_action = re.split(r'\.action\(', section)[0]
cmd_options = extract_options(section_until_action, OPTION_PATTERN)
result['subcommands'].append({
'name': cmd_name,
'description': cmd_desc,
'arguments': cmd_args,
'options': cmd_options
})
return result
def find_command_file(cmd_name: str) -> Path | None:
"""Find the command file for a given command name."""
standard_path = CLI_COMMANDS_DIR / cmd_name / f"{cmd_name}.ts"
if standard_path.exists():
return standard_path
return None
def extract_cli_commands() -> dict:
"""
Extract CLI command metadata from the CLI source files.
Returns a dict with global_options and commands.
"""
result = {
'version': '',
'global_options': [],
'commands': []
}
if not CLI_MAIN.exists():
print(f"Warning: CLI main file not found at {CLI_MAIN}")
return result
main_content = CLI_MAIN.read_text()
# Extract version
version_match = re.search(r'export\s+const\s+VERSION\s*=\s*["\']([^"\']+)["\']', main_content)
if version_match:
result['version'] = version_match.group(1)
# Extract global options from main.ts
global_opt_pattern = re.compile(
r'\.globalOption\(\s*["\']([^"\']+)["\']\s*,\s*["\']([^"\']+)["\']\s*\)',
re.MULTILINE
)
for match in global_opt_pattern.finditer(main_content):
flag, desc = match.groups()
result['global_options'].append({'flag': flag, 'description': desc})
# Extract command registrations from main.ts
cmd_reg_pattern = re.compile(
r'\.command\(\s*["\']([^"\']+)["\']\s*,\s*(\w+)\s*\)',
re.MULTILINE
)
inline_cmd_pattern = re.compile(
r'\.command\(\s*["\']([^"\']+)["\']\s*,\s*["\']([^"\']+)["\']\s*\)',
re.MULTILINE
)
registered_commands = []
for match in cmd_reg_pattern.finditer(main_content):
cmd_name = match.group(1).split()[0]
registered_commands.append(cmd_name)
# Process each registered command
for cmd_name in registered_commands:
cmd_file = find_command_file(cmd_name)
if cmd_file:
try:
cmd_content = cmd_file.read_text()
cmd_data = parse_command_block(cmd_content, cmd_file)
cmd_data['name'] = cmd_name
result['commands'].append(cmd_data)
except Exception as e:
print(f"Warning: Could not parse command file for {cmd_name}: {e}")
# Handle special inline commands from main.ts
for match in inline_cmd_pattern.finditer(main_content):
cmd_name = match.group(1).split()[0]
cmd_desc = match.group(2)
if cmd_name not in [c['name'] for c in result['commands']]:
result['commands'].append({
'name': cmd_name,
'description': cmd_desc,
'options': [],
'subcommands': [],
'arguments': '',
'alias': ''
})
return result
# =============================================================================
# Markdown Generation
# =============================================================================
def generate_cli_commands_markdown(cli_data: dict) -> str:
"""Generate markdown documentation from extracted CLI command data."""
md = "# Windmill CLI Commands\n\n"
md += "The Windmill CLI (`wmill`) provides commands for managing scripts, flows, apps, and other resources.\n\n"
# Global options
if cli_data.get('global_options'):
md += "## Global Options\n\n"
for opt in cli_data['global_options']:
flag = opt['flag']
desc = opt['description']
md += f"- `{flag}` - {desc}\n"
md += "\n"
# Commands
if cli_data.get('commands'):
md += "## Commands\n\n"
for cmd in sorted(cli_data['commands'], key=lambda x: x['name']):
md += f"### {cmd['name']}\n\n"
if cmd.get('description'):
md += f"{cmd['description']}\n\n"
if cmd.get('alias'):
md += f"**Alias:** `{cmd['alias']}`\n\n"
if cmd.get('arguments'):
md += f"**Arguments:** `{cmd['arguments']}`\n\n"
# Top-level options for this command
if cmd.get('options'):
md += "**Options:**\n"
for opt in cmd['options']:
md += f"- `{opt['flag']}` - {opt['description']}\n"
md += "\n"
# Subcommands
if cmd.get('subcommands'):
md += "**Subcommands:**\n\n"
for sub in cmd['subcommands']:
sub_name = sub['name']
sub_args = f" {sub['arguments']}" if sub.get('arguments') else ""
sub_desc = sub.get('description', '')
md += f"- `{cmd['name']} {sub_name}{sub_args}`"
if sub_desc:
md += f" - {sub_desc}"
md += "\n"
# Subcommand options
if sub.get('options'):
for opt in sub['options']:
md += f" - `{opt['flag']}` - {opt['description']}\n"
# Nested sub-subcommands (e.g. `datatable migrate new`)
for subsub in sub.get('subcommands', []):
ss_args = f" {subsub['arguments']}" if subsub.get('arguments') else ""
md += f" - `{cmd['name']} {sub_name} {subsub['name']}{ss_args}`"
if subsub.get('description'):
md += f" - {subsub['description']}"
md += "\n"
for opt in subsub.get('options', []):
md += f" - `{opt['flag']}` - {opt['description']}\n"
md += "\n"
return md
# Who is running the script is answered by contextual variables, not by an SDK call, so the
# SDK reference has to say so: it is where an agent looks for a `usernameToEmail`-style helper.
IDENTITY_OF_THE_RUN_TS = """To know who is running the script, read the contextual variables rather than calling the API:
`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL`. WM_END_USER_EMAIL is the app viewer when
the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL
is the user the job is permissioned as. WM_USERNAME is the matching username."""
IDENTITY_OF_THE_RUN_PY = """To know who is running the script, read the contextual variables rather than calling the API:
`os.environ.get("WM_END_USER_EMAIL") or os.environ.get("WM_EMAIL")`. WM_END_USER_EMAIL is the app
viewer when the run was triggered from an app and empty otherwise (both variables are always
defined), WM_EMAIL is the user the job is permissioned as. WM_USERNAME is the matching username."""
# `setClient` reads BASE_INTERNAL_URL/BASE_URL and WM_TOKEN itself; agents that miss this rebuild
# that logic by hand and get it wrong.
PRECONFIGURED_CLIENT = """The client configures itself from the job's environment — base URL, token and credentials mode
are all set before your code runs, so there is nothing to initialize and no reason to read
WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only
reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw
HTTP for third-party APIs."""
# The listing is the helper surface, not the whole API, so each language needs its own escape
# hatch: a flat "not listed means it does not exist" is false and leaves an uncovered endpoint
# with no legal move.
UNLISTED_ENDPOINTS_TS = """The helpers below are the surface to prefer. For an endpoint none of them covers, import the
generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not
listed here but they do exist. What does not exist is a helper name you guessed at: if it is
neither listed below nor a service method, do not call it."""
UNLISTED_ENDPOINTS_PY = """The functions below are the surface to prefer. For an endpoint none of them covers,
wmill.Windmill().get(endpoint) and .post(endpoint) issue an authenticated request against this
instance. What does not exist is a function name you guessed at: if it is not listed below, do
not call it."""
def generate_ts_sdk_markdown(functions: list[dict], _types: list[dict]) -> str:
"""Generate compact documentation for TypeScript SDK."""
md = "# TypeScript SDK (windmill-client)\n\n"
md += "Import: import * as wmill from 'windmill-client'\n\n"
md += PRECONFIGURED_CLIENT + "\n\n"
md += UNLISTED_ENDPOINTS_TS + "\n\n"
md += IDENTITY_OF_THE_RUN_TS + "\n\n"
for i, func in enumerate(functions):
if func.get('docstring'):
# Format docstrings with JSDoc /** */ syntax
md += "/**\n"
docstring_lines = func['docstring'].split('\n')
for line in docstring_lines:
md += f" * {line}\n"
md += " */\n"
async_prefix = 'async ' if func['async'] else ''
md += f"{async_prefix}{func['name']}{func['generic']}({func['params']}): {func['return_type']}"
md += "\n"
if i < len(functions) - 1:
md += "\n"
return md
def generate_py_sdk_markdown(functions: list[dict], _classes: list[dict]) -> str:
"""Generate compact documentation for Python SDK."""
md = "# Python SDK (wmill)\n\n"
md += "Import: import wmill\n\n"
md += PRECONFIGURED_CLIENT + "\n\n"
md += UNLISTED_ENDPOINTS_PY + "\n\n"
md += IDENTITY_OF_THE_RUN_PY + "\n\n"
for func in functions:
# Skip private functions
if func['name'].startswith('_'):
continue
docstring = func.get('docstring')
if docstring:
# Format multi-line docstrings with # prefix on each line
docstring_lines = docstring.split('\n')
for line in docstring_lines:
md += f"# {line}\n"
async_prefix = 'async ' if func['async'] else ''
return_annotation = f" -> {func['return_type']}" if func['return_type'] else ''
md += f"{async_prefix}def {func['name']}({func['params']}){return_annotation}\n"
md += "\n"
return md
def generate_ts_exports(prompts: dict[str, str]) -> str:
"""Generate TypeScript file that exports all prompts."""
ts = "// Auto-generated by generate.py - DO NOT EDIT\n\n"
for name, content in prompts.items():
escaped = escape_for_ts(content)
ts += f"export const {name} = `{escaped}`;\n\n"
return ts
def generate_ts_declarations(prompts: dict[str, str]) -> str:
"""Generate the .d.ts for prompts.ts.
Each export is declared as a plain `string` rather than a string-literal
type so the declaration file does not embed (and drift against) the prompt
contents — those live only in prompts.ts.
"""
dts = "// Auto-generated by generate.py - DO NOT EDIT\n\n"
for name in prompts.keys():
dts += f"export declare const {name}: string;\n"
return dts
# =============================================================================
# Schema File Generation
# =============================================================================
def generate_schema_files(cli_schemas: dict[str, dict]) -> dict[str, str]:
"""
Generate standalone YAML schema files for triggers and schedules.
Returns a dict mapping schema keys (e.g., 'http_trigger') to YAML content.
"""
print("Generating standalone schema files...")
# Ensure schemas directory exists
OUTPUT_SCHEMAS_DIR.mkdir(parents=True, exist_ok=True)
schema_yaml_content = {}
# Collect all schema types from SCHEMA_MAPPINGS
for skill_name, schema_types in SCHEMA_MAPPINGS.items():
for schema_name, file_suffix in schema_types:
if schema_name not in cli_schemas:
print(f" Warning: Schema '{schema_name}' not found, skipping")
continue
# Convert the schema to JSON Schema format
json_schema = format_schema_as_json(cli_schemas[schema_name])
if not json_schema:
print(f" Warning: Empty schema for '{schema_name}', skipping")
continue
# Convert to YAML
schema_yaml = yaml.dump(json_schema, default_flow_style=False, sort_keys=False, allow_unicode=True)
# Write to file
schema_file = OUTPUT_SCHEMAS_DIR / f"{file_suffix}.schema.yaml"
schema_file.write_text(schema_yaml)
# Store for return
schema_yaml_content[file_suffix] = schema_yaml
print(f" Generated {len(schema_yaml_content)} schema files")
return schema_yaml_content
# =============================================================================
# Workspace Tool Zod Schema Generation
# =============================================================================
WORKSPACE_TOOL_ZOD_SCHEMAS = [
('NewSchedule', 'scheduleRequestSchema'),
('NewHttpTrigger', 'httpTriggerRequestSchema'),
('NewWebsocketTrigger', 'websocketTriggerRequestSchema'),
('NewKafkaTrigger', 'kafkaTriggerRequestSchema'),
('NewNatsTrigger', 'natsTriggerRequestSchema'),
('NewPostgresTrigger', 'postgresTriggerRequestSchema'),
('NewMqttTrigger', 'mqttTriggerRequestSchema'),
('NewAmqpTrigger', 'amqpTriggerRequestSchema'),
('NewSqsTrigger', 'sqsTriggerRequestSchema'),
('GcpTriggerData', 'gcpTriggerRequestSchema'),
('AzureTriggerData', 'azureTriggerRequestSchema'),
('NewEmailTrigger', 'emailTriggerRequestSchema'),
('CreateVariable', 'variableRequestSchema'),
('CreateResource', 'resourceRequestSchema'),
]
WORKSPACE_TOOL_TRIGGER_SCHEMAS = [
('http', 'httpTriggerRequestSchema'),
('websocket', 'websocketTriggerRequestSchema'),
('kafka', 'kafkaTriggerRequestSchema'),
('nats', 'natsTriggerRequestSchema'),
('postgres', 'postgresTriggerRequestSchema'),
('mqtt', 'mqttTriggerRequestSchema'),
('amqp', 'amqpTriggerRequestSchema'),
('sqs', 'sqsTriggerRequestSchema'),
('gcp', 'gcpTriggerRequestSchema'),
('azure', 'azureTriggerRequestSchema'),
('email', 'emailTriggerRequestSchema'),
]
WORKSPACE_TOOL_ZOD_OUTPUT_PATH = (
SCRIPT_DIR.parent
/ 'frontend'
/ 'src'
/ 'lib'
/ 'components'
/ 'copilot'
/ 'chat'
/ 'workspaceToolsZod.gen.ts'
)
def _resolve_schema_refs(schema: dict, backend_schemas: dict, openflow_schemas: dict, seen: tuple[str, ...] = ()) -> dict:
"""Resolve OpenAPI refs so json-schema-to-zod emits concrete enums/objects."""
if isinstance(schema, list):
return [_resolve_schema_refs(item, backend_schemas, openflow_schemas, seen) for item in schema]
if not isinstance(schema, dict):
return schema
if '$ref' in schema:
ref = schema['$ref']
ref_name = ref.split('/')[-1]
if ref_name in seen:
# Zod cannot express the recursion inline; stay permissive so the nested
# payload survives parsing instead of being stripped as unknown keys.
return {'type': 'object', 'additionalProperties': True}
source = openflow_schemas if 'openflow.openapi.yaml' in ref or ref_name not in backend_schemas else backend_schemas
ref_schema = source.get(ref_name)
if not ref_schema:
return {'type': 'object'}
resolved = _resolve_schema_refs(copy.deepcopy(ref_schema), backend_schemas, openflow_schemas, (*seen, ref_name))
for key, value in schema.items():
if key != '$ref':
resolved[key] = _resolve_schema_refs(value, backend_schemas, openflow_schemas, seen)
return resolved
return {
key: _resolve_schema_refs(value, backend_schemas, openflow_schemas, seen)
for key, value in schema.items()
}
def _ts_string(value: str) -> str:
return json.dumps(value)
def _zod_literal(value) -> str:
return json.dumps(value)
def _apply_zod_metadata(expr: str, schema: dict) -> str:
if schema.get('description'):
expr += f".describe({_ts_string(schema['description'])})"
if schema.get('nullable'):
expr += ".nullable()"
if 'default' in schema:
expr += f".default({_zod_literal(schema['default'])})"
return expr
def _json_schema_to_zod(schema: dict, indent: int = 0) -> str:
schema = schema or {}
if 'allOf' in schema:
raise ValueError('Unsupported allOf in workspace tool Zod schema generation')
variants = schema.get('anyOf') or schema.get('oneOf')
if variants:
expr = "z.union([{}])".format(
', '.join(_json_schema_to_zod(item, indent) for item in variants)
)
return _apply_zod_metadata(expr, schema)
if 'enum' in schema:
enum_values = ', '.join(_zod_literal(value) for value in schema['enum'])
expr = f"z.enum([{enum_values}])"
return _apply_zod_metadata(expr, schema)
schema_type = schema.get('type')
if schema_type == 'string':
expr = 'z.string()'
if schema.get('format') == 'date-time':
expr += '.datetime({ offset: true })'
elif schema_type == 'boolean':
expr = 'z.boolean()'
elif schema_type in ('number', 'integer'):
expr = 'z.number()'
if schema_type == 'integer':
expr += '.int()'
if 'minimum' in schema:
expr += f".gte({_zod_literal(schema['minimum'])})"
if 'maximum' in schema:
expr += f".lte({_zod_literal(schema['maximum'])})"
elif schema_type == 'array':
expr = f"z.array({_json_schema_to_zod(schema.get('items', {}), indent)})"
elif schema_type == 'object' or schema.get('properties') is not None or schema.get('additionalProperties') is not None:
properties = schema.get('properties') or {}
if not properties and schema.get('additionalProperties'):
expr = 'z.record(z.string(), z.any())'
else:
required = set(schema.get('required') or [])
prop_lines = []
child_indent = '\t' * (indent + 1)
closing_indent = '\t' * indent
for key, value in properties.items():
prop_expr = _json_schema_to_zod(value, indent + 1)
if key not in required:
prop_expr += '.optional()'
prop_lines.append(f"{child_indent}{_ts_string(key)}: {prop_expr}")
if prop_lines:
expr = "z.object({\n" + ",\n".join(prop_lines) + f"\n{closing_indent}}})"
else:
expr = 'z.object({})'
else:
expr = 'z.any()'
return _apply_zod_metadata(expr, schema)
def generate_workspace_tool_zod_schemas(backend_schemas: dict, openflow_schemas: dict) -> None:
"""Generate Zod schemas used by frontend AI chat workspace mutation tools."""
print("Generating workspace tool Zod schemas...")
missing = [schema_name for schema_name, _ in WORKSPACE_TOOL_ZOD_SCHEMAS if schema_name not in backend_schemas]
if missing:
print(f" Warning: Missing schemas for workspace tool Zod generation: {', '.join(missing)}")
return
trigger_path_description = (
backend_schemas.get('NewHttpTrigger', {})
.get('properties', {})
.get('path', {})
.get('description')
or "The new trigger's Windmill path"
)
lines = [
"// Auto-generated by generate.py - DO NOT EDIT",
"",
"import { z } from 'zod'",
"",
]
for schema_name, export_name in WORKSPACE_TOOL_ZOD_SCHEMAS:
schema = _resolve_schema_refs(
copy.deepcopy(backend_schemas[schema_name]),
backend_schemas,
openflow_schemas,
)
lines.append(f"export const {export_name} = {_json_schema_to_zod(schema)}")
lines.append("")
lines.extend([
"export const triggerRequestSchemas = {",
*[
f"\t{kind}: {schema_name},"
for kind, schema_name in WORKSPACE_TOOL_TRIGGER_SCHEMAS
],
"} as const",
"",
f"const triggerPathSchema = z.string().min(1).describe({_ts_string(trigger_path_description)})",
"",
"// The kind-specific fields of a trigger config, with the three the tool supplies",
"// itself removed. Fetched one at a time through get_trigger_schema rather than",
"// inlined into create_trigger: as a union of all eleven this serialized to ~39k",
"// characters of JSON Schema, resent on every request of every chat.",
"export const triggerConfigSchemas = {",
*[
f"\t{kind}: {schema_name}.omit({{ path: true, script_path: true, is_flow: true }}),"
for kind, schema_name in WORKSPACE_TOOL_TRIGGER_SCHEMAS
],
"} as const",
"",
"export const createTriggerToolSchema = z.object({",
"\tkind: z.enum([",
*[
f"\t\t{_ts_string(kind)},"
for kind, _ in WORKSPACE_TOOL_TRIGGER_SCHEMAS
],
"\t]),",
"\tpath: triggerPathSchema,",
"\tconfig: z",
"\t\t.record(z.string(), z.any())",
"\t\t.describe(",
"\t\t\t'The kind-specific trigger configuration. Call get_trigger_schema with the same kind first to get its exact fields.'",
"\t\t)",
"})",
])
lines.append("")
WORKSPACE_TOOL_ZOD_OUTPUT_PATH.write_text("\n".join(lines))
print(" Generated workspaceToolsZod.gen.ts")
# =============================================================================
# Datatable SDK Extraction
# =============================================================================
TS_SQL_UTILS_PATH = TS_SDK_DIR / "sqlUtils.ts"
def extract_datatable_ts_sdk() -> str:
"""Extract datatable-specific type definitions from TypeScript SDK (sqlUtils.ts).
Reads the source file and extracts the public API surface:
- SqlStatement<T> type (fetch, fetchOne, fetchOneScalar, execute methods)
- DatatableSqlTemplateFunction interface (template tag + query method)
- datatable() function signature
"""
if not TS_SQL_UTILS_PATH.exists():
print(f" Warning: sqlUtils.ts not found at {TS_SQL_UTILS_PATH}")
return ''
content = TS_SQL_UTILS_PATH.read_text()
md = "## TypeScript Datatable API (windmill-client)\n\n"
md += "Import: `import * as wmill from 'windmill-client'`\n\n"
# Extract exported type/interface/function definitions from sqlUtils.ts
# We use extract_balanced to handle nested braces correctly
# 1. Extract SqlStatement<T> type
match = re.search(r'(\/\*\*(?:[^*]|\*(?!\/))*\*\/\s*)?export\s+type\s+SqlStatement<T>\s*=\s*', content)
if match:
jsdoc_raw = match.group(1)
brace_start = content.index('{', match.end() - 1)
body, end = extract_balanced(content, brace_start, '{', '}')
if end != -1:
if jsdoc_raw:
md += clean_jsdoc(jsdoc_raw) + "\n"
md += "```typescript\n"
md += f"type SqlStatement<T> = {{\n{_indent_body(body)}\n}};\n"
md += "```\n\n"
# 2. Extract DatatableSqlTemplateFunction interface
match = re.search(
r'(\/\*\*(?:[^*]|\*(?!\/))*\*\/\s*)?export\s+interface\s+DatatableSqlTemplateFunction\s+extends\s+SqlTemplateFunction\s*',
content
)
if match:
brace_start = content.index('{', match.end() - 1)
body, end = extract_balanced(content, brace_start, '{', '}')
if end != -1:
md += "```typescript\n"
md += "// Template tag function: sql`SELECT * FROM table WHERE id = ${id}`.fetch()\n"
md += f"interface DatatableSqlTemplateFunction {{\n"
md += f" // Tagged template usage:\n"
md += f" <T = any>(strings: TemplateStringsArray, ...values: any[]): SqlStatement<T>;\n"
md += f"{_indent_body(body)}\n"
md += "};\n"
md += "```\n\n"
# 3. Extract datatable() function
match = re.search(
r'(\/\*\*(?:[^*]|\*(?!\/))*\*\/\s*)?export\s+function\s+datatable\s*\(([^)]*)\)\s*:\s*(\S+)',
content
)
if match:
jsdoc_raw, params, return_type = match.groups()
if jsdoc_raw:
md += clean_jsdoc(jsdoc_raw) + "\n"
md += "```typescript\n"
md += f"function datatable({params.strip()}): {return_type}\n"
md += "```\n"
return md
def extract_datatable_py_sdk(py_content: str) -> str:
"""Extract datatable-specific class/function definitions from Python SDK.
Uses Python AST to extract:
- datatable() function
- DataTableClient class with query() method
- SqlQuery class with fetch(), fetch_one(), fetch_one_scalar(), execute() methods
"""
if not py_content:
return ''
try:
tree = ast.parse(py_content)
except SyntaxError as e:
print(f" Warning: Could not parse Python SDK for datatable extraction: {e}")
return ''
md = "## Python Datatable API (wmill)\n\n"
md += "Import: `import wmill`\n\n"
# Target classes and the top-level datatable function
target_classes = {'DataTableClient', 'SqlQuery'}
# 1. Extract datatable() top-level function
for node in tree.body:
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == 'datatable':
docstring = ast.get_docstring(node) or ''
params = _format_py_params(node)
return_ann = f" -> {ast.unparse(node.returns)}" if node.returns else ''
if docstring:
for line in docstring.split('\n'):
md += f"# {line}\n"
md += f"def datatable({params}){return_ann}\n\n"
break
# 2. Extract target classes with their public methods
for node in tree.body:
if isinstance(node, ast.ClassDef) and node.name in target_classes:
class_doc = ast.get_docstring(node) or ''
if class_doc:
for line in class_doc.split('\n'):
md += f"# {line}\n"
md += f"class {node.name}:\n"
for item in node.body:
if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)):
if item.name.startswith('_') and item.name != '__init__':
continue
docstring = ast.get_docstring(item) or ''
params = _format_py_params(item, skip_self=True)
return_ann = f" -> {ast.unparse(item.returns)}" if item.returns else ''
async_prefix = 'async ' if isinstance(item, ast.AsyncFunctionDef) else ''
if docstring:
for line in docstring.split('\n'):
md += f" # {line}\n"
md += f" {async_prefix}def {item.name}({params}){return_ann}\n\n"
md += "\n"
return md
def _format_py_params(node: ast.FunctionDef, skip_self: bool = False) -> str:
"""Format function parameters from AST node."""
params = []
args = node.args
num_defaults = len(args.defaults)
num_args = len(args.args)
for i, arg in enumerate(args.args):
if skip_self and arg.arg == 'self':
continue
param_str = arg.arg
if arg.annotation:
param_str += f": {ast.unparse(arg.annotation)}"
default_idx = i - (num_args - num_defaults)
if default_idx >= 0:
default = args.defaults[default_idx]
param_str += f" = {ast.unparse(default)}"
params.append(param_str)
if args.vararg:
vararg_str = f"*{args.vararg.arg}"
if args.vararg.annotation:
vararg_str += f": {ast.unparse(args.vararg.annotation)}"
params.append(vararg_str)
# A bare `*` before the keyword-only args, when nothing else already introduced them.
# Without it the rendered signature reads as all-positional, and a caller written against
# these docs passes a keyword-only argument positionally and gets a TypeError.
if args.kwonlyargs and not args.vararg:
params.append('*')
for i, arg in enumerate(args.kwonlyargs):
param_str = arg.arg
if arg.annotation:
param_str += f": {ast.unparse(arg.annotation)}"
if args.kw_defaults[i]:
param_str += f" = {ast.unparse(args.kw_defaults[i])}"
params.append(param_str)
if args.kwarg:
kwarg_str = f"**{args.kwarg.arg}"
if args.kwarg.annotation:
kwarg_str += f": {ast.unparse(args.kwarg.annotation)}"
params.append(kwarg_str)
return ', '.join(params)
def _indent_body(body: str) -> str:
"""Clean and re-indent a type body for readable output."""
lines = body.strip().split('\n')
result = []
for line in lines:
stripped = line.strip()
if stripped:
# Keep JSDoc comments and method signatures with consistent indentation
if not stripped.startswith('//') and not stripped.startswith('/*') and not stripped.startswith('*'):
result.append(f" {stripped}")
else:
result.append(f" {stripped}")
else:
result.append('')
return '\n'.join(result)
# =============================================================================
# Workflow-as-Code SDK Extraction
# =============================================================================
WAC_TS_FUNCTIONS = [
'getResumeUrls',
'task',
'taskScript',
'taskFlow',
'workflow',
'step',
'sleep',
'waitForApproval',
'getApprovalUrls',
'parallel',
]
WAC_PY_FUNCTIONS = [
'get_resume_urls',
'task',
'task_script',
'task_flow',
'workflow',
'step',
'sleep',
'wait_for_approval',
'get_approval_urls',
'parallel',
]
def _extract_ts_angle_params(content: str, start_pos: int) -> tuple[str, int]:
"""Extract TypeScript generic parameters, ignoring arrow `=>` tokens."""
if start_pos >= len(content) or content[start_pos] != '<':
return '', start_pos
depth = 0
i = start_pos
quote: str | None = None
while i < len(content):
char = content[i]
prev = content[i - 1] if i > 0 else ''
if quote:
if char == '\\':
i += 2
continue
if char == quote:
quote = None
i += 1
continue
if char in ('"', "'", '`'):
quote = char
elif char == '<':
depth += 1
elif char == '>' and prev != '=':
depth -= 1
if depth == 0:
return content[start_pos:i + 1], i + 1
i += 1
return '', -1
def _render_ts_jsdoc(jsdoc_raw: str | None) -> str:
if not jsdoc_raw:
return ''
docstring = clean_jsdoc(jsdoc_raw)
if not docstring:
return ''
lines = ["/**"]
for line in docstring.split('\n'):
lines.append(f" * {line}" if line else " *")
lines.append(" */")
return '\n'.join(lines)
def _extract_ts_interface(content: str, name: str) -> str:
pattern = re.compile(
r'(?:(/\*\*(?:[^*]|\*(?!/))*\*/)\s*)?'
rf'export\s+interface\s+{re.escape(name)}\s*',
re.MULTILINE
)
match = pattern.search(content)
if not match:
return ''
try:
brace_start = content.index('{', match.end() - 1)
except ValueError:
return ''
body, end = extract_balanced(content, brace_start, '{', '}')
if end == -1:
return ''
parts = []
jsdoc = _render_ts_jsdoc(match.group(1))
if jsdoc:
parts.append(jsdoc)
parts.append(f"export interface {name} {{\n{_indent_body(body)}\n}}")
return '\n'.join(parts)
def _extract_ts_exported_function(content: str, name: str) -> str:
pattern = re.compile(
r'(?:(/\*\*(?:[^*]|\*(?!/))*\*/)\s*)?'
rf'export\s+(async\s+)?function\s+{re.escape(name)}\s*',
re.MULTILINE
)
match = pattern.search(content)
if not match:
return ''
jsdoc_raw, is_async = match.groups()
pos = match.end()
while pos < len(content) and content[pos] in ' \t\n':
pos += 1
generic = ''
if pos < len(content) and content[pos] == '<':
generic, pos = _extract_ts_angle_params(content, pos)
if pos == -1:
return ''
while pos < len(content) and content[pos] in ' \t\n':
pos += 1
if pos >= len(content) or content[pos] != '(':
return ''
params, paren_end = extract_balanced(content, pos, '(', ')')
if paren_end == -1:
return ''
return_type, _ = extract_return_type(content, paren_end + 1)
async_prefix = 'async ' if is_async else ''
signature = f"export {async_prefix}function {name}{generic}({clean_params(params)})"
if return_type:
signature += f": {clean_params(return_type)}"
parts = []
jsdoc = _render_ts_jsdoc(jsdoc_raw)
if jsdoc:
parts.append(jsdoc)
parts.append(signature)
return '\n'.join(parts)
def extract_wac_ts_sdk(ts_content: str) -> str:
"""Extract Workflow-as-Code API signatures from the TypeScript SDK."""
if not ts_content:
return ''
declarations = []
for interface_name in ('TaskRetry', 'TaskOptions'):
interface = _extract_ts_interface(ts_content, interface_name)
if interface:
declarations.append(interface)
for function_name in WAC_TS_FUNCTIONS:
signature = _extract_ts_exported_function(ts_content, function_name)
if signature:
declarations.append(signature)
else:
print(f" Warning: TypeScript WAC function '{function_name}' not found")
if not declarations:
return ''
md = "## TypeScript Workflow-as-Code API (windmill-client)\n\n"
md += 'Import: `import { workflow, task, taskScript, taskFlow, step, sleep, waitForApproval, getApprovalUrls, getResumeUrls, parallel } from "windmill-client"`\n\n'
md += "```typescript\n"
md += "\n\n".join(declarations)
md += "\n```\n"
return md
def _format_py_params_exact(node, skip_self: bool = False) -> str:
"""Format Python parameters from AST, preserving bare * for keyword-only args."""
params = []
args = node.args
positional = list(args.posonlyargs) + list(args.args)
num_defaults = len(args.defaults)
num_positional = len(positional)
for i, arg in enumerate(positional):
if skip_self and arg.arg == 'self':
continue
param_str = arg.arg
if arg.annotation:
param_str += f": {ast.unparse(arg.annotation)}"
default_idx = i - (num_positional - num_defaults)
if default_idx >= 0:
param_str += f" = {ast.unparse(args.defaults[default_idx])}"
params.append(param_str)
if args.vararg:
vararg_str = f"*{args.vararg.arg}"
if args.vararg.annotation:
vararg_str += f": {ast.unparse(args.vararg.annotation)}"
params.append(vararg_str)
elif args.kwonlyargs:
params.append('*')
for i, arg in enumerate(args.kwonlyargs):
param_str = arg.arg
if arg.annotation:
param_str += f": {ast.unparse(arg.annotation)}"
if args.kw_defaults[i] is not None:
param_str += f" = {ast.unparse(args.kw_defaults[i])}"
params.append(param_str)
if args.kwarg:
kwarg_str = f"**{args.kwarg.arg}"
if args.kwarg.annotation:
kwarg_str += f": {ast.unparse(args.kwarg.annotation)}"
params.append(kwarg_str)
return ', '.join(params)
def _render_py_docstring(docstring: str, indent: str = '') -> str:
if not docstring:
return ''
return '\n'.join(f"{indent}# {line}" if line else f"{indent}#" for line in docstring.split('\n'))
def _extract_py_function_signature(tree: ast.Module, name: str) -> str:
for node in tree.body:
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == name:
docstring = ast.get_docstring(node) or ''
params = _format_py_params_exact(node)
return_ann = f" -> {ast.unparse(node.returns)}" if node.returns else ''
async_prefix = 'async ' if isinstance(node, ast.AsyncFunctionDef) else ''
parts = []
rendered_docstring = _render_py_docstring(docstring)
if rendered_docstring:
parts.append(rendered_docstring)
parts.append(f"{async_prefix}def {node.name}({params}){return_ann}")
return '\n'.join(parts)
return ''
def _extract_py_class_signature(tree: ast.Module, name: str) -> str:
for node in tree.body:
if isinstance(node, ast.ClassDef) and node.name == name:
parts = []
docstring = _render_py_docstring(ast.get_docstring(node) or '')
if docstring:
parts.append(docstring)
bases = f"({', '.join(ast.unparse(base) for base in node.bases)})" if node.bases else ''
parts.append(f"class {node.name}{bases}:")
for item in node.body:
if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef)) and item.name == '__init__':
init_docstring = _render_py_docstring(ast.get_docstring(item) or '', indent=' ')
if init_docstring:
parts.append(init_docstring)
params = _format_py_params_exact(item)
parts.append(f" def __init__({params})")
break
return '\n'.join(parts)
return ''
def extract_wac_py_sdk(py_content: str) -> str:
"""Extract Workflow-as-Code API signatures from the Python SDK."""
if not py_content:
return ''
try:
tree = ast.parse(py_content)
except SyntaxError as e:
print(f" Warning: Could not parse Python SDK for WAC extraction: {e}")
return ''
declarations = []
task_error = _extract_py_class_signature(tree, 'TaskError')
if task_error:
declarations.append(task_error)
for function_name in WAC_PY_FUNCTIONS:
signature = _extract_py_function_signature(tree, function_name)
if signature:
declarations.append(signature)
else:
print(f" Warning: Python WAC function '{function_name}' not found")
if not declarations:
return ''
md = "## Python Workflow-as-Code API (wmill)\n\n"
md += "Import: `from wmill import workflow, task, task_script, task_flow, step, sleep, wait_for_approval, get_approval_urls, get_resume_urls, parallel, TaskError`\n\n"
md += "```python\n"
md += "\n\n".join(declarations)
md += "\n```\n"
return md
# =============================================================================
# Skill Generation
# =============================================================================
def generate_skill_content(
skill_name: str,
description: str,
intro: str,
content: str,
sdk_content: str = ''
) -> str:
"""Generate a skill file with YAML frontmatter."""
parts = [
"---",
f"name: {skill_name}",
f"description: {description}",
"---",
"",
]
if intro:
parts.extend([intro, ""])
parts.append(content)
if sdk_content:
parts.extend(["", sdk_content])
return '\n'.join(parts)
# Skill definitions for config-driven generation
SKILL_DEFINITIONS = [
{
'name': 'write-flow',
'description': 'MUST use when creating flows.',
'content_key': 'flow',
},
{
'name': 'raw-app',
'description': 'MUST use when creating raw apps.',
'content_key': 'raw_app',
},
{
'name': 'triggers',
'description': 'MUST use when configuring triggers.',
'content_key': 'triggers',
'schema_types': [
('HttpTrigger', 'http_trigger'),
('WebsocketTrigger', 'websocket_trigger'),
('KafkaTrigger', 'kafka_trigger'),
('NatsTrigger', 'nats_trigger'),
('PostgresTrigger', 'postgres_trigger'),
('MqttTrigger', 'mqtt_trigger'),
('AmqpTrigger', 'amqp_trigger'),
('SqsTrigger', 'sqs_trigger'),
('GcpTrigger', 'gcp_trigger'),
('AzureTrigger', 'azure_trigger'),
('EmailTrigger', 'email_trigger'),
],
},
{
'name': 'schedules',
'description': 'MUST use when configuring schedules.',
'content_key': 'schedules',
'schema_types': [('Schedule', 'schedule')],
},
{
'name': 'resources',
'description': 'MUST use when managing resources.',
'content_key': 'resources',
},
{
'name': 'write-workflow-as-code',
'description': 'MUST use when writing or modifying Windmill Workflow-as-Code scripts using workflow, task, step, sleep, approvals, taskScript, taskFlow, task_script, or task_flow.',
'content_key': 'workflow_as_code',
'intro_key': 'wac_cli',
'sdk_content_key': 'wac',
},
{
'name': 'cli-commands',
'description': 'MUST use when using the CLI, including debugging job failures and inspecting run history via `wmill job`.',
'content_key': 'cli_commands',
},
{
'name': 'preview',
'description': 'MUST use when opening the Windmill dev page / visual preview of a flow, script, or app. Triggers on words like preview, open, navigate to, visualize, see the flow/app/script, and after writing a flow/script/app for visual verification.',
'content_key': 'preview',
},
]
def generate_skills(
languages: dict[str, str],
ts_sdk_md: str,
py_sdk_md: str,
wac_ts_md: str,
wac_py_md: str,
flow_cli: str,
flow_base: str,
openflow_content: str,
cli_commands: str,
cli_schemas: dict[str, dict] | None = None
):
"""Generate individual skill files for Claude Code."""
print("Generating skill files...")
cli_schemas = cli_schemas or {}
# Ensure skills directory exists
OUTPUT_SKILLS_DIR.mkdir(parents=True, exist_ok=True)
# Read base files for additional skills.
# Note: raw-app.md is the chat-relevant authoring guide. The CLI workflow
# (wmill app new wizard, on-disk layout, sql_to_apply/, CLI commands) lives
# in raw-app-cli.md. Concatenated here for the skill so CLI users see CLI
# guidance first, then the platform shape.
base_dir = SCRIPT_DIR / "base"
raw_app_cli_md = read_markdown_file(base_dir / "raw-app-cli.md")
raw_app_authoring_md = read_markdown_file(base_dir / "raw-app.md")
base_content = {
'flow': f"{flow_cli}\n\n{flow_base}\n\n{openflow_content}",
'raw_app': f"{raw_app_cli_md}\n\n{raw_app_authoring_md}",
'triggers': read_markdown_file(base_dir / "triggers.md"),
'schedules': read_markdown_file(base_dir / "schedules.md"),
'resources': read_markdown_file(base_dir / "resources.md"),
'workflow_as_code': read_markdown_file(base_dir / "workflow-as-code.md"),
'cli_commands': cli_commands,
'preview': read_markdown_file(base_dir / "preview.md"),
}
# CLI intro for script skills
script_cli_intro = """## CLI Commands
Place scripts in a folder.
After writing, tell the user which command fits what they want to do:
- `wmill script preview <script_path>` — **default when iterating on a local script.** Runs the local file without deploying.
- `wmill script run <path>` — runs the script **already deployed** in the workspace. Use only when the user explicitly wants to test the deployed version, not local edits.
- `wmill generate-metadata` — regenerate the local `.script.yaml` (input schema) and `.lock` (resolved dependencies) for scripts you changed, and refresh their content hashes in `wmill-lock.yaml`. Local files only — **not** a deploy. See "Keep metadata in sync" below.
- Deploy local changes to the workspace — via `git push` or `wmill sync push` depending on how the repo is wired (see the **Deploying** section in `AGENTS.wmill.md`). Only suggest/run a deploy when the user explicitly asks to deploy/publish/push — not when they say "run", "try", or "test".
### Preview vs run — choose by intent, not habit
If the user says "run the script", "try it", "test it", "does it work" while there are **local edits to the script file**, use `script preview`. Do NOT push the script to then `script run` it — pushing is a deploy, and deploying just to test overwrites the workspace version with untested changes.
Only use `script run` when:
- The user explicitly says "run the deployed version" / "run what's on the server".
- There is no local script being edited (you're just invoking an existing script).
Only use `sync push` when:
- The user explicitly asks to deploy, publish, push, or ship.
- The preview has already validated the change and the user wants it in the workspace.
### Keep metadata in sync after editing
`wmill-lock.yaml` tracks a content hash for each item. Editing a script's content — most importantly **adding or removing an import** or **changing `main`'s arguments** — invalidates that hash and leaves the `.lock`, the `.script.yaml` input schema, and the hash row out of date. Run `wmill generate-metadata` (scoped to what you touched) after such edits so the resolved lock, the auto-generated args UI (driven by `.script.yaml`), and `wmill-lock.yaml` all match the code. Leaving them stale produces spurious diffs in git-sync and CI.
This only writes local files (it is **not** a deploy), but it re-resolves dependencies, so it can bump unpinned versions (the same as deploying from the UI; expected, not a bug). So by default offer it and run it once the user agrees, rather than running it silently after every edit — unless the project's `AGENTS.md` opts into running metadata automatically (see the "Keeping metadata in sync" preference there). Either way YOU run the command, not the user. After running it, diff the regenerated `.lock` / `.script.lock` files and tell the user which dependency versions changed (e.g. `requests 2.31.0 → 2.32.0`), so they can catch an unwanted bump before deploying — even under `Metadata: auto`, since it's information, not a confirmation gate. Pin versions in code to keep them fixed.
With no path argument, `generate-metadata` regenerates only the items whose content hash drifted — not everything. Imports propagate: editing a script that others import marks every importer stale too, so a one-line change to a shared module can regenerate many locks (by design — their locks must reflect the imported code). If it touches more than you expect, run `wmill generate-metadata --dry-run` — it lists each stale item with a reason (`content changed` or `depends on <path>`) without changing anything — then narrow with a path argument (`wmill generate-metadata f/foo`) or `--strict-folder-boundaries`.
If the on-disk `.lock` and `.script.yaml` are already correct and only `wmill-lock.yaml` needs its hashes refreshed (hash drift, or bootstrapping missing entries), use `wmill generate-metadata rehash` — it re-records hashes from disk with no backend round-trip and no dependency changes.
### After writing — offer to test, don't wait passively
If the user hasn't already told you to run/test/preview the script, offer it as a one-sentence next step (e.g. "Want me to run `wmill script preview` with sample args?"). Do not present a multi-option menu.
If the user already asked to test/run/try the script in their original request, skip the offer and just execute `wmill script preview <path> -d '<args>'` directly — pick plausible args from the script's declared parameters. The shape varies by language: `main(...)` for code languages, the SQL dialect's own placeholder syntax (`$1` for PostgreSQL, `?` for MySQL/Snowflake, `@P1` for MSSQL, `@name` for BigQuery, etc.), positional `$1`, `$2`, … for Bash, `param(...)` for PowerShell.
`wmill script preview` does not deploy, but it still executes script code and may cause side effects; run it yourself when the user asked to test/preview (or after confirming that execution is intended). `wmill generate-metadata` does not deploy either — it only writes local files (locks, schemas, hashes) — but offer it before running (or run automatically if the project's `AGENTS.md` opts in), per "Keep metadata in sync" above. Deploying to the workspace (`git push` or `wmill sync push` depending on how the repo is wired — see the **Deploying** section) is the only step that mutates remote state — do it only when the user explicitly asks to deploy/publish/push.
For a **visual** open-the-script-in-the-dev-page preview (rather than `script preview`'s run-and-print-result), use the `preview` skill.
Use `wmill resource-type list --schema` to discover available resource types."""
wac_cli_intro = f"""{script_cli_intro}
Workflow-as-Code files use the normal script CLI workflow. There are no separate WAC deploy commands."""
intro_content = {
'wac_cli': wac_cli_intro,
}
extra_sdk_content = {
'wac': "\n\n".join(filter(None, [wac_ts_md, wac_py_md])),
}
skills_generated = []
# Generate script skills for each language
for lang_key, lang_content in languages.items():
if lang_key not in LANGUAGE_METADATA:
print(f" Warning: No metadata for language '{lang_key}', skipping")
continue
metadata = LANGUAGE_METADATA[lang_key]
skill_name = f"write-script-{lang_key}"
skill_dir = OUTPUT_SKILLS_DIR / skill_name
skill_dir.mkdir(parents=True, exist_ok=True)
# Determine which SDK to include
language_sdk_content = ''
if lang_key in TS_SDK_LANGUAGES:
language_sdk_content = ts_sdk_md
elif lang_key in PY_SDK_LANGUAGES:
language_sdk_content = py_sdk_md
skill_content = generate_skill_content(
skill_name=skill_name,
description=metadata['description'],
intro=script_cli_intro,
content=lang_content,
sdk_content=language_sdk_content
)
(skill_dir / "SKILL.md").write_text(skill_content)
skills_generated.append(skill_name)
# Generate other skills from definitions
# Note: Skills with schema_types (triggers, schedules) get base content only.
# Schemas are stored separately and combined at CLI init time.
for skill_def in SKILL_DEFINITIONS:
content = base_content.get(skill_def['content_key'], '')
if not content:
continue
skill_name = skill_def['name']
skill_dir = OUTPUT_SKILLS_DIR / skill_name
skill_dir.mkdir(parents=True, exist_ok=True)
# Note: We no longer append schemas here. Skills with 'schema_types'
# will have schemas combined at CLI init time from SCHEMAS export.
skill_content = generate_skill_content(
skill_name=skill_name,
description=skill_def['description'],
intro=intro_content.get(skill_def.get('intro_key', ''), ''),
content=content,
sdk_content=extra_sdk_content.get(skill_def.get('sdk_content_key', ''), '')
)
(skill_dir / "SKILL.md").write_text(skill_content)
skills_generated.append(skill_name)
print(f" Generated {len(skills_generated)} skills")
return skills_generated
def generate_skills_ts_export(skills: list[str], schema_yaml_content: dict[str, str] | None = None) -> str:
"""Generate TypeScript file that exports skill metadata for the CLI.
Args:
skills: List of skill names
schema_yaml_content: Dict mapping schema keys (e.g., 'http_trigger') to YAML content
"""
schema_yaml_content = schema_yaml_content or {}
ts = "// Auto-generated by generate.py - DO NOT EDIT\n\n"
ts += "export interface SkillMetadata {\n"
ts += " name: string;\n"
ts += " description: string;\n"
ts += " languageKey?: string;\n"
ts += "}\n\n"
ts += "export const SKILLS: SkillMetadata[] = [\n"
skill_desc_map = {s['name']: s['description'] for s in SKILL_DEFINITIONS}
for skill in skills:
if skill.startswith('write-script-'):
lang_key = skill.replace('write-script-', '')
if lang_key in LANGUAGE_METADATA:
metadata = LANGUAGE_METADATA[lang_key]
ts += f' {{ name: "{skill}", description: "{metadata["description"]}", languageKey: "{lang_key}" }},\n'
elif skill in skill_desc_map:
ts += f' {{ name: "{skill}", description: "{skill_desc_map[skill]}" }},\n'
ts += "];\n\n"
# Generate the skills content inline for bundling
ts += "// Skill content for each skill (loaded inline for bundling)\n"
ts += "export const SKILL_CONTENT: Record<string, string> = {\n"
# We'll read the generated files and embed them
for skill in skills:
skill_path = OUTPUT_SKILLS_DIR / skill / "SKILL.md"
if skill_path.exists():
content = skill_path.read_text()
escaped = escape_for_ts(content)
ts += f' "{skill}": `{escaped}`,\n'
ts += "};\n\n"
# Generate SCHEMAS export (YAML content for each schema)
ts += "// YAML schema content for triggers and schedules\n"
ts += "export const SCHEMAS: Record<string, string> = {\n"
for schema_key, yaml_content in sorted(schema_yaml_content.items()):
escaped = escape_for_ts(yaml_content)
ts += f' "{schema_key}": `{escaped}`,\n'
ts += "};\n\n"
# Generate SCHEMA_MAPPINGS export (maps skill names to their schemas)
ts += "// Maps skill names to their schema types and file patterns\n"
ts += "export interface SchemaMapping {\n"
ts += " name: string;\n"
ts += " schemaKey: string;\n"
ts += " filePattern: string;\n"
ts += "}\n\n"
ts += "export const SCHEMA_MAPPINGS: Record<string, SchemaMapping[]> = {\n"
for skill_name, schema_types in SCHEMA_MAPPINGS.items():
ts += f' "{skill_name}": [\n'
for schema_name, file_suffix in schema_types:
ts += f' {{ name: "{schema_name}", schemaKey: "{file_suffix}", filePattern: "*.{file_suffix}.yaml" }},\n'
ts += " ],\n"
ts += "};\n"
return ts
def format_schema_for_markdown(schema_yaml: str, schema_name: str, file_pattern: str) -> str:
"""Format a standalone schema block for plugin skill files."""
return f"""## {schema_name} (`{file_pattern}`)
Must be a YAML file that adheres to the following schema:
```yaml
{schema_yaml.strip()}
```"""
def render_plugin_skill_content(skill_name: str, schema_yaml_content: dict[str, str]) -> str:
"""Render plugin-ready skill content from generated base skill files."""
skill_path = OUTPUT_SKILLS_DIR / skill_name / "SKILL.md"
if not skill_path.exists():
raise FileNotFoundError(f"Missing generated skill content for {skill_name}: {skill_path}")
skill_content = skill_path.read_text()
schema_mappings = SCHEMA_MAPPINGS.get(skill_name, [])
if not schema_mappings:
return skill_content
schema_docs = []
for schema_name, schema_key in schema_mappings:
schema_yaml = schema_yaml_content.get(schema_key)
if not schema_yaml:
continue
schema_docs.append(
format_schema_for_markdown(
schema_yaml=schema_yaml,
schema_name=schema_name,
file_pattern=f"*.{schema_key}.yaml",
)
)
if not schema_docs:
return skill_content
return f"{skill_content}\n\n" + "\n\n".join(schema_docs)
def resolve_plugin_skills_dir(plugin_dir: Path) -> Path:
"""Resolve the plugin skills directory from a repo root, plugin root, or skills dir."""
plugin_dir = plugin_dir.expanduser().resolve()
plugin_root = plugin_dir / "plugins" / "windmill"
if (plugin_root / ".claude-plugin" / "plugin.json").exists():
return plugin_root / "skills"
plugin_skills_dir = plugin_dir / "skills"
plugin_json = plugin_dir / ".claude-plugin" / "plugin.json"
if plugin_json.exists():
return plugin_skills_dir
if plugin_dir.name == "skills":
return plugin_dir
return plugin_skills_dir
def generate_plugin_skills(
plugin_dir: Path,
skills: list[str],
schema_yaml_content: dict[str, str],
) -> Path:
"""Generate standalone skills in a Claude plugin checkout."""
skills_dir = resolve_plugin_skills_dir(plugin_dir)
skills_dir.mkdir(parents=True, exist_ok=True)
expected_skills = set(skills)
for existing in skills_dir.iterdir():
if existing.is_dir() and existing.name not in expected_skills:
shutil.rmtree(existing)
for skill_name in skills:
skill_dir = skills_dir / skill_name
skill_dir.mkdir(parents=True, exist_ok=True)
(skill_dir / "SKILL.md").write_text(
render_plugin_skill_content(skill_name, schema_yaml_content)
)
print(f"\nGenerated for plugin:")
print(f" - {skills_dir} ({len(skills)} skills)")
return skills_dir
# =============================================================================
# Context7 Docs Repo Generation
# =============================================================================
# Files in the context7 target directory that must survive a regeneration
# (everything else is wiped to keep the export deterministic).
CONTEXT7_PRESERVE = frozenset(
{
".git",
".github",
".gitignore",
".gitattributes",
"CODEOWNERS",
"LICENSE",
"LICENSE.md",
"context7.json",
}
)
# Name written into manifest.json — also used to recognise the docs repo
# when re-generating into an existing checkout.
CONTEXT7_REPO_NAME = "windmill-cli-docs"
def extract_agents_md_template() -> str:
"""Extract the AGENTS.wmill.md template string from cli/src/guidance/core.ts.
Keeping a single source of truth in TypeScript avoids drift between what
`wmill init` writes locally and what we publish for context7 ingestion.
"""
core_ts_path = SCRIPT_DIR.parent / "cli" / "src" / "guidance" / "core.ts"
content = core_ts_path.read_text()
# Anchor on the function name so adding other template-literal-returning
# functions to core.ts can't silently re-target the regex. The function
# was renamed from `generateAgentsMdContent` → `generateAgentsCliMdContent`
# when the managed file split out of AGENTS.md into AGENTS.cli.md.
match = re.search(
r"function\s+generateAgentsCliMdContent\b[\s\S]*?return\s+`([\s\S]*?)`;",
content,
)
if not match:
raise RuntimeError(
f"Could not extract AGENTS.wmill.md template from {core_ts_path}"
)
return _unescape_ts_template_literal(match.group(1))
def _unescape_ts_template_literal(raw: str) -> str:
"""Decode TS template-literal escapes in one pass.
Multi-pass `.replace()` would mangle e.g. `\\\\` -> `\\` -> `` ` `` if the
template ever contained a literal backslash followed by a backtick. A
single-pass scan is order-independent.
"""
return re.sub(
r"\\(.)",
lambda m: {"`": "`", "$": "$", "\\": "\\"}.get(m.group(1), m.group(0)),
raw,
)
def render_agents_md_for_docs(
skills: list[str], skill_desc_map: dict[str, str]
) -> str:
"""Render AGENTS.wmill.md exactly as `wmill init` would, for the docs repo.
The skill reference paths point at `.agents/skills/` (the canonical tree
that Codex/Pi read directly and that Claude Code mirrors under
`.claude/skills/`) — matching `buildSkillsReference` in
`cli/src/guidance/writer.ts`.
"""
template = extract_agents_md_template()
skills_reference = "\n".join(
f"- `.agents/skills/{name}/SKILL.md` - {skill_desc_map[name]}"
for name in skills
if name in skill_desc_map
)
return template.replace("${skillsReference}", skills_reference)
def build_skill_desc_map(skills: list[str]) -> dict[str, str]:
"""Map each skill name to its user-facing description.
Mirrors the logic in `generate_skills_ts_export`: language skills draw from
LANGUAGE_METADATA, everything else from SKILL_DEFINITIONS.
"""
desc_map = {s["name"]: s["description"] for s in SKILL_DEFINITIONS}
for skill in skills:
if skill.startswith("write-script-"):
lang_key = skill.replace("write-script-", "")
metadata = LANGUAGE_METADATA.get(lang_key)
if metadata:
desc_map[skill] = metadata["description"]
return desc_map
def _looks_like_windmill_manifest(path: Path) -> bool:
"""Return True iff `path` is a JSON file whose top-level `name` is ours.
Used to distinguish a previously-generated docs repo from an unrelated
project that happens to have a `manifest.json` (Chrome extensions, npm
packages, web app manifests, etc.).
"""
try:
data = json.loads(path.read_text())
except (OSError, json.JSONDecodeError):
return False
return isinstance(data, dict) and data.get("name") == CONTEXT7_REPO_NAME
def _verify_context7_target(target_dir: Path) -> None:
"""Refuse to wipe a non-empty dir that doesn't look like the docs repo.
A typo such as `--context7-dir .`, `~`, or the wrong checkout could
otherwise nuke unrelated files. We accept the target if it's empty/new,
if it has our ownership file, if its `manifest.json` self-identifies as
the windmill-cli-docs repo, or if its git origin points at one.
"""
if not target_dir.exists() or not any(target_dir.iterdir()):
return
if (target_dir / "context7.json").exists():
return
manifest_path = target_dir / "manifest.json"
if manifest_path.exists() and _looks_like_windmill_manifest(manifest_path):
return
git_dir = target_dir / ".git"
if git_dir.exists():
import subprocess
try:
origin = subprocess.run(
["git", "-C", str(target_dir), "config", "--get", "remote.origin.url"],
capture_output=True,
text=True,
check=True,
).stdout.strip()
if CONTEXT7_REPO_NAME in origin:
return
except subprocess.CalledProcessError:
pass
raise RuntimeError(
f"Refusing to overwrite {target_dir}: target does not look like the "
f"{CONTEXT7_REPO_NAME} docs repo.\n"
f"Expected one of:\n"
f" - a `context7.json` at the top level,\n"
f" - a `manifest.json` whose top-level `name` is {CONTEXT7_REPO_NAME!r},\n"
f" - a git remote `origin` containing '{CONTEXT7_REPO_NAME}'.\n"
f"If this is the right directory, add a `context7.json` and retry."
)
def clear_context7_dir(target_dir: Path) -> None:
"""Wipe the docs repo dir of previously generated content.
Preserves a small allowlist (.git, .github, LICENSE, context7.json, etc.)
so this can run against a real checkout without nuking version control or
CI config.
"""
if not target_dir.exists():
return
for entry in target_dir.iterdir():
if entry.name in CONTEXT7_PRESERVE:
continue
if entry.is_dir():
shutil.rmtree(entry)
else:
entry.unlink()
def _read_windmill_version() -> str | None:
"""Return the Windmill release version (e.g. '1.700.2'), or None if absent.
Sourced from `version.txt` at the repo root — the same file release-please
updates on every release.
"""
version_file = SCRIPT_DIR.parent / "version.txt"
if not version_file.exists():
return None
return version_file.read_text().strip() or None
def generate_context7_repo(
target_dir: Path,
skills: list[str],
schema_yaml_content: dict[str, str],
cli_commands_md: str,
) -> Path:
"""Generate a fully-rendered docs repo suitable for context7 ingestion.
Layout written to `target_dir`:
AGENTS.md # the prompt agents see in their projects
README.md # stable intro for humans / context7
manifest.json # version + skill list (for indexing)
cli-commands.md # full CLI flag reference
skills/<name>/SKILL.md # one rendered skill per file
"""
target_dir = target_dir.expanduser().resolve()
target_dir.mkdir(parents=True, exist_ok=True)
_verify_context7_target(target_dir)
clear_context7_dir(target_dir)
skill_desc_map = build_skill_desc_map(skills)
# AGENTS.md — the managed CLI guidance (what `wmill init` writes as
# AGENTS.wmill.md locally). Kept under the `AGENTS.md` filename here to
# preserve the existing context7 ingest path; docs consumers read this
# as the canonical AGENTS file.
(target_dir / "AGENTS.md").write_text(
render_agents_md_for_docs(skills, skill_desc_map)
)
# Full CLI reference at top level.
(target_dir / "cli-commands.md").write_text(cli_commands_md)
# One markdown per skill, with schemas inlined (no template placeholders).
skills_dir = target_dir / "skills"
skills_dir.mkdir(parents=True, exist_ok=True)
for skill_name in skills:
skill_dir = skills_dir / skill_name
skill_dir.mkdir(parents=True, exist_ok=True)
(skill_dir / "SKILL.md").write_text(
render_plugin_skill_content(skill_name, schema_yaml_content)
)
# Stable README so the GitHub repo landing page tells readers (and
# context7's crawler) what they're looking at.
(target_dir / "README.md").write_text(_context7_readme(skills))
# Machine-readable index for context7 / downstream consumers.
# Note: the `name` field is also the marker `_verify_context7_target`
# uses to distinguish our `manifest.json` from generic ones.
manifest = {
"name": CONTEXT7_REPO_NAME,
"description": (
"Auto-generated Windmill CLI docs: agent prompt, skills, and "
"full CLI reference. Source: github.com/windmill-labs/windmill."
),
"skills": [
{"name": name, "description": skill_desc_map.get(name, "")}
for name in skills
],
}
version = _read_windmill_version()
if version:
manifest["version"] = version
(target_dir / "manifest.json").write_text(
json.dumps(manifest, indent=2) + "\n"
)
print(f"\nGenerated for context7 docs repo:")
print(f" - {target_dir} ({len(skills)} skills + AGENTS.md + cli-commands.md)")
return target_dir
def _context7_readme(skills: list[str]) -> str:
"""Render the README that ships at the root of the docs repo.
Doubles as a CLI quickstart for humans landing on the GitHub page and as
the top-level entry point context7 indexes first — keep it actionable.
"""
skill_lines = "\n".join(f"- `skills/{name}/SKILL.md`" for name in skills)
return f"""# Windmill CLI Quickstart
[`wmill`](https://www.windmill.dev/docs/advanced/cli) is the official command
line interface for [Windmill](https://www.windmill.dev) — an open-source
platform for internal tools, workflows, API integrations, background jobs, and
UIs. Use it to authenticate against a workspace, scaffold local projects,
sync scripts/flows/apps between your filesystem and a workspace, and run or
debug jobs from your terminal.
## Install
```sh
npm install -g windmill-cli
wmill --version
```
Upgrade later with `wmill upgrade`.
## Connect to a workspace
```sh
wmill workspace add
```
This walks you through adding a workspace profile — a `(name, remote URL,
workspace id, token)` tuple stored under `~/.config/windmill`. You can have
multiple profiles and switch between them with `wmill workspace switch <name>`.
A workspace token is created from the Windmill UI under
`User Settings → Tokens`. For self-hosted instances, point the remote at your
own URL (e.g. `https://windmill.example.com`).
## Initialize a project directory
```sh
wmill init
```
`wmill init` creates:
- `wmill.yaml` — sync configuration (which folders/types to track).
- `AGENTS.md` + `CLAUDE.md` — the agent prompt published in this repo.
- `.claude/skills/` and `.agents/skills/` — per-task guides used by AI coding
assistants (Claude Code, Codex, Pi). These are the same `SKILL.md` files
you'll find under `skills/` in this repo.
It also offers to bind a workspace profile to the current git branch and to
import git-sync settings from the backend if any are configured.
## Sync between local files and a workspace
```sh
wmill sync pull # workspace → local (writes flows, scripts, apps, etc.)
wmill sync push # local → workspace
```
Sync is idempotent and diff-aware: `wmill sync push --dry-run` previews the
changes without applying them. Use `--yaml` (recommended) to keep specs as
YAML rather than JSON.
For individual entities you can also use the type-specific commands:
```sh
wmill script push path/to/script.ts
wmill flow push path/to/flow.yaml
wmill app push path/to/app.yaml
wmill resource push path/to/resource.yaml
```
## Run, inspect, and debug jobs
```sh
wmill script run u/me/my_script --data '{{"foo": "bar"}}'
wmill flow run u/me/my_flow --data @inputs.json
wmill job list --failed --limit 20
wmill job get <job_id>
wmill job logs <job_id>
```
Logs and flow steps stream as the job runs. For flow failures, `wmill job get`
shows the step tree with each sub-job's id so you can drill in with
`wmill job logs <sub_job_id>`.
## Scaffold new entities
```sh
wmill script new u/me/path --language bun
wmill flow new u/me/path --summary "..."
wmill app new u/me/path --summary "..." --framework svelte
```
These create the correct folder layout and a minimal spec file, then print
next-step hints. Prefer them over hand-creating the folders — they pick the
right naming conventions for your workspace.
## Triggers and schedules
Triggers (HTTP routes, WebSocket, Kafka, NATS, MQTT, SQS, GCP Pub/Sub, Azure
Event Hubs, Email, Postgres CDC) and cron schedules are tracked as YAML files
synced alongside your scripts and flows. See `skills/triggers/SKILL.md` and
`skills/schedules/SKILL.md` for the full schemas.
## Completion
```sh
source <(wmill completions bash) # bash, zsh: source <(wmill completions zsh)
source (wmill completions fish | psub) # fish
```
## Reference
- `cli-commands.md` — every `wmill` command and flag, generated from the
source.
- `AGENTS.md` — the top-level prompt the CLI installs into each project (and
the same instructions AI coding assistants follow when working in a
Windmill repo).
- `skills/<name>/SKILL.md` — one self-contained guide per common task.
### Skills index
{skill_lines}
## About this repo
Auto-generated mirror of the Windmill CLI's bundled AI-agent guidance and
command reference, published for ingestion by docs aggregators such as
[context7](https://context7.com).
**Do not edit by hand.** This repo is regenerated from
[windmill-labs/windmill](https://github.com/windmill-labs/windmill) on every
release. Open issues and PRs in the source repo, not here. The generator is
`system_prompts/generate.py --context7-dir`.
"""
# =============================================================================
# Main Entry Point
# =============================================================================
def parse_args() -> argparse.Namespace:
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description=(
"Generate Windmill system prompts, CLI guidance, and optionally "
"plugin-ready standalone skills."
)
)
parser.add_argument(
"--plugin-dir",
type=Path,
help=(
"Optional plugin target. Accepts a windmill-claude-plugin repo root, "
"a plugin root, or a skills directory, and refreshes standalone skills there."
),
)
parser.add_argument(
"--context7-dir",
type=Path,
help=(
"Optional path to a docs-repo checkout (e.g. windmill-cli-docs). "
"Writes AGENTS.md, cli-commands.md, skills/, README.md, and manifest.json "
"with all placeholders resolved, suitable for context7 ingestion."
),
)
return parser.parse_args()
def main():
"""Main generation function."""
args = parse_args()
print("Generating system prompts documentation...")
# Ensure output directories exist
OUTPUT_SDKS_DIR.mkdir(parents=True, exist_ok=True)
OUTPUT_GENERATED_DIR.mkdir(parents=True, exist_ok=True)
# Read SDK files
ts_content = ''
if TS_SDK_DIR.exists():
for ts_file in sorted(TS_SDK_DIR.glob('*.ts')):
if not ts_file.name.endswith('.d.ts'):
ts_content += ts_file.read_text() + '\n'
py_content = PY_SDK_PATH.read_text() if PY_SDK_PATH.exists() else ''
openflow_raw = OPENFLOW_SCHEMA_PATH.read_text() if OPENFLOW_SCHEMA_PATH.exists() else ''
# Extract only components.schemas from OpenFlow and convert to minified JSON
openflow_yaml = yaml.safe_load(openflow_raw) if openflow_raw else {}
openflow_schemas = openflow_yaml.get('components', {}).get('schemas', {})
openflow_schemas_json = json.dumps(openflow_schemas, separators=(',', ':'))
openflow_content = f"## OpenFlow Schema\n\n{openflow_schemas_json}"
# Extract TypeScript SDK info
print("Parsing TypeScript SDK...")
ts_functions = extract_ts_functions(ts_content)
ts_types = extract_ts_types(ts_content)
ts_sdk_md = generate_ts_sdk_markdown(ts_functions, ts_types)
(OUTPUT_SDKS_DIR / "typescript.md").write_text(ts_sdk_md)
print(f" Found {len(ts_functions)} functions, {len(ts_types)} types")
# Extract Python SDK info
print("Parsing Python SDK...")
py_functions = extract_py_functions(py_content)
py_classes = extract_py_classes(py_content)
py_sdk_md = generate_py_sdk_markdown(py_functions, py_classes)
(OUTPUT_SDKS_DIR / "python.md").write_text(py_sdk_md)
print(f" Found {len(py_functions)} functions, {len(py_classes)} classes")
# Extract datatable-specific SDK docs (for app mode system prompt)
print("Extracting datatable SDK docs...")
datatable_ts_md = extract_datatable_ts_sdk()
datatable_py_md = extract_datatable_py_sdk(py_content)
(OUTPUT_SDKS_DIR / "datatable-typescript.md").write_text(datatable_ts_md)
(OUTPUT_SDKS_DIR / "datatable-python.md").write_text(datatable_py_md)
# Extract Workflow-as-Code SDK docs (for WAC skills and prompt helpers)
print("Extracting Workflow-as-Code SDK docs...")
wac_ts_md = extract_wac_ts_sdk(ts_content)
wac_py_md = extract_wac_py_sdk(py_content)
(OUTPUT_SDKS_DIR / "wac-typescript.md").write_text(wac_ts_md)
(OUTPUT_SDKS_DIR / "wac-python.md").write_text(wac_py_md)
# Read base prompts
print("Assembling complete prompts...")
base_dir = SCRIPT_DIR / "base"
languages_dir = SCRIPT_DIR / "languages"
script_base = read_markdown_file(base_dir / "script-base.md")
flow_base = read_markdown_file(base_dir / "flow-base.md")
resources_base = read_markdown_file(base_dir / "resources.md")
raw_app_base = read_markdown_file(base_dir / "raw-app.md")
pipeline_base = read_markdown_file(base_dir / "pipeline-base.md")
workflow_as_code_base = read_markdown_file(base_dir / "workflow-as-code.md")
flow_cli = read_markdown_file(base_dir / "flow-cli.md")
flow_chat_special_modules = read_markdown_file(base_dir / "flow-chat-special-modules.md")
# Read language files
languages = {}
for lang_file in sorted(languages_dir.glob("*.md")):
languages[lang_file.stem] = lang_file.read_text()
# Extract and generate CLI commands documentation
print("Extracting CLI commands...")
cli_data = extract_cli_commands()
cli_commands = generate_cli_commands_markdown(cli_data)
# Append hand-written CLI guidance covering bits that aren't obvious from
# the auto-generated per-command --help (file_key semantics, --storage,
# workspace scope). The cli-commands skill is the entry point agents read
# to learn about `wmill`, so non-obvious usage notes belong here.
object_storage_cli = read_markdown_file(base_dir / "object-storage-cli.md")
if object_storage_cli:
cli_commands = f"{cli_commands}\n\n{object_storage_cli}"
OUTPUT_CLI_DIR.mkdir(parents=True, exist_ok=True)
(OUTPUT_CLI_DIR / "cli-commands.md").write_text(cli_commands)
print(f" Found {len(cli_data['commands'])} commands, {len(cli_data['global_options'])} global options")
# Extract schemas from backend OpenAPI for CLI format documentation
print("Extracting backend OpenAPI schemas...")
cli_schemas = {}
backend_schemas = {}
if BACKEND_OPENAPI_PATH.exists():
backend_openapi_raw = BACKEND_OPENAPI_PATH.read_text()
backend_openapi = yaml.safe_load(backend_openapi_raw)
backend_schemas = backend_openapi.get('components', {}).get('schemas', {})
# Extract and transform schemas for CLI format (removing server-managed fields)
schema_names = [
'Schedule', 'NewSchedule',
'HttpTrigger', 'NewHttpTrigger',
'WebsocketTrigger', 'NewWebsocketTrigger',
'KafkaTrigger', 'NewKafkaTrigger',
'NatsTrigger', 'NewNatsTrigger',
'PostgresTrigger', 'NewPostgresTrigger',
'MqttTrigger', 'NewMqttTrigger',
'AmqpTrigger', 'NewAmqpTrigger',
'SqsTrigger', 'NewSqsTrigger',
'GcpTrigger',
'AzureTrigger',
'EmailTrigger', 'NewEmailTrigger',
]
for schema_name in schema_names:
if schema_name in backend_schemas:
cli_schemas[schema_name] = extract_cli_schema(backend_schemas[schema_name], backend_schemas, openflow_schemas)
print(f" Extracted {len(cli_schemas)} schemas for CLI format")
else:
print(f" Warning: Backend OpenAPI file not found at {BACKEND_OPENAPI_PATH}")
# Generate standalone schema files for triggers and schedules
schema_yaml_content = generate_schema_files(cli_schemas)
generate_workspace_tool_zod_schemas(backend_schemas, openflow_schemas)
# Assemble prompts for export
prompts = {
# Base prompts
'SCRIPT_BASE': script_base,
'FLOW_BASE': flow_base,
'RESOURCES_BASE': resources_base,
'RAW_APP_BASE': raw_app_base,
'PIPELINE_BASE': pipeline_base,
'WORKFLOW_AS_CODE_BASE': workflow_as_code_base,
'FLOW_CHAT_SPECIAL_MODULES': flow_chat_special_modules,
# SDKs
'SDK_TYPESCRIPT': ts_sdk_md,
'SDK_PYTHON': py_sdk_md,
'WAC_SDK_TYPESCRIPT': wac_ts_md,
'WAC_SDK_PYTHON': wac_py_md,
# Datatable-specific SDK docs (for app mode)
'DATATABLE_SDK_TYPESCRIPT': datatable_ts_md,
'DATATABLE_SDK_PYTHON': datatable_py_md,
# Schema (raw YAML content)
'OPENFLOW_SCHEMA': openflow_content,
# CLI
'CLI_COMMANDS': cli_commands,
}
# Add language prompts
for lang_name, lang_content in languages.items():
prompts[f'LANG_{lang_name.upper()}'] = lang_content
# Generate TypeScript exports
ts_exports = generate_ts_exports(prompts)
(OUTPUT_GENERATED_DIR / "prompts.ts").write_text(ts_exports)
(OUTPUT_GENERATED_DIR / "prompts.d.ts").write_text(generate_ts_declarations(prompts))
# Generate complete script.md (all languages combined)
script_md_parts = [script_base]
for lang_name in sorted(languages.keys()):
script_md_parts.append(languages[lang_name])
script_md_parts.extend([ts_sdk_md, py_sdk_md])
script_md = "\n\n".join(filter(None, script_md_parts))
(OUTPUT_GENERATED_DIR / "script.md").write_text(script_md)
# Generate complete flow.md
flow_md_parts = [flow_base, openflow_content]
flow_md = "\n\n".join(filter(None, flow_md_parts))
(OUTPUT_GENERATED_DIR / "flow.md").write_text(flow_md)
# Generate an index file
index_content = """// Auto-generated by generate.py - DO NOT EDIT
// Re-export all prompts
export * from './prompts';
import * as prompts from './prompts';
// Languages that use the TypeScript SDK
const TS_SDK_LANGUAGES = ['bun', 'deno', 'nativets', 'bunnative'];
// Languages that use the Python SDK
const PY_SDK_LANGUAGES = ['python3'];
// Languages that use the TypeScript Workflow-as-Code SDK
const WAC_TS_SDK_LANGUAGES = ['bun'];
// Languages that use the Python Workflow-as-Code SDK
const WAC_PY_SDK_LANGUAGES = PY_SDK_LANGUAGES;
// Helper to combine prompts for scripts
export function getScriptPrompt(language: string): string {
const langKey = `LANG_${language.toUpperCase()}` as keyof typeof prompts;
const langPrompt = (prompts as Record<string, string>)[langKey] || '';
// Determine which SDK to include based on language
let sdkPrompt = '';
if (TS_SDK_LANGUAGES.includes(language)) {
sdkPrompt = prompts.SDK_TYPESCRIPT;
} else if (PY_SDK_LANGUAGES.includes(language)) {
sdkPrompt = prompts.SDK_PYTHON;
}
return [
prompts.SCRIPT_BASE,
langPrompt,
sdkPrompt
].filter(Boolean).join('\\n\\n');
}
// Helper to combine prompts for flows
export function getFlowPrompt(): string {
return [
prompts.FLOW_BASE,
prompts.OPENFLOW_SCHEMA
].filter(Boolean).join('\\n\\n');
}
// Helper for resource & variable authoring
export function getResourcePrompt(): string {
return prompts.RESOURCES_BASE;
}
// Helper for raw app authoring (chat consumers). Inline backend runnables are
// ordinary Windmill jobs, so the reference has to carry the SDK the runnable
// calls — without it an agent invents client functions and hand-rolls HTTP.
// Only one SDK is returned: both would double an already large tool result.
export function getRawAppPrompt(language?: string): string {
const sdkPrompt = PY_SDK_LANGUAGES.includes(language ?? '')
? prompts.SDK_PYTHON
: prompts.SDK_TYPESCRIPT;
return [
prompts.RAW_APP_BASE,
sdkPrompt
].filter(Boolean).join('\\n\\n');
}
// Helper for data pipeline authoring (chat consumers)
export function getPipelinePrompt(): string {
return prompts.PIPELINE_BASE;
}
// Helper to get the datatable SQL SDK reference (wmill.datatable()).
// Pass a language to get only that SDK; omit it to get both.
export function getDatatableSdkReference(language?: string): string {
if (language == null) {
return [
prompts.DATATABLE_SDK_TYPESCRIPT,
prompts.DATATABLE_SDK_PYTHON
].filter(Boolean).join('\\n\\n');
}
if (TS_SDK_LANGUAGES.includes(language)) {
return prompts.DATATABLE_SDK_TYPESCRIPT;
}
if (PY_SDK_LANGUAGES.includes(language)) {
return prompts.DATATABLE_SDK_PYTHON;
}
// Unknown language: return both rather than nothing.
return [
prompts.DATATABLE_SDK_TYPESCRIPT,
prompts.DATATABLE_SDK_PYTHON
].filter(Boolean).join('\\n\\n');
}
// Helper to combine prompts for Workflow-as-Code scripts
export function getWorkflowAsCodePrompt(language?: string): string {
let sdkPrompt = '';
if (language == null) {
sdkPrompt = [
prompts.WAC_SDK_TYPESCRIPT,
prompts.WAC_SDK_PYTHON
].filter(Boolean).join('\\n\\n');
} else if (WAC_TS_SDK_LANGUAGES.includes(language)) {
sdkPrompt = prompts.WAC_SDK_TYPESCRIPT;
} else if (WAC_PY_SDK_LANGUAGES.includes(language)) {
sdkPrompt = prompts.WAC_SDK_PYTHON;
} else {
return '';
}
return [
prompts.WORKFLOW_AS_CODE_BASE,
sdkPrompt
].filter(Boolean).join('\\n\\n');
}
"""
(OUTPUT_GENERATED_DIR / "index.ts").write_text(index_content)
index_dts_content = """export * from './prompts';
export declare function getScriptPrompt(language: string): string;
export declare function getFlowPrompt(): string;
export declare function getResourcePrompt(): string;
export declare function getRawAppPrompt(language?: string): string;
export declare function getPipelinePrompt(): string;
export declare function getDatatableSdkReference(language?: string): string;
export declare function getWorkflowAsCodePrompt(language?: string): string;
"""
(OUTPUT_GENERATED_DIR / "index.d.ts").write_text(index_dts_content)
# Generate skill files for Claude Code
CLI_GUIDANCE_DIR.mkdir(parents=True, exist_ok=True)
skills = generate_skills(
languages=languages,
ts_sdk_md=ts_sdk_md,
py_sdk_md=py_sdk_md,
wac_ts_md=wac_ts_md,
wac_py_md=wac_py_md,
flow_cli=flow_cli,
flow_base=flow_base,
cli_commands=cli_commands,
openflow_content=openflow_content,
cli_schemas=cli_schemas
)
# Generate skills TypeScript export for CLI
skills_ts = generate_skills_ts_export(skills, schema_yaml_content)
# Replace hardcoded path conventions with placeholders for CLI runtime resolution.
# init.ts resolves these based on the nonDottedPaths setting in wmill.yaml.
# (Frontend auto-generated files keep the default non-dotted conventions.)
skills_ts = (skills_ts
.replace("\\`__flow\\`", "\\`{{FLOW_SUFFIX}}\\`")
.replace(
"Inline script files should NOT include \\`.inline_script.\\`"
" in their names (e.g. use \\`a.ts\\`, not \\`a.inline_script.ts\\`).",
"{{INLINE_SCRIPT_NAMING}}"
)
.replace("my_flow__flow", "my_flow{{FLOW_SUFFIX}}")
.replace("my_app__raw_app/", "my_app{{RAW_APP_SUFFIX}}/")
)
(CLI_GUIDANCE_DIR / "skills.gen.ts").write_text(skills_ts)
print(f"\nGenerated files:")
print(f" - auto-generated/sdks/typescript.md")
print(f" - auto-generated/sdks/python.md")
print(f" - auto-generated/sdks/wac-typescript.md")
print(f" - auto-generated/sdks/wac-python.md")
print(f" - auto-generated/cli/cli-commands.md (auto-generated from CLI source)")
print(f" - auto-generated/prompts.ts")
print(f" - auto-generated/prompts.d.ts")
print(f" - auto-generated/index.ts")
print(f" - auto-generated/script.md")
print(f" - auto-generated/flow.md")
print(f" - auto-generated/skills/ ({len(skills)} skills)")
print(f" - auto-generated/schemas/ ({len(schema_yaml_content)} schema files)")
print(f"\nGenerated for CLI:")
print(f" - cli/src/guidance/skills.gen.ts")
if args.plugin_dir:
generate_plugin_skills(args.plugin_dir, skills, schema_yaml_content)
if args.context7_dir:
generate_context7_repo(
args.context7_dir, skills, schema_yaml_content, cli_commands
)
print("\nDone!")
if __name__ == '__main__':
main()