mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-21 08:02:38 +00:00
0df128cc93cd314bcab112ebd32424fdd40b7d14
78
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  hugocasaandClaude Opus 5
|
37e493ae66 |
feat: add an instance setting to refuse a token in MCP URLs (#11162)
* feat: add an instance setting to refuse a token in MCP URLs
MCP clients are commonly configured with the token in the URL
(`/api/mcp/w/{workspace}/mcp?token=...`). A URL-borne credential ends up in
browser history, proxy logs and referrers, so an instance can now turn that
channel off with the `mcp_disable_token_query_param` global setting and leave
the Authorization header as the only way in, which sends MCP clients through
the OAuth flow the endpoints already advertise.
The rejection is a middleware on both the workspaced and the gateway MCP
mounts, layered outside everything that reads a token and inside the
WWW-Authenticate layer, so the 401 carries the resource pointer a client needs
to start OAuth discovery.
Off by default. With it on, the token drawer and the home connect drawer stop
offering to mint a token for an MCP URL and hand over the bare URL instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: read the MCP URL policy when a URL is asked for, and drop the all-workspaces option
Two review findings on the token drawer:
The policy was read once per page load and cached for the browser session, so a
superadmin turning the setting on left every open tab handing out `?token=` URLs
the server now refuses. Both entry points now read it when the user actually asks
for an MCP URL: when MCP mode is entered, and when the connect drawer opens.
The workspace picker offered "All workspaces / Multi-workspace", but the gateway's
consent screen binds the token it issues to the one workspace picked there, so
OAuth has no multi-workspace grant to hand out. That entry is now token-only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: don't guess the MCP URL policy, and say where the switch lands on restart
Review findings:
The comment on the settings load claimed `MODE=mcp` as the target deployment, but
that mode joins no monitor loop, so the startup pass is its only read and a change
lands on restart. That is true of every global setting there, `base_url` included;
the comment now says so, and the setting description tells an operator running
dedicated MCP servers what to expect.
A failed settings probe resolved to "tokens allowed", so with the switch on the
drawer would mint a non-expiring token and hand over a URL the server refuses for
as long as it exists. The probe now propagates its error and the panel reports it
with a retry, creating nothing until the answer is known.
The test passed a valid token, so it could not tell a rejection before
authentication from one after it. It now also sends a token that was never valid
and asserts the middleware's own message, which fails if the layer moves inward.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: withhold the MCP URL until a workspace is picked
With no persisted workspace the store starts undefined, so opening the drawer from
/user/workspaces before choosing one rendered a copyable
`/api/mcp/w/undefined/mcp`. It reads like a real URL and a client pointed at it
would never connect. The panel now asks for a workspace instead, matching the guard
the token branch already has on its generate button.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: drop the coverage-status note from the MCP switch test
It documented what the test does not reach rather than a constraint the next
reader could break; that belongs in the PR, not the module doc. The layer-order
rationale, which is what a future edit would break, stays.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor: fall back to the bare MCP URL instead of alerting on a failed read
When the setting read fails, show the bare URL rather than an error with a retry.
It works whichever way the setting is, so no alert is needed, and it still never
mints a token for a URL the server may refuse. The connect drawer's wording falls
back the same way so the blurb matches the panel.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor: move the MCP URL token setting to Core
It sat in the Auth/OAuth/SAML list, which the settings sidebar shows under SSO,
suggesting a dependency on SSO that does not exist: MCP OAuth has Windmill act as
the authorization server, and any login method, password included, completes it.
It is an instance-wide credential policy, so it now lives with the other ones in
Core, kept out of quick setup like its neighbours.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
hugocasaandClaude Opus 5
|
9320312eac |
feat: cap user token expiration with an instance setting (#11159)
* feat: cap user token expiration with an instance setting Adds `max_token_expiration_days`, an instance-wide ceiling on how far ahead a token created through `POST /users/tokens/create` may expire. With it set, that route refuses a token with no expiration and one that expires past the window; absent or non-positive, nothing changes. Only the user-facing handler enforces it. Server-side mints (native trigger webhook tokens, app embed tokens, sessions) pick a lifetime the caller never chooses and go straight to `create_token_internal`, so they stay uncapped, as does the superadmin `impersonate` route. Service accounts are exempt, in the workspace the token targets or in any workspace for a global token, so unattended automation can keep longer-lived credentials. The token form now surfaces the API error instead of only logging it, and offers "Expires In" in MCP mode as well: that mode always sent no expiration, which the cap refuses, leaving MCP URLs impossible to generate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: shorten over-long token expirations instead of refusing them Refusing a non-compliant request breaks the callers that cannot comply. The CLI authorization page, `wmill user create-token` and the editor's language-server token each pick a lifetime — usually none at all — with no way to read the setting, so a cap made browser login hang and the editor lose its LSP root rather than stopping the long-lived tokens the setting is aimed at. `cap_token_expiration` now returns the expiration to store, shortening a request that asks for too long or for none. The policy still holds absolutely, no caller can break, and there is no clock-skew boundary where an expiration exactly at the ceiling flips to an error. The token form needed no changes at all, so its MCP and error-toast edits are gone with it. Also drops the Enterprise badge on the setting, which nothing enforced, notes the mint paths in docs/auth-surface.md, and pins that `tokens/impersonate` and the second-workspace case stay outside the exemption. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: drop the unintended token-form change and correct the exemption docs The token form needed no change once the ceiling shortens rather than refuses, but the earlier revert restored from the index, which already held the staged edit, so the MCP expiration field and the error toast stayed on the branch with a comment justifying them by a refusal that no longer happens. docs/auth-surface.md claimed a service-account row in any workspace exempts outright; that only holds for a workspace-less token, which has no workspace to match. A ceiling written as a string, which the YAML instance config and config sync can both produce, now has a test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: offer only expirations within the ceiling in the token form The server shortens a token that asks for longer than `max_token_expiration_days` or for no expiration, which the token form could not tell anyone: a user picking "No expiration" got the ceiling silently. The form now reads the setting and, with one set, drops "No expiration" and every choice above it, adds the ceiling itself as "N days (maximum)" and selects it, and says the instance limits tokens to N days. MCP mode hides the expiration field and always sent none, so with a ceiling the field now shows there too and keeps its value across the toggle. Without a ceiling the form is unchanged. Reading it needs no superadmin: the setting joins the keys any logged-in user can read through `GET /settings/global/{key}`. It holds a policy, not a secret. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: make the token form and the server agree on what counts as a ceiling The form parsed `max_token_expiration_days` more loosely than `cap_token_expiration`, so the two could disagree on whether a ceiling exists at all. A `7.0` from the YAML instance config, or a string such as "7.0" or "1e1", made the form hide "No expiration" and announce a 7-day limit while the server capped nothing; a value between chrono's and JavaScript's date limits preselected an expiration the server could not parse. Both now read the same thing as a ceiling: a whole number of days from 1 to 1,000,000, stored as an integer, an integral float or a string of digits. `parseMaxTokenExpirationDays` holds the frontend's copy, and the instance settings validation uses it too, so the settings page no longer accepts a value the server would ignore. The bound replaces the date-range guard on both sides. Also corrects the rationale for shortening rather than refusing: the setting is now readable by any logged-in user, so those callers do not read it rather than cannot, and CLIs already installed never will. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: cap service-account tokens like everyone else's The ticket exempted service accounts from `max_token_expiration_days`, but their tokens are the long-lived ones a rotation policy is meant to bound, and the exemption let any workspace admin get an uncapped token by impersonating one. It also left the token form unable to agree with the server: an admin impersonating a service account was offered only capped choices while the server would have kept any. `cap_token_expiration` now takes just the requested expiration, with no per-caller lookup, and the service-account query and its cache entry are gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: cap superadmin impersonation tokens and pin the frontend parser `POST /users/tokens/impersonate` wrote its own token row with whatever expiration the superadmin sent, so it was the one route left that could mint a token that never expires with `max_token_expiration_days` set. The ceiling only decides the stored expiration (the auth lookup never reads the setting), so leaving it uncapped meant exactly that. It now goes through `cap_token_expiration` like `create_token`; nothing in Windmill calls it, so no caller changes. Also adds `tokenExpiration.test.ts`, pinning which stored values `parseMaxTokenExpirationDays` reads as a ceiling against the server's reading, and documents that tokens existing when the setting is turned on or lowered keep their expiration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: reject a max_token_expiration_days the token routes cannot read The settings API and config sync stored any value for the key, and the token routes can only read an unparseable one as no ceiling. A typo such as `7.5` or "7.0" was accepted and silently turned the policy off. `parse_max_token_expiration_days` in windmill-common is now the single server reading of the setting: null or empty clears it, a whole number of days within the bound is the ceiling, anything else is an error. The settings write hook and `sync_global_settings_declarative` reject that error, and `cap_token_expiration` reads through the same function, logging a value written around both. Tests: the parser's accept/clear/reject table (the same table as the frontend parser's), the settings API refusing 7.5, and config sync refusing "7.0". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: reserve the CLI login token label so its expiry does not email the user With a token expiration ceiling, the token the CLI authorization page mints now expires, so every `wmill` login earned an "expiring soon" and an "expired and deleted" email and critical alert. The CLI already signs in again on its own when that token stops working, so those notifications ask the user to do nothing. The page now labels it `cli-login:<username>` (previously `cli-<username>`), reserved in `is_user_token` and its SQL and Svelte mirrors: no expiry notifications, and the label cannot be edited. A colon-terminated namespace like `embed_app:` and `impersonation:` keeps hand-made labels clear of it. Not in `is_server_minted_label`, since the page mints through `/users/tokens/create`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: skip the expiring-soon warning for tokens that were short-lived from the start A token whose whole lifetime fits in the 7-day warning window got its "expiring soon" email (and critical alert, when enabled) minutes after it was created, about a lifetime its creator had just picked. With an expiration ceiling of 7 days or less that is every token created from the form or `wmill token create`. `register_token_expiry_notification` no longer queues a warning for such a token. The window is now `TOKEN_EXPIRY_WARNING_DAYS`, shared with `check_expiring_tokens`, so shortening the warning window can never leave tokens of an intermediate lifetime with no warning at all. The "expired and deleted" notice still goes out for every user token. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: exempt service-account tokens from the expiration ceiling again Service accounts are the identity automation that needs a long-lived credential runs as, so their tokens are exempt from `max_token_expiration_days` once more: a service account in the workspace the token names, or in any workspace for a workspace-less token. `tokens/impersonate` checks the impersonated account, so a superadmin minting a token for a service account gets the same exemption. The token form applies the same rule for the account it is running as, when that account is a service account in the current workspace, which is what an admin impersonating one sees; otherwise it would offer only capped choices while the server keeps any. Any workspace admin can create and impersonate a service account to hold an uncapped token, so the ceiling bounds personal tokens; the doc comment and docs/auth-surface.md say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: decide the token form's service-account exemption from the token's workspace The form treated the account as a service account only when it was one in the workspace the app was on, while the server checks the workspace the token is for, or any workspace for a workspace-less token. With an email that is a service account in one workspace and an ordinary member of another, picking the other workspace in MCP mode offered "No expiration" and the server silently stored the ceiling; the reverse hid the exemption. `GET /workspaces/users` now returns each membership's `is_service_account` (its query already joins the `usr` row), and the form applies the server's rule to the token's own workspace. The selection becomes a derived value held within the ceiling, so switching to a capped workspace never leaves an unoffered choice selected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
 
|
0e807fb1dd |
feat: put a data table's connection under Postgres roles (#11020)
* feat(datatables): put a data table's connection under Postgres roles
A data table backed by the instance database resolved to exactly one Postgres connection,
`custom_instance_user`, for everyone who could reach it at all. There was no way to say
this job reads, that one writes, this one never sees the salaries table.
A data table role is now a real Postgres login on the cluster, defined once for the
instance by a superadmin and named exactly as they named it. A script that declares
`-- role analytics` connects as `analytics`, and Postgres decides what it may touch —
grants are ordinary SQL. Windmill answers only "may this caller ask for this role", from
the tenant lists on the data table entry: `u/alice`, `g/analysts`, `f/finance` or `*`.
A data table with no `permissions` block behaves exactly as before.
Everything that opens a connection on someone's behalf goes through one chokepoint,
`get_datatable_resource_from_db`, which takes the identity explicitly and fails closed when
there is none. The role logs in as itself — never `SET ROLE`, which a script could
`RESET ROLE` its way out of.
A fork's data table entry becomes a pointer at the workspace that governs it rather than a
copy of it. The settings clone used to hand a fork a byte-identical entry naming the
parent's database, which a fork admin could edit to grant themselves `admin` there; a
pointer has nothing local to edit, and its tenants are evaluated as a member of the
governing workspace, by email. `permissions` is stripped from the workspace export and
ignored on import: tenants name principals of one workspace, and a settings push is not
where an access decision should be made.
Operations that see the whole database whatever the roles grant stay with the governing
workspace's admins: editing the roles, a migration that declares none, and opening a
replication stream for a Postgres trigger or capture.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): gate the paths that reach a whole database as admin
Auditing what still resolved through the unchecked resolver turned up three that act for a
caller and hand back the admin connection: `resolve_pg_source_checked` (behind schema
export, the full-schema read, database creation, import and the forked-database drop), the
connection test, and the schema snapshot a fork clone takes of its parent. On a data table
under roles each let any workspace member — or a fork admin who is nobody in the governing
workspace — read or copy the whole database whatever its roles grant.
All three now require admin reach on the governing workspace. A dump taken under a
restricted role would be a silently truncated copy rather than an error, so refusing is the
only right answer for the copy paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): confine roles to the instance database, and stop a fork reaching the parent's bookkeeping
A data table role is a login on Windmill's own Postgres. Nothing stopped a workspace admin
putting a *resource-backed* data table under roles, at which point the executor dialled the
host that resource names — one the admin chose — with the role's real cluster password, and
`CONNECT` is granted to every registered instance database. Both ends now refuse: the
permissions endpoint rejects the save, and the chokepoint refuses to substitute credentials
on a non-instance entry rather than trusting the record it read.
Two more places reached the governing database without answering to it. The initial-migration
generator returned a `pg_dump` of the whole schema to any member. And the migration
rename/delete cascade followed a fork's pointer into the parent, so a fork admin renaming or
removing their own local entry relabelled or wiped the parent's `_wm_migrations` — after
which the parent re-runs every migration from zero. The remote half is now skipped when the
entry resolves into another workspace, which is also just correct: a fork renaming what it
calls a data table changes nothing about the data table.
Also: revoking a tenant now bounces the replication streams of every workspace holding an
entry that resolves here, not only the governing one, so a fork's trigger stops rather than
living on inside its open connection; the instance role catalog and the governing workspace's
tenant lists are no longer returned to someone who cannot edit them; and the tenant rename
dedup collapses non-adjacent duplicates, per role rather than once any role changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): fail loudly where a role or a pointer can be left half-recorded
Three ways the feature could end up in a state nobody could see or undo.
Creating a role writes the cluster first and the catalog second, but the catalog write was an
`UPDATE` that matched nothing when the instance Postgres settings row was absent — leaving a
live login with a password nobody recorded: invisible to the catalog, un-recreatable because
the name is taken, and un-deletable because there is no entry to delete. It now errors, so
the operation is retryable once the row is restored.
Deleting a workspace only nulls the fork lineage; the data table entries pointing at it are
left resolving to nothing. Sweeping them is not an option — turning a pointer back into a copy
would hand each fork the database outright — so the delete now names the data tables it
stranded, and resolving one says which workspace is missing rather than reporting a data table
this workspace never had.
`InstanceDatatableRole` derived `Debug` while holding a Postgres password; it is now
hand-written so `{:?}` on the catalog cannot put a live credential in a log line.
Adds the two branches the reviews found unpinned: a caller who is not a member of the
governing workspace at all, and `NoIdentity` — the compatibility path for an agent worker that
predates this and sends no job id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): unbreak two operator messages and two comments that described other code
The two strings this branch added for states an operator hits once — the catalog write that
matched nothing, and the delete that stranded a pointer — were collapsed from their multi-line
form with the indentation left in, so both rendered with a fourteen-space gap mid-sentence.
`list_datatables` claimed to report a chain it cannot follow and then dropped it; it does drop
it, and the comment now says why that is the right place to stay quiet. The non-superadmin
check in `edit_datatable_config` was introduced as also covering references, which it does not
and need not: `reference` is overwritten from the stored entry for every caller before the
check runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): serialize role catalog mutations, and state each helper's authorization contract
The catalog is one JSON document, so create, rename, enable and delete are all
read-modify-write. Two concurrent creates read the same snapshot, both succeed in the
cluster, and the second write drops the first — leaving a live Postgres login with a password
nobody recorded, which is the exact state the delete path exists to prevent. Every mutation
now runs in one transaction holding an advisory lock across the read, the cluster DDL and the
write, so a lost update cannot happen and a failure rolls the whole thing back. The DDL
helpers take that transaction rather than the pool, which is what makes the lock cover them.
Their statements moved off `sqlx::raw_sql`: the simple protocol is only needed for genuinely
multi-statement SQL, and its future is not `Send`, which an axum handler holding the
transaction requires. Each of these is one statement anyway.
The new cross-crate surface now says what callers must do. `read_role_catalog` returns
plaintext credentials; `create`/`rename`/`set_login`/`drop_instance_role` and
`converge_connect_grants` mutate cluster-wide state; `read_datatable_entry` reads a workspace's
raw config. All of them are superadmin-gated by their current handlers, but nothing said so at
the definition, which is where the next caller looks.
Also: the roles table reloads after a failed login toggle instead of leaving it claiming a flip
that did not land; the rename affordance is the design-system `Button`, not a raw one; and
`resolve_datatable_pg_as_caller` drops a `role` parameter no caller ever filled — browsing
resolves as the data table's default until the database manager grows a picker.
Why role passwords stay a plain `String` while the instance user's password beside them is a
`StringOrSecretRef`, asked three times across reviews: that one is a secret ref because an
operator supplies it and may want it from their own backend, while these are minted here and
never entered by anyone, so there is nothing for a ref to point at. Encrypting generated
secrets at rest is a separate change that would take the replication password with it. Now
said at the field.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): give the role catalog its own row, out of reach of the config machinery
Putting it inside `custom_instance_pg_databases` was the wrong call, and it cost two ways.
The catalog serializes a generated Postgres password per role, and that row is the
operator-facing instance config, so the passwords reached `get_instance_config` and its YAML
editor — a live cluster credential in a response body, a UI field and any log of either.
Worse in the other direction: `to_settings_map` strips the catalog, so a full-row upsert of
that key writes the row back without it and the catalog is gone, while the cluster keeps every
login it described.
`custom_instance_replication_pwd` is the precedent and says exactly why — a generated secret,
written only by the server, never operator-authored, hidden so the config machinery cannot
read, rewrite or drop it. The catalog is the same thing, so it now has the same shape:
`datatable_roles`, in `HIDDEN_SETTINGS`, `PROTECTED_SETTINGS` and the agent-worker denylist.
No redaction to keep in step with three code paths, and no way for a neighbouring write to
take it out.
Two races on the same shared documents. `edit_datatable_config` read the stored data tables
outside its transaction and then wrote the whole `datatable` document, so a permissions save
committing in between was silently rolled back; it now reads under `FOR UPDATE`. And
`set_datatable_permissions` validated role ids against the catalog before opening its
transaction, so a deletion in between let it write a deleted role back — including as the
default, which every later job then fails on; it now holds the catalog lock and the settings
row across validation and write.
Completes the authorization contracts the previous commit claimed but did not finish:
`read_datatable_entry` (which it named and missed), `resolve_governing_datatable`, whose whole
job is to answer for a workspace the caller may not belong to, and
`converge_connect_grants_with`, which had not inherited its wrapper's.
Also the generic Python SDK reference: `_format_py_params` learned the bare `*` last time, but
`extract_py_functions` is a second formatter and still rendered `datatable(name, role)`, so
code written from that page passed a keyword-only argument positionally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): make the concurrency test pin the handlers, and the contracts describe what is enforced
The concurrency test reimplemented the read-modify-write inline, so deleting the lock from all
three handlers left it green — it pinned Postgres, not the code it was written for. It now
drives `create_datatable_role` twice concurrently and asserts the catalog kept both names.
Checked the way the last one should have been: removing the lock from the handler makes it
fail with "wmtest_a_… is a live cluster login the catalog forgot".
The contracts added last commit were stricter than this PR's own callers, which is worse than
none — the next reader sees a rule already broken and learns to ignore it.
`read_role_catalog` said superadmin-only while two of its four callers are open to any
workspace member, and `converge_connect_grants` said superadmin while
`set_datatable_permissions` reaches it as a workspace admin. Both were fine on substance: the
rule that actually holds is about the credential never reaching a response, log, audit record
or export, not about who may call. They now say that. `read_datatable_entry` gets the same
treatment rather than the one the earlier message claimed for it: it is the primitive every
resolution goes through, so it is deliberately open, and what must not escape is `permissions`
— it names the governing workspace's users, groups and folders.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): close the last ways a role or a pointer can be left pointing at nothing
The raw settings readers hand back whatever is in the row, so moving the catalog into its own
`global_settings` key protected the config machinery and left `GET /settings/global/datatable_roles`
and the settings listing returning every live password. Both now filter that one key. The
neighbouring `custom_instance_replication_pwd` has the same shape and is not touched here: it
predates this and widening the fix to it is a decision about an operator workflow, not a
consequence of this change.
Three ways a save could leave something resolving to nothing:
A permissioned data table could be moved to a PostgreSQL resource. The block was carried across
as a server-owned field, the runtime refuses roles on a resource-backed table, so the save
succeeded and every job afterwards failed. Refused instead — turning roles off first is one step,
and it keeps discarding an access decision something somebody chose.
Renaming a governing data table left every fork pointing at the old name: the data table
disappears from their pickers and their jobs stop, with nothing in the renaming workspace to
suggest why. The rename now follows into the pointers in the same transaction.
Deleting one cannot be followed the same way, so it is reported instead — the response names what
it stranded, the way deleting a workspace does, and the fork's own error already says which
workspace is gone.
Also: `ensure_instance_db_grant_options_unchecked` claimed superadmin while the permissions
handler reaches it as a workspace admin (the same class fixed last commit, one instance missed);
the role entry kept an `instance_config_schema` derive it no longer needs; `write_role_catalog`
was the one writer of that table not stamping `updated_at`; and the concurrency test dropped its
roles only on success — a failing run is exactly the one that creates them without recording them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* refactor(datatables): put the role catalog in its own table, not in global_settings
Five findings across three rounds were all the same choice. A set of live Postgres credentials
was living in `global_settings`, which has generic read, list, write, config-export and CLI
round-trip paths that know nothing about what they carry: the passwords reached the instance
config and its YAML editor, a full-row upsert of a neighbouring key erased the catalog,
`GET /settings/global/{key}` and the settings listing returned them raw, and this round the
redaction that fixed the last two turned `wmill instance push` into something that wipes every
password — a fix breaking the assumption the previous fix made. `POST /settings/global/datatable_roles`
could also empty it outside the lock.
The approved plan offered a table or `global_settings`, so this is the other option it already
allowed rather than a new design. `datatable_role` is a table: no generic settings path can read
it, list it, export it, write it or round-trip it, so none of the five needs a guard. The
redaction, the hidden/protected/agent-denylist entries and the JSON document all go with it.
One row per role also removes the read-modify-write the concurrency work was about: two
concurrent creates are two inserts, and the unique index on `name` is what settles a collision.
The advisory lock stays for the one window rows do not cover — `CREATE ROLE` is invisible to
another transaction until commit, so without it both creates pass their `pg_roles` check.
Also from this round: rename mappings are checked against the configuration they claim to
describe, since fork pointers are rewritten from them — a caller could otherwise submit
`main -> missing` against an unchanged config and repoint every fork of `main` at a name nothing
has, and `A -> B` plus `B -> C` moved what pointed at `A` all the way to `C`. And the warning
naming forks a delete stranded reached the response but not the screen: both the data table
settings save and the workspace delete now show it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): validate a rename against the save it describes, and re-check under the locks
Three from the round, all about deciding on state that could already have moved.
A permission save resolved the data table and checked it was instance-backed before taking any
lock, then wrote under one. A config save committing in between could move the table onto a
PostgreSQL resource — recreating exactly what the transition guard refuses — or rename it, in
which case the write targeted a key that no longer existed and reported success having changed
nothing. It now re-resolves and re-checks on the locked state.
Rename validation checked that the source existed before and the target existed after, which
still accepts `main -> decoy` against a save that keeps both: every fork of `main` then follows
onto a different data table, silently, because it keeps resolving. The rule is now the actual
old-to-new key transition — a source may only survive if another rename took its name, and a
target may only pre-exist if another rename freed it. That also stops two sources sharing one
target, and it admits a swap, which the previous guard refused: `datatables` is keyed by name, so
a swap cannot be done one save at a time, and refusing it was a regression against main. The
pointer cascade now runs in two passes through a temporary name, the way the migration cascade
one layer down already handles the same shape, so `A -> B` with `B -> C` moves each pointer once
from what it named before the save.
The tenant mutators say what they are for: they write an access decision for any workspace named,
with an arbitrary mutation, and exist for the transaction that frees or renames a principal.
Editing a decision on purpose belongs in the permissions endpoint.
Carried in the same change: the stranded-fork list is a field rather than a phrase to grep out of
a success string; the pointer cascade matches with `EXISTS` instead of a `LIKE` over the whole
document, so a workspace whose pointers name something else is not rewritten to a byte-identical
value under an exclusive lock; and `InstanceDatatableRole` drops the serde derives left over from
the JSON document, one of which would emit `pwd`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): cascade on the leave route that is used, gate migrations before the admin connection, and drop a role atomically
The tenant cascade on leaving went onto `/users/leave`. The UI and the generated client call
`/workspaces/leave` — a different handler in a different crate with the same name — which
deleted the membership and left `u/<username>` in the tenant lists. Leaving and rejoining
therefore restored the access the leave was supposed to end, and a later account taking the
username would have inherited it. The regression test drives the route the client actually
calls; without the fix it fails with "leaving kept the tenant".
The migration endpoints authorized too late. `run_datatable_migrations` opened the data table's
admin connection, created `_wm_migrations` and read it before reaching the per-migration role
check — so with nothing pending, nothing was checked at all. Rollback returned before its check
when nothing was applied, and the status endpoint had none. All three now ask, before any
connection is opened, whether the caller can reach the data table as any role at all; which role
a given migration runs as is still decided per migration, and by the executor after that.
Deleting a role committed the cluster drop and the catalog row, then swept the tenant lists in
separate transactions. A sweep failing part-way left workspaces naming a role nothing can connect
as, while the retry answered `NotFound` because the catalog entry was already gone. The sweep now
runs in the same transaction, so the drop, the row and every tenant list commit together.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): refuse to copy a data table that is under roles
pg_dump carries no roles and the import runs with --no-privileges, so a copied
data table arrives owned by the admin connection with no GRANT for any role.
The settings clone brings `permissions` across, so the fork's tenants pass
Windmill's check, connect as the role they were given, and are denied by
Postgres on everything: an entry that reads as configured and answers nothing.
Refuse the copy — in the import endpoint before any data moves, and in the fork
path the CLI takes. Replaying the source's owners and ACLs into the clone is
what lifts this, and is a change of its own. Dropping `permissions` from the
copy instead would be the unsafe half, since the copy holds the parent's rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): refuse the clone's database too, not only its data
A clone is two endpoints: `create_pg_database` then `import_pg_database`. Only
the second refused a data table under roles, so a fork asking to clone one
created and registered an empty `wm_fork_…` instance database and then failed —
and nothing collects it, since `drop_forked_datatable_databases` only drops
entries carrying `forked_from` and no entry names this one.
Refuse in both, so the clone stops before a database exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* nit worker error msg
* fix pg_dump stuck on version 17 on nix
* fix(datatables): refuse a malformed role annotation instead of ignoring it
`-- Role operator`, `-- role operator;` and `-- role operator -- why` all failed
the annotation parser's exact-match rule, so the query fell through to the data
table's default role and ran, silently, under a login the author did not choose.
Naming a role exists precisely to not do that.
A leading comment whose first word is `role` is now an annotation attempt: the
keyword matches case-insensitively, one trailing `;` is tolerated, and anything
else is an error naming the line. Only callers that already know the target is a
`datatable://` reference ever run this, so ordinary SQL keeps its comments.
Also bumps the dev shell's postgres client to 18 — it trailed the server the dev
database runs, which takes out every data table export, clone and fork-with-data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): refuse a malformed role query string instead of ignoring it
`?Role=analytics`, `?role=` and `?x=1&role=…` all fell through the reference
parser's exact-match rule, so the connection resolved to the data table's default
role and ran under a login the caller never asked for — the URI half of the same
trap as a malformed `-- role` annotation.
The key now matches case-insensitively, and anything else in the query string is
an error naming it; `role` is the only parameter a reference takes. Callers that
only need the entry keep a lenient `datatable_ref_name`, since they never act on
the role. The DuckDB `ATTACH` parser propagates it rather than attaching under
the default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* fix(datatables): carry the role annotation into the row_to_json retry
The retry rebuilds its SQL from `pruneComments(code)`, so the leading comment
block never reached the second attempt — and with it the `-- role <name>` line
that decides which login the query runs as. The retry connected as the data
table's default role instead, so a query the first attempt was denied could
succeed on the second, reported as "recovered with the row_to_json fix".
Carry the leading comment block over. The retry itself is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* chore(datatables): don't mount the roles UI until the ACL editor lands
Enforcement ships first. The permissions drawer is what turns roles on, and the
catalog section is what creates them — both are only useful once there is a way
to grant a role the privileges it needs, which arrives with the ACL editor. Left
mounted they would offer a feature whose other half does not exist.
The two components are complete and reviewed; only their call sites here are
commented out, with a note pointing the follow-up PRs at them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* fix(datatables): honour `-- role: x`, and fix the DuckDB attach test
Two review findings, both real.
`attach_datatable_parses_name_and_role` never compiled: `parse_attach_datatable`
returns `Result<Option<_>>` now and one call site kept a single `unwrap`. Its
`?Role=analytics` case also asserted a refusal, contradicting the parser in the
same commit, which matches the key case-insensitively. Replaced with the cases
that are genuinely malformed, and a positive one for the cased key.
`-- role: analytics` fell through to the default role — the silent fallback the
strict parser exists to remove, for the spelling most likely to be typed. The
keyword now accepts an optional colon, attached or spaced, while a word that
merely starts with it (`rolebased`) is still not an attempt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* fix(datatables): clone a fork's pointer instead of failing after the copy
Forking a fork with cloning left an orphan database. The preflight resolves the
pointer and sees the governing entry, so both endpoints ran and filled the new
database; `apply_forked_datatable` then refused the inherited pointer and rolled
the fork back, stranding a registered `wm_fork_*` that no entry names and whose
name blocks the retry.
Refusing earlier would have been the smaller change, but forking a fork and
cloning worked before pointers existed, so it would trade an orphan for a
regression. Resolve what the pointer names and write the terminal entry the
clone needs: the whole `database` object rather than a patch of its
`resource_path`, since a pointer has none, and `reference` removed with it.
Also accepts `-- role=x` and `-- Role = x`, two more spellings that fell through
to the default role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* fix(datatables): refuse to roll back the catalog while roles exist
The down migration dropped the table and left every role behind: live Postgres
logins whose passwords only that table carried, so after a revert Windmill could
neither use, disable nor delete them, and re-applying could not recreate them
because the names were taken. Cleaning up here is not possible either — dropping
a role means reassigning what it owns in every instance database, and a
migration runs in one — so it now refuses while the catalog is non-empty and
says to delete the roles through instance settings, which does the cluster work.
Also enforces the instance-only invariant the resolved-pointer clone relies on
rather than only asserting it in a comment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* refactor(datatables): settle clonability in one place, before anything is created
A clone is three stages a workspace apart — `create_pg_database`, then
`import_pg_database`, then `apply_forked_datatable` inside the fork transaction.
Only the third can roll back, and `CREATE DATABASE` is not transactional, so any
refusal that lives there strands a registered `wm_fork_*` that no entry names
and whose name blocks the retry.
That orphan has now been fixed three times, most recently reintroduced by a
guard added one commit ago. Patching each new refusal into the first endpoint is
not the fix; having two places that can refuse is. `ensure_datatable_is_clonable`
now answers every reason a copy can be refused and returns what it resolved, and
the stage that writes the entry only does the work.
Also takes an ACCESS EXCLUSIVE lock before the rollback guard counts, so a role
created concurrently cannot slip between the check and the drop.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
* fix(datatables): let a retried clone reclaim its own leftover database
A clone creates its target database one request before it copies into it, and
the fork that would name it is written a request after that. Any failure in
between — a pg_dump error, a bad restore, a dropped connection, the source's
roles changing mid-flow — left a registered `wm_fork_*` that no entry names,
and every retry then failed on its name. This predates data table roles.
`create_pg_database` now reclaims such a leftover before creating: only a
`wm_fork_*` database Windmill registered as a data table database and that no
data table or ducklake entry names, in any workspace, archived ones included.
The drop never terminates connections, so a clone still copying into it makes
the reclaim fail instead of being cut off. It is limited to callers who
administer the source — reaching it is not enough, since on a data table
without roles every member reaches it — and anyone else gets the refusal an
existing database always got.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Revert "fix(datatables): let a retried clone reclaim its own leftover database"
This reverts commit
|
||
|
hugocasaandClaude Opus 5
|
42f489685b |
feat: store resource type display names and label hub integrations (#11113)
* feat: label resource types and integrations with hub display names Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: load hub integration names in the app and flow pickers Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: load hub resource type names where drawers title a type Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: store resource type display names and drop the hardcoded list Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: leave display_name out of the fork comparison Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: ignore over-long synced display names, move name loaders Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: share the hub integration list cache, backfill admins only Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep a name over a nameless duplicate, retry failed hub reads Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 5
|
d8d7332eb6 |
feat: add per-route CORS origin allowlist for HTTP triggers (#10833)
* feat: add per-route CORS origin allowlist for HTTP triggers Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: fail closed on cold router cache and invalid origin input Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: resolve CORS route from the decoded path like the request handler Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: add instance-wide default allowed origins for HTTP routes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: let non-superadmins read the default allowed origins setting Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: badge the advanced section when a route's origins are restricted Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: state inherited origins on the control and use one hint row Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: trim the origins tooltip and relabel the toggle when a default exists Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: keep the origins format hint visible until an entry is wrong Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: state the at-least-one requirement in the origins hint Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: import the origins validator in the trigger-http tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: make an empty allowlist deny rather than fall back to the default Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: address review nits on origin validation and the CORS editor * fix: derive the origins error from the stored list and tighten host validation * fix: parse real IPv6 hosts and refuse a newly emptied allowlist * refactor: make origin validation advisory except for null and non-ascii * feat: let an empty allowlist be saved as deny every origin * docs: document the empty allowlist as deny every origin * fix: bound allowlists, reject commas, and decide cors after the handler * chore: revert unrelated rustfmt churn in windmill-common tests * chore: revert unrelated rustfmt churn in windmill-common * chore: drop the route types the cors restructure replaced * fix: take the stricter cors decision from before and after the handler * fix: strip runnable cors headers when the routers are unavailable * docs: document the allowlist bounds in the openapi schema * fix: let an unavailable cors read defer to one that resolved * refactor: carry the resolved cors policy from the handler to the middleware * docs: describe why an unavailable read fails closed on the paths that reach it * fix: validate the default origins on the declarative settings path * test: keep the webhook doc comment with the test it describes * fix: warn on impossible schemes and ports, and validate the instance setting * feat: treat an empty allowlist as unset at both levels * perf: decode the cors path only when the fallback needs it * docs: document the empty allowlist as unset in the api schema * docs: describe an empty allowlist as unset in the frontend comments * docs: say what a null allowlist resolves to, not what it meant before the default existed * docs: state what the validator refuses and why methods stay broad * feat: exempt static asset routes from the origin allowlist * fix: hide the origin control for every static target, not just websites * fix: exempt only static websites, not single-file static assets * fix: warn on an unclosed ipv6 host in the origins advisory * fix: require assets present, not just the static website flag --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
 Ruben FiszelandClaude Opus 5
|
45102c8265 |
fix: let the hub_sync job read the uid and hub_base_url settings (#11106)
Claude-Session: https://claude.ai/code/session_01Q6triDksvGJ4YK2gA1acEc Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5
|
f8f7c0009f |
fix: serve instance env settings at the documented /settings/local path (#11075)
Claude-Session: https://claude.ai/code/session_01RbLzVfFDZ9pjBGHSzCSrNZ Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5
|
abf4c6c234 |
feat: add a dismissible instance-wide announcement banner (#11037)
* feat: add a dismissible instance-wide announcement banner Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ExBp57hUoB8hQm36bJuUEs * fix: harden instance banner validation and mandatory-banner visibility Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ExBp57hUoB8hQm36bJuUEs * fix: sequence instance banner loads and match the backend character cap Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ExBp57hUoB8hQm36bJuUEs * fix: gate the settings save on a valid instance banner link Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ExBp57hUoB8hQm36bJuUEs * feat: restrict the announcement banner to the managed cloud Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ExBp57hUoB8hQm36bJuUEs --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Fable 5.1
|
4fef1195ad |
fix: apply object-storage test SSRF validation to all non-super-admins (#10933)
* fix: apply object-storage test SSRF validation to all non-super-admins Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJLqsE5br9r5e7qy8ULwUg * fix: name the job-token case in object-storage test rejections Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJLqsE5br9r5e7qy8ULwUg * fix: run object-storage connection tests with a short-lived user token Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJLqsE5br9r5e7qy8ULwUg * fix: test object-storage resources from the browser, mint a token only for the worker test Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJLqsE5br9r5e7qy8ULwUg * fix: resolve variable and resource references before the browser-side object-storage test Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJLqsE5br9r5e7qy8ULwUg * fix: bound the browser-side object-storage test to 15s Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJLqsE5br9r5e7qy8ULwUg * fix: explain object-storage test rejections and name the way out Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJLqsE5br9r5e7qy8ULwUg * fix: keep the server-resolved address out of the object-storage test rejection Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FJLqsE5br9r5e7qy8ULwUg --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Ruben Fiszelandwindmill-internal-app[bot]
|
cfcfe298dd |
feat(ai-chat): make reusable skills ai_skill resources you select per workspace (#10914)
* feat(ai-chat): make reusable skills ai_skill resources you select per workspace * chore: pin the ee ref to the skill telemetry counters * fix: address review findings on skill authoring, import and migration * fix: enforce skill selection in read_skill and stop imports clobbering resources * feat: carry format_extension from the hub into synced resource types * fix: let an edit set or clear a resource type's format_extension * fix: regenerate the sqlx cache and close the review round findings * fix: close the round-2 findings on folder ACLs, cached sync and truncation * refactor: make the skills migration non-destructive and use design-system inputs * fix: close the round-4 findings on folder owners, startup sync and truncation * fix: clear obsolete extensions, guard folder owners, and report skipped skills * fix: honor explicit-null extensions and report same-type migration conflicts * fix: scope skill actions to the committed workspace and paginate the listing * fix: keep the drawer scoped to the live workspace and surface truncation * fix: discard a skills refresh for a workspace the chat has left * chore: update ee-repo-ref to 6efe7a73c745c2e1377a34498523c00d89010a3d This commit updates the EE repository reference after PR #764 was merged in windmill-ee-private. Previous ee-repo-ref: 55998c142bc72edd08532748af1974b16035658d New ee-repo-ref: 6efe7a73c745c2e1377a34498523c00d89010a3d Automated by sync-ee-ref workflow. --------- Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
|
815de49e23 |
feat: make the service log retention period an instance setting (#10889)
* feat: make the service log retention period an instance setting Service log retention was a hardcoded 14 days with no override, unlike job retention. It becomes the `service_log_retention_secs` global setting (env `SERVICE_LOG_RETENTION_SECS`, default unchanged at 14 days), reloaded on change like the other retention settings. The constant becomes `DEFAULT_SERVICE_LOG_RETENTION_SECS` and every reader goes through `service_log_retention_secs()`, so the `log_file` sweep, the object-storage orphan scan, the columnar store's compaction and pruning, the retrieval clamp and the search index's trim window all follow the configured value. Loaded outside `initial_load`'s `server_mode` guard: a dedicated indexer trims the search index to a window derived from this value and is not a server. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: never let a non-positive service log retention expire every log Every service log cutoff is `now - retention`, so a `0` or negative window puts the cutoff at or after `now` and the next sweep reads the whole history as expired — deleting the `log_file` rows and their object-storage files irreversibly. `0` is reachable two ways now that the window is configurable: it is what an operator types by analogy with the job retention period sitting directly above it, where `0` does mean keep forever; and `SecondsInput` writes a `0` into a field that was merely focused, so saving the Jobs panel is enough. Service logs always have a window, so clamp an unusable value back to the default in the accessor every reader already goes through. The upper bound is where `chrono::Duration::seconds` panics, which would abort the sweep that reads it. The settings field rejects a non-positive value rather than silently correcting it, and its description now names the database rows too — they are swept on every instance, including one with no object storage configured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: address review findings on the service log retention setting - Bound the monitor's `log_file` sweep. Every process rotates a log file a minute, so lowering the retention can make one ordinary setting change expire millions of rows; the unbounded `DELETE ... RETURNING` materialized all of them, and their deletion futures, in a single tick. Batched like the settings-page cleanup on the same table. - Make the retention atomic private and give it one writer, so a value that would expire every service log cannot reach a cutoff by any path, and say so in the log when one is rejected rather than falling back silently. - Cap the retention at a century. The previous ceiling only bounded `TimeDelta` construction, while consumers compute `now - retention`, which panics past year 262143, and build a Postgres interval that overflows well before the old cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: cap an oversized service log retention instead of shortening it The two unusable directions were landing on the same fallback, so configuring a retention above the ceiling silently produced 14 days — deleting logs the operator had asked to keep for longer. Too large now caps at the maximum, which preserves that intent; only a non-positive value, which would expire everything and has no upward reading, falls back to the default. Also bound the `log_file` drain to ten batches per pass: `monitor_db` runs under a 600s timeout that cancels every maintenance future in the same `join!` and reports a critical error, so a backlog large enough to need batching has to drain across ticks, the way the neighbouring sweeps already do. The settings field carries the upper bound too, and the superseded query's offline entry is dropped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: route the new log-file registration cutoff through the retention accessor `send_log_files_to_object_store` arrived on main while this branch was open and reads the retention directly. The atomic behind it is private now, so it goes through the accessor like every other consumer — which also means the cutoff it uses to skip registering already-expired files follows the configured retention rather than a fixed two weeks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: say why every mode loads the service log retention setting A worker registers its rotated log files against the retention cutoff, so the comment naming only the indexer no longer covers why the setting sits outside the `server_mode` guard. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: file service log retention under Monitoring, not Jobs Service logs are the Windmill processes' own logs — every process rotates and registers its own, no job involved — so the Jobs panel was grouping by the shape of the widget rather than by the subject. It sits under Monitoring now, beside the Indexer panel that holds the other service-log window. Its own section rather than inside that panel: the panel is badged EE, while this governs the database sweep that runs on every instance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * chore: update ee-repo-ref to a6e3533b26195918a17fea58646f71d2bbcde288 This commit updates the EE repository reference after PR #752 was merged in windmill-ee-private. Previous ee-repo-ref: 1d93da24bd166b9a5a5cc204034a1d35ffc88474 New ee-repo-ref: a6e3533b26195918a17fea58646f71d2bbcde288 Automated by sync-ee-ref workflow. * feat: say on the service logs page where the logs actually are The retention number alone does not tell an operator what it governs, and the answer differs by instance. Two states are worth calling out because they are the ones where retention does not mean what it looks like: Without instance object storage, each process keeps its files on its own disk. The page lists what every host wrote, since the rows are in the shared database, but can only open the files of the replica serving the request, and a host's files go with it when it is replaced. With object storage but "Delete logs from s3 periodically" off — the backend default, since uploads are gated on a store existing while deletions are gated on that toggle — expiring a log removes the row and the local file and leaves the uploaded copy behind for good. The retention field itself now names every copy it covers and says that full-text search reaches back at most that far, and less when the indexer's own window is shorter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * fix: describe raw log files as the transient copy they became Retiring the raw files landed while this was being written: the indexer now deletes each one as soon as it is ingested, and the log viewer rebuilds a file from the columnar store once the raw copy is gone. So the durable copy is the store, and warning that an uploaded file is kept forever when periodic s3 deletion is off only holds where no indexer runs to ingest it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN * chore: point ee-repo-ref at the EE compile fix EE main does not build on its own: extracting the index-window expression and adding a fourth copy of it landed in separate PRs that never conflicted textually. windmill-ee-private#756 is the one-line fix; this pins it so CI has a tree that compiles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WsnpNSM6K3oyjwntRwJtVN --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasaandClaude Opus 5
|
b6e059116a |
feat: track token cost in AI sessions and chats (#10688)
* feat: track token cost in AI sessions and chats * fix: address review findings on AI cost tracking * fix: price inherited and overridden models at their real rates * fix: stop newer model revisions inheriting an older price * fix: stop a sub-model inheriting its family's price * fix: keep alias suffixes resolving to their model's price * fix: count OpenRouter cache writes and drop unverifiable rates * refactor: move AI spend out of the chat into workspace and user settings * fix: pin the usage workspace per turn and stop inventing cache rates * fix: leave Sonnet 5 unpriced while its promotional rate runs * docs: record the new table in the schema summary and tighten comments * fix: mark estimated AI costs with ~ and drop session grouping Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: name the workspace in the self-scoped AI usage title Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: state that overrides never replace a provider-returned cost Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: let a cleared cache rate inherit again and flag partial totals Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: clear a refused rate's error when the input snaps back Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop a revision variant inheriting its base family's rate Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: report AI usage before tools run and price self usage consistently Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: key pricing rows on the model id usage is reported under Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: surface Bedrock and Gemini usage the chat proxy was dropping Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: count Gemini tool-use prompt tokens as input Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: price flat-rate Gemini Flash models Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: state the tool-use token invariant once Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c2deea13b7 |
fix(security): a WM_TOKEN job token can never be a global superadmin (GHSA-hfh4-cx4h-3fcr) (#10124)
* fix(security): a WM_TOKEN job token can never be a global superadmin (GHSA-hfh4-cx4h-3fcr)
Privilege escalation: an app/flow/schedule/trigger execution policy's `on_behalf_of`
(which a `wm_deployers` member can set) could point at a superadmin email. The
resulting job `WM_TOKEN` then passed the email-based superadmin checks, granting
instance superadmin. `forbid_superadmin_job_token` only guarded ~15 of ~75 routes.
Fix at the token layer: a WM_TOKEN must never satisfy a superadmin gate,
regardless of whose email it runs as (sentinel OR a real superadmin).
- `ApiAuthed` gains a `job_id` field, stamped once in `AuthCache::get_opt_job_authed`
from the resolved token's job_id (correct even on cache hits).
- `require_super_admin(db, email)` -> `require_super_admin(db, &ApiAuthed)`, rejects
`authed.job_id.is_some()`. `require_super_admin_email` kept for the few internal
callers without an ApiAuthed.
- `is_super_admin_authed(db, &ApiAuthed)` for the boolean `is_super_admin_email`
authorization branches on request handlers (workspace deletion, fork drops,
dev-workspace attach/archive, object-storage SSRF exemption, custom dbname, EE GHES
+ connected repositories, ...). Migrate ~75 sites (OSS + EE).
- CUSTOM_INSTANCE_DB reads the *authenticated* job_id, not the caller-supplied
`?job_id` query param. Worker-tag check takes a precomputed job-aware `is_super_admin`
on the request path.
Execution-time on-behalf checks (scheduled/flow worker-tag, Cloud enqueue quota,
is_devops_email) are hardened in a follow-up — see
docs/followup-onbehalf-execution-privilege-hardening.md.
Regression tests: a superadmin-email WM_TOKEN is rejected on `require_super_admin`
routes, on `DELETE /workspaces/delete/{w}` (403, workspace preserved), and on the
CUSTOM_INSTANCE_DB lookup with no `?job_id` (401); real superadmin tokens still succeed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: cap devops role at workspace admin and reject reserved on_behalf_of identities
Extends the job-token cap with three pieces:
- `require_devops_role` takes `&ApiAuthed` and rejects job tokens.
`is_devops_email` is true for superadmin emails, so every worker-management,
instance-config and service-log route was reachable by the same superadmin
`WM_TOKEN` that `require_super_admin` already rejects.
- A `job_id` claim that does not parse as a uuid rejects the token rather than
resolving to `None`, which would clear the job provenance and uncap it. Applies
to the internal JWT and the external `jwt_ext_` path.
- Defense in depth at store time: `validate_on_behalf_of` refuses the reserved
internal sentinels as an `on_behalf_of` on apps/flows/scripts/schedules/triggers,
and app execution refuses a policy carrying one — covering already-persisted and
forked-app rows that predate the cap. Deploying on behalf of a real user,
including a real superadmin, stays allowed; the cap handles that at execution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(mcp): preserve job-token provenance when minting the proxy JWT
The MCP endpoint-tool proxy re-mints a JWT from the caller's ApiAuthed to
forward the proxied request, but passed job_id: None. A job's WM_TOKEN is
capped at workspace admin (GHSA-hfh4-cx4h-3fcr); dropping the job_id here
re-minted an uncapped token that satisfies require_super_admin /
require_devops_role on the proxied route (e.g. listWorkers exposing worker
IPs, job/workspace IDs, and sensitive tags).
Carry api_authed.job_id into create_jwt_token. Adds an in-module regression
that decodes the forwarded JWT and asserts the job_id is preserved for a job
caller and absent for a non-job caller.
Reported by Codex CI review (P1) on #10124.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: cap the admin-or-devops gate at workspace admin for job tokens
require_admin_or_devops (the EE critical-alerts endpoints) grants when the
caller is a workspace admin OR an instance devops. is_devops_email is true
for superadmins, so a WM_TOKEN running on-behalf of a superadmin who is not a
member of the target workspace could clear the devops branch and read/ack that
workspace's critical alerts (GHSA-hfh4-cx4h-3fcr). This gate takes a bare
email, not an ApiAuthed, so the token-layer cap could not see it.
Thread the caller's job-token provenance and reject the devops branch for job
tokens, matching require_devops_role. The workspace-admin branch stays allowed
— that is the cap ceiling. Adds an enterprise-gated regression proving the
bypass is closed and a real superadmin token still clears the gate.
Found while auditing the PR for bare-email gates the choke-point cap misses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: cap instance-global is_admin gates at workspace admin for job tokens
Three instance-global routes gate on the caller's own `is_admin` claim, which
`ApiAuthed.is_admin` carries into a WM_TOKEN (it is a workspace-admin claim,
true for superadmins too). A job token is capped at workspace admin
(GHSA-hfh4-cx4h-3fcr), so its is_admin claim must not authorize instance
actions on a route with no workspace binding:
- `unarchive_workspace` — unarchive an arbitrary workspace by id
- `prune_concurrency_group` — delete a global concurrency group
- `list_worker_groups` — return unobfuscated `env_vars_static` (may hold secrets)
Add job-token-aware `is_instance_admin` / `require_instance_admin` helpers (the
same shape as `require_super_admin` / `require_devops_role`) and use them at
these three sites. Workspace-scoped `require_admin(authed.is_admin, ...)` gates
are intentionally left unchanged — a workspace-admin job token is within the
cap there. Regression added covering all three; verified it lets a WM_TOKEN
unarchive/leak without the fix and is blocked with it.
Reported by Codex CI review (P1) on #10124.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(mcp): drop orphaned path_field_renames from EndpointTool test helper
The merge with main adopted main's mcp path-substitution refactor (#10162),
which removed the `path_field_renames` field from `EndpointTool` and its
consumer (`substitute_path_params` no longer takes per-field path renames).
main's `runner.rs` `ep` test helper still constructed the struct with
`path_field_renames: None`, so the workspace test build (cargo test --all,
which compiles windmill-mcp's own #[cfg(test)] module under the `server`
feature) failed with E0560. A plain `cargo check` does not compile that test
module, so it only surfaced in CI's cargo_test.
Remove the orphaned field to match the struct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: describe the sentinel-rejection policy the forged-identity test asserts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: complete ApiAuthed initializers in feature-gated tests after merge
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: stop job tokens minting credentials that shed their provenance
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: cap the MCP OAuth approval mint at the same elevated-job-token gate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: cap the self-service password reset at the elevated-job-token gate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: cap app embed/SDK mints and scope widening at the elevated-job-token gate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: keep job tokens from destroying the account they run on behalf of
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: deny job tokens a foreign-workspace admin claim and workspace ejection
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: keep the follow-up inventory in the PR instead of the repo
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: make the session workspace status gate job-token aware
session_workspace_status derived its superadmin branch from a bare email
check, so a job token carrying a superadmin identity resolved the existence
of workspaces it has no relationship with rather than seeing them as
deleted. Switch to is_super_admin_authed, matching every other instance
gate reached from a request ApiAuthed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* revert: leave the global concurrency-group listing on the plain admin gate
The listing exposes concurrency keys across workspaces, which is metadata
rather than a capability, and it 401s rather than degrading. Keep the guard
on the prune route next to it, which is the destructive one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: keep the instance-admin gate on the global concurrency listing
The listing spans every workspace's concurrency keys, and the gate rejects
only job tokens: the !is_admin branch is the pre-existing check, so
workspaced tokens and interactive admins are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to d30af67d38954f9012f7bad08da23e347344b4c6
This commit updates the EE repository reference after PR #664 was merged in windmill-ee-private.
Previous ee-repo-ref: 7870573dbc3360f99bada143f094c67dce0d9e9c
New ee-repo-ref: d30af67d38954f9012f7bad08da23e347344b4c6
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: hugocasa <hugo@casademont.ch>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
|
5b0a159a01 |
fix(smtp): explain why a test email failed instead of 'deadline has elapsed' (#10620)
* fix(smtp): explain why a test email failed instead of 'deadline has elapsed' Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(smtp): keep non-SMTP error codes and retire a stale test alert Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to f0df8b82c4c089d384423ed64b8504506084820d This commit updates the EE repository reference after PR #721 was merged in windmill-ee-private. Previous ee-repo-ref: 1ffaf3dea81e007c6c11146c1e12e97e83f5b938 New ee-repo-ref: f0df8b82c4c089d384423ed64b8504506084820d Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 5
|
552ad9c859 |
fix: reflect custom tag add/remove in the manage-tags drawer immediately (#10526)
* fix: reflect custom tag add/remove in the manage-tags drawer immediately Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: do not fail the custom_tags write when the cache refresh errors Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
318c9f0073 |
feat(git-sync): dedicated base url for GitHub webhook delivery (#10411)
* feat(git-sync): let GitHub webhooks register a dedicated base url Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): validate the webhook base url and apply it on change Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: pin ee ref for the git-sync webhook base url change Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): validate and reconcile the webhook base url on every write path Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): route every declarative settings writer through the same rules Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): let the reconciler own the webhook field write-back Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): make the webhook base url validators agree across UI and server Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): lock the workspace row across git_sync read-modify-writes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: pin the webhook base url validator to its server counterpart Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): retry a failed webhook move on every re-apply of the setting Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): retry pending webhook moves on every declarative re-apply Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): reject non-string webhook base urls and bound the sweep Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): reject credential-bearing webhook base urls Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): keep credentials out of webhook base url validation errors Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): redact through the last authority @ when reporting a bad url Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): stop echoing unparsed webhook base urls instead of scrubbing them Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): never echo a submitted webhook base url in validation errors Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): keep the submitted scheme out of validation errors Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(git-sync): drop the webhook sweep, surface stale receivers in settings Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): refresh the stale webhook list when settings are saved Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): mark registered_url nullable and drop the duplicated field error Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: pin ee ref after dropping the reconcile lock and CAS Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(git-sync): refresh the stale webhook list on category saves too Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to aa05ca8e97fc8265cd724753a80db37f83243254 This commit updates the EE repository reference after PR #695 was merged in windmill-ee-private. Previous ee-repo-ref: 3e6cd9226b68707233ae2434511fe5131dce808b New ee-repo-ref: aa05ca8e97fc8265cd724753a80db37f83243254 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 5
|
fbf9f04e10 |
fix: surface the real postgres error when data table migrations fail (#10371)
* fix: surface the real postgres error when data table migrations fail Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address review nits on the data table migration error fix Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: name the exact grant a data table migration needs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: quote both identifiers in the data table grant hint Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: add a data table connection and privilege check to workspace settings Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: report data table privileges from the capability fields, not the grant list Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: read grant targets from the server and drop the public schema guess Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: render the search_path suggestion server-side and pin the granted database Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: key the connection check on request identity, not the data table name Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: declare the data table check schema field nullable and required Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
1b6b2aa859 |
fix(datatable): provision the replication user on managed postgres (#10375)
* fix(datatable): provision the replication user on managed postgres * fix(datatable): serialize replication user provisioning and sync config schema * fix(datatable): keep replication cleanup best-effort and self-heal a null password |
||
|
Ruben Fiszel
|
78e115bee5 |
fix: datatable full schema hangs behind a transaction-pooling postgres proxy (#10352)
* fix: datatable full schema hangs behind a transaction-pooling postgres proxy * test: pause the clock in the pg connection shutdown test |
||
|
Diego ImbertandClaude Fable 5
|
68daed8501 |
refactor: custom-instance datatable connection handling (#10271)
Attach custom-instance datatables in the DuckDB executor through a DuckDB secret instead of an inline connection string, and route postgres triggers on custom-instance datatables through a dedicated custom_instance_replication_user role (with its own auto-generated password in global_settings). Normalize custom_instance_user attributes on server boot. Claude-Session: https://claude.ai/code/session_01Tp6NNNinCB8dwWqGaFXDRF Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
07d4b674f1 |
fix: manual resource type sync fetches from hub first, cache as fallback (#10269)
* fix: manual resource type sync fetches from hub first, cache as fallback The superadmin "Synchronize resource types" endpoint (POST /api/settings/sync_cached_resource_types) was cache-first: it read the on-disk hub_rt cache and only fell back to the hub when no cache file existed. Since that cache is refreshed by a daily cron, a newly-published hub type could not be pulled on demand, the button replayed the stale cache and reported "Synced 0", so the type never landed in the admins workspace. The manual endpoint is now hub-first: it fetches the live list and upserts it into admins, falling back to reading the on-disk cache only when the hub is unreachable (airgapped install / network error), logging which path it took. The startup/offline sync in main.rs (SYNC_CACHED_RT + the cache-rt cron) stays cache-based and owns writing the cache, so this endpoint never touches it. Adds an optional `name` query param: when a specific type is requested and is still absent from the hub after syncing, the endpoint returns an explicit not-found instead of a silent "Synced 0". The three not-found frontend call sites (ResourceForm, AppConnectInner, ApiConnectForm) thread the type name through SyncResourceTypes; the global instance-settings button stays name-less. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: run whole-list sync before the targeted not-found check, word 404 by source Address review nits: the optional `name` check ran before the upsert loop, so a `?name=<absent>` request skipped the whole-list refresh; move it after the loop so the sync always happens. Also word the not-found 404 by source, the cache-fallback path (hub unreachable) no longer claims it checked the hub. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
ec6324409d |
fix(s3): support instance-policy credentials in object storage tests (#10238)
* fix(s3): cache ambient aws credentials and surface credential chain errors Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SCczeVBDhDnLXWqxPWw6od * fix(frontend): clarify that object storage connection tests run on the server Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SCczeVBDhDnLXWqxPWw6od * fix: render test connection tooltip in popup and harden cache test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SCczeVBDhDnLXWqxPWw6od * better doc * fix(frontend): correct tooltip wording, tests run from the executing worker Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SCczeVBDhDnLXWqxPWw6od * fix(s3): single-flight ambient credential provider creation * fix(frontend): clarify object storage tests run on the server process * fix(frontend): make instance object storage test tooltip provider-neutral --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
|
ff774c46bf |
feat: add per-workspace job-retention override (#10050)
* feat: add per-workspace job-retention override (EE) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 2ba6a2a75b6fc97858b306b2c98ada481e363c10 This commit updates the EE repository reference after PR #658 was merged in windmill-ee-private. Previous ee-repo-ref: e7fb36acd813cd717bcf05f5aafbf81de271d618 New ee-repo-ref: 2ba6a2a75b6fc97858b306b2c98ada481e363c10 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
e77b7523a5 | nit enterprise implies license feature | ||
|
Ruben FiszelandClaude Opus 4.8
|
76a9523009 |
feat: use derived username instead of email for non-member superadmins (#9857)
* feat: use derived username instead of email for non-member superadmins Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address review - drop redundant username cache, guard whoami membership by email Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: use explicit non_member boolean instead of role string for superadmin banner Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve email from password table for non-member superadmin permissioned_as Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve non-member superadmin drafts via shared username->email resolver Adds resolve_username_to_email (usr, then super_admin password fallback for both derived-username and email modes) and uses it in get_email_from_permissioned_as and the drafts get/list endpoints, so a non-member superadmin's drafts resolve and no email leaks into the drafts payload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: superadmin-not-in-workspace schedule uses derived username as permissioned_as Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve non-member superadmin identity in draft owner-circles, username_to_email, and home filter Applies the password-fallback username resolution to the script/flow/app/draft owner-circle subqueries and the username_to_email endpoint (was an admins-workspace 'username == email' hack), and switches the home items-list user-folder filter to the non_member flag instead of the now-broken username-contains-@ heuristic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: backfill non-member superadmin favorites from email to derived username Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: propagate DB errors in username resolution instead of leaking email (CI review) Addresses cubic-dev-ai P2: get_instance_username_or_fallback_to_email now returns Result and only falls back to the email for a genuine 'no derived username'; a query error propagates so callers fail closed rather than leaking the raw email as the acting username. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: clarify non-member superadmin popover (username used + admin permissions) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep username_to_email endpoint member-only to not disclose non-member superadmin email (CI review) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: forbid disabling automate_username_creation once usernames assigned (CI review) Makes the setting effectively one-way once instance-wide usernames exist, so the global-uniqueness invariant that keeps stored u/<username> identities (schedules/triggers/drafts/superadmin ownership) unambiguous can never be dropped back to workspace-local uniqueness. Re-saving false on an already-disabled instance stays a no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
83ed011e26 |
feat(object-store): make GCS service account key optional for Workload Identity (#9842)
build_gcs_client always called `.with_service_account_key(...)`, so an
absent key (the settings UI stores "no key" as the empty JSON object `{}`)
was handed to the builder and failed to parse instead of falling through
to the object_store crate's InstanceCredentialProvider. Skip the call when
the key is blank so GCS uses the instance's ambient credentials (GKE
Workload Identity / the GCP metadata server).
"Blank" (empty/whitespace/`{}`/`null`) is centralized in a shared
`gcs_service_account_key_is_blank` predicate so the build path and the
non-super-admin connectivity-test SSRF guard (`validate_object_storage_test`)
agree on what counts as "no key" — otherwise a blank key would bypass the
guard yet still trigger the ambient-credential fallback, letting an
untrusted caller probe arbitrary buckets with the server's instance role.
Also clarify the settings UI hint that the key may be left empty for
ambient credentials, and add regression tests for the blank-key build path
and the guard.
Fixes WIN-2110
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
75ba81b2d2 |
fix(audit): don't read pg_authid from an elevated context in S3 export migration (#9832)
* fix(audit): don't read pg_authid from an elevated context in S3 export migration Migration 20260626132251 aborted instance startup on managed Postgres (e.g. Cloud SQL) with "Modifying pg_authid or pg_auth_members is not allowed in elevated context": the audit S3 export "oldest in-flight xact_start" floor probe calls pg_has_role(...), which reads pg_authid, and managed providers forbid that read from an elevated context. The migration ran the probe inline in its UPDATE, so the whole migration — and the instance boot — failed. Extract the probe into a shared SQL function audit_logs_s3_oldest_inflight_ts() that returns the oldest in-flight xact_start (when cluster-wide stats are visible) or NULL otherwise. The pg_has_role read is wrapped in a plpgsql BEGIN/EXCEPTION subtransaction, so a pg_authid failure returns NULL (callers fall back to a conservative 7-day window / reject) instead of aborting. is_superuser (a GUC, no catalog read) is checked first to short-circuit. The migration's trigger and UPDATE, the OSS backfill try_start, and the EE exporter/startup anchor (companion windmill-ee-private PR) all route through it. Because 20260626132251 already shipped, it is added to the potentially_stale list in windmill-api/src/db.rs: on startup the stale _sqlx_migrations row (checksum mismatch) is deleted and the fixed, idempotent migration re-applies, so already-migrated instances upgrade without a checksum-mismatch boot failure. Fixes WIN-2108 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 95352c13c4c82247d8cfd80936f9203aeb079802 This commit updates the EE repository reference after PR #635 was merged in windmill-ee-private. Previous ee-repo-ref: 136f49a52af922868acac33abf8198913a9e835c New ee-repo-ref: 95352c13c4c82247d8cfd80936f9203aeb079802 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
|
577ceeee86 |
perf(audit): re-anchor S3 audit export on enable + opt-in backfill (#9818)
* [ee] perf(audit): re-anchor S3 audit export on enable + opt-in backfill
The S3/GCS audit-log export's steady-state query filters by `age(xmin)`
(unindexable), so the only scan bound is the timestamp floor. On a fresh
enable the floor was epoch, and on a re-enable the cursor resumed from its
pre-disable position — either way the first run scanned the whole
`audit_partitioned` table. Under a `statement_timeout` (e.g. Aiven) that scan
never completes: the cursor never advances, nothing is exported, and the
repeated full scans saturate the database.
Re-anchor on enable (EE companion, windmill-ee-private#634):
- New trigger migration records a recent timestamp floor instead of the epoch
sentinel and `DO UPDATE`s the cursor to the current snapshot xmin on
re-enable, so the export always resumes from ~now and never rescans history.
Includes a one-time fixup for legacy epoch-sentinel checkpoints on upgrade.
Opt-in historical backfill (new `audit_logs_s3_backfill` module + endpoints):
- Exports a chosen `[from, to)` window on demand, scanning strictly by
`timestamp` (the partition key) in bounded keyset pages — each query is an
index scan capped at one page (verified via EXPLAIN: later partitions
`never executed`, ~11ms/page), so it stays well under any statement timeout
regardless of window size. Writes alongside the steady-state objects under
logs/audit/, without touching the xmin cursor.
- POST /settings/audit_logs_s3_backfill {from,to} (super-admin + Enterprise),
GET /settings/audit_logs_s3_backfill_status.
Also repurposes the status endpoint's `bootstrapping` flag to mean "draining a
backlog" (the cursor is capped and catching up), and updates the setting
description to point operators at the backfill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): heartbeat backfill lease per object; bump EE ref
Address review (cubic): persist progress (refreshing the lease heartbeat) after
every object PUT in the backfill page loop, not only once per page, so the gap
between heartbeats stays well under STALE_HEARTBEAT_SECS even on slow uploads
and another replica can't re-claim mid-page and run a concurrent backfill.
Bumps ee-repo-ref.txt to pull in the EE test-race fix (folding the backlog-drain
regression into the single audit e2e test).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject unstable backfill windows; bump EE ref
Address review (P1): the backfill keyset-pages over rows visible at scan time
and declares completion when the scan runs dry, but a row's `timestamp` is its
inserting transaction's `xact_start`. A window whose upper bound is recent or in
the future could silently omit a transaction that started inside `[from, to)`
but commits after the scan passed that timestamp. `try_start` now rejects any
`to` newer than the oldest in-flight `xact_start` (everything strictly older
than the oldest running transaction is committed and stable), using the same
trustworthy stats gating as the exporter's floor (restricted role / 2PC → a
7-day-old cutoff).
Bumps ee-repo-ref.txt for the EE monotonic-checkpoint fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): re-anchor legacy epoch checkpoints instead of synthetic floor
Address review (P1): the legacy-checkpoint fixup stamped last_oldest_inflight_ts
to now()-7d while leaving the old last_xmin in place. On an instance that
enabled export on the old code >7 days ago and got stuck before the first
successful batch, the next run would filter post-enable rows older than 7 days
out via `timestamp >= ts_floor` while still advancing last_xmin over the
interval — silently dropping them (the same floor-vs-cursor loss class fixed
elsewhere in this PR), and contradicting the "nothing committed after enabling
is skipped" guarantee.
A stuck epoch-sentinel checkpoint cannot be safely resumed (its backlog can be
arbitrarily old, so any recent floor prunes rows the cursor then skips, and an
epoch floor reintroduces the full scan). Re-anchor it to the migration's current
snapshot xmin instead — exactly like a fresh enable — so the export resumes
cleanly from ~now and the never-exported pre-upgrade window is recovered via the
opt-in backfill rather than silently dropped. Reword the setting description so
it no longer implies the disabled/legacy window is covered by the cursor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(audit): end-to-end integration tests for the object-store backfill
The backfill previously had only SQL-level/EXPLAIN validation. Add real
integration tests (in-memory object store, sqlx::test) exercising the public
path:
- backfill_exports_window_in_pages: with the page size forced to 2 rows, a
settled 3-day window is exported across multiple keyset pages; asserts every
in-window row lands exactly once, rows outside [from,to) are excluded, a day
that straddles a page boundary yields more than one object, progress counts
match, and a re-run is idempotent (deterministic keys overwritten, no dupes).
- backfill_rejects_unstable_window: a future/live `to` is rejected as unstable,
a window safely in the past is accepted.
Adds a test-only PAGE_ROWS override so multi-page behaviour is exercised with a
handful of rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(audit): note backfill scope is audit_partitioned only
Make explicit that, like the steady-state export, the backfill reads only
audit_partitioned; the pre-partitioning `audit` table is intentionally out of
scope (not a missed case).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject backfill windows before the partitioned boundary
Address review (Codex P1): the backfill reads only audit_partitioned, but
pre-partitioning history lives in the legacy `audit` table (still read by audit
list/get via UNION ALL, and retained for the configured period — 365 days by
default on EE). Since the setting text points operators at this API for
"pre-existing history", a window overlapping legacy rows would report completion
while silently omitting them.
Per the decision to not export the legacy table, reject instead of silently
omit: try_start now rejects a `from` earlier than the oldest audit_partitioned
timestamp (every legacy row predates the partition cutover, so a `from` at/after
that boundary can never overlap them). Reworded the setting text to scope the
backfill to the partitioned era. Added a regression test, plus an RAII guard
(cubic P2) so the test-only globals are restored even if an assertion panics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): backfill object keys per-window; require trustworthy settled cutoff
Address review (two P1s):
- Object-key overwrite loss: keys were `dt=<day>/audit_backfill_<min_id>.ndjson`.
A narrower, overlapping backfill can start a day's page at the same first row
(same min_id) but hold fewer rows, and `put` would overwrite a broader run's
object — silently dropping the rows only that object held. Include the
requested window in the key so different ranges write disjoint objects (same
window re-runs stay idempotent; consumers dedupe overlapping rows by id). New
regression test (verified red→green).
- Untrustworthy settled cutoff: when min(xact_start) isn't trustworthy (role
lacks pg_read_all_stats/superuser, or a prepared 2PC txn exists), the old
now()-7d fallback could still let an old transaction commit rows inside an
accepted window after the scan, so a "complete" backfill silently missed them.
Since a backfill asserts completeness, reject in those cases instead of
falling back. (The continuous exporter keeps its 7-day fallback — it only
claims bounded lag.)
Also makes the tests robust under the parallel runner: run_backfill takes the
store as a param, so tests pass a local in-memory store (no global
OBJECT_STORE_SETTINGS race) and serialize on the PAGE_ROWS override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject backfill overlapping legacy table; regen deref openapi; trim migration comment
Address review (1 P1 + 2 P2):
- Empty-partition backfill (P1): the min(audit_partitioned) guard no-ops when
audit_partitioned is empty, so an upgraded instance with legacy `audit` rows
but no partitioned rows yet would accept a window and complete with zero rows,
silently omitting the legacy rows. Check the legacy `audit` table directly:
reject any window that overlaps a legacy row (subsumes the boundary check and
covers the empty-partitioned case). Test updated accordingly.
- openapi-deref (P2): regenerate openapi-deref.yaml/json (served via include_str!)
so /openapi.{yaml,json} expose the new backfill endpoints.
- Migration comment (P2): trim the PR-history narration to the durable
constraints (why a recent floor and a monotonic cursor are required), per
AGENTS.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to b821fecccbcba2efed544890576bf2b84321d70d
This commit updates the EE repository reference after PR #634 was merged in windmill-ee-private.
Previous ee-repo-ref: 6b191b77aabcf77658ad4f9031576e0d7b66bf89
New ee-repo-ref: b821fecccbcba2efed544890576bf2b84321d70d
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
Ruben FiszelandClaude Opus 4.8
|
aa098c70c0 |
perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes (#9786)
* perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: document delete_jobs auth contract and workspace-scope jobs_export purge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
12f92e3ab7 |
[ee] feat(backend): native script retry without one-step-flow wrapping (#9688)
* feat(backend): native script retry without one-step-flow wrapping Schedules and data pipelines that retry a single script previously wrapped it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a v2_job_status row, and UI projection complexity. This adds native retry on a plain JobKind::Script job. - RetrySettings: flatten Retry into a deduped retry_settings table, carried via the existing runnable_settings_handle (lazy, off the hot path). - push() materializes a bare-script-with-retry SingleStepFlow into a native Script job (gated on min-version + no handlers/retry_if). - add_completed_job re-pushes the next attempt on failure with backoff, tracking the attempt counter in v2_job_queue.extras and the chain via parent_job; schedule completion handlers fire only on the terminal attempt. - frontend: ScriptRetryChain shows the attempt chain on the run page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): native retry_if eval + per-occurrence schedule handlers Extends native script retry to the two cases that previously stayed on the one-step-flow path: - retry_if: evaluated natively on the failure path via a feature-gated windmill-jseval dep (quickjs) over the failure result + flow_input; push materializes such policies natively only when quickjs is available. - on_failure_times / on_recovery: apply_schedule_handlers now resolves each past scheduled occurrence's terminal status across its native-retry chain (root OR any parent_job=root child succeeded) and excludes the current occurrence, so the counting is per-occurrence rather than per-attempt. All scheduled-script retries now go native (schedule.rs gate removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): always materialize retry_if natively; unsupported without quickjs retry_if is evaluated by the worker (which always has quickjs), not the pusher, so gating materialization on the pusher's feature was wrong. The flow path was never a real fallback either — the flow runtime needs quickjs to evaluate retry_if too. retry_if now always goes native; on a worker without quickjs it is unsupported and fails closed (no retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): un-park asset-cascade (pipeline) retry Native retry resolves the blocker that parked pipeline retry: a retried subscriber is now a Script job (not a one-step flow / flow step), so it stays eligible for asset dispatch and can trigger its own downstream on recovery. - scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset edges (was dropped with a TODO warning). - asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job, so native-retry attempts dispatch on success while flow steps stay excluded. - tests: retry-bearing subscriber now dispatches as a native Script carrying the policy in runnable_settings_handle; native-retry attempt is eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): cap native retry interval, lazy result serialization, idempotent retry push Hardening from a self-review of the native retry path: - Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path (evaluate_retry); the exponential formula could otherwise schedule up to ~18h vs the flow path's 6h. - Serialize the failure result lazily (only when a retry_if policy needs it), so the common failure no longer pays the serialization on the failure path. - Push each retry with a deterministic id per (root, attempt). If a worker dies between enqueueing the retry and finalizing the current attempt, the reaper re-handles the attempt and lands here again — push rejects the duplicate id, so the retry is enqueued exactly once (no double-retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): defer schedule handlers idempotently on retry-push replay (review P1) Address local-review findings: - P1: retry_pending was derived from the retry push *result*, so on a worker crash + reaper replay the duplicate-id push returned Err → retry_pending flipped to false → apply_schedule_handlers fired for the non-terminal attempt (and the terminal attempt later fired them again). Pre-check whether the deterministic retry id already exists and report it as pending without re-pushing, so the handler-deferral invariant is crash-idempotent too. - P2: refresh the stale 'wrap the script in a one-step flow' comment in the asset-cascade retry push — it now materializes a native Script. - Add RetrySettings <-> Retry round-trip unit tests (clamping edges). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): native retry chain + per-occurrence status sqlx tests Close the two integration-test gaps flagged in local review: - chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts) and asserts crash-replay idempotency (the P1 fix: a replayed completion reports pending without double-enqueueing). - per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but- recovered occurrence counts as success, retries (parent_job set) are excluded from occurrence counting, and the current occurrence is excluded. - canceled_job_does_not_retry: cancellation wins over a pending retry. Runtime sqlx API (no .sqlx cache entry needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): exclude schedule handlers from the retry-attempt chain The retry chain listed all script children of the root by parent_job, but schedule completion handlers (on_failure/on_recovery/on_success) are also script children — when the occurrence has no retries, the handler's parent is the root itself, so a successful, never-retried job rendered a bogus 'Retries (1)' badge pointing at the handler. Filter children to re-runs of the same script (matching script_hash); real retries keep the root's hash, handlers run a different script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): surface schedule handlers on the run page Extend the run-page chain component with schedule completion handlers: - A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/ on_success runs that fired for that occurrence (found as children of the terminal attempt, identified by their synthetic created_by). - A handler's own run page now shows a 'Failure/Recovery/Success handler' label with a link back to the run it handled and its schedule. on_recovery and on_success share created_by, disambiguated by the recovery-only error_started_at arg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare An earlier `cargo sqlx prepare` on this branch ran before #8801's folder_default_permissioned_as test merged in, so it pruned the 3 query caches that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main. * fix(backend): only cascade assets from native retry attempts, not handlers (review P1) is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script child became asset-eligible — including schedule/error/recovery handlers (Script jobs with parent_job set and no flow_step_id). A handler that declares assets would then trigger a cascade the old parent_job IS NULL guard prevented. Gate parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as its chain parent (handlers run a different script). Runtime query, no sqlx cache. * fix(backend): cache the private-gated retry_setting asset-dispatch test query The same prepare-without-private that dropped the folder_default caches also pruned the cache for the retry_setting_dispatches_subscriber_as_native_script test query (asset_trigger_dispatch.rs:721). Regenerated with --features private. * fix(backend): exclude handler children from per-occurrence recovery (review) A scheduled occurrence's on_failure/on_success handler runs as a successful child (parent_job = occurrence), and the per-occurrence success EXISTS counted ANY successful child — so a failed occurrence whose error handler succeeded was marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in the merge) and on_failure_times counting. EE query now scopes the EXISTS to same-runnable children (only native retry attempts); regenerate sqlx cache + bump ee-repo-ref. native_retry_test gains a handler-child regression case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): scheduled-script retry is a native Script, not SingleStepFlow test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a native Script. Update both to assert kind='script' and that the retry policy is carried via runnable_settings_handle. * fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic) Address cubic CI review: - P1: the SingleStepFlow->native Script materialization dropped dedicated_worker, so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve it from the script row in push so the materialized Script keeps the dedicated tag. - P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the u32->i16 // retry count narrowing (scripts.rs) instead of wrapping. * fix(backend): use a retry-specific signal, not runnable equality (codex review) Address Codex CI review: - P1: is_native_retry_attempt treated any same-runnable parented Script child as a retry. WAC v2 inline children have that exact shape, so an inline child of an asset producer would cascade. Use a retry-specific signal instead: the job carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no flow_innermost_root_job. Apply the same flow_innermost guard to the EE per-occurrence EXISTS (WAC inline children must not count as a recovery). - P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate now resolves as 'retry pending' (re-check on the duplicate-id error) instead of flipping retry_pending to false and firing handlers early. - Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): explicit native_retry_attempt marker, drop heuristics Replace the per-site "is this a retry?" inference (parent_job + runnable match + flow_innermost / retry_settings) with one explicit marker: a sparse native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker also carries the attempt counter (previously in v2_job_queue.extras), so it's the single source of truth. - asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker. - EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost. - maybe_enqueue: reads/writes the marker (persistent) instead of queue extras. - Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap. - Eliminates handler / WAC-inline-child misclassification by construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex) The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but the periodic monitor.rs path deleted v2_job rows without it, orphaning markers. Add the same WHERE job_id = ANY(...) sweep there. * fix(backend): widen native_retry_attempt.attempt to integer (cubic) The smallint column was cast to/from u32 and could wrap a retry chain longer than i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32 attempt count, so no narrowing occurs on the maybe_enqueue read/write path. * feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT - Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi), computed from the native_retry_attempt marker. The run-page chain now filters retry attempts by is_retry instead of the script_hash heuristic, so WAC v2 inline children (same script, parent_job) no longer render as retries (codex). - Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep the plain catch-and-continue matching the other side-table deletes; the table is created by a startup migration so it always exists when cleanup runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but does not select is_retry, which errored with "no column found". Only the list endpoint populates the marker; #[sqlx(default)] lets every other query omit the column and default to None. * feat(backend): select is_retry in single-job GET too for consistency The list endpoint already exposes the marker; populate it on the single-job GET (both completed and queued variants) as well so a run loaded directly reflects its retry status. #[sqlx(default)] stays as a safety net for any other query. * feat(backend): reap orphaned native_retry_attempt markers via periodic sweep The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so direct job deletions (workspace/job delete, schedule clearing) would leave marker rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which must then be remembered for every future path), reap orphans in the periodic delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the job). The table is sparse so the anti-join drives off it and probes v2_job by PK — cheap. Retention still sweeps markers inline (keeps the table small so this stays cheap); a transient orphan is harmless (nothing reads is_retry for a gone job). * fix(frontend): include flow handlers in retry chain handler row (codex) Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...), whose handler job is a flow, not a script. The chain fetched children with jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts are still selected by is_retry and handlers by created_by, so both kinds surface. * fix(backend): carry concurrency/debouncing settings into native retries maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings ::default(), dropping the script/pipeline concurrency settings the failed job carried in its runnable_settings_handle. A retry of a concurrency-limited script then inserted no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and pass them in the payload, which push forwards to the materialized retry. Adds a regression test asserting the retry's handle resolves to the concurrency settings. * fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex) P1a (schedule.rs): the scheduled-retry materialization fetched the script's concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings ::default() into the SingleStepFlow payload, so the root attempt's handle held only the retry policy and the whole chain ran unbounded. Pass the fetched settings. Regression test asserts the root handle resolves to retry + concurrency. P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract — it is pub only for the integration test; the sole production caller is the worker completion path passing a DB-derived, already-authorized MiniCompletedJob. * docs(backend): attach native-retry auth contract to the function itself (codex) The doc block was merged with eval_retry_if's doc and bound to that function, leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if keeps its own doc; the native-retry + authorization contract now sits directly above maybe_enqueue_native_script_retry. * docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex) - Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since 1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the served spec couldn't see the field. Now current at 1.739.0. - schedule.rs: a retry_if gate is evaluated at failure time and fails closed without quickjs (no retry); it does not fall back to a flow path. - windmill-types jobs.rs: is_retry is selected by both the list and single-job GET endpoints (not list-only). * docs(backend): fix remaining stale retry_if/quickjs comments (codex) The retry_if block and the push materialization comments claimed push keeps retry_if on a flow path / the worker always has quickjs. The code always materializes native retry and the no-quickjs eval_retry_if path fails closed — correct the comments to that constraint. * docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex) - Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated and the job does not retry (no one-step-flow fallback). - jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence failure/recovery status across the retry chain, so the old 'restricted to schedules whose handlers don't need per-occurrence counting' claim is dropped. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Diego ImbertandClaude Opus 4.8
|
170cd79aaf |
fix: allow hyphens in postgresql database name validation (#9782)
* fix: allow hyphens in postgresql database name validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: cover hyphen acceptance in validate_dbname Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 4.8
|
24446e8009 |
fix: allow object storage test for non-super-admins, harden on cloud (#9739)
* fix: allow non-super-admin object storage test, harden SSRF surface on cloud Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: validate effective object storage host to close region/bucket SSRF bypass Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: validate gcs_base_url/token_uri in GCS service account key to close SSRF bypass Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: match url scheme case-insensitively in object storage host validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
e90b2be8fa |
perf(monitor): skip protected prefix in retention delete via cross-batch watermark (WIN-2088) (#9744)
The expired-job retention loop re-scanned the same oldest rows on every batch. When the oldest completed jobs are undeletable (children of a still-active root flow), the ORDER BY completed_at ASC scan walked that protected prefix on each of the up-to-20 batches, doing a v2_job PK lookup per row — quadratic in prefix size (measured ~9s/batch, ~180s/cleanup-cycle on a 1.5M-row prefix). Carry a completed_at watermark (max deleted) across batches and re-apply it as completed_at >= floor so each batch resumes past the already-processed prefix. Also skip the v2_job join entirely when no old root flow is active (the common case), since nothing is protected then. Measured: subsequent batches 9000ms -> 159ms; empty-set path 154 -> 36ms. The watermark only ever skips rows the current run already deleted, was protecting, or skip-locked — all deferred to the next run, identical to the unbounded scan's row set (verified: union of batched deletes == single delete, 0 diff). Mirrored in windmill-api-settings log_cleanup. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
75bafabeee |
perf(monitor): hash active-root exclusion in retention delete (WIN-2088) (#9732)
* perf(monitor): hash active-root exclusion in retention delete The expired-job retention delete (delete_expired_jobs_batch) excluded jobs belonging to still-active root flows with `COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)`. That ScalarArrayOp is evaluated per candidate row as a linear scan of $3, so cost grows with the number of active root jobs. Express the exclusion as `NOT IN (SELECT u FROM unnest($3) u WHERE u IS NOT NULL)` instead. The subquery form lets Postgres build a one-time hashed SubPlan and apply it as a filter on the ordered index scan, giving O(1) membership per candidate while preserving the `ORDER BY completed_at ASC LIMIT` early termination. The `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids). Measured on a 2M-row synthetic v2_job_completed (batch LIMIT 20000, 5-run min): active roots | != ALL (before) | NOT IN hashed (after) -------------|-----------------|---------------------- 100 | 108 ms | 104 ms 1000 | 168 ms | 105 ms 10000 | 719 ms | 131 ms Both forms return identical row sets (verified via EXCEPT, 0 diff). Neutral at small active-root counts, ~5.5x faster when many flows are active. Relates to WIN-2088 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(monitor): apply hashed active-root exclusion to log_cleanup mirror windmill-api-settings/log_cleanup.rs::delete_expired_jobs_batch carries a byte-identical copy of the retention delete and shared its prepared-query cache. Updating only monitor.rs removed that shared cache entry and broke the SQLX_OFFLINE build of the mirror. Apply the same NOT IN (hashed SubPlan) rewrite so both copies converge on one cached query and the mirror gets the same speedup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
8a0b0abead |
fix: ignore NotFound errors when deleting log files from object store (#9707)
* fix: ignore NotFound errors when deleting log files from object store Periodic and manual log cleanup delete log files from instance object storage. S3's DeleteObjects silently ignores missing keys, but GCS returns a 404 for each individual delete, which the object_store crate's default delete_stream surfaces as Error::NotFound. This produced noisy error/warning logs on every cleanup cycle even though the cleanup succeeded (DB records are removed regardless). Treat a NotFound delete as a successful no-op in both delete handlers: - monitor.rs: skip logging NotFound errors - log_cleanup.rs: count NotFound as deleted instead of an error Fixes WIN-2081 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: report 404 (already-absent) count in object store log cleanup Track delete calls that returned 404 (object already absent) separately from real deletes so operators can see how many of the attempted deletes were no-ops, instead of those numbers silently folding into s3_deleted. - monitor.rs: emit a final info summary per cleanup cycle: "N deleted, M already absent (404), K failed" (only when work occurred) - log_cleanup.rs: add s3_not_found to LogCleanupProgress (serde default for backward-compatible deserialization of in-flight rows), thread it through s3_bulk_delete and all call sites, and log a final summary on release - openapi.yaml + generated client + ObjectStoreConfigSettings.svelte: surface the 404 count in the manual cleanup status UI Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: import ObjectStoreError directly from object_store_reexports The object_store_reexports module already re-exports object_store::Error under the name ObjectStoreError, so `Error as ObjectStoreError` failed to resolve (no `Error` in that module). This compiles only behind the parquet feature, which the local dev `cargo watch` doesn't enable, so it was caught by CI's full-feature check rather than locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 4.8
|
fb44fe7af2 |
fix: require super admin for object storage config test endpoint (#9683)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Diego Imbert
|
a9e5140995 |
feat: warn when custom instance db is shared across workspaces (#9359)
* feat: warn when custom instance db is shared across workspaces * Fix leaking workspace names * sqlx prepare |
||
|
Ruben Fiszel
|
8bf7fd2c92 | feat(queue): stochastic admission + EE availability of workspace fairness algorithm (#9321) | ||
|
Ruben FiszelandClaude Opus 4.7
|
e218d60919 |
skip workspaced-route duplicate checks on cloud (#9305)
* fix(settings): skip workspaced-route duplicate checks on cloud The pre-write validation hooks for `app_workspaced_route` and `http_route_workspaced_route` query the DB for cross-workspace duplicates and fail the save when any are found. On cloud both `custom_path_exists` (apps) and `route_path_key_exists` (HTTP triggers) already scope lookups by `workspace_id` regardless of these settings, so duplicates across workspaces are expected and the validation has no runtime meaning. The result was that any cloud super-admin attempting to save instance settings with these toggles set to false received `Duplicate HTTP route paths detected` even though the setting has no effect on cloud routing. Fixes WIN-1983 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(error): render JsonErr as readable text and return 400 `Error::JsonErr` previously rendered through `#[error("Error: {0:#?}")]`, leaking Rust's `Debug` output (`Object { "error": String(...), "details": Array [...] }`) into the HTTP response body, and was bucketed into the catch-all 500 branch in `IntoResponse`. The result was a 500 status with a wall of Rust debug syntax in the toast — confusing and user-hostile. - Bucket `JsonErr` into 400 (Bad Request): every current call site (workspaced-route duplicate checks, OAuth client errors, etc.) is a client/validation issue, not an internal server fault. - Add `format_json_err_message` which surfaces the `error` field as the headline, summarises `details` (with a `- key=value` per entry), and pretty-prints the rest as JSON for unknown shapes. The frontend toast now reads e.g. Duplicate HTTP route paths detected - route_path=a, workspace_id=admins, http_method=post - route_path=a, workspace_id=starter, http_method=post Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(toast): preserve newlines and escape HTML in multi-line errors The toast renders via `{@html processMessage(message)}`, so server-side error bodies that span multiple lines (e.g. the duplicate-route response from the settings endpoint) collapsed into a single line because HTML treats consecutive whitespace (including `\n`) as a single space. When the message contains a newline, escape HTML first (defends against injected markup in server error bodies) and convert `\n` to `<br />` so multi-line errors stay readable in the toast. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fixup: address CI review feedback - toast.ts: escape HTML unconditionally. The previous gate on `\n` left single-line server error bodies unsafe under {@html}, which cubic flagged as P0. The path regex below only inserts a `<span>` around a `u/...` or `f/...` capture that can't contain HTML metacharacters, so escaping the whole input is the simpler and correct fix. - error.rs: add unit tests pinning the rendered shape of `format_json_err_message` (error+details, error-only, truncation cap, non-object fallback to pretty JSON). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
de2e243313 |
feat(queue): per-workspace fairness cap on the shared cloud worker pool (#9303)
* feat(queue): cloud-only per-workspace fairness cap on the shared worker pool
On `app.windmill.dev` the cluster runs a single default worker group, so a
single workspace flooding the queue can degrade quality of service for
everyone else. This adds an opt-in mechanism that caps any single workspace
at a configurable share of the shared worker pool when it has been
dominating cluster activity for more than a configurable window.
Detection signal counts both currently-running jobs and jobs completed in
the rolling window, so it catches workspaces hogging slots with long jobs
**and** workspaces spamming many tiny jobs (where no individual job's
started_at is old, but throughput share dominates).
Refresh is coordinated cluster-wide via a single UPDATE on
`background_task_state`: the `WHERE updated_at < now() - interval` predicate
combined with row-level locking means only one process per refresh cycle
actually runs the aggregation, regardless of fleet size. Every other
process gets the freshly written value in the same round trip via
`UNION ALL ... LIMIT 1`. Heavy aggregation rate stays at ~0.2-0.5 qps for
the whole cluster.
Pull queries are split: the existing query string and its bind shape stay
bit-identical to today, so the planner keeps using the same indexes when
fairness is off or no workspace is currently capped. A separate
`WORKER_PULL_QUERIES_FAIRNESS` adds `AND workspace_id <> ALL($2::text[])`
and is only materialized while the feature is enabled.
Hard-gated to `CLOUD_HOSTED=true` + BASE_URL host == app.windmill.dev at
three layers: frontend `cloudonly: true`, API setter rejection in
`set_global_setting_internal`, runtime check in `fairness_active`. Settings
are exposed under Jobs in the instance-settings UI; defaults are off so
the change is a no-op for self-hosted.
Two-pass pull guarantees no worker idling: if every queued job belongs to
a capped workspace, the second pass uses the unmodified pull queries.
Cap re-asserts on the next refresh.
Fixes WIN-1982
* fix(queue): address CI review findings on workspace fairness
Six fixes from the four-reviewer cross-check on #9303:
1. **Aggregation evaluation (Codex P1).** The previous `INSERT ... ON CONFLICT
DO UPDATE WHERE updated_at < ...` had the heavy `v2_job_queue ∪
v2_job_completed` aggregation inlined into `VALUES`, which Postgres
evaluates for every contender to build the proposed row — losing the
"one heavy aggregation per cycle cluster-wide" property the design
advertises. Split into three small statements: (a) cheap claim with
constant `VALUES`, (b) winner-only `UPDATE ... SET value = jsonb_build_object('overloaded', <agg>)`
(Postgres only evaluates `SET` per row matching `WHERE`, so losers never
compute the aggregation), (c) read for everyone. Heavy query now truly
runs ~0.2-0.5 qps cluster-wide regardless of fleet size.
2. **Numeric setting wraparound (cubic P1).** `u64 as u32` and downstream
`u32 as i32` could silently flip sign and feed `make_interval(secs => -N)`,
making `now() - interval` a future timestamp and disabling the
completed-jobs half of the activity signal. Clamp `duration_secs` to
[1, 86400] and `min_total_jobs` to [0, u32::MAX] before storing.
3. **`/instance_config` bypass (cubic/Claude/Codex P2).** Bulk config endpoint
sidestepped `set_global_setting_internal`'s gate; a self-hosted superadmin
could persist `workspace_fairness_*` rows via the bulk path. Mirror the
per-key check in `set_instance_config` upsert flow.
4. **DB error coerced to false (Claude P2).** `load_workspace_fairness_enabled`
collapsed `Err(_)` to `false` and unconditionally swapped the atomic — a
transient DB blip during notify-event propagation toggled the feature off
cluster-wide (and triggered a `store_pull_query` rebuild precisely when load
is highest). Now propagates the error so the atomic stays at its prior value.
5. **Refresh failure cooldown (Claude P2).** Storing `0` removed the rate
limit entirely; every subsequent pull spawned a new refresh task. Leave
`LAST_REFRESH_MICROS` at `now_us` (already written by the CAS) so the
natural interval acts as the cooldown.
6. **Visibility + duplication (Pi P2).** Mark `make_pull_query_fairness` as
`pub(crate)`. Move the duplicated `BASE_URL host == app.windmill.dev`
parser into `windmill-common::worker::is_cloud_production_host` and share
it between the API setter and the runtime path.
Verified locally:
- `POST /api/settings/global/workspace_fairness_enabled` → 400 (per-key gate)
- `PUT /api/settings/instance_config` with fairness key → 400 (bulk gate)
- `cargo check --workspace --features=private,enterprise,quickjs` — clean
Refs WIN-1982.
* fix(queue): second round of CI review nits on workspace fairness
Three issues raised by the Codex/Claude re-review of commit
|
||
|
Ruben Fiszelandwindmill-internal-app[bot]
|
ba6fb7021b |
feat: export audit logs to a dedicated object store folder (#9207)
* feat: export audit logs to dedicated object store folder * fix: gap-free audit export via snapshot-xmin gate and stable object keys * test: add integration test for audit log object store exporter * fix: cursor audit export on snapshot xmin to prevent id-leapfrog loss * fix: protect audit s3 checkpoint from config sync and bound export interval * fix: anchor audit s3 checkpoint at enable time to not skip first-window rows * fix: anchor first audit export at the enable transaction's xid * fix: use epoch timestamp floor on first audit export run to not drop old backlog * fix: anchor audit export at startup for env-var enable path * fix: anchor audit export via enabling-txn snapshot xmin trigger * fix: bound the bootstrap audit export to MAX_XID_INTERVAL per tick * refactor: store audit export cursor in background_task_state, add status endpoint * docs: align store_audit_logs_s3 setting text with the actual enable-boundary contract * [ee] refactor: move audit s3 export core logic to EE, gate on Enterprise license * chore: update ee-repo-ref to ec3cd353245e1cdf6a290528dbd7f2ac2498386c This commit updates the EE repository reference after PR #579 was merged in windmill-ee-private. Previous ee-repo-ref: 4ffc6d5f874e64d7dc4a147b4e73baa6c44867a5 New ee-repo-ref: ec3cd353245e1cdf6a290528dbd7f2ac2498386c Automated by sync-ee-ref workflow. --------- Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
|
9c6cd8c852 |
offline (URL-bound) license keys (#9089)
* [ee] feat(license): offline (URL-bound) license keys Offline keys are a 4-segment variant for air-gapped customers — no phone-home, embedded seat/CU caps, locked to the instance's base_url. Existing 3-segment online keys are unchanged. Companion PRs: - windmill-labs/windmill-ee-private (full design + EE impl) - windmill-labs/windmill-customer-service (issuance + portal) - windmill-labs/windmill-cf-worker-keygen (signing) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] refactor(license): bind offline keys via instance hash; simpler CU enforcement - /settings/license_status now surfaces an `instance_hash` superadmins share with support when requesting an offline key - OfflineMetadata: `hash` replaces `base_url`; OfflineCapStatus reports `current_cu` (last 2min) and drops the grace-period fields - verify_license_key now takes a db so EE can recheck the hash - InstanceSetting.svelte: hash copy-block + simpler status panel - Bump ee-repo-ref Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] chore(license): bump ee-repo-ref Pulls in the current_cu clamp + prod public key restoration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] refactor(license): split instance_hash endpoint; minimal cap UI; restore workers expiry toast - `instance_hash` is no longer part of /settings/license_status responses; it lives at GET /settings/instance_hash (super-admin only) so it isn't re-emitted on every status poll. The UI doesn't show it — admins fetch it explicitly when requesting a key from support. - InstanceSetting offline cap UI is now two compact green/red status lines (Seats X.X/Y and CUs X.X/Y) placed above the action buttons, matching the existing "Latest key renewal" badge style. The block-panel is gone. - "Latest key renewal" line and the "Renew key" button are now hidden when an offline key is loaded (renewal is server-disabled for offline keys). - Restore parseLicenseKey + checkLicenseExpiration toast on /workers (works for both 3- and 4-segment keys). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] chore(license): bump ee-repo-ref Pulls in the plain-SHA256 instance hash + stats_ee revert. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] chore(license): bump ee-repo-ref Picks up the alert wording change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] chore(license): bump ee-repo-ref Picks up the instance_uid cache so the periodic verify_license_key cycle no longer hits global_settings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] refactor(license): rename /settings/license_status → /offline_license_status The endpoint was only used by the offline-license UI; the other fields it returned (license_key_id, license_key_valid, kind, offline metadata) were unused. Rename to clarify scope and flatten the response — it now returns just the OfflineCapStatus (or null when no offline license is loaded). Frontend uses `offlineCapStatus != null` as the "is offline" check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] fix(ci): regenerate sqlx cache for the inline worker_ping query After reverting unused stats_ee helpers (fetch_worker_pings*), the inline `sqlx::query_as!(WorkerPingRecord, ...)` in get_stats_payload lost its cache entry — CI's check_ee_full + cargo_test were failing under SQLX_OFFLINE=true with E0282 type-inference errors. Re-running update_sqlx.sh regenerates the cache file under its current hash and prunes a couple of stale entries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] fix(license): address cubic-bot review - get_offline_license_status: propagate enforce_offline_caps errors as 500 instead of swallowing into a "no offline license" (Option::None) response - canonical_base_url: rewrite the doc to match the actual fallback behavior (lowercase + trailing-slash strip on URL parse failure); the original cross-service contract is gone since the customer-service no longer canonicalizes (treats the instance hash as opaque) - check_seat_cap_for_new_user: take an email and short-circuit when the email is already in `usr ∪ workspace_invite` so net-zero invite upserts and invite→user transitions aren't spuriously blocked at cap. Mirrors the dedup rule the count itself uses. - Bump ee-repo-ref to pull in the EE-side change Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] chore(license): bump ee-repo-ref Picks up the exact-delta seat-cap check (replaces the simple existence short-circuit). Regenerates the new sqlx cache for the bool_and query. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * [ee] fix(license): propagate get_instance_hash errors; bump ee-repo-ref - get_instance_hash: replace `.ok().flatten()` with map_err+? so DB errors during instance_uid lookup surface as 500 instead of silently returning `{"instance_hash": null}` (same pattern get_offline_license_status already uses) - Bump ee-repo-ref to pull in the enforce_offline_caps cached-state preservation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to c6cd1afe2d9e04809b30751cd1687b28a65e62b1 This commit updates the EE repository reference after PR #566 was merged in windmill-ee-private. Previous ee-repo-ref: a6d91016ae0d43c46604313aecae3aa9c778c8e0 New ee-repo-ref: c6cd1afe2d9e04809b30751cd1687b28a65e62b1 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.5
|
392888d113 |
docs: add SAFETY comments to all dynamic SQL call sites (#9009)
* docs: add SAFETY comments to all dynamic SQL call sites Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: address review feedback on SAFETY comments - Fix missed comment for obo_triggers loop in offboarding.rs - Fix variable name in comment (table -> table_name) in offboarding.rs - Fix api-settings comment to reference inline VALID_NAME regex, not validate_dbname() - Add SAFETY comments to batch_execute calls in api-settings - Fix db.rs comment: PG_SCHEMA is env var, not compile-time constant - Add doc comments on RunnableSettingsTraitInternal constants * docs: remove misleading SAFETY comment on static SQL --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
 centdixandClaude Opus 4.6
|
26a6d1e4ce |
refactor: create windmill-ai crate (part 1 — types, traits, base modules) (#8530)
* refactor: create windmill-ai crate and move base AI types from windmill-common Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: move worker AI types to windmill-ai crate Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: move QueryBuilder trait and StreamEventSink abstraction to windmill-ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add base64 dependency to windmill-ai for bedrock PDF support Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add windmill-ai refactor plan Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: address PR review — remove dead bedrock feature, add boxed_sink helper, move plan to docs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.7
|
8514347784 |
fix: serve populated jwks at /.well-known/jwks.json for vault (#8865)
* fix: serve populated jwks at /.well-known/jwks.json for vault Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: gate jwks route on private feature and use oidc_oss Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
5b3913052e |
refactor: convert read-hot globals to AtomicBool/I64 and ArcSwap (#8815)
* refactor: extract load helpers from reload_setting family Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: convert atomic primitive globals to AtomicBool/AtomicI64 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: convert CRITICAL_*/HUB_API_SECRET/INSTANCE_EVENTS_WEBHOOK/JWT_SECRET to ArcSwap Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: pin ee-repo-ref to arcswap-refactor EE branch commit * refactor: convert BASE_URL/HUB_BASE_URL/MIN_VERSION/LICENSE_KEY*/LICENSE_KEY_ID to ArcSwap Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: convert worker hot-path globals to ArcSwap (WORKER_CONFIG et al) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: pin ee-repo-ref to combined arcswap-urls+worker EE commit * chore: update ee-repo-ref to d8be8f88cb8898c8f6b27421989d53528223815d This commit updates the EE repository reference after PR #532 was merged in windmill-ee-private. Previous ee-repo-ref: c375aaaac9ec0fc0480993627d0defc8054c31a4 New ee-repo-ref: d8be8f88cb8898c8f6b27421989d53528223815d Automated by sync-ee-ref workflow. * fix: cleanup unused imports + fix 2 missed WORKER_CONFIG readers Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to ce0f8fbbbde09c4a858312d2d8716d224e99042c This commit updates the EE repository reference after PR #534 was merged in windmill-ee-private. Previous ee-repo-ref: 450b601b5aba0ca0b2045f4b5071aa8701b4bfb7 New ee-repo-ref: ce0f8fbbbde09c4a858312d2d8716d224e99042c Automated by sync-ee-ref workflow. * fix: secret_backend_integration test — BASE_URL.write().await → .store() Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: convert APP_WORKSPACED_ROUTE to AtomicBool for symmetry with HTTP_ROUTE_WORKSPACED_ROUTE Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to e587df8 (post-#535 merge) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
3f5841f84d |
feat: instance-level ruff config auto-pulled by LSP container (#8803)
* feat: add instance-level ruff config auto-pulled by LSP container Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: move ruff config to new LSP tab in instance settings Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Diego Imbert
|
3d43d31aba |
fix: refresh custom instance user password if auth failed (#8787)
* Refresh custom instance user pwd if connection failed * No longer need to check on startup * nit: unneeded inner function * fix |
||
|
Ruben FiszelandClaude Opus 4.6
|
ec9cec1d02 |
fix: treat empty global setting strings as unset (#8793)
* fix: treat empty global setting strings as unset Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: close protected-setting whitespace gap in diff and preserve empty ws override Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
3d4f4c6c38 |
feat: Fork datatables (#8339)
* export_datatable_schema * Propose to fork the datatable on ws fork * dump datatable * Dockerfile * Fix import_datatable_dump * datatable schema fork works! * Option to copy both schema and data * Datatable fork behavior * nit ui * use psql instead * remove fork_datatable route * feat: add fork_pg_database and export_pg_schema routes with DB Manager UI Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: pluralize "schema" to "schemas" in DB Manager export/import UI Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add import mode select (schema only vs schema + data) to DB Manager import Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Select schema or schema+data when important database * fix: prepend $res: prefix to resource paths in DB Manager import/export Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * style: dynamic import button label based on selected mode Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * nits * feat: add warning alert when schema+data import mode is selected Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * nit hide on cloud hosted * refactor: remove fork_behavior from datatable settings Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: split CreateWorkspace into layout wrapper and CreateWorkspaceInner Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: instantiate CreateWorkspaceInner in globalForkModal Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * nit icons * Data table fork UI * feat: pass per-datatable fork behaviors from UI to backend during workspace fork Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix fork overwriting all datatables * UI nits * custom instance db refactor * custom instance db wizard btn for all in dropdown * nit * Delete custom instance database button * Disable forking for resource datatables * Big import buttons when db empty * Revert "Disable forking for resource datatables" This reverts commit |