mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-27 08:02:39 +00:00
7a29abc7eec491c35217445c1c08486ff030a22c
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
   
|
7a29abc7ee |
feat: run a workspace handler when a variable's value expires (#10837)
* feat: run a workspace handler when a secret variable expires Secret variables gain an optional `value_expires_at`, and a workspace can point at a script or flow to run once that date passes, so a rotation is driven by the platform instead of tracked out of band. The secret never leaves the database. The sweep selects the path, description and expiry and nothing else, so the handler is given a reference to rotate rather than a value to handle. Job results are stored in cleartext and shown in Runs, which is what makes that a boundary rather than a preference; a test asserts the job's args never contain the value. Each row is claimed by writing `expiration_dispatched_for` in the same transaction as the push, so a failed push un-claims it and a lost race claims nothing. Moving the date re-arms the row; re-saving the same date does not. A pass takes at most 100 rows under a transaction-scoped advisory lock, and orders so that one broken workspace cannot starve the rest. Dispatch runs as `g/variable_expiration_handler`, exempt from the CE group limit and from non-premium exec metering the way the error and schedule handlers already are. A handler that keeps failing backs off and raises a critical alert instead of retrying every pass, and the back-off ends when the handler is repointed. The expiry travels with the variable through workspace clone, tarball export and workspace comparison. The dispatch marker deliberately does not, so a fork re-arms on its own copy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(frontend): expose variable value expiration in the UI The variable drawer takes an optional expiry on a secret value, and the variables list badges it, turning red once the date has passed. A workspace settings tab picks the handler, its extra args, and whether variables under `u/` are skipped. That mute flag is stored inside the handler setting, so the toggle stays disabled until a handler is picked: saving it on its own would clear the setting it lives in and still report success. The expiry is carried as a tri-state through the draft deploy and copilot paths — absent leaves the remote value alone, an explicit null clears it — so deploying a draft cannot silently drop a date someone set elsewhere. The settings tab names the one case where the configured identity does not apply: a handler script carrying `on_behalf_of` runs as that user rather than as `g/variable_expiration_handler`, which matters because the same tab asks the admin to grant that group write access to the variables it rotates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(cli): sync variable expiry and the workspace expiration handler `wmill sync` carries `value_expires_at` on variables in both directions, and pushes the workspace handler alongside the error and success handlers in the grouped settings format. Dropping the key from a variable's yaml clears the remote expiry, so the file stays canonical rather than accumulating fields that can only ever be added. The handler follows the same rule, with the caveat recorded where the type is declared: an omitted handler only reaches the clear when some other setting differs, because the push short-circuits on a comparison that walks the local object's keys and so never sees a missing one. That gate is shared with the two existing handlers and is left as it is here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(frontend): open the variable expiration handler template in a new tab The "Create a handler" button hands its script template to /scripts/add through the URL hash, which `load` reads from `window.location`. A client-side navigation leaves the previous page's hash in place, so the payload was dropped and the editor opened on the default template at a `u/` path. `target="_blank"` forces a full page load, matching the success handler's "Create from template" button. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(frontend): make the variable expiry field state its own status The expiry field rendered a DateTimeInput whose time defaults to 12:00 when it holds no value, so a variable with no expiry looked like one already configured, and "cleared" looked identical to "never set". Setting an expiry is optional, so the field has to say which it is. Rebuild it on the shape ScriptAdvancedSettings already uses for Timeout, Cache and Concurrency limits: a toggle whose checked state is the stored value being present, seeding a date on enable rather than revealing an empty picker, with the input under its own label. Drop `clearable` with it, so the toggle is the single way to mean "no expiry" instead of competing with an X. Stop the surrounding copy promising a run that may not happen. A handler is only dispatched if the workspace has one configured and does not skip u/ paths, so the list tooltip now states the date alone, and the form hedges both conditions, naming the u/ mute only on personal paths where it can apply. Also anchor the mute toggle's tooltip: Popover's trigger is a span around a block-level label, which stretched to the full width of its SettingCard and left the popup floating beside the card rather than under the control. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(rust-client): initialize value_expires_at in the variable models The generated `CreateVariable` / `EditVariable` structs gain the field from openapi.yaml, and client.rs builds both with exhaustive literals. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(frontend): drop extra_args when the variable expiration handler is cleared Clearing the handler nulls the whole setting server-side, extra_args included, but the tab kept its loaded copy. A handler picked afterwards without reloading was persisted with the cleared one's arguments, which the tab does not display. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: back the variable expiration handler's built-in Slack and Teams destinations Backfills the Slack bot token grant for g/variable_expiration_handler, mirroring 20231016142659 for g/error_handler: the slack_bot folder's extra_perms is written once at Slack connect and never refreshed, and disconnecting leaves the folder behind, so workspaces that already connected Slack cannot pick the grant up from the creation-time list alone. Adds run_variable_expiration_test_job so an admin can exercise the handler on demand rather than waiting for a real expiry, which may be months away. It goes through push_variable_expiration_handler so the test run proves what matters: the handler resolves, and the group can read what it needs, under the same identity and argument shape the sweep uses. Also covers extra_args on the edit endpoint, which the settings tab now writes for the built-ins' channel. The endpoint rebuilds the setting from the request, so an omitted extra_args erases it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG * feat(frontend): offer Slack and Teams as variable expiration handlers Gives the variable expiration tab the same no-code destinations the workspace error handler has, so "tell #ops before this credential expires" no longer requires writing a script. A sibling component rather than a fourth mode of ErrorOrRecoveryHandler: that one is shared with the workspace error handler and the three schedule handlers, and its stripped-args list, hub path suffixes and Email tab are all error-shaped. This reuses the pieces that are already generic (SlackConnectionStatus, TeamsConnectionStatus, ChannelSelector, ScriptPicker, SchemaForm) and leaves those five call sites untouched. extra_args stops being an API/CLI-only escape hatch now that the built-ins write their channel into it, so it joins the unsaved-changes comparison and the discard path, normalised through the existing helper that drops the two auto-managed keys. The two hub ids are placeholders until the scripts are pushed to the hub. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG * fix(frontend): scope the handler identity notice to the chosen destination The notice told every admin to grant g/variable_expiration_handler write access to the variables it rotates. That is right for a custom handler and wrong for the built-ins, which only post a message and are granted the Slack bot token by migration. Following it for a Slack handler would hand a group write access to the workspace's secrets for no reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG * feat(frontend): flag the Slack and Teams handlers as EE Connecting a workspace to Slack or Teams is EE-only (oauth2_oss.rs returns "Slack only available on enterprise"), so on CE those two tabs can never be completed. Says so up front, the way the error/success handler tab does, instead of leaving the user at a "not connected" badge with no working path. Only on those two tabs: a custom script or flow handler is CE. Drops the "Connect Slack first" alert, now that the banner and the connection badge above it already say it twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG * chore(backend): regenerate the sqlx cache after the merge The merged workspace_settings column list in change_workspace_id is a new query text, so it needs a new cached entry. Drops the two pre-merge entries it supersedes; both had been left with conflict markers by the merge, since rename/rename markers are eight characters and escape the usual scan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG * fix(cli): narrow the expiry-clear check to a variable that exists remotely Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG * feat(frontend): point the built-in handlers at their published hub scripts Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(backend): regenerate the sqlx cache after the merge Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(frontend): disable the Slack and Teams handler controls without a license Matches the workspace error handler, which greys out its built-in destinations and hides the test button on an unlicensed instance. The custom destination stays editable: unlike the error handler, the variable expiration handler itself is not an EE feature. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: record which destination a variable expiration handler uses handler_configured now counts slack, teams, custom or cleared instead of set or cleared, so usage shows whether the built-in destinations are picked. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(frontend): let the variable expiration handler template run on any worker Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: guard the variable expiration test endpoint and correct its help text * fix: reuse the error handler's slack arg and built-in path helpers * docs: trim variable expiration sweep and backfill comments * feat: put the variable value expiration under an advanced section * fix: restrict the expiration test job to built-ins and keep expiry on conflict reload * fix: link the expiry hint to the variable's workspace and document built-in test paths * fix: give the expiration sweep its own lock id and reuse double_option * fix: pin the variable expiration presets so the hub-paths test accepts them * feat: let the variable expiration handler set how far ahead it runs * fix: give the notify before expiry setting its own card * feat: remind the variable expiration handler at a set interval * fix: tighten the gap between the Slack channel field and its test button * feat: refuse a value expiry on variables linked to an OAuth account * refactor: reuse the shared handler form for variable expiration * refactor: sweep expiring variables round-robin and check the handler exists * chore: trim comments to the constraints they record * fix: compare expiry dates by moment when deploying a variable draft * refactor: rename value_expires_at to value_valid_until * fix: neutralize variable fields in expiration handler args * fix: reserve the variable_expiration_handler group name for admins * fix: send first expiry notices before reminders * test: pin that a handler failing to start leaves its rows unclaimed * chore: say any hub handler path skips the existence check * fix: empty a pre-existing handler group before granting it the Slack token * docs: correct the registered feature usage counts * fix: let a workspace whose handler failed to start sit out for an hour * fix: send first notices before reminders across workspaces * fix: end a failed handler's wait as soon as it is redeployed * fix: show an expiry-only variable change in the AI fork diff * chore: regenerate workspace tool schemas * test: keep one variable expiration guard per behavior * fix: use the renamed hub expiration handlers and template * feat: carry the variable expiry in windmill-utils-internal 1.8.4 * fix: keep the variable expiry through MCP drafts * chore: bump windmill-utils-internal to 1.8.4 * fix: keep the variable expiry through mcp deploys * fix: disable saving a slack or teams expiry handler without ee * fix: skip the failure mark when the expiry handler was saved mid-sweep * fix: count expiry telemetry only where the expiry changed * test: assert the exact variable expiry in export and cli push tests * fix: stamp an expiry handler failure with the pass's read time * fix: wait for a pending handler lock instead of marking it failed * test: isolate expiry handler tests from the deployed-script cache * fix: retry a failing expiry handler every pass instead of waiting an hour * chore: update ee-repo-ref to b6e23d229966cf0019c0a1926a254fda658d8def This commit updates the EE repository reference after PR #744 was merged in windmill-ee-private. Previous ee-repo-ref: c52c06d145b13a6fb04459e85ca5818bf3c1bff4 New ee-repo-ref: b6e23d229966cf0019c0a1926a254fda658d8def Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: hugocasa <hugo@casademont.ch> |
||
|
4942149b05 |
feat: encrypt workspace secrets with AES-256-GCM (#11678)
* feat: encrypt workspace secrets with AES-256-GCM Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs: note the aead gate covers workers only and is one-way Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: skip secrets the upgrade cannot store and clear its guard on panic Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: start aead secret writes one release after read support Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 06610ecc950ed64a2709d7f1aa8c7b49a95fdbae This commit updates the EE repository reference after PR #869 was merged in windmill-ee-private. Previous ee-repo-ref: 599eae0034b34c5e0d3d42df37cf18bbe01f9b49 New ee-repo-ref: 06610ecc950ed64a2709d7f1aa8c7b49a95fdbae Automated by sync-ee-ref workflow. * feat: widen variable.value to fit AES-GCM ciphertext of the longest legacy secret Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: refuse legacy secrets longer than 15000 chars so every secret can be upgraded Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: cap legacy secrets at 15000 chars with a check constraint so every writer is covered Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
 Ruben FiszelandClaude Opus 5
|
ee9e550a48 |
feat(git-sync): sync extra_perms for variables (#11004)
* feat(git-sync): sync extra_perms for variables Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hvv5B8VP5Di4dbcCiVyZyE * refactor: trim the variable ACL-sync comment to the 4-line limit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hvv5B8VP5Di4dbcCiVyZyE * test: cover the revoke direction of variable extra_perms sync Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hvv5B8VP5Di4dbcCiVyZyE --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 4.8
|
ba4b368706 |
fix: prevent variable push from corrupting is_secret variables (#9705)
* fix: prevent variable push from corrupting is_secret variables Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(cli): unit-test looksLikeWorkspaceCiphertext shape detection Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): scope is_secret downgrade to single-file push, not sync push Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): warn when variable push stores a secret value as already-encrypted Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): route workspace-resolution and auth diagnostics to stderr Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(cli): rephrase comments to describe current behavior, not history Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Fable 5
|
4e9e0c024b |
feat(cli): add --yes, --secret/--no-secret and --description to variable add (#9548)
* feat(cli): add --yes, --secret/--no-secret and --description to variable add Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(cli): cover variable add create/update flag semantics Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): warn on secret downgrade in variable add and pin preserve semantics in test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
 Diego ImbertandClaude Opus 4.6
|
4427a3d37f |
feat: add workspace-specific flag for resources and variables (#8836)
* feat: add workspace-specific flag for resources and variables Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove set_ws_specific endpoint and fix rust-client compilation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: fall back to workspace name for ws_specific file naming When wsNameForFiles is not set (no wmill.yaml workspace config), ws_specific items would not get workspace-suffixed filenames during pull. Now falls back to workspace.name/workspaceId. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use workspace ID instead of CLI name for ws_specific file naming Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: pass workspace ID fallback to elementsToMap for ws_specific push Without this, workspace-specific files (e.g., a.admins.resource.yaml) were not recognized during push when no wmill.yaml or git branch was available, causing spurious deletions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ui nits * nit * Fix variable edit when only editing ws_specific * mark_linked_variables_ws_specific * Helper label * Support json format alongside yaml * Fix file naming push/pull asymetry & ws_specific orphans * Revert all CLI diffs * CLI now appends the remote ws_specific list to the local specificItems * UI for Env switcher * Refactor Resource/Variable editors to use dumb component * Refactor side effects * Editor works with multi workspaces * Fix can_save * Fix As JSON * nit * UI nits * list_ws_specific_versions as pl sql function to avoid round trips * UI Nits * Per-workspace version read-only check * fix: reset session context in list_ws_specific_versions to prevent RLS leakage The function calls set_session_context() in a loop. Although SET LOCAL is transaction-scoped (so settings revert at autocommit), defending against the function being invoked inside a longer outer transaction: - wrap the loop in a sub-block with EXCEPTION WHEN OTHERS that resets the session to a deny-default (windmill_user, empty session.* GUCs) before re-raising, - on the happy path, reset to the same deny-default at the end of the function. * feat: audit auto-marked ws_specific variables When a resource is saved as ws_specific, every variable referenced via $var: inside its value is auto-INSERTed into ws_specific. Previously this happened silently. Now: - mark_linked_variables_ws_specific takes the authed user, - the INSERT uses RETURNING path so we know exactly which variables were freshly flipped (not the ones already ws_specific), - each newly flipped variable gets a 'variables.set_ws_specific' audit entry pointing at the resource that triggered it. * perf: skip mark_linked_variables_ws_specific when nothing relevant changed update_resource was calling mark_linked_variables_ws_specific on every save when the resource was ws_specific, even on a description-only or label-only edit. Gate the call on `ns.value.is_some() || ns.ws_specific == Some(true)` so we only re-mark when the $var: refs could actually have changed or ws_specific was freshly enabled. * docs: explain asymmetric ws_specific toggle in resource tooltip Enabling the resource's 'Workspace specific' toggle silently marks every variable referenced via $var: inside the value as ws_specific, but disabling it does not un-mark those variables (they may be referenced by other resources). Surface this in the tooltip so users know what to expect. * fix: surface non-404 errors when fetching ws_specific items in CLI sync mergeWsSpecificFromServer was catching every error from listWsSpecific and logging it at debug. That's correct for old servers without the endpoint (404), but a 401/403/network failure would silently produce an incomplete sync. Now distinguish 404 (debug, expected) from everything else (warn with status + message) so users notice when the merge fails for real reasons. * perf: collapse compare_two_variables presence checks into one round-trip The early-return path was issuing four sequential EXISTS queries (ws_specific × {source, fork}, variable × {source, fork}). Combine them into a single SELECT so the per-variable diff cost drops ~4x. * sqlx prepare * docs: clarify has_sql_updates invariant in update_variable The else branch of the npath resolution is only reachable for non-rename edits (labels-only, ws_specific-only) because ns.path being Some always forces has_sql_updates=true at the top of the function. Add a debug_assert and a comment explaining the invariant so a future change that decouples ns.path from has_sql_updates trips immediately. Also use `path` directly instead of unwrap_or_default-ing ns.path, since we know it's None here. * chore: drop redundant ws_specific type augmentations ListableResource and ListableVariable from $lib/gen now include `ws_specific?: boolean` after the openapi.yaml additions in this branch were regenerated. The intersection types in resources/+page and variables/+page were duplicating the field — drop them. * Put WsSpecificVersions toggle in top drawer bar * nit size * feat: detect local-only ws_specific items on sync push When wmill.yaml lists a resource/variable in specificItems but the remote isn't yet marked ws_specific for that item, sync push silently dropped the flag because: 1. file-content diff alone never noticed (ws_specific is metadata, not YAML body) — push{Resource,Variable} were never called for those items; 2. even when called, isSuperset(local, remote) returned true and the early-return skipped the API call. Now: - mergeWsSpecificFromServer returns the raw server list alongside the merged config so push can compare 'in local' vs 'in server'; - a new computeWsSpecificFlagOnlyPushes helper walks the local file map, finds ws_specific-flagged paths absent from the server list, and the push function injects them as synthetic 'edited' changes (same before and after content) so the standard display + apply pipeline picks them up; - push{Resource,Variable} no longer early-return when content matches but the ws_specific flag differs. Pull is unaffected — only the push-side caller of mergeWsSpecific takes the new (merged, serverItems) tuple. * getDeployTo for selected ws * refactor: ws_specific kind handling, support .json files The ws_specific helpers had two warts: 1. computeWsSpecificFlagOnlyPushes hardcoded `.resource.{yaml,json}` / `.variable.{yaml,json}` magic strings, even though the existing getTypeStrFromPath / removeType helpers already do that work and already cover both extensions. 2. isSpecificItem / isItemTypeConfigured only matched `.yaml` paths, so users with opts.json local files got no specificItems coverage at all — patterns from wmill.yaml (and from mergeWsSpecificFromServer) are expressed with `.yaml`, and a `.json` file never matched. Changes: - Replace WS_SPECIFIC_KIND_MAP (a closed enum of resource+variable) with configKeyForItemKind, a generic kind→SpecificItemsConfig key mapping. Triggers fold into 'triggers' via the `_trigger` suffix, so adding a kind to the backend's list_ws_specific_versions doesn't require a CLI change. - mergeWsSpecificFromServer now appends `${item.path}.${item.item_kind}.yaml` through the same helper. - computeWsSpecificFlagOnlyPushes uses getTypeStrFromPath + removeType, gated by configKeyForItemKind. No more magic strings. - isSpecificItem and isItemTypeConfigured normalize trailing `.json` to `.yaml` once at the entry, so a single set of patterns covers both extensions for the same logical item. * refactor: dedicated change type for ws_specific flag-only pushes Previously the sync push code injected a synthetic 'edited' Change with before === after to nudge the apply loop into calling pushResource / pushVariable for ws_specific-flag-only diffs, and a guard inside those two functions skipped the early-return when the flag differed. The contract was implicit and easy to break — any future 'skip identical edits' optimization in the change pipeline would silently drop these pushes. Replace with an explicit Change variant: type WsSpecificFlag = { name: 'ws_specific_flag'; path: string; kind: string; wsSpecific: boolean; }; The push apply loop now has a dedicated branch for it that calls wmill.updateResource / updateVariable with just the ws_specific flag. prettyChanges renders it on its own line. The dry-run JSON output picks it up via the existing change.name / change.path passthrough. The defensive wsSpecificMatches check inside push{Resource,Variable} is no longer needed (sync push doesn't go through them for flag-only diffs) and is reverted. * drop folders * feat(cli): warn on remote ws_specific items missing from local config When 'wmill sync pull' fetches the server's ws_specific list, items the server marks as ws_specific but that aren't matched by the local wmill.yaml's specificItems patterns now produce a warning. The merge already preserves correctness (those items are still treated as ws_specific during this pull), but the user's config drifts from the remote — and a later push from another machine without that config would push the item as non-ws_specific. Surface the drift so the user can update wmill.yaml. Also filter ws_specific_flag changes out before preCheckPermissionedAs (it expects added/edited/deleted only and they have no content payload so on_behalf_of resolution doesn't apply). * fix(cli): scope ws_specific drift warning to items in this pull's changes Previously the warning iterated every ws_specific item the server returned, producing log spam for items unrelated to the current pull (items that exist locally with no change, or items the user has nothing to do with this round). Move the loop after compareDynFSElement and only warn for items whose path appears in the changes list — i.e., items the user is actually pulling right now. * fix: clean up linked-side ws_specific rows on resource/variable delete Three places left orphaned ws_specific rows behind: 1. delete_resource deleted the resource's own ws_specific row and the linked variables, but never the ws_specific 'variable' rows that mark_linked_variables_ws_specific had auto-inserted for those variable paths. 2. delete_variable deleted its own ws_specific row and the linked resource at the same path, but never a ws_specific 'resource' row at that path. 3. delete_resources_bulk didn't even cascade to linked variables, let alone clean up their ws_specific rows. A new resource or variable later created at one of those paths would silently inherit a stale ws_specific flag — list_ws_specific would report it as workspace-specific, workspace diffs would treat it as 'no changes', and CLI sync would skip it. Fix: - delete_resource: DELETE FROM ws_specific WHERE item_kind = 'variable' AND path = ANY(linked_var_paths) before the linked-variable delete. - delete_variable: DELETE FROM ws_specific WHERE item_kind = 'resource' AND path = path before the linked-resource delete. - delete_resources_bulk: collect $var: refs from each bulk-deleted resource (mirror of single delete), then delete ws_specific 'variable' rows AND the variable rows themselves. Brings bulk delete in line with single delete semantics, including the orphan cleanup. * fix: gate list_ws_specific by resource/variable RLS The endpoint queried ws_specific directly under user_db, but ws_specific itself has no per-item RLS — only a workspace-level column. Any workspace member could enumerate every ws_specific path including those in folders they lack read access to (e.g. f/finance/prod_db_creds), revealing path existence that list_resources / list_variables would have hidden. Add EXISTS clauses against resource and variable so the same path-based RLS policies that govern those tables (see_own / see_member / see_extra_perms_user / see_extra_perms_groups / see_folder_extra_perms_user) also gate visibility here. The user transaction already establishes the session context; the joins make the policies apply. * only resources and variables * fix(cli): make workspace-specific path mapping handle .json files isSpecificItem() was extended to normalize .json -> .yaml so .json files could be matched against patterns, but the surrounding helpers remained yaml-only: - toWorkspaceSpecificPath only mapped folder.meta.yaml / settings.yaml / .X.yaml — a foo.resource.json went through unchanged, so the workspace-specific filename was never produced. - fromWorkspaceSpecificPath only matched .yaml extensions — pushing foo.dev.resource.json could not map back to foo.resource.json. - isCurrentWorkspaceFile / isWorkspaceSpecificFile regexes ended in \.yaml$, missing every branch-specific .json file. Replace the literal '.yaml' anchors with '(yaml|json)' alternations, preserve the actual extension on round-trips, and rename the helper buildYamlTypePattern -> buildItemTypePattern (it never had anything extension-specific in it). getFileTypeSuffix now returns the matching suffix for either extension. Changed: - getFileTypeSuffix - toWorkspaceSpecificPath / fromWorkspaceSpecificPath - isCurrentWorkspaceFile / isWorkspaceSpecificFile - isTriggerFile / isScheduleFile isItemTypeConfigured / isSpecificItem don't need touching — their checks run after normalizeJsonToYaml(), which already collapses both extensions to .yaml at the entry. * fix: create_resource?update_if_exists=true honors ws_specific=false The upsert path matched on `unwrap_or(false)`, so an explicit `ws_specific: false` and an absent flag were indistinguishable — both fell through with no DELETE on the existing ws_specific row. Callers trying to clear the flag via PUT-with-update_if_exists silently saw their request ignored. Mirror update_resource's three-way handling: Some(true) -> INSERT (+ mark linked variables) Some(false) -> DELETE (only when update_if_exists, since a pure create has no existing row anyway) None -> leave the existing flag alone create_variable doesn't have an upsert path (no ON CONFLICT), so the same bug doesn't apply there. * sqlx prepare * test: cover ws_specific cleanup, RLS filtering, upsert clearing, and CLI .json paths Backend (backend/tests/ws_specific.rs + fixture): - test_linked_delete_cleanup: creates a ws_specific resource that references a variable via $var:, deletes the resource, asserts the cross-kind ws_specific row for the auto-marked variable is also removed. Then does the inverse for delete_variable, verifying the ws_specific 'resource' row at the same path is cleaned by variable delete. - test_list_ws_specific_filters_by_rls: admin creates ws_specific items in u/test-user/ and u/test-user-2/; verifies admin sees both via list_ws_specific while a non-admin (test-user-2) only sees their own path — the RLS see_own policy on the joined resource/variable tables hides the other. - test_create_resource_upsert_clears_ws_specific: covers the three-way Option<bool> handling on the upsert path: Some(true) inserts, Some(false) clears the existing row, None leaves it alone. CLI: - specific_items_unit.test.ts: add 14 tests covering toWorkspaceSpecificPath / fromWorkspaceSpecificPath / isWorkspaceSpecificFile / isCurrentWorkspaceFile / isSpecificItem / isItemTypeConfigured for .json files (variable, resource, trigger, schedule, folder.meta, settings). - ws_specific_flag_only_unit.test.ts (new): covers computeWsSpecificFlagOnlyPushes — emits flag-only changes only for resource and variable kinds (the backend's list_ws_specific_versions scope), does not emit for schedules or triggers, returns empty when serverItems is null (older server), respects existing server entries, preserves .json extension on filePath. - Export computeWsSpecificFlagOnlyPushes so it can be unit-tested. * perf: index workspace_settings.deploy_to for the recursive CTE list_ws_specific_versions's recursive CTE probes WHERE ws.deploy_to = r.ws_id every iteration; without an index on workspace_settings.deploy_to each iteration seq-scans the table — at 10M workspaces with the depth cap of 32 that's up to 320M row reads per call. deploy_to is sparse (most workspaces don't deploy anywhere), so a partial index WHERE deploy_to IS NOT NULL stays small while still covering every probe. Tucked into the existing migration since the function and the index ship together. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
38acaa3653 |
fix(cli): fix 13 CLI bugs — exit codes, sync tar fallback, variable encryption, JSON output (#8582)
* fix(cli): fix 13 CLI bugs — exit codes, sync tar fallback, variable encryption, JSON output, parent dirs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): address PR review — TarAsZip.folder(), retry timeout, stderr hint Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): update resource-type list test to handle empty state message Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.5
|
501a4ff2a9 |
fix: Improve CLI developer experience: error handling, sync workflow, JSON output, workspace forks (#8578)
* fix(cli): address 28 DX friction points across CLI commands Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore(cli): regenerate system prompts after help text updates Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address PR review feedback Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): update removeType tests to match lenient behavior Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address CE/EE sync friction and improve JSON output Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): revert instance config masking to avoid breaking push flow Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): mask instance secrets by default with interactive prompt Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore(cli): regenerate system prompts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): use stderr for errors, optimize skipped-files scan, rename --auto to --auto-metadata Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(cli): improve workspace fork lifecycle — delete-fork fallback, list-forks, --workspace override Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): update fork merge instructions to reference all merge methods Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): clarify skipped-files warning comment re DynFSElement traversal Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
4fedfdfd11 |
feat(cli): add consistent get/list/new subcommands for all item types (#8047)
* feat(cli): add consistent get/list/new subcommands for all item types Make the CLI consistent so every item type (script, flow, app, resource, resource-type, variable, schedule, folder, trigger) supports get/list/new subcommands, enabling the CLI to be used as a full API client in bash scripts with jq piping. - Add --json flag to all list commands for machine-readable output - Register explicit "list" subcommand alongside default action - Add "get <path> [--json]" subcommand to fetch single items from API - Rename "bootstrap" to "new" for script/flow, keep "bootstrap" as alias - Add "new" subcommand for resource, resource-type, variable, schedule, folder, and trigger to create local template YAML files - Update cli-commands skill documentation for wmill init - Add integration tests for all new commands Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * feat: install wmill CLI in Docker images and use it for bash variable/resource access - Install windmill-cli via bun in all Dockerfiles that include bun - DockerfileCli: switch from node:slim to oven/bun:slim - CLI: auto-configure from WM_WORKSPACE/WM_TOKEN/BASE_INTERNAL_URL env vars as last-resort fallback when no workspace is configured - Frontend: replace curl-based bash snippets with wmill variable/resource get - Add backend integration tests for wmill CLI in bash scripts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): install windmill-cli in backend test workflow Ensures wmill is available on PATH for bash integration tests that use `wmill variable get` and `wmill resource get`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(cli): replace @std/* Deno dependencies with Node.js equivalents Replace @std/log with a lightweight custom logger (core/log.ts), @std/path with node:path, and @std/yaml with the yaml npm package. Also fix process hang on exit, add --node option to install_dev.sh, and add missing hasRequiredPermissions to NpmProvider. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * all * all * refactor(cli): replace @ayonli/jsext and @std/encoding with lightweight alternatives Replace @ayonli/jsext (8.4MB) with tar-stream (32kB) for tar creation, replace @std/encoding with Node.js Buffer.toString("hex"), and fix @windmill-labs/shared-utils to use direct npm instead of JSR mirror. Also resolve merge conflicts in sync.ts and fix pre-existing type errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): use singleQuote YAML output and pass yamlOptions in gitsync pull The yaml library defaults to double quotes, but the codebase (and tests) expect single-quoted strings. Add singleQuote: true to yamlOptions and pass yamlOptions to gitsync-settings pull writeFile calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * all * fix(cli): address code review feedback - Install CLI from source in backend tests instead of npm - Fix script bootstrap catch block to re-throw "File already exists" - Add type-safe local variable after trigger kind validation - Use created_by instead of policy.on_behalf_of for app get output - Note --kind is recommended for faster trigger lookup in help text - Document node symlink purpose in Dockerfiles Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): use /usr/bin for wmill wrapper to ensure it's in PATH Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(ci): install wmill to ~/.local/bin to avoid permission issues Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci(backend): switch to Blacksmith runner and add cargo caching - Switch from ubicloud-standard-16 to blacksmith-16vcpu-ubuntu-2404 for faster NVMe-backed builds - Add stickydisk for cargo target directory (persistent NVMe cache across runs) - Add cache for cargo registry and git dependencies - Upgrade DuckDB FFI cache from actions/cache@v3 to useblacksmith/cache@v1 - Enable CARGO_INCREMENTAL=1 to benefit from persistent target cache Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix ci --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
a2cefdf0a2 |
refactor(cli): migrate CLI from Deno to Bun/Node.js (#8041)
* fix: only enable EE features in test backend when license key is available Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: skip EE tests without license key and exclude test-skills from test discovery Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: unskip passing tests and add duplicate (remote, workspaceId) check in addWorkspace Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(cli): migrate from Deno APIs to Node.js/Bun-compatible APIs Replace Deno-specific APIs with Node.js equivalents across the entire CLI codebase to enable running on Node.js/Bun. Switch build system from dnt to bun, update imports from jsr:/npm: prefixed to bare specifiers, and add package.json/tsconfig.json for the Node.js ecosystem. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * test(cli): expand test coverage with new integration and unit tests Add standalone_commands.test.ts covering folder list, schedule list, resource-type list/push/update, script show/run/bootstrap, and user commands. Add unit tests for filePathExtensionFromContentType and removeExtensionToPath. Add git_unit, local_encryption_unit, resource_folders_unit, and settings_unit test files. Fix schedule cron expressions (6-field format), add includeSchedules flag, improve test setup with pre-build and auto-cleanup, and support TEST_CLI_RUNTIME=node. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): replace Deno.readFile with node:fs in WASM loaders and add schema parsing tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(cli): switch WASM parsers from local files to npm packages Use published windmill-parser-wasm-* npm packages instead of local wasm/ files. A loadParser() helper uses createRequire to resolve the .wasm binary from node_modules and passes it to init() via readFileSync, avoiding fetch() and Deno.readFile() patches. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(cli): add coverage for --locks-required lint feature Add 15 tests covering the lock-checking functionality merged from main: - checkMissingLocks: standalone scripts (python, bun, bash), inline lock file resolution (valid, empty, missing), flow inline rawscripts (with/without locks, nested forloopflow), app inline scripts, raw apps without backend folder - runLint --locks-required integration: reports issues when locks missing, skips checks when flag absent, passes when locks exist Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci(cli): replace Deno with Bun in CI workflows - cli-tests.yml: remove Deno setup, use `bun test` instead of `deno test`, add `bun install` step for dependency installation - npm_on_release.yml: replace Deno setup with Bun setup for CLI publishing - build.sh: add `bun install` before building so CI has dependencies Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): pre-start backend in test preload and remove Deno test leftovers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): normalize path separators for Windows compatibility Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * more tests + windows * ci(cli): use Blacksmith runner for Windows tests Switch test-windows job from windows-latest to blacksmith-16vcpu-windows-2025 for faster CI execution. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): fix Windows path separator expectations in unit tests buildMetadataPath and extractResourceName normalize to forward slashes internally, so tests should not expect platform-specific separators in their output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): fix Windows CI test failures for dev_server and script_run Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): set BUN_PATH and NODE_BIN_PATH for backend worker on Windows Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci(cli): add SSH debug step on Windows test failure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): use native path separators for ignore check in dev mode on Windows Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
 centdixandAlexander Petric
|
69c2a7c1c8 |
chore(cli): better folder structure + add config utils (#6319)
* organize in folders * add config command * fix * cleaning * move utility functions * merge * only show token with option * only show token with option * fix * remove config command * add config utils * change paths * nit * clean path assigner --------- Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com> |