starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-08-18 16:02:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
14,433 Commits 725 Branches 1,509 Tags
main
Commit Graph

3 Commits

Author SHA1 Message Date
Diego Imbert 4427a3d37f feat: add workspace-specific flag for resources and variables (#8836) 
* feat: add workspace-specific flag for resources and variables

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove set_ws_specific endpoint and fix rust-client compilation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: fall back to workspace name for ws_specific file naming

When wsNameForFiles is not set (no wmill.yaml workspace config),
ws_specific items would not get workspace-suffixed filenames during
pull. Now falls back to workspace.name/workspaceId.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use workspace ID instead of CLI name for ws_specific file naming

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: pass workspace ID fallback to elementsToMap for ws_specific push

Without this, workspace-specific files (e.g., a.admins.resource.yaml)
were not recognized during push when no wmill.yaml or git branch was
available, causing spurious deletions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* ui nits

* nit

* Fix variable edit when only editing ws_specific

* mark_linked_variables_ws_specific

* Helper label

* Support json format alongside yaml

* Fix file naming push/pull asymetry & ws_specific orphans

* Revert all CLI diffs

* CLI now appends the remote ws_specific list to the local specificItems

* UI for Env switcher

* Refactor Resource/Variable editors to use dumb component

* Refactor side effects

* Editor works with multi workspaces

* Fix can_save

* Fix As JSON

* nit

* UI nits

* list_ws_specific_versions as pl sql function to avoid round trips

* UI Nits

* Per-workspace version read-only check

* fix: reset session context in list_ws_specific_versions to prevent RLS leakage

The function calls set_session_context() in a loop. Although SET LOCAL is
transaction-scoped (so settings revert at autocommit), defending against
the function being invoked inside a longer outer transaction:

- wrap the loop in a sub-block with EXCEPTION WHEN OTHERS that resets
  the session to a deny-default (windmill_user, empty session.* GUCs)
  before re-raising,
- on the happy path, reset to the same deny-default at the end of the
  function.

* feat: audit auto-marked ws_specific variables

When a resource is saved as ws_specific, every variable referenced via
$var: inside its value is auto-INSERTed into ws_specific. Previously
this happened silently. Now:

- mark_linked_variables_ws_specific takes the authed user,
- the INSERT uses RETURNING path so we know exactly which variables
  were freshly flipped (not the ones already ws_specific),
- each newly flipped variable gets a 'variables.set_ws_specific' audit
  entry pointing at the resource that triggered it.

* perf: skip mark_linked_variables_ws_specific when nothing relevant changed

update_resource was calling mark_linked_variables_ws_specific on every
save when the resource was ws_specific, even on a description-only or
label-only edit. Gate the call on `ns.value.is_some() || ns.ws_specific
== Some(true)` so we only re-mark when the $var: refs could actually
have changed or ws_specific was freshly enabled.

* docs: explain asymmetric ws_specific toggle in resource tooltip

Enabling the resource's 'Workspace specific' toggle silently marks
every variable referenced via $var: inside the value as ws_specific,
but disabling it does not un-mark those variables (they may be
referenced by other resources). Surface this in the tooltip so users
know what to expect.

* fix: surface non-404 errors when fetching ws_specific items in CLI sync

mergeWsSpecificFromServer was catching every error from listWsSpecific
and logging it at debug. That's correct for old servers without the
endpoint (404), but a 401/403/network failure would silently produce an
incomplete sync. Now distinguish 404 (debug, expected) from everything
else (warn with status + message) so users notice when the merge fails
for real reasons.

* perf: collapse compare_two_variables presence checks into one round-trip

The early-return path was issuing four sequential EXISTS queries
(ws_specific × {source, fork}, variable × {source, fork}). Combine
them into a single SELECT so the per-variable diff cost drops ~4x.

* sqlx prepare

* docs: clarify has_sql_updates invariant in update_variable

The else branch of the npath resolution is only reachable for non-rename
edits (labels-only, ws_specific-only) because ns.path being Some always
forces has_sql_updates=true at the top of the function. Add a debug_assert
and a comment explaining the invariant so a future change that decouples
ns.path from has_sql_updates trips immediately. Also use `path` directly
instead of unwrap_or_default-ing ns.path, since we know it's None here.

* chore: drop redundant ws_specific type augmentations

ListableResource and ListableVariable from $lib/gen now include
`ws_specific?: boolean` after the openapi.yaml additions in this
branch were regenerated. The intersection types in resources/+page
and variables/+page were duplicating the field — drop them.

* Put WsSpecificVersions toggle in top drawer bar

* nit size

* feat: detect local-only ws_specific items on sync push

When wmill.yaml lists a resource/variable in specificItems but the
remote isn't yet marked ws_specific for that item, sync push silently
dropped the flag because:

1. file-content diff alone never noticed (ws_specific is metadata, not
   YAML body) — push{Resource,Variable} were never called for those
   items;
2. even when called, isSuperset(local, remote) returned true and the
   early-return skipped the API call.

Now:

- mergeWsSpecificFromServer returns the raw server list alongside the
  merged config so push can compare 'in local' vs 'in server';
- a new computeWsSpecificFlagOnlyPushes helper walks the local file map,
  finds ws_specific-flagged paths absent from the server list, and the
  push function injects them as synthetic 'edited' changes (same before
  and after content) so the standard display + apply pipeline picks
  them up;
- push{Resource,Variable} no longer early-return when content matches
  but the ws_specific flag differs.

Pull is unaffected — only the push-side caller of mergeWsSpecific takes
the new (merged, serverItems) tuple.

* getDeployTo for selected ws

* refactor: ws_specific kind handling, support .json files

The ws_specific helpers had two warts:

1. computeWsSpecificFlagOnlyPushes hardcoded `.resource.{yaml,json}` /
   `.variable.{yaml,json}` magic strings, even though the existing
   getTypeStrFromPath / removeType helpers already do that work and
   already cover both extensions.
2. isSpecificItem / isItemTypeConfigured only matched `.yaml` paths,
   so users with opts.json local files got no specificItems coverage
   at all — patterns from wmill.yaml (and from mergeWsSpecificFromServer)
   are expressed with `.yaml`, and a `.json` file never matched.

Changes:

- Replace WS_SPECIFIC_KIND_MAP (a closed enum of resource+variable)
  with configKeyForItemKind, a generic kind→SpecificItemsConfig key
  mapping. Triggers fold into 'triggers' via the `_trigger` suffix,
  so adding a kind to the backend's list_ws_specific_versions doesn't
  require a CLI change.
- mergeWsSpecificFromServer now appends `${item.path}.${item.item_kind}.yaml`
  through the same helper.
- computeWsSpecificFlagOnlyPushes uses getTypeStrFromPath + removeType,
  gated by configKeyForItemKind. No more magic strings.
- isSpecificItem and isItemTypeConfigured normalize trailing `.json` to
  `.yaml` once at the entry, so a single set of patterns covers both
  extensions for the same logical item.

* refactor: dedicated change type for ws_specific flag-only pushes

Previously the sync push code injected a synthetic 'edited' Change with
before === after to nudge the apply loop into calling pushResource /
pushVariable for ws_specific-flag-only diffs, and a guard inside those
two functions skipped the early-return when the flag differed. The
contract was implicit and easy to break — any future 'skip identical
edits' optimization in the change pipeline would silently drop these
pushes.

Replace with an explicit Change variant:

  type WsSpecificFlag = {
    name: 'ws_specific_flag';
    path: string;
    kind: string;
    wsSpecific: boolean;
  };

The push apply loop now has a dedicated branch for it that calls
wmill.updateResource / updateVariable with just the ws_specific flag.
prettyChanges renders it on its own line. The dry-run JSON output picks
it up via the existing change.name / change.path passthrough.

The defensive wsSpecificMatches check inside push{Resource,Variable} is
no longer needed (sync push doesn't go through them for flag-only
diffs) and is reverted.

* drop folders

* feat(cli): warn on remote ws_specific items missing from local config

When 'wmill sync pull' fetches the server's ws_specific list, items the
server marks as ws_specific but that aren't matched by the local
wmill.yaml's specificItems patterns now produce a warning. The merge
already preserves correctness (those items are still treated as ws_specific
during this pull), but the user's config drifts from the remote — and a
later push from another machine without that config would push the item
as non-ws_specific. Surface the drift so the user can update wmill.yaml.

Also filter ws_specific_flag changes out before preCheckPermissionedAs
(it expects added/edited/deleted only and they have no content payload
so on_behalf_of resolution doesn't apply).

* fix(cli): scope ws_specific drift warning to items in this pull's changes

Previously the warning iterated every ws_specific item the server returned,
producing log spam for items unrelated to the current pull (items that
exist locally with no change, or items the user has nothing to do with
this round). Move the loop after compareDynFSElement and only warn for
items whose path appears in the changes list — i.e., items the user is
actually pulling right now.

* fix: clean up linked-side ws_specific rows on resource/variable delete

Three places left orphaned ws_specific rows behind:

1. delete_resource deleted the resource's own ws_specific row and the
   linked variables, but never the ws_specific 'variable' rows that
   mark_linked_variables_ws_specific had auto-inserted for those
   variable paths.
2. delete_variable deleted its own ws_specific row and the linked
   resource at the same path, but never a ws_specific 'resource' row at
   that path.
3. delete_resources_bulk didn't even cascade to linked variables, let
   alone clean up their ws_specific rows.

A new resource or variable later created at one of those paths would
silently inherit a stale ws_specific flag — list_ws_specific would
report it as workspace-specific, workspace diffs would treat it as
'no changes', and CLI sync would skip it.

Fix:

- delete_resource: DELETE FROM ws_specific WHERE item_kind = 'variable'
  AND path = ANY(linked_var_paths) before the linked-variable delete.
- delete_variable: DELETE FROM ws_specific WHERE item_kind = 'resource'
  AND path = path before the linked-resource delete.
- delete_resources_bulk: collect $var: refs from each bulk-deleted
  resource (mirror of single delete), then delete ws_specific 'variable'
  rows AND the variable rows themselves. Brings bulk delete in line with
  single delete semantics, including the orphan cleanup.

* fix: gate list_ws_specific by resource/variable RLS

The endpoint queried ws_specific directly under user_db, but ws_specific
itself has no per-item RLS — only a workspace-level column. Any workspace
member could enumerate every ws_specific path including those in folders
they lack read access to (e.g. f/finance/prod_db_creds), revealing path
existence that list_resources / list_variables would have hidden.

Add EXISTS clauses against resource and variable so the same path-based
RLS policies that govern those tables (see_own / see_member /
see_extra_perms_user / see_extra_perms_groups / see_folder_extra_perms_user)
also gate visibility here. The user transaction already establishes the
session context; the joins make the policies apply.

* only resources and variables

* fix(cli): make workspace-specific path mapping handle .json files

isSpecificItem() was extended to normalize .json -> .yaml so .json
files could be matched against patterns, but the surrounding helpers
remained yaml-only:

- toWorkspaceSpecificPath only mapped folder.meta.yaml / settings.yaml
  / .X.yaml — a foo.resource.json went through unchanged, so the
  workspace-specific filename was never produced.
- fromWorkspaceSpecificPath only matched .yaml extensions — pushing
  foo.dev.resource.json could not map back to foo.resource.json.
- isCurrentWorkspaceFile / isWorkspaceSpecificFile regexes ended in
  \.yaml$, missing every branch-specific .json file.

Replace the literal '.yaml' anchors with '(yaml|json)' alternations,
preserve the actual extension on round-trips, and rename the helper
buildYamlTypePattern -> buildItemTypePattern (it never had anything
extension-specific in it). getFileTypeSuffix now returns the matching
suffix for either extension. Changed:

- getFileTypeSuffix
- toWorkspaceSpecificPath / fromWorkspaceSpecificPath
- isCurrentWorkspaceFile / isWorkspaceSpecificFile
- isTriggerFile / isScheduleFile

isItemTypeConfigured / isSpecificItem don't need touching — their
checks run after normalizeJsonToYaml(), which already collapses both
extensions to .yaml at the entry.

* fix: create_resource?update_if_exists=true honors ws_specific=false

The upsert path matched on `unwrap_or(false)`, so an explicit
`ws_specific: false` and an absent flag were indistinguishable — both
fell through with no DELETE on the existing ws_specific row. Callers
trying to clear the flag via PUT-with-update_if_exists silently saw
their request ignored.

Mirror update_resource's three-way handling:

  Some(true)  -> INSERT (+ mark linked variables)
  Some(false) -> DELETE (only when update_if_exists, since a pure
                 create has no existing row anyway)
  None        -> leave the existing flag alone

create_variable doesn't have an upsert path (no ON CONFLICT), so the
same bug doesn't apply there.

* sqlx prepare

* test: cover ws_specific cleanup, RLS filtering, upsert clearing, and CLI .json paths

Backend (backend/tests/ws_specific.rs + fixture):

- test_linked_delete_cleanup: creates a ws_specific resource that
  references a variable via $var:, deletes the resource, asserts the
  cross-kind ws_specific row for the auto-marked variable is also
  removed. Then does the inverse for delete_variable, verifying the
  ws_specific 'resource' row at the same path is cleaned by variable
  delete.
- test_list_ws_specific_filters_by_rls: admin creates ws_specific items
  in u/test-user/ and u/test-user-2/; verifies admin sees both via
  list_ws_specific while a non-admin (test-user-2) only sees their own
  path — the RLS see_own policy on the joined resource/variable tables
  hides the other.
- test_create_resource_upsert_clears_ws_specific: covers the three-way
  Option<bool> handling on the upsert path: Some(true) inserts,
  Some(false) clears the existing row, None leaves it alone.

CLI:

- specific_items_unit.test.ts: add 14 tests covering toWorkspaceSpecificPath
  / fromWorkspaceSpecificPath / isWorkspaceSpecificFile /
  isCurrentWorkspaceFile / isSpecificItem / isItemTypeConfigured for
  .json files (variable, resource, trigger, schedule, folder.meta,
  settings).
- ws_specific_flag_only_unit.test.ts (new): covers
  computeWsSpecificFlagOnlyPushes — emits flag-only changes only for
  resource and variable kinds (the backend's list_ws_specific_versions
  scope), does not emit for schedules or triggers, returns empty when
  serverItems is null (older server), respects existing server entries,
  preserves .json extension on filePath.
- Export computeWsSpecificFlagOnlyPushes so it can be unit-tested.

* perf: index workspace_settings.deploy_to for the recursive CTE

list_ws_specific_versions's recursive CTE probes WHERE ws.deploy_to =
r.ws_id every iteration; without an index on workspace_settings.deploy_to
each iteration seq-scans the table — at 10M workspaces with the depth
cap of 32 that's up to 320M row reads per call.

deploy_to is sparse (most workspaces don't deploy anywhere), so a
partial index WHERE deploy_to IS NOT NULL stays small while still
covering every probe. Tucked into the existing migration since the
function and the index ship together.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-07 13:35:54 +00:00
Ruben Fiszel 5b97092997 feat: unify CLI config to workspaces, deprecate gitBranches/environments (#8767) 
* refactor: unify CLI config to workspaces, deprecate gitBranches/environments

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: update frontend examples and regenerate system prompts for workspaces config

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test: update test files to use workspaces config instead of gitBranches

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: handle --branch with --base-url correctly in sync pull/push

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: warn when --workspace overrides auto-detected branch or misses config entry

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: show reason why workspace was selected in log message

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs: clarify specificItems file naming uses gitBranch as suffix

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: rename branch-specific to workspace-specific, use workspace name as file suffix

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: rename branch-specific to workspace-specific, add comprehensive integration tests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: simplify bind and init to be workspace-centric

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: make bind/unbind interactive with --workspace and --branch flags

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: make bind interactive with profile selection, workspace name, and optional branch

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: init offers to bind workspace using same flow as wmill workspace bind

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: skip backend git-sync check in init when no workspace was bound

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: skip all API calls in init when no workspace was bound

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: log when RT namespace is skipped, offer to generate it after bind

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: warn when no workspace bound during init

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: init git-sync check uses bound workspace, not active profile

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: init uses selected profile directly, avoids re-resolving and duplicate prompt

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: init skips requireLogin, uses bound profile token directly

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: auto-pick or prompt workspace from config when no branch matches

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: show configured workspaces list and bind hint in resolution messages

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: cache bound profile to avoid duplicate profile selection prompts in init

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: hoist boundProfile scope, add 2 comprehensive integration tests covering all flows

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: rt.d.ts prompt defaults to no when file exists, better description

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: remove empty overrides from generated config, add specificItems hint

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: add inline comments for non-trivial fields, add overrides/promotionOverrides hints to bound workspaces

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: regenerate system prompts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
 2026-04-09 19:46:34 +00:00
centdix 5fd2c1a129 chore(cli): separate unit tests from integration tests and fix test cleanup (#8562) 
* fix(cli): separate unit tests from integration tests and fix test cleanup

- Rename 14 non-backend test files to *_unit.test.ts convention
- Add UNIT_ONLY env var guard in setup.ts to skip cargo build/backend startup
- Add test:unit and test:integration scripts to package.json
- Use setsid on Linux for process group management so stop() kills both
  cargo and the windmill child process
- Fix exit handler to kill process group instead of just the direct child
- Add cleanupStaleTestResources() to drop orphaned windmill_test_* databases
  and kill orphaned backend processes on startup
- Rewrite TESTING.md with current bun-based instructions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): fix process group approach - kill by db name instead of setsid

The setsid approach didn't work because setsid forks, making the PID
we get from Bun.spawn ephemeral. Instead, kill orphaned windmill child
processes by matching our unique database name in /proc/pid/environ.

Also add afterAll hook in setup.ts so full async cleanup (process kill
+ database drop) runs when all tests complete normally, not just on
SIGINT/SIGTERM.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): address PR review feedback

- Remove duplicate cleanupStaleTestResources() call in getTestBackend()
  (already called in setup.ts)
- Add regex guard on database names before SQL interpolation
- Extract shared killWindmillProcessesByEnvMatch() helper to deduplicate
  process-killing logic
- Remove redundant test:integration script (test already runs everything)
- Flip setup.ts to if/else pattern for readability

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-27 16:13:33 +00:00