From 9444049d6013c77a5f25f01a736eb5cd741fb3e6 Mon Sep 17 00:00:00 2001 From: hugocasa Date: Tue, 8 Sep 2026 11:54:43 +0200 Subject: [PATCH] feat: bring gitlab repositories to parity for git sync (#10938) * feat: track and rotate gitlab git-sync repository tokens * chore: point ee-repo-ref at the gitlab credential branch * fix: strip server-owned credential status and correct expiry copy * fix: gate credential maintenance on enterprise and alert on stalled renewal * fix: alert on an auto-renewed token only once it has actually expired * feat: receive gitlab push webhooks for instant git sync pull * feat: open gitlab merge requests and post diff previews on them Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: keep gitlab merge request previews out of the project's own pipeline Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: bound the credential maintenance pass and gate the gitlab picker on a license Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: create the gitlab picker's variable in the edited workspace Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: make the gitlab picker's variable path collision-resistant Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: state the gitlab scope and rotation facts the code relies on Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: resolve the check marker's repository from its path, not a stored url Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: refuse to finish a check whose repository has been repointed Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: trust a check marker's captured url when it carries no identity Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: recreate a missing webhook from credential maintenance Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: state that relative-url gitlab installs are out of scope Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: keep credential status out of exports and clear stale webhook warnings Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: refuse an unprovable check and guard the picker on the stored repository Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: re-check the picker's target path at the moment it is written Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: snapshot the picker's inputs before it starts writing Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: recommend a project access token per repository Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * [ee] feat: keep the git-sync credential in workspace settings * [ee] fix: drop a removed repo's credential and honor the workspace override * [ee] fix: resolve a fork's git credential from its whole ancestry * [ee] refactor: reuse fork_ancestor_chain instead of a second ancestry walk * [ee] fix: resolve an app installation from the whole ancestry, not the parent * [ee] revert: keep the app installation fallback at one level * fix: store the git credential only once the resource is saved * fix: keep a repository's credential when it leaves git sync settings * docs: cut the gitlab picker's token guidance down to what it needs * feat: mark a repository whose credential windmill holds * fix: ignore the managed-credential marker when the url carries a token * docs: drop the picker's setup alert for a line by the token field * feat: replace a repository's stored token from its resource * fix: store a picked credential for its own workspace, before the resource * refactor: key a stored git credential by its repository, not its resource Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: refresh the sqlx cache for the repository-keyed credential queries Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: gate the credential pass budget on the features that use it Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: decide credential rotation ownership by repository, not resource path Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * refactor: renew only the credentials windmill holds, not tokens in a repo url Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: authenticate the fork-branch poll and correct the renewal guidance Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: do not claim a managed credential for a url the client cannot resolve Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: define the credential facade for private builds without enterprise Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: pin the listed token before the await and name the real renewal blocker Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: pin the token the replace flow checked, and derive the scope test once Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: classify the renewal state once so the card cannot contradict itself Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * refactor: ask only whether the token gets renewed, not why it does not Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * refactor: replace the managed-credential marker with a server answer Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: read renewal from the credential and its origin, not a removed field Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: read the provider for url-token repos, await the origin before defaulting, and visit unchecked repos last The maintenance pass sorted repositories with no recorded check first on the premise that they cost nothing, but a token-in-URL remote on a host that is not GitLab is probed every pass and never records a check, so it held the head of the list ahead of the tokens that expire. Such repositories now sort last. The card decided its delivery defaults before the origin lookup landed, so a freshly picked GitLab repository never got webhook delivery; the two lookups are awaited together. The resource editor offers to replace a token only where it is held, not in a fork that borrows it, and the replace flow refuses a URL it cannot parse instead of keying the token to it. Attaching a stored credential to a commit-hash probe now requires admin, matching the installation credential beside it. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: describe the gitlab listing token the way the picker and the setup guide do Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * refactor: a token in the repository url is a plain remote, not a tracked credential Drops the status fingerprint that told one URL token from another, the docs' promise that such a token's expiry is reported, and the test's expectation that a URL-token repository declares a host. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: the card reads the credential origin for managed controls and honours the licence for a borrowed token Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: bump the ee ref Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: hide a repository's credential line once nothing is held for the repository it names Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: describe the exported credential status as it is Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * refactor: run the credential maintenance pass as its own task, without a budget The pass ran inside the monitor's join, whose deadline cancels every future in it, and a rotation cancelled between GitLab issuing a token and Windmill storing it loses the token family. A wall-clock budget with a least-recently-checked ordering kept it under the deadline. Spawning the pass instead makes the deadline irrelevant, so the budget, the ordering and the counter go; the advisory lock keeps a slow pass from overlapping the next, as it already did. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: say what detaching the maintenance pass buys, and what it does not Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: run git sync on the hub script version that reads a stored credential Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: run the deploy push and the connection test on the hub versions that read a stored credential Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: keep App repositories and plain remotes out of the stored-credential paths Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: host-neutral deploy preview wording, drop the project filter from the GitLab picker Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: bump ee ref, rotation no longer retains a second connection per repository Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: bump ee ref, the rotation write-back holds a single connection Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: hold the credential maintenance lock in a transaction so a dead sweep releases it Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: describe the credential-stored callback as it fires Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: keep the credential maintenance lock past the pool's idle-in-transaction timeout Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: update ee-repo-ref to e092518ee60e33160fee9ae91a4d109566f7b0ee This commit updates the EE repository reference after PR #771 was merged in windmill-ee-private. Previous ee-repo-ref: 74481f7cc345757aebb2a8b04d3a22978328c348 New ee-repo-ref: e092518ee60e33160fee9ae91a4d109566f7b0ee Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 Co-authored-by: windmill-internal-app[bot] --- ...83dce0c9986f3847227c2f66daa84d4109d7d.json | 15 ++ ...e72dc33ca421338e04277ac91c745351f89f3.json | 23 ++ ...1d3f9e80375b52748c458854a4883a6ecc8f6.json | 24 ++ ...31c79281d5cb2b0e81f1e832eac6eb3cc32e2.json | 16 ++ ...260d5ead34a6a3783dd21db08c7ee049eb232.json | 15 ++ ...a4d446b8ea00cf51e02138a06c245ab33252e.json | 16 ++ ...fd2a14b2e9e613bcf4bc80abbfd495637ab2f.json | 16 ++ ...6e96cd3a980411d80c6b10bae949cf2c42385.json | 22 ++ ...496c9448cffba5786b51e748bec0165a6519c.json | 17 ++ ...05d53bc756d00c8d28462097717502d82cc15.json | 23 ++ ...ced12d26f67f1b059c7c2afe66f8e4b3a9749.json | 22 ++ ...c90d7d82bd20ea30f1dc08f2e2066fb8d6f4b.json | 26 ++ ...18313f0434758413dbf83f8575d09de98fc25.json | 23 ++ backend/ee-repo-ref.txt | 2 +- ...it_credentials_workspace_settings.down.sql | 1 + ..._git_credentials_workspace_settings.up.sql | 8 + backend/src/monitor.rs | 189 ++++++++++++- .../fixtures/git_sync_fork_credential.sql | 55 ++++ backend/tests/git_sync_fork_credential.rs | 252 ++++++++++++++++++ .../windmill-api-workspaces/src/workspaces.rs | 38 ++- .../src/workspaces_extra.rs | 2 +- backend/windmill-api/openapi.yaml | 178 ++++++++++++- backend/windmill-api/src/git_sync_oss.rs | 5 + backend/windmill-api/src/lib.rs | 9 + backend/windmill-api/src/workspaces_export.rs | 16 +- backend/windmill-common/src/git_sync_oss.rs | 24 +- backend/windmill-common/src/workspaces.rs | 53 +++- backend/windmill-store/src/resources.rs | 19 ++ .../windmill-worker/src/result_processor.rs | 130 ++++++--- docs/git-sync-gitlab-setup.md | 170 ++++++++++++ .../src/lib/components/ApiConnectForm.svelte | 23 ++ .../src/lib/components/AppConnectInner.svelte | 9 + .../components/GitHubAppIntegration.svelte | 4 +- .../lib/components/GitLabIntegration.svelte | 221 +++++++++++++++ .../src/lib/components/ResourceEditor.svelte | 62 ++++- .../src/lib/components/ResourceForm.svelte | 27 +- .../git_sync/GitSyncRepositoryCard.svelte | 181 +++++++++++-- .../git_sync/ReplaceGitCredential.svelte | 134 ++++++++++ frontend/src/lib/hubPaths.json | 4 +- 39 files changed, 1975 insertions(+), 99 deletions(-) create mode 100644 backend/.sqlx/query-1d8ccd32266637d7f7915f92a8483dce0c9986f3847227c2f66daa84d4109d7d.json create mode 100644 backend/.sqlx/query-1fc9a0aeabb0a33efc37167be3de72dc33ca421338e04277ac91c745351f89f3.json create mode 100644 backend/.sqlx/query-48055203c97499ab4fc4dcbe9271d3f9e80375b52748c458854a4883a6ecc8f6.json create mode 100644 backend/.sqlx/query-64f72d7477f7c82d1596560ff6c31c79281d5cb2b0e81f1e832eac6eb3cc32e2.json create mode 100644 backend/.sqlx/query-73a3417f5331032a9b2afcf5b1c260d5ead34a6a3783dd21db08c7ee049eb232.json create mode 100644 backend/.sqlx/query-9c0ebee945eee03d667a4f38d18a4d446b8ea00cf51e02138a06c245ab33252e.json create mode 100644 backend/.sqlx/query-af5609185c6218d12fcb4a4234cfd2a14b2e9e613bcf4bc80abbfd495637ab2f.json create mode 100644 backend/.sqlx/query-b2f91eb32edd8db1605aeb5a17b6e96cd3a980411d80c6b10bae949cf2c42385.json create mode 100644 backend/.sqlx/query-bbb8331348216892e4714e4338c496c9448cffba5786b51e748bec0165a6519c.json create mode 100644 backend/.sqlx/query-ce3340a43a141cc7527211db16905d53bc756d00c8d28462097717502d82cc15.json create mode 100644 backend/.sqlx/query-d0eeb992a826d26376e9802a8c4ced12d26f67f1b059c7c2afe66f8e4b3a9749.json create mode 100644 backend/.sqlx/query-ddd41de79b23b0436bbc7997751c90d7d82bd20ea30f1dc08f2e2066fb8d6f4b.json create mode 100644 backend/.sqlx/query-f3a27f7781d986d9917a7022e1118313f0434758413dbf83f8575d09de98fc25.json create mode 100644 backend/migrations/20260904110943_git_credentials_workspace_settings.down.sql create mode 100644 backend/migrations/20260904110943_git_credentials_workspace_settings.up.sql create mode 100644 backend/tests/fixtures/git_sync_fork_credential.sql create mode 100644 backend/tests/git_sync_fork_credential.rs create mode 100644 docs/git-sync-gitlab-setup.md create mode 100644 frontend/src/lib/components/GitLabIntegration.svelte create mode 100644 frontend/src/lib/components/git_sync/ReplaceGitCredential.svelte diff --git a/backend/.sqlx/query-1d8ccd32266637d7f7915f92a8483dce0c9986f3847227c2f66daa84d4109d7d.json b/backend/.sqlx/query-1d8ccd32266637d7f7915f92a8483dce0c9986f3847227c2f66daa84d4109d7d.json new file mode 100644 index 0000000000..b9cbc6c382 --- /dev/null +++ b/backend/.sqlx/query-1d8ccd32266637d7f7915f92a8483dce0c9986f3847227c2f66daa84d4109d7d.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_settings (workspace_id, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, git_credentials, ducklake, dbt_warehouses, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts, guest_access_enabled, guest_jwt_public_key, guest_jwt_jwks_url) SELECT $1, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, git_credentials, ducklake, dbt_warehouses, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts, guest_access_enabled, guest_jwt_public_key, guest_jwt_jwks_url FROM workspace_settings WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "1d8ccd32266637d7f7915f92a8483dce0c9986f3847227c2f66daa84d4109d7d" +} diff --git a/backend/.sqlx/query-1fc9a0aeabb0a33efc37167be3de72dc33ca421338e04277ac91c745351f89f3.json b/backend/.sqlx/query-1fc9a0aeabb0a33efc37167be3de72dc33ca421338e04277ac91c745351f89f3.json new file mode 100644 index 0000000000..6b4324af4c --- /dev/null +++ b/backend/.sqlx/query-1fc9a0aeabb0a33efc37167be3de72dc33ca421338e04277ac91c745351f89f3.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT (\n SELECT elem\n FROM jsonb_array_elements(git_credentials) AS elem\n WHERE elem->>'repo_identity' = $2\n )\n FROM workspace_settings WHERE workspace_id = $1 FOR UPDATE\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "elem", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "1fc9a0aeabb0a33efc37167be3de72dc33ca421338e04277ac91c745351f89f3" +} diff --git a/backend/.sqlx/query-48055203c97499ab4fc4dcbe9271d3f9e80375b52748c458854a4883a6ecc8f6.json b/backend/.sqlx/query-48055203c97499ab4fc4dcbe9271d3f9e80375b52748c458854a4883a6ecc8f6.json new file mode 100644 index 0000000000..b2cfb1cd89 --- /dev/null +++ b/backend/.sqlx/query-48055203c97499ab4fc4dcbe9271d3f9e80375b52748c458854a4883a6ecc8f6.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT elem->'credential'\n FROM workspace_settings, jsonb_array_elements(git_sync->'repositories') AS elem\n WHERE workspace_id = $1 AND elem->>'git_repo_resource_path' IN ($2, $3)\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "48055203c97499ab4fc4dcbe9271d3f9e80375b52748c458854a4883a6ecc8f6" +} diff --git a/backend/.sqlx/query-64f72d7477f7c82d1596560ff6c31c79281d5cb2b0e81f1e832eac6eb3cc32e2.json b/backend/.sqlx/query-64f72d7477f7c82d1596560ff6c31c79281d5cb2b0e81f1e832eac6eb3cc32e2.json new file mode 100644 index 0000000000..dc1f6ffbb1 --- /dev/null +++ b/backend/.sqlx/query-64f72d7477f7c82d1596560ff6c31c79281d5cb2b0e81f1e832eac6eb3cc32e2.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE workspace_settings\n SET git_credentials = (\n SELECT COALESCE(jsonb_agg(elem), '[]'::jsonb)\n FROM jsonb_array_elements(git_credentials) AS elem\n WHERE elem->>'repo_identity' IS DISTINCT FROM $2\n ) || jsonb_build_array($3::jsonb)\n WHERE workspace_id = $1\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "64f72d7477f7c82d1596560ff6c31c79281d5cb2b0e81f1e832eac6eb3cc32e2" +} diff --git a/backend/.sqlx/query-73a3417f5331032a9b2afcf5b1c260d5ead34a6a3783dd21db08c7ee049eb232.json b/backend/.sqlx/query-73a3417f5331032a9b2afcf5b1c260d5ead34a6a3783dd21db08c7ee049eb232.json new file mode 100644 index 0000000000..7c11dc80c0 --- /dev/null +++ b/backend/.sqlx/query-73a3417f5331032a9b2afcf5b1c260d5ead34a6a3783dd21db08c7ee049eb232.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_settings (workspace_id, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, git_credentials, ducklake, dbt_warehouses, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts) SELECT $1, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, git_credentials, ducklake, dbt_warehouses, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts FROM workspace_settings WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "73a3417f5331032a9b2afcf5b1c260d5ead34a6a3783dd21db08c7ee049eb232" +} diff --git a/backend/.sqlx/query-9c0ebee945eee03d667a4f38d18a4d446b8ea00cf51e02138a06c245ab33252e.json b/backend/.sqlx/query-9c0ebee945eee03d667a4f38d18a4d446b8ea00cf51e02138a06c245ab33252e.json new file mode 100644 index 0000000000..6ef7b20a4f --- /dev/null +++ b/backend/.sqlx/query-9c0ebee945eee03d667a4f38d18a4d446b8ea00cf51e02138a06c245ab33252e.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE workspace_settings\n SET git_credentials = (\n SELECT COALESCE(jsonb_agg(elem), '[]'::jsonb)\n FROM jsonb_array_elements(git_credentials) AS elem\n WHERE elem->>'repo_identity' IS DISTINCT FROM $2\n ) || jsonb_build_array($3::jsonb)\n WHERE workspace_id = $1\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "9c0ebee945eee03d667a4f38d18a4d446b8ea00cf51e02138a06c245ab33252e" +} diff --git a/backend/.sqlx/query-af5609185c6218d12fcb4a4234cfd2a14b2e9e613bcf4bc80abbfd495637ab2f.json b/backend/.sqlx/query-af5609185c6218d12fcb4a4234cfd2a14b2e9e613bcf4bc80abbfd495637ab2f.json new file mode 100644 index 0000000000..41481e2ec7 --- /dev/null +++ b/backend/.sqlx/query-af5609185c6218d12fcb4a4234cfd2a14b2e9e613bcf4bc80abbfd495637ab2f.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE workspace_settings\n SET git_credentials = (\n SELECT COALESCE(jsonb_agg(elem), '[]'::jsonb)\n FROM jsonb_array_elements(git_credentials) AS elem\n WHERE elem->>'repo_identity' IS DISTINCT FROM $2\n ) || jsonb_build_array($3::jsonb)\n WHERE workspace_id = $1\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "af5609185c6218d12fcb4a4234cfd2a14b2e9e613bcf4bc80abbfd495637ab2f" +} diff --git a/backend/.sqlx/query-b2f91eb32edd8db1605aeb5a17b6e96cd3a980411d80c6b10bae949cf2c42385.json b/backend/.sqlx/query-b2f91eb32edd8db1605aeb5a17b6e96cd3a980411d80c6b10bae949cf2c42385.json new file mode 100644 index 0000000000..222bfc0f82 --- /dev/null +++ b/backend/.sqlx/query-b2f91eb32edd8db1605aeb5a17b6e96cd3a980411d80c6b10bae949cf2c42385.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_unlock(hashtext($1)::bigint)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_unlock", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b2f91eb32edd8db1605aeb5a17b6e96cd3a980411d80c6b10bae949cf2c42385" +} diff --git a/backend/.sqlx/query-bbb8331348216892e4714e4338c496c9448cffba5786b51e748bec0165a6519c.json b/backend/.sqlx/query-bbb8331348216892e4714e4338c496c9448cffba5786b51e748bec0165a6519c.json new file mode 100644 index 0000000000..980ba7a36c --- /dev/null +++ b/backend/.sqlx/query-bbb8331348216892e4714e4338c496c9448cffba5786b51e748bec0165a6519c.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE workspace_settings\n SET git_sync = jsonb_set(git_sync, '{repositories}',\n COALESCE((SELECT jsonb_agg(\n CASE WHEN elem->>'git_repo_resource_path' IN ($2, $3)\n THEN CASE WHEN $4::jsonb = 'null'::jsonb\n THEN elem - 'credential'\n ELSE jsonb_set(elem, '{credential}', $4) END\n ELSE elem END)\n FROM jsonb_array_elements(git_sync->'repositories') AS elem), '[]'::jsonb)\n )\n WHERE workspace_id = $1\n AND jsonb_typeof(git_sync->'repositories') = 'array'\n AND EXISTS (\n SELECT 1 FROM jsonb_array_elements(git_sync->'repositories') AS e\n WHERE e->>'git_repo_resource_path' IN ($2, $3)\n )\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "bbb8331348216892e4714e4338c496c9448cffba5786b51e748bec0165a6519c" +} diff --git a/backend/.sqlx/query-ce3340a43a141cc7527211db16905d53bc756d00c8d28462097717502d82cc15.json b/backend/.sqlx/query-ce3340a43a141cc7527211db16905d53bc756d00c8d28462097717502d82cc15.json new file mode 100644 index 0000000000..58300795af --- /dev/null +++ b/backend/.sqlx/query-ce3340a43a141cc7527211db16905d53bc756d00c8d28462097717502d82cc15.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT elem\n FROM workspace_settings, jsonb_array_elements(git_credentials) AS elem\n WHERE workspace_id = $1 AND elem->>'repo_identity' = $2\n LIMIT 1\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "elem", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "ce3340a43a141cc7527211db16905d53bc756d00c8d28462097717502d82cc15" +} diff --git a/backend/.sqlx/query-d0eeb992a826d26376e9802a8c4ced12d26f67f1b059c7c2afe66f8e4b3a9749.json b/backend/.sqlx/query-d0eeb992a826d26376e9802a8c4ced12d26f67f1b059c7c2afe66f8e4b3a9749.json new file mode 100644 index 0000000000..cbcb7f18f7 --- /dev/null +++ b/backend/.sqlx/query-d0eeb992a826d26376e9802a8c4ced12d26f67f1b059c7c2afe66f8e4b3a9749.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_try_advisory_lock(hashtext($1)::bigint)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_try_advisory_lock", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "d0eeb992a826d26376e9802a8c4ced12d26f67f1b059c7c2afe66f8e4b3a9749" +} diff --git a/backend/.sqlx/query-ddd41de79b23b0436bbc7997751c90d7d82bd20ea30f1dc08f2e2066fb8d6f4b.json b/backend/.sqlx/query-ddd41de79b23b0436bbc7997751c90d7d82bd20ea30f1dc08f2e2066fb8d6f4b.json new file mode 100644 index 0000000000..9622e63a56 --- /dev/null +++ b/backend/.sqlx/query-ddd41de79b23b0436bbc7997751c90d7d82bd20ea30f1dc08f2e2066fb8d6f4b.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ws.workspace_id, ws.git_sync\n FROM workspace_settings ws\n JOIN workspace w ON w.id = ws.workspace_id\n WHERE NOT w.deleted\n AND ws.git_sync IS NOT NULL\n AND jsonb_typeof(ws.git_sync->'repositories') = 'array'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "git_sync", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + true + ] + }, + "hash": "ddd41de79b23b0436bbc7997751c90d7d82bd20ea30f1dc08f2e2066fb8d6f4b" +} \ No newline at end of file diff --git a/backend/.sqlx/query-f3a27f7781d986d9917a7022e1118313f0434758413dbf83f8575d09de98fc25.json b/backend/.sqlx/query-f3a27f7781d986d9917a7022e1118313f0434758413dbf83f8575d09de98fc25.json new file mode 100644 index 0000000000..1a71c74e9e --- /dev/null +++ b/backend/.sqlx/query-f3a27f7781d986d9917a7022e1118313f0434758413dbf83f8575d09de98fc25.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT (\n SELECT elem\n FROM jsonb_array_elements(git_credentials) AS elem\n WHERE elem->>'repo_identity' = $2\n )\n FROM workspace_settings WHERE workspace_id = $1 FOR UPDATE\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "elem", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "f3a27f7781d986d9917a7022e1118313f0434758413dbf83f8575d09de98fc25" +} diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index 0ebbb1b5ea..8340c8d90e 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -d33ea730c550cdbc7d050aeb6d40dcef3d134e07 +e092518ee60e33160fee9ae91a4d109566f7b0ee diff --git a/backend/migrations/20260904110943_git_credentials_workspace_settings.down.sql b/backend/migrations/20260904110943_git_credentials_workspace_settings.down.sql new file mode 100644 index 0000000000..ca88ec0f1e --- /dev/null +++ b/backend/migrations/20260904110943_git_credentials_workspace_settings.down.sql @@ -0,0 +1 @@ +ALTER TABLE workspace_settings DROP COLUMN IF EXISTS git_credentials; diff --git a/backend/migrations/20260904110943_git_credentials_workspace_settings.up.sql b/backend/migrations/20260904110943_git_credentials_workspace_settings.up.sql new file mode 100644 index 0000000000..7ece575cc1 --- /dev/null +++ b/backend/migrations/20260904110943_git_credentials_workspace_settings.up.sql @@ -0,0 +1,8 @@ +-- Server-owned git-sync credentials, one entry per repository the workspace +-- holds a token for, keyed by that repository rather than by a resource naming +-- it: a resource's URL is writable, and its path is not settled while it is +-- being created. +-- Kept out of `git_sync` because that column is copied into forks and returned +-- by the workspace settings API; this one is copied by neither. +ALTER TABLE workspace_settings + ADD COLUMN IF NOT EXISTS git_credentials JSONB NOT NULL DEFAULT '[]'::jsonb; diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index a11ed55882..9aae2cca06 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -354,7 +354,9 @@ pub async fn initial_load( ) } }); - pass.action(windmill_common::min_version::store_min_keep_alive_version(db)); + pass.action(windmill_common::min_version::store_min_keep_alive_version( + db, + )); pass.setting( windmill_common::global_settings::INSTANCE_EVENTS_WEBHOOK_SETTING, false, @@ -709,7 +711,6 @@ pub async fn initial_load( pass.run(conn).await; } - pub fn apply_metrics_enabled(value: Option) { if let Some(serde_json::Value::Bool(t)) = value { METRICS_ENABLED.store(t, Ordering::Relaxed) @@ -1066,8 +1067,8 @@ pub fn apply_fork_workspace_tag_append_fork_suffix(value: Option error::Result<()> { - let v = - load_value_from_global_settings_with_conn(conn, CRITICAL_ALERT_MUTE_UI_SETTING, true).await?; + let v = load_value_from_global_settings_with_conn(conn, CRITICAL_ALERT_MUTE_UI_SETTING, true) + .await?; apply_critical_alert_mute_ui_setting(v); Ok(()) } @@ -2732,7 +2733,6 @@ pub async fn reload_timeout_wait_result_setting(conn: &Connection) { .await; } - pub async fn reload_extra_pip_index_url_setting(conn: &Connection) { reload_option_setting_with_tracing( conn, @@ -2823,7 +2823,6 @@ pub async fn reload_bunfig_install_scopes_setting(conn: &Connection) { .await; } - pub async fn reload_nuget_config_setting(conn: &Connection) { reload_option_setting_with_tracing( conn, @@ -2931,7 +2930,6 @@ pub async fn reload_ruby_repos_setting(conn: &Connection) { .await; } - pub async fn reload_workspace_registries_setting(conn: &Connection) { match load_value_from_global_settings_with_conn( conn, @@ -3185,7 +3183,6 @@ pub async fn apply_job_isolation_setting(value: Option) { } } - async fn resolve_license_key_value(conn: &Connection, quiet: bool) -> anyhow::Result { let q = load_value_from_global_settings_with_conn(conn, LICENSE_KEY_SETTING, true) .await @@ -3480,7 +3477,10 @@ impl<'a> SettingsPass<'a> { // on compile-time defaults until the next full reload. Only the single-query transport // can fail this way; over HTTP the batch already is the per-setting read. if matches!(conn, Connection::Sql(_)) && values.is_empty() && !names.is_empty() { - tracing::warn!("Falling back to per-setting reads for {} settings", names.len()); + tracing::warn!( + "Falling back to per-setting reads for {} settings", + names.len() + ); values = fetch_settings_individually(conn, &names).await; } for (name, http) in &declared { @@ -3872,7 +3872,6 @@ pub fn parse_setting_value( value } - #[cfg(feature = "prometheus")] pub async fn monitor_pool(db: &DB) { if METRICS_ENABLED.load(Ordering::Relaxed) { @@ -4316,6 +4315,24 @@ pub async fn monitor_db( } }; + // Re-check what each git-sync repository's own credential says about its expiry, + // and rotate the ones close to it. Every ~40 min: the values move over days, and + // `should_run` counts iterations in a u8. Spawned rather than joined: the join + // below is cancelled at its deadline, which a long sweep would reach, and a + // rotation cut off between GitLab issuing a token and Windmill storing it + // loses the token family. Detached, only process shutdown can cut it off, + // which a rotation almost never coincides with. The pass's advisory lock + // keeps a slow one from overlapping the next. + let git_credential_maintenance_f = async { + #[cfg(all(feature = "enterprise", feature = "private"))] + if server_mode && iteration.is_some() && iteration.as_ref().unwrap().should_run(240) { + if let Some(db) = conn.as_sql() { + let db = db.clone(); + tokio::spawn(async move { maintain_git_credentials(&db).await }); + } + } + }; + // run every 2 iterations (~20s at the default LISTEN_NEW_EVENTS_INTERVAL_SEC). // Enterprise feature: the active `// freshness` backstop lives in // windmill-queue's `freshness_watchdog` (`private`); OSS gets a no-op stub. @@ -4369,6 +4386,7 @@ pub async fn monitor_db( export_audit_logs_to_object_store_f, cleanup_scheduled_job_deletions_f, git_auto_pull_f, + git_credential_maintenance_f, pipeline_freshness_watchdog_f, reconcile_unarmed_schedules_f, ); @@ -4691,6 +4709,156 @@ lazy_static::lazy_static! { #[cfg(feature = "private")] const AUTO_PULL_POLL_SLACK_S: i64 = 30; +/// Advisory lock id ensuring only one server replica maintains git credentials at +/// a time (adjacent to GIT_AUTO_PULL_LOCK_ID). +#[cfg(all(feature = "enterprise", feature = "private"))] +const GIT_CREDENTIAL_LOCK_ID: i64 = 737_483_923; + +/// Refresh every git-sync repository's credential status and rotate the ones near +/// expiry, so a token dies visibly (and usually not at all) rather than taking +/// sync down on its expiry date. +#[cfg(all(feature = "enterprise", feature = "private"))] +async fn maintain_git_credentials(db: &Pool) { + use windmill_common::ee_oss::{get_license_plan, LicensePlan}; + + if !matches!(get_license_plan().await, LicensePlan::Enterprise) { + return; + } + + // Transaction-scoped advisory lock, as for the schedule reconcile above: a + // session lock on a pooled connection would ride back into the pool still + // held if the sweep died before unlocking, and wedge the pass on every + // replica until a restart. The transaction only owns the lock; the sweep + // commits each status and each rotated token on its own as it goes. + let mut lock_tx = match db.begin().await { + Ok(tx) => tx, + Err(e) => { + tracing::error!("git credentials: failed to begin lock tx: {e:#}"); + return; + } + }; + // The transaction stays idle while the sweep talks to git hosts, and the + // pool's ten-minute idle-in-transaction timeout would end it, lock included, + // partway through a sweep over enough slow hosts. Lifted for this + // transaction only; it dies with the connection either way. + if let Err(e) = sqlx::query("SET LOCAL idle_in_transaction_session_timeout = 0") + .execute(&mut *lock_tx) + .await + { + tracing::error!("git credentials: failed to lift the idle timeout: {e:#}"); + return; + } + let locked: bool = match sqlx::query_scalar("SELECT pg_try_advisory_xact_lock($1)") + .bind(GIT_CREDENTIAL_LOCK_ID) + .fetch_one(&mut *lock_tx) + .await + { + Ok(v) => v, + Err(e) => { + tracing::error!("git credentials: advisory lock failed: {e:#}"); + return; + } + }; + if !locked { + return; + } + + if let Err(e) = maintain_git_credentials_inner(db).await { + tracing::error!("git credentials: maintenance error: {e:#}"); + } + drop(lock_tx); +} + +#[cfg(all(feature = "enterprise", feature = "private"))] +async fn maintain_git_credentials_inner(db: &Pool) -> error::Result<()> { + use windmill_common::workspaces::WorkspaceGitSyncSettings; + + // Same deleted/archived exclusion as the auto-pull poller: a dead workspace's + // settings row survives, and rotating a token for one would be pure damage. + let rows = sqlx::query!( + r#"SELECT ws.workspace_id, ws.git_sync + FROM workspace_settings ws + JOIN workspace w ON w.id = ws.workspace_id + WHERE NOT w.deleted + AND ws.git_sync IS NOT NULL + AND jsonb_typeof(ws.git_sync->'repositories') = 'array'"# + ) + .fetch_all(db) + .await?; + + for row in rows { + let Some(git_sync) = row.git_sync else { + continue; + }; + let settings: WorkspaceGitSyncSettings = match serde_json::from_value(git_sync) { + Ok(s) => s, + Err(e) => { + tracing::warn!( + "git credentials: invalid git_sync settings for workspace {}: {e}", + row.workspace_id + ); + continue; + } + }; + + for repo in settings.repositories.iter() { + let path = &repo.git_repo_resource_path; + // This refreshes and records the status on every repository it looks at, + // rotating only the ones near expiry, so it is the whole maintenance pass + // rather than just the rotation half. + if let Err(e) = windmill_common::git_sync_ee::rotate_git_credential_if_due( + db, + &row.workspace_id, + path, + ) + .await + { + tracing::error!( + "git credentials: maintenance failed for {path} in workspace {}: {e:#}", + row.workspace_id + ); + } + + // A repository that wants webhook delivery but holds no hook never + // gets one otherwise: the reconcile runs on a settings save, so a + // credential that was unusable when the hook should have been created + // would leave it missing until an admin saved again. Checking stored + // state costs nothing, and only the repositories actually missing a + // hook reach the host. + use windmill_common::workspaces::AutoPullMode; + // Also when a hook exists but carries a warning: a save during a GitLab + // outage keeps the hook and records why it could not be confirmed, and + // that warning is only cleared by a reconcile that confirms it again. + // Only repositories with a checked credential of their own: for + // everything else a settings save stays the one place hooks are + // reconciled, so an App repository or a plain remote is never touched + // here, and a recorded delivery mode nobody saved is never normalized. + let needs_hook = repo.credential.is_some() + && repo.auto_pull.as_ref().is_some_and(|a| { + a.enabled + && matches!(a.mode, AutoPullMode::Auto | AutoPullMode::Webhook) + && (a.webhook_id.is_none() || a.webhook_error.is_some()) + }); + if needs_hook { + let mut repo = repo.clone(); + if let Err(e) = windmill_common::git_sync_ee::sync_repo_webhook( + db, + &row.workspace_id, + &mut repo, + ) + .await + { + tracing::warn!( + "git credentials: could not reconcile the webhook for {path} in workspace {}: {e:#}", + row.workspace_id + ); + } + } + } + } + Ok(()) +} + #[cfg(feature = "private")] async fn poll_git_auto_pull_inner(db: &Pool) -> error::Result<()> { use windmill_common::workspaces::{AutoPullMode, WorkspaceGitSyncSettings}; @@ -6588,7 +6756,6 @@ pub async fn reload_critical_alerts_on_db_oversize(conn: &DB) -> error::Result<( Ok(()) } - pub async fn reload_jwt_secret_setting(db: &DB) -> error::Result<()> { let v = load_value_from_global_settings(db, JWT_SECRET_SETTING).await?; apply_jwt_secret_setting(db, v).await diff --git a/backend/tests/fixtures/git_sync_fork_credential.sql b/backend/tests/fixtures/git_sync_fork_credential.sql new file mode 100644 index 0000000000..1c1d8bd1e1 --- /dev/null +++ b/backend/tests/fixtures/git_sync_fork_credential.sql @@ -0,0 +1,55 @@ +-- A parent whose git-sync repository has a recorded credential, and the workspace +-- shapes the credential lookup and the qualification predicate have to tell apart. +-- +-- The credential itself is shared down the fork chain; the recorded *status* is +-- not, because it describes one repository and a fork can repoint its copy of the +-- resource. Forks get a status by fork creation copying it, which no fixture here +-- simulates, so a fork without one is a workspace nothing has checked yet. + +INSERT INTO workspace (id, name, owner, parent_workspace_id) VALUES + ('parent-ws', 'parent-ws', 'test-user', NULL), + ('fork-ws', 'fork-ws', 'test-user', 'parent-ws'), + -- A fork of a fork: the shape a fork of a dev workspace takes, and the one a + -- parent-only lookup misses. + ('deep-fork-ws', 'deep-fork-ws', 'test-user', 'fork-ws'), + ('errored-fork-ws', 'errored-fork-ws', 'test-user', 'parent-ws'), + ('orphan-ws', 'orphan-ws', 'test-user', NULL); + +-- A stored credential is encrypted with its own workspace's key. +INSERT INTO workspace_key (workspace_id, kind, key) VALUES + ('parent-ws', 'cloud', 'parent-key'), + ('fork-ws', 'cloud', 'fork-key'), + ('deep-fork-ws', 'cloud', 'deep-fork-key'), + ('errored-fork-ws', 'cloud', 'errored-fork-key'), + ('orphan-ws', 'cloud', 'orphan-key'); + +-- The parent holds the credential. +INSERT INTO workspace_settings (workspace_id, git_sync) VALUES + ('parent-ws', '{"repositories":[{"git_repo_resource_path":"$res:u/admin/repo", + "credential":{"provider":"gitlab","rotatable":true,"checked_at":1788500000}}]}'), + +-- A fork inherits the repository but not the credential: this is what +-- clone_workspace_data leaves behind. + ('fork-ws', '{"repositories":[{"git_repo_resource_path":"$res:u/admin/repo"}]}'), + +-- Two levels down, so neither the credential nor its status is one hop away. + ('deep-fork-ws', '{"repositories":[{"git_repo_resource_path":"$res:u/admin/repo"}]}'), + +-- A fork whose own credential has since failed. Its own standing must win over +-- the parent's healthy record rather than being papered over. + ('errored-fork-ws', '{"repositories":[{"git_repo_resource_path":"$res:u/admin/repo", + "credential":{"provider":"gitlab","rotatable":false,"checked_at":1788500000, + "error":"GitLab no longer accepts this token"}}]}'), + +-- No credential and no parent to borrow one from. + ('orphan-ws', '{"repositories":[{"git_repo_resource_path":"$res:u/admin/repo"}]}'); + +-- The resource each repository entry names, all pointing at the same repository. +-- The errored fork carries its token in the URL, the way a repository configured +-- by hand does: a plain remote, whatever the chain above it holds. +INSERT INTO resource (workspace_id, path, value, resource_type) VALUES + ('parent-ws', 'u/admin/repo', '{"url":"https://gitlab.com/grp/proj.git"}', 'git_repository'), + ('fork-ws', 'u/admin/repo', '{"url":"https://gitlab.com/grp/proj.git"}', 'git_repository'), + ('deep-fork-ws', 'u/admin/repo', '{"url":"https://gitlab.com/grp/proj.git"}', 'git_repository'), + ('errored-fork-ws', 'u/admin/repo', '{"url":"https://oauth2:glpat-inline@gitlab.com/grp/proj.git"}', 'git_repository'), + ('orphan-ws', 'u/admin/repo', '{"url":"https://gitlab.com/grp/proj.git"}', 'git_repository'); diff --git a/backend/tests/git_sync_fork_credential.rs b/backend/tests/git_sync_fork_credential.rs new file mode 100644 index 0000000000..7e31136727 --- /dev/null +++ b/backend/tests/git_sync_fork_credential.rs @@ -0,0 +1,252 @@ +//! A fork reaches the git credential held above it in its fork chain. +//! +//! Fork creation copies the parent's git-sync repositories but not the credential, +//! which is stored per workspace so that rotation has one owner. Chains nest (a +//! fork of a dev workspace, a fork of that), so the depth-2 cases here are what +//! keep the lookup from regressing to the parent. +//! +//! The recorded *status* is not shared the same way: it describes one repository, +//! and a fork can repoint its copy of the resource, so each workspace answers from +//! its own record and gets one by fork creation copying it down. +#![cfg(all(feature = "enterprise", feature = "private"))] + +use sqlx::{Pool, Postgres}; +use windmill_common::git_sync_ee::{ + git_credential_for_url, repo_provider, repo_supports_managed_git_features, set_git_credential, + GitProvider, +}; +use windmill_common::workspaces::GitCredentialProvider; + +const REPO: &str = "$res:u/admin/repo"; +const URL: &str = "https://gitlab.com/grp/proj.git"; + +/// A repository is managed when a credential is held for the repository its +/// URL names now and the last check found it healthy. The recorded status is +/// keyed by resource path, so alone it would outlive a repoint; the held +/// credential alone says nothing about whether the host still accepts it. +#[sqlx::test(fixtures("git_sync_fork_credential"))] +async fn credential_status_is_a_workspaces_own(db: Pool) -> anyhow::Result<()> { + assert!( + !repo_supports_managed_git_features(&db, "parent-ws", REPO).await, + "a healthy status with nothing held behind it does not qualify" + ); + set_git_credential( + &db, + "parent-ws", + URL, + "glpat-secret", + GitCredentialProvider::Gitlab, + ) + .await?; + assert!( + repo_supports_managed_git_features(&db, "parent-ws", REPO).await, + "the workspace holding both the credential and the recorded status qualifies" + ); + assert!( + !repo_supports_managed_git_features(&db, "fork-ws", REPO).await, + "a fork borrowing the credential with no record of its own does not: the \ + status describes one repository, and this fork's resource could name another" + ); + assert!( + !repo_supports_managed_git_features(&db, "errored-fork-ws", REPO).await, + "a workspace whose own credential failed stays disqualified" + ); + Ok(()) +} + +/// The host a repository talks to is declared when its credential is stored, and +/// travels with the credential down the fork chain. +/// +/// Read from the recorded status instead, a fork answered with the default +/// provider until its own check ran, which is long enough to register a webhook +/// against the wrong receiver. +#[sqlx::test(fixtures("git_sync_fork_credential"))] +async fn the_provider_comes_from_the_credential_and_reaches_forks( + db: Pool, +) -> anyhow::Result<()> { + assert_eq!( + repo_provider(&db, "parent-ws", REPO).await, + GitProvider::GitHub, + "with nothing stored there is no declaration to read, so the default stands" + ); + + set_git_credential( + &db, + "parent-ws", + URL, + "glpat-secret", + GitCredentialProvider::Gitlab, + ) + .await?; + + assert_eq!( + repo_provider(&db, "parent-ws", REPO).await, + GitProvider::GitLab, + "the workspace that stored it reads its own declaration" + ); + assert_eq!( + repo_provider(&db, "fork-ws", REPO).await, + GitProvider::GitLab, + "and a fork resolving that credential reads it too, without a check of its own" + ); + assert_eq!( + repo_provider(&db, "deep-fork-ws", REPO).await, + GitProvider::GitLab, + "two levels down as well" + ); + assert_eq!( + repo_provider(&db, "orphan-ws", REPO).await, + GitProvider::GitHub, + "a workspace outside the chain resolves no credential and no declaration" + ); + assert_eq!( + repo_provider(&db, "errored-fork-ws", REPO).await, + GitProvider::GitHub, + "a token written into the URL makes the repository a plain remote: the \ + parent's credential is not consulted and no host is declared" + ); + Ok(()) +} + +/// The stored credential is shared with forks and keyed by one repository. +/// +/// Both properties are the point of keeping it in `workspace_settings` under the +/// repository's identity: sharing is what stops a rotation from stranding every +/// fork on a revoked token, and the key is what stops a rewritten resource URL +/// from carrying the token to a host of the writer's choosing. +#[sqlx::test(fixtures("git_sync_fork_credential"))] +async fn a_fork_reads_an_ancestors_credential_for_the_bound_repository_only( + db: Pool, +) -> anyhow::Result<()> { + set_git_credential( + &db, + "parent-ws", + URL, + "glpat-secret", + GitCredentialProvider::Gitlab, + ) + .await?; + + assert_eq!( + git_credential_for_url(&db, "parent-ws", URL) + .await? + .as_deref(), + Some("glpat-secret"), + "the workspace that stored it reads it back" + ); + assert_eq!( + git_credential_for_url(&db, "fork-ws", URL) + .await? + .as_deref(), + Some("glpat-secret"), + "a fork stores none of its own and resolves the parent's" + ); + assert_eq!( + git_credential_for_url(&db, "deep-fork-ws", URL) + .await? + .as_deref(), + Some("glpat-secret"), + "a fork of a fork resolves the root's, two levels up" + ); + assert_eq!( + git_credential_for_url(&db, "fork-ws", "https://evil.example/grp/proj.git").await?, + None, + "a resource repointed at another repository asks for that one's \ + credential and finds none" + ); + assert_eq!( + git_credential_for_url(&db, "orphan-ws", URL).await?, + None, + "a workspace with no credential and no parent resolves nothing" + ); + Ok(()) +} + +/// One repository's credential is untouched by another's. +/// +/// The key is the repository, so picking a second repository stores beside the +/// first rather than over it. Keyed by the resource instead, a workspace editing +/// one repository's resource to point somewhere else would replace the token the +/// original repository was still syncing with. +#[sqlx::test(fixtures("git_sync_fork_credential"))] +async fn each_repository_keeps_its_own_credential(db: Pool) -> anyhow::Result<()> { + const OTHER_URL: &str = "https://gitlab.com/grp/other.git"; + + set_git_credential( + &db, + "parent-ws", + URL, + "glpat-first", + GitCredentialProvider::Gitlab, + ) + .await?; + set_git_credential( + &db, + "parent-ws", + OTHER_URL, + "glpat-second", + GitCredentialProvider::Gitlab, + ) + .await?; + + assert_eq!( + git_credential_for_url(&db, "parent-ws", URL) + .await? + .as_deref(), + Some("glpat-first"), + "storing a second repository's token leaves the first's in place" + ); + assert_eq!( + git_credential_for_url(&db, "parent-ws", OTHER_URL) + .await? + .as_deref(), + Some("glpat-second") + ); + + set_git_credential( + &db, + "parent-ws", + URL, + "glpat-replacement", + GitCredentialProvider::Gitlab, + ) + .await?; + assert_eq!( + git_credential_for_url(&db, "parent-ws", URL) + .await? + .as_deref(), + Some("glpat-replacement"), + "storing the same repository again replaces rather than duplicates" + ); + assert_eq!( + git_credential_for_url(&db, "parent-ws", OTHER_URL) + .await? + .as_deref(), + Some("glpat-second"), + "and still leaves the other repository alone" + ); + Ok(()) +} + +/// A credential issued for `https` is not served for the `http` spelling. +/// +/// The resource holding the URL is writable by anyone with write on its path, so +/// without the scheme in the key that edit would send the token over cleartext. +#[sqlx::test(fixtures("git_sync_fork_credential"))] +async fn a_credential_is_not_served_over_a_downgraded_transport( + db: Pool, +) -> anyhow::Result<()> { + set_git_credential( + &db, + "parent-ws", + URL, + "glpat-secret", + GitCredentialProvider::Gitlab, + ) + .await?; + assert_eq!( + git_credential_for_url(&db, "parent-ws", "http://gitlab.com/grp/proj.git").await?, + None + ); + Ok(()) +} diff --git a/backend/windmill-api-workspaces/src/workspaces.rs b/backend/windmill-api-workspaces/src/workspaces.rs index 93e0ad5ca8..85226ddf2c 100644 --- a/backend/windmill-api-workspaces/src/workspaces.rs +++ b/backend/windmill-api-workspaces/src/workspaces.rs @@ -3986,6 +3986,7 @@ async fn edit_git_sync_config( clear_client_supplied_auto_pull_state(ap); } repo.open_pr_error = None; + repo.credential = None; } reject_parent_only_git_sync_settings_on_fork( &db, @@ -4086,6 +4087,7 @@ async fn edit_git_sync_config( continue; }; repo.open_pr_error = old.open_pr_error.clone(); + repo.credential = old.credential.clone(); if let (Some(new_ap), Some(old_ap)) = (repo.auto_pull.as_mut(), old.auto_pull.as_ref()) { @@ -4131,6 +4133,7 @@ async fn edit_git_sync_config( .flatten() .and_then(|v| serde_json::from_value(v).ok()); let removed_webhooks: Vec<(String, i64)> = existing + .as_ref() .map(|e| { e.repositories .iter() @@ -4164,6 +4167,19 @@ async fn edit_git_sync_config( // `sync_repo_webhook` writes back the webhook fields it changes itself: // the remote hook and the record of it have to move together, so // persisting them out here would let one land without the other. + // Before the webhook reconcile, which decides whether this repo can have + // one from the credential this records. Also puts a short-lived or + // under-scoped token in front of the operator while they are still on the + // settings page, rather than when it expires. + if let Err(e) = windmill_common::git_sync_ee::refresh_git_credential_status( + &db, + &w_id, + &repo.git_repo_resource_path, + ) + .await + { + tracing::warn!("git credential check error: {}", e); + } if let Err(e) = windmill_common::git_sync_ee::sync_repo_webhook(&db, &w_id, repo).await { tracing::warn!("git auto-pull: webhook sync error: {}", e); @@ -4217,6 +4233,7 @@ async fn edit_git_sync_repository( clear_client_supplied_auto_pull_state(ap); } new_config.repository.open_pr_error = None; + new_config.repository.credential = None; reject_parent_only_git_sync_settings_on_fork( &db, &w_id, @@ -4341,6 +4358,7 @@ async fn edit_git_sync_repository( // from the UI cannot revert what the poller/webhook layer wrote. let mut updated = new_config.repository; updated.open_pr_error = existing_repo.open_pr_error.clone(); + updated.credential = existing_repo.credential.clone(); match (updated.auto_pull.as_mut(), existing_repo.auto_pull.as_ref()) { (Some(new_ap), Some(old_ap)) => { new_ap.last_synced_sha = old_ap.last_synced_sha.clone(); @@ -4398,6 +4416,15 @@ async fn edit_git_sync_repository( .iter_mut() .find(|r| r.git_repo_resource_path == new_config.git_repo_resource_path) { + if let Err(e) = windmill_common::git_sync_ee::refresh_git_credential_status( + &db, + &w_id, + &repo.git_repo_resource_path, + ) + .await + { + tracing::warn!("git credential check error: {}", e); + } if let Err(e) = windmill_common::git_sync_ee::sync_repo_webhook(&db, &w_id, repo).await { tracing::warn!("git auto-pull: webhook sync error: {}", e); } @@ -4537,6 +4564,10 @@ async fn delete_git_sync_repository( } } + // The stored credential is deliberately left alone: it belongs to the git + // repository resource, which this endpoint does not delete, and the resource + // still authenticates with it for connection tests and commit lookups. + // Trigger git sync for repository deletion handle_deployment_metadata( &authed.email, @@ -6306,9 +6337,10 @@ async fn update_workspace_settings( // Auto-pull and fork PRs are parent-owned and must not be inherited: // the fork would otherwise carry the parent's webhook id (turning off // auto-pull on the fork would delete the parent's webhook). A fork - // still inherits the push-direction config and the installation. - // Repo → fork sync is driven by the parent's webhook/poller - // (`sync_forks`), which routes the fork's `wm-fork/**` branch into it. + // still inherits the push-direction config, the installation, and the + // recorded credential status, which describes the repository rather + // than belonging to either workspace and would otherwise leave the + // fork unqualified for managed features until its first check. r.auto_pull = None; r.fork_open_prs = false; r.open_pr_error = None; diff --git a/backend/windmill-api-workspaces/src/workspaces_extra.rs b/backend/windmill-api-workspaces/src/workspaces_extra.rs index b2883e5f9a..e75f0d1ef2 100644 --- a/backend/windmill-api-workspaces/src/workspaces_extra.rs +++ b/backend/windmill-api-workspaces/src/workspaces_extra.rs @@ -113,7 +113,7 @@ pub(crate) async fn change_workspace_id( // Duplicate workspace settings (keep copy in old workspace for reference) info!("Duplicating workspace_settings table"); sqlx::query!( - "INSERT INTO workspace_settings (workspace_id, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, ducklake, dbt_warehouses, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts, guest_access_enabled, guest_jwt_public_key, guest_jwt_jwks_url) SELECT $1, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, ducklake, dbt_warehouses, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts, guest_access_enabled, guest_jwt_public_key, guest_jwt_jwks_url FROM workspace_settings WHERE workspace_id = $2", + "INSERT INTO workspace_settings (workspace_id, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, git_credentials, ducklake, dbt_warehouses, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts, guest_access_enabled, guest_jwt_public_key, guest_jwt_jwks_url) SELECT $1, slack_team_id, slack_name, slack_command_script, slack_email, customer_id, plan, webhook, ai_config, large_file_storage, git_sync, default_app, default_scripts, deploy_ui, mute_critical_alerts, color, operator_settings, teams_command_script, teams_team_id, teams_team_name, git_app_installations, git_credentials, ducklake, dbt_warehouses, slack_oauth_client_id, slack_oauth_client_secret, datatable, teams_team_guid, auto_invite, error_handler, success_handler, public_app_execution_limit_per_minute, error_handler_fallback_to_instance_alerts, guest_access_enabled, guest_jwt_public_key, guest_jwt_jwks_url FROM workspace_settings WHERE workspace_id = $2", &rw.new_id, &old_id ) diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index 4abd5f4dff..65a29c141d 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -2741,7 +2741,7 @@ paths: /w/{workspace}/github_app/token: post: - summary: get github app token + summary: get the git credential for a git-sync job (GitHub App token, or the credential stored for the repository) operationId: getGithubAppToken tags: - workspace @@ -2761,7 +2761,7 @@ paths: - job_token responses: "200": - description: github app token + description: git credential content: application/json: schema: @@ -2905,6 +2905,131 @@ paths: "200": description: Successfully imported the installation + /w/{workspace}/git_sync/gitlab/projects: + post: + tags: + - Git Sync + summary: List the GitLab projects a token can sync + description: >- + Lists the projects the supplied GitLab token can push to, so a git + repository resource can be filled in without hand-writing a project + path. The token is used for this call only and is never stored. + Requires workspace admin. + operationId: listGitlabProjects + parameters: + - $ref: "#/components/parameters/WorkspaceId" + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + base_url: + type: string + description: The GitLab instance, e.g. https://gitlab.com + token: + type: string + description: A project access token with the api scope, or a group token that reaches the project + search: + type: string + description: Narrow the list to projects matching this text + required: + - base_url + - token + responses: + "200": + description: the projects the token can sync + content: + application/json: + schema: + type: array + items: + $ref: "#/components/schemas/GitlabProject" + + /w/{workspace}/git_sync/credential/origin: + get: + tags: + - Git Sync + summary: Where a repository's credential comes from + description: >- + Whether Windmill holds this repository's access token, and which host it + talks to. `held` means this workspace stores it, `borrowed` means an + ancestor does and it is not this workspace's to replace. Both absent + means the repository authenticates with whatever its URL carries. + Returns no secret. Requires workspace admin. + operationId: getCredentialOrigin + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: path + in: query + required: true + description: >- + Path of the git repository resource, with or without the `$res:` + prefix. A path rather than a URL, because a resource URL may carry a + token and a URL in a query string lands in logs. + schema: + type: string + responses: + "200": + description: where the credential comes from + content: + application/json: + schema: + type: object + properties: + origin: + type: string + enum: + - held + - borrowed + provider: + type: string + enum: + - gitlab + + /w/{workspace}/git_sync/credential: + post: + tags: + - Git Sync + summary: Store the credential for a git repository + description: >- + Stores the access token a git repository authenticates with, so it does + not have to be written into the repository URL or a workspace variable. + The token is write-only: it is served only to a git-sync job that + presents its own job token, and a fork of this workspace reads this + copy instead of holding one of its own. Requires workspace admin. + operationId: setGitCredential + parameters: + - $ref: "#/components/parameters/WorkspaceId" + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + repo_url: + type: string + description: >- + The repository the credential is for, and the key it is + stored under. It is served for this repository and no other, + so repointing a resource elsewhere cannot carry the token + along. + token: + type: string + description: The access token, as pasted + required: + - repo_url + - token + responses: + "200": + description: the credential was stored + content: + text/plain: + schema: + type: string + /w/{workspace}/github_app/ghes_installation_callback: post: summary: GHES installation callback @@ -34484,9 +34609,58 @@ components: open_pr_error: type: string description: server-owned, last failure opening a PR for a deploy branch of this repo + credential: + $ref: "#/components/schemas/GitCredentialStatus" required: - git_repo_resource_path + GitCredentialStatus: + type: object + description: server-owned, what the repo's own credential reports about itself + properties: + provider: + type: string + enum: + - gitlab + token_id: + type: integer + format: int64 + expires_at: + type: string + format: date + description: absent for a non-expiring token + scopes: + type: array + items: + type: string + rotatable: + type: boolean + description: whether this workspace renews the credential itself + checked_at: + type: integer + format: int64 + error: + type: string + required: + - provider + - rotatable + - checked_at + + GitlabProject: + type: object + description: a GitLab project a token can sync, as the resource form needs it + properties: + path_with_namespace: + type: string + description: nested group path plus project name, which is also GitLab's project id + http_url_to_repo: + type: string + default_branch: + type: string + required: + - path_with_namespace + - http_url_to_repo + AutoPullMode: type: string enum: diff --git a/backend/windmill-api/src/git_sync_oss.rs b/backend/windmill-api/src/git_sync_oss.rs index 0451d88699..d1653d9aa9 100644 --- a/backend/windmill-api/src/git_sync_oss.rs +++ b/backend/windmill-api/src/git_sync_oss.rs @@ -10,6 +10,11 @@ pub fn workspaced_service() -> Router { Router::new() } +#[cfg(not(feature = "private"))] +pub fn workspaced_git_sync_service() -> Router { + Router::new() +} + #[cfg(not(feature = "private"))] pub fn global_service() -> Router { Router::new() diff --git a/backend/windmill-api/src/lib.rs b/backend/windmill-api/src/lib.rs index 4ba60428c9..f404d7be9b 100644 --- a/backend/windmill-api/src/lib.rs +++ b/backend/windmill-api/src/lib.rs @@ -959,6 +959,15 @@ pub async fn run_server( #[cfg(not(feature = "enterprise"))] Router::new() }) + .nest("/w/{workspace_id}/git_sync", { + #[cfg(feature = "enterprise")] + { + git_sync_oss::workspaced_git_sync_service() + } + + #[cfg(not(feature = "enterprise"))] + Router::new() + }) .nest( "/w/{workspace_id}/resources_u", public_service().layer(cors.clone()), diff --git a/backend/windmill-api/src/workspaces_export.rs b/backend/windmill-api/src/workspaces_export.rs index 1c22981ee0..3a2a608369 100644 --- a/backend/windmill-api/src/workspaces_export.rs +++ b/backend/windmill-api/src/workspaces_export.rs @@ -1586,10 +1586,11 @@ pub(crate) async fn tarball_workspace( .await?; // Use v2 format only if explicitly requested, otherwise use v1 (legacy) for backward compatibility - // Server-owned auto-pull state (the HMAC webhook secret + hook id/error and - // the synced-sha / last-pull status) must never leave the server: keep it out - // of export archives and synced repos, and don't let a re-imported workspace - // inherit another install's hook/sync state. Mirrors the GET-settings redaction. + // Server-owned state (the HMAC webhook secret + hook id/error, the + // synced-sha / last-pull status, and what the credential check observed) + // must never leave the server: keep it out of export archives and synced + // repos, and don't let a re-imported workspace inherit another install's + // hook/sync state. Mirrors the GET-settings redaction. fn redact_git_sync_for_export(git_sync: Option) -> Option { let mut git_sync = git_sync?; if let Some(repos) = git_sync @@ -1610,6 +1611,13 @@ pub(crate) async fn tarball_workspace( auto_pull.remove(field); } } + // What this install observed about its own credential: a token + // id and expiry, and a `checked_at` that moves on its own. + // None of it describes the workspace, and in a git-synced + // `wmill.yaml` it would churn the file for no reason. + if let Some(repo) = repo.as_object_mut() { + repo.remove("credential"); + } } } Some(git_sync) diff --git a/backend/windmill-common/src/git_sync_oss.rs b/backend/windmill-common/src/git_sync_oss.rs index 5f8ba771fb..4c7de4fa7b 100644 --- a/backend/windmill-common/src/git_sync_oss.rs +++ b/backend/windmill-common/src/git_sync_oss.rs @@ -1,11 +1,16 @@ #[cfg(feature = "private")] #[allow(unused)] pub use crate::git_sync_ee::*; -#[cfg(not(feature = "private"))] +#[cfg(not(all(feature = "private", feature = "enterprise")))] use sqlx::{Pool, Postgres}; use url::Url; -#[cfg(not(feature = "private"))] +/// Gated on the pair to match [`with_stored_credential`] below, whose callers +/// reach it through this facade un-gated and so depend on it. Nothing routes +/// here today (the one caller imports the enterprise item directly), so this is +/// for uniformity: the next plain caller would otherwise find no definition +/// under `private` without `enterprise`. +#[cfg(not(all(feature = "private", feature = "enterprise")))] pub async fn get_github_app_token_internal( _db: &Pool, _job_token: &str, @@ -15,6 +20,21 @@ pub async fn get_github_app_token_internal( )); } +/// Server-held git credentials are an enterprise feature, so on this build a +/// repository URL authenticates with whatever it already carries. +/// +/// Gated on the pair rather than on `private` alone: `private` does not imply +/// `enterprise`, and the callers are plain (no `#[cfg]`), so a build with one +/// and not the other would find neither this nor the enterprise definition. +#[cfg(not(all(feature = "private", feature = "enterprise")))] +pub async fn with_stored_credential( + _db: &Pool, + _w_id: &str, + url: String, +) -> crate::error::Result { + Ok(url) +} + lazy_static::lazy_static! { /// Matches a `user:password@` (or `user@`) userinfo component right after the URL scheme. static ref GIT_URL_USERINFO_RE: regex::Regex = diff --git a/backend/windmill-common/src/workspaces.rs b/backend/windmill-common/src/workspaces.rs index b6d10cb7df..443c3b6b63 100644 --- a/backend/windmill-common/src/workspaces.rs +++ b/backend/windmill-common/src/workspaces.rs @@ -183,7 +183,7 @@ pub enum ObjectType { DatatableMigration, } -pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28931/sync-script-to-git-repo-windmill"; +pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28949/sync-script-to-git-repo-windmill"; /// Hub script that applies a repository's state back into a workspace /// (the repo → Windmill / "pull" direction). Same script the UI runs from @@ -191,7 +191,7 @@ pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28931/sync-script-to-git-repo /// ignores the slug, so the slug is kept free of characters that would be /// percent-encoded into the run URL (a `:` becomes `%3A`, which some hardened /// reverse proxies reject as double-encoding when the client re-encodes it). -pub const GIT_SYNC_PULL_SCRIPT_PATH: &str = "hub/28930/git-sync-init-repository-windmill"; +pub const GIT_SYNC_PULL_SCRIPT_PATH: &str = "hub/28948/git-sync-init-repository-windmill"; /// Prefix used to identify fork workspaces. A workspace whose id starts with this string is a /// fork of another workspace. @@ -345,13 +345,14 @@ pub struct GitRepositorySettings { #[serde(default, skip_serializing_if = "Option::is_none")] pub auto_pull: Option, /// Open a PR when a deploy pushes a `wm_deploy/**` branch of this promotion - /// repo (app-backed only; runs from the deploy callback so it works without + /// repo (needs a credential the server holds — a GitHub App installation or + /// a checked GitLab token; runs from the deploy callback so it works without /// inbound webhooks). Off by default so upgrades don't change behavior. #[serde(default, skip_serializing_if = "is_false")] pub promotion_open_prs: bool, /// Parent-level: open a PR when a fork of this workspace deploys to its - /// `wm-fork/**` branch (app-backed only; the fork's deploy callback reads - /// this from the parent). Off by default. + /// `wm-fork/**` branch (needs a credential the server holds; the fork's + /// deploy callback reads this from the parent). Off by default. #[serde(default, skip_serializing_if = "is_false")] pub fork_open_prs: bool, /// Server-owned: the last failure opening a PR for a deploy branch of this @@ -360,6 +361,10 @@ pub struct GitRepositorySettings { /// successful PR; never accepted from clients. #[serde(default, skip_serializing_if = "Option::is_none")] pub open_pr_error: Option, + /// Server-owned: what the repo's credential says about its own expiry and + /// scopes. Written by the credential check, never accepted from clients. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub credential: Option, } impl GitRepositorySettings { @@ -405,6 +410,44 @@ pub enum AutoPullMode { Polling, } +/// Host whose credential lifecycle Windmill can manage from the repo URL. +#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)] +#[serde(rename_all = "lowercase")] +pub enum GitCredentialProvider { + Gitlab, +} + +/// What the repo's own credential says about itself, refreshed by asking the +/// host. Server-owned: written by the credential check, never accepted from a +/// client. +/// +/// Absent means the check has not run or the repo carries no credential we can +/// introspect (a GitHub App repo mints tokens per call and has nothing to expire). +#[derive(Serialize, Deserialize, Debug, Clone)] +pub struct GitCredentialStatus { + pub provider: GitCredentialProvider, + /// Changes on every rotation, so it identifies the current token, not the + /// credential's whole history. + #[serde(skip_serializing_if = "Option::is_none")] + pub token_id: Option, + /// `None` is a non-expiring token, which only self-managed GitLab can issue + /// (and only for a service account). It means no warning and no rotation. + #[serde(skip_serializing_if = "Option::is_none")] + pub expires_at: Option, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub scopes: Vec, + /// Whether *this workspace* renews the credential. That needs a scope which + /// permits it (`api` or `self_rotate`) and a credential this workspace holds: + /// a token carried in the repository URL is the operator's to manage, and one + /// resolved from an ancestor is the ancestor's, so neither is renewed here. + pub rotatable: bool, + /// Unix timestamp (seconds) of the last check. + pub checked_at: i64, + /// Why the last check or rotation failed, cleared by the next success. + #[serde(skip_serializing_if = "Option::is_none")] + pub error: Option, +} + /// Outcome of the most recent auto-pull attempt, surfaced in the UI. #[derive(Serialize, Deserialize, Debug, Clone)] pub struct AutoPullStatus { diff --git a/backend/windmill-store/src/resources.rs b/backend/windmill-store/src/resources.rs index cf3c73f8da..1ea5545b68 100644 --- a/backend/windmill-store/src/resources.rs +++ b/backend/windmill-store/src/resources.rs @@ -3629,6 +3629,16 @@ async fn get_git_commit_hash( })?; git_resource.url = resolve_azure_devops_url(&db_with_opt_authed, &w_id, &git_resource.url, false).await?; + // A credential is stored under the repository it was issued for, so a + // resource repointed elsewhere finds none. Which credential can be attached + // is bounded by that; who may use it is bounded here, on the same terms as + // the installation credential above. + let plain_url = git_resource.url.clone(); + git_resource.url = + windmill_common::git_sync_oss::with_stored_credential(&db, &w_id, git_resource.url).await?; + if git_resource.url != plain_url { + require_admin(authed.is_admin, &authed.username)?; + } let identities: Vec = query .git_ssh_identity @@ -4307,6 +4317,10 @@ pub async fn get_git_repo_head_for_autopull( } git_resource.url = resolve_azure_devops_url(&git_sync_system_dba(db), w_id, &git_resource.url, true).await?; + // A repo whose credential Windmill holds carries none in its URL, so the + // poller has to attach it here or every probe would be unauthenticated. + git_resource.url = + windmill_common::git_sync_oss::with_stored_credential(db, w_id, git_resource.url).await?; if let Some(branch) = git_resource.branch.as_deref().filter(|s| !s.is_empty()) { let branch = branch.to_string(); @@ -4413,6 +4427,11 @@ pub async fn get_git_repo_fork_heads_for_autopull( )); } git_resource.url = resolve_azure_devops_url(&dba, w_id, &git_resource.url, true).await?; + // Same reason as the head probe above: a repository whose credential Windmill + // holds carries none in its URL, and listing the fork branches is the half of + // polling that would otherwise go out unauthenticated. + git_resource.url = + windmill_common::git_sync_oss::with_stored_credential(db, w_id, git_resource.url).await?; validate_git_url(&git_resource.url).await?; validate_git_ref(base_branch)?; diff --git a/backend/windmill-worker/src/result_processor.rs b/backend/windmill-worker/src/result_processor.rs index 647d569cbe..fd4f08f5fb 100644 --- a/backend/windmill-worker/src/result_processor.rs +++ b/backend/windmill-worker/src/result_processor.rs @@ -816,8 +816,19 @@ pub async fn handle_receive_completed_job( #[cfg(all(feature = "enterprise", feature = "private"))] #[derive(serde::Deserialize)] struct GitSyncCheck { - check_run_id: i64, - repo_url: String, + /// Absent when the repository's host has no check surface (GitLab): the + /// result then reaches the pull request through the managed comment alone. + #[serde(default)] + check_run_id: Option, + /// Only markers written before the repository URL moved out of job args + /// carry one; the resource path on the job is what is used now. + #[serde(default)] + repo_url: Option, + /// Host and path of the repository the check was created on, with no + /// credential in it. The resource path is mutable, so this is what proves + /// the resource still points where the check lives. + #[serde(default)] + repo: Option, #[serde(default)] pr_number: Option, #[serde(default)] @@ -1162,18 +1173,18 @@ async fn maybe_open_git_sync_deploy_pr( }; // Base = the tracked branch (resource branch, else the repo default). Also - // acts as the app-backed gate: PR creation needs the installation token. - let base = match windmill_common::git_sync_ee::get_app_repo_head_for_autopull( + // acts as the gate: PR creation needs a credential the server itself holds. + let base = match windmill_common::git_sync_ee::managed_pr_base_branch( db, workspace_id, &repo_path, ) .await { - Ok(Some((branch, _))) => branch, + Ok(Some(branch)) => branch, Ok(None) => { tracing::warn!( - "git sync PR: repo {repo_path} in {workspace_id} has a PR-on-deploy toggle set but is not GitHub-App-backed; skipping (connect the repo through the GitHub App, or use the open-pr-on-commit workflow)" + "git sync PR: repo {repo_path} in {workspace_id} has a PR-on-deploy toggle set but the server holds no credential for it; skipping (connect the repo through the GitHub App or a GitLab token, or use the open-pr-on-commit workflow)" ); return; } @@ -1373,22 +1384,59 @@ async fn maybe_post_git_sync_check( (None, Some(deploy)) => (true, deploy), (None, None) => return, }; - let Ok(mut check) = serde_json::from_value::(marker) else { + let Ok(check) = serde_json::from_value::(marker) else { return; }; - // Markers carry the literal resource URL (job args are persisted, so a - // `$var:`-resolved URL must not land there); interpolate before calling - // GitHub. - check.repo_url = - match windmill_common::variables::get_variable_or_self(check.repo_url, db, workspace_id) - .await - { - Ok(u) => u, - Err(e) => { - tracing::error!("git sync-check: cannot interpolate repo url: {e:#}"); - return; - } - }; + // Job args are persisted, so the repository URL is not among them: it is + // re-resolved here from the resource path the pull job carries. A marker + // written before that change still has the URL, and is honoured until the + // last such job has drained. + // The resource path is mutable, so it is only trusted when the marker also + // carries the identity to check it against. A marker written before that + // identity existed keeps using the URL it captured at enqueue, which cannot + // have been repointed since. + let repo_url = match ( + check.repo.is_some(), + row.repo_path.as_deref(), + check.repo_url.clone(), + ) { + // The resource path is mutable, so following it is only safe when the + // marker also carries the identity to check the result against. + (true, Some(path), _) => { + windmill_common::git_sync_ee::resolve_repo_url_interpolated(db, workspace_id, path) + .await + } + // A marker written before that identity existed captured the URL itself, + // which cannot have been repointed since. + (_, _, Some(url)) => { + windmill_common::variables::get_variable_or_self(url, db, workspace_id).await + } + // Neither: nothing here can prove which repository this check belongs to, + // and resolving the path anyway is how a preview reaches the wrong one. + // Leaving the check unfinished is the safe failure. + _ => { + tracing::error!( + "git sync-check: the marker carries neither a repository identity nor a url; not acting on it" + ); + return; + } + }; + let repo_url = match repo_url { + Ok(u) => u, + Err(e) => { + tracing::error!("git sync-check: cannot resolve repo url: {e:#}"); + return; + } + }; + // A resource repointed while the diff was running would otherwise close a + // check, or post a preview, on a repository that has nothing to do with it. + if check.repo.is_some() && windmill_common::git_sync_ee::repo_identity(&repo_url) != check.repo + { + tracing::warn!( + "git sync-check: the repository moved since the check was created; leaving it alone" + ); + return; + } // "In sync" on a PR that visibly changes files reads as a bug when those // files are outside the repo's sync filters — say what the scope is. let scope_note = if !is_deploy && success { @@ -1451,7 +1499,7 @@ async fn maybe_post_git_sync_check( ( "failure", "Merge conflicts with the base branch".to_string(), - "This PR cannot be merged cleanly, so there is no deploy diff to compute. Resolve the conflicts and push again to re-run this check." + "This branch cannot be merged cleanly, so there is no deploy diff to compute. Resolve the conflicts and push again to re-run this check." .to_string(), ) } else if pr_check_error.as_deref() == Some("PR_HEAD_REF_UNAVAILABLE") { @@ -1461,7 +1509,7 @@ async fn maybe_post_git_sync_check( ( "neutral", "Could not compute the deploy diff".to_string(), - "Windmill could not fetch this PR's head or enough history from GitHub to compute its merge with the base. Push again to re-run this check." + "Windmill could not fetch this branch's head, or enough history, to compute its merge with the base. Push again to re-run this check." .to_string(), ) } else if pr_check_error.is_some() { @@ -1484,20 +1532,20 @@ async fn maybe_post_git_sync_check( "success", "In sync".to_string(), format!( - "Merging this PR would make no changes to the workspace.{}", + "Merging this branch would make no changes to the workspace.{}", scope_note.as_deref().unwrap_or_default() ), ), Some((changes, settings_changed)) => { let mut lines = vec![format!( - "Merging this PR would apply {} change(s) to the workspace:\n", + "Merging this branch would apply {} change(s) to the workspace:\n", changes.len() )]; lines.extend(format_change_list(&changes)); if settings_changed { lines.push(match check.wmill_yaml_changed { - Some(true) => "\nThis PR changes wmill.yaml: pulling also applies the updated workspace settings.".to_string(), - Some(false) => "\nIndependent of this PR, the workspace's git-sync settings differ from the repo's wmill.yaml and a pull updates them to match.".to_string(), + Some(true) => "\nThis branch changes wmill.yaml: pulling also applies the updated workspace settings.".to_string(), + Some(false) => "\nIndependent of this branch, the workspace's git-sync settings differ from the repo's wmill.yaml and a pull updates them to match.".to_string(), None => "\nA pull also updates the workspace's git-sync settings to match the repo's wmill.yaml.".to_string(), }); } @@ -1520,19 +1568,21 @@ async fn maybe_post_git_sync_check( Some(url) => format!("{summary}\n\n[See the job in Windmill]({url})"), None => summary.clone(), }; - if let Err(e) = windmill_common::git_sync_ee::update_check_run( - db, - workspace_id, - &check.repo_url, - check.check_run_id, - conclusion, - &title, - &check_summary, - job_url.as_deref(), - ) - .await - { - tracing::error!("git sync-check: failed to update check run: {e:#}"); + if let Some(check_run_id) = check.check_run_id { + if let Err(e) = windmill_common::git_sync_ee::update_check_run( + db, + workspace_id, + &repo_url, + check_run_id, + conclusion, + &title, + &check_summary, + job_url.as_deref(), + ) + .await + { + tracing::error!("git sync-check: failed to update check run: {e:#}"); + } } // Phase 4 also maintains ONE managed comment on the PR (Cloudflare @@ -1556,7 +1606,7 @@ async fn maybe_post_git_sync_check( if let Err(e) = windmill_common::git_sync_ee::upsert_pr_comment( db, workspace_id, - &check.repo_url, + &repo_url, pr_number, marker, &body, diff --git a/docs/git-sync-gitlab-setup.md b/docs/git-sync-gitlab-setup.md new file mode 100644 index 0000000000..6e736dee37 --- /dev/null +++ b/docs/git-sync-gitlab-setup.md @@ -0,0 +1,170 @@ +# Git sync with GitLab + +GitLab has no equivalent of a GitHub App, so there is nothing to install and no +consent screen. What Windmill needs instead is one credential you create in +GitLab and paste once. With it, a GitLab repository gets the same managed +features an app-backed GitHub repository has: instant pull over a webhook, merge +requests opened on deploy, and a diff preview posted onto the merge request. + +## The credential + +Create a **project access token** on the project you are syncing (Settings → +Access tokens). It is a bot identity that outlives the person who created it, +which is what you want for a credential the instance uses unattended, and it +reaches exactly the one project. + +| | | +| --- | --- | +| Scope | `api` | +| Role | Developer to push deploy branches; **Maintainer** to also manage the webhook and open merge requests | +| Expiry | Required. A group service account PAT can be non-expiring on self-managed (see below); an access token cannot | + +**Use a separate token per repository.** A group access token works too and +reaches every project in the group, which is convenient for a lot of +repositories — but Windmill stores the credential per repository, and renewal +rewrites the repository it renewed for. Any other repository holding that same +token keeps the revoked one and stops syncing until you paste a new token there. +Each stranded repository says so on its card, so it is visible rather than +silent, but a token per repository avoids it entirely. + +`api` is a superset: it authorizes Git over HTTPS as well, so no separate +`write_repository` is needed to clone and push, and it is also what makes the +token renewable. A `write_repository`-only token can still push, but Windmill +cannot inspect or renew it and reports that in the workspace's git sync settings. + +### The identity Windmill acts as + +GitLab issues an access token to a bot user it creates for it — `project__bot_…` +for a project token, `group__bot_…` for a group one — and the bot's display +name is **the name you gave the token**. That name is the byline on everything +Windmill does: the author of deploy commits, of the merge requests it opens, and +of the preview notes it writes. Name it for what it is, `windmill-sync` or +similar, rather than something only you will recognise. + +Each token you create adds another bot member to the project or group. Renewal +does not — it keeps the same bot — so a repository accumulates one bot, not one +per year. + +Renewal goes through GitLab's own self-rotation endpoint. Both kinds of access +token are held as their bot user's personal access token, so the token rotates +itself and Windmill never needs a credential with rights over the project or +group. + +## Connecting a repository + +In the resource form for a `git_repository` resource, use the **GitLab** button: +paste the instance URL and the token, pick a project from the list, and Windmill +keeps the token for you. The resource itself gets the plain remote URL +(`"url": "https://gitlab.com/group/project.git"`), with no credential in it. + +The token is stored encrypted on the workspace, keyed by the repository it was +issued for rather than by the resource naming it, the same way a GitHub App +installation is held against the account it covers. Nothing reads it back out +over the API: the server attaches it when it talks to GitLab, and a sync job +receives it only against its own job token. Repointing a resource's `url` asks +for a different repository's token and finds none, so the edit carries nothing +with it; a repository that genuinely moved needs its token entered again. + +Because the repository is the key, the token is stored the moment you pick the +project, before the resource is saved. Renaming the resource later keeps it, and +cancelling the edit leaves a stored token that nothing uses until some resource +points at that repository again. + +Forks of the workspace read this one copy rather than getting their own, so +renewal reaches all of them at once and the token is not duplicated into every +descendant workspace. + +Treat workspace admin as equivalent to holding the token. An admin of the +workspace, or of any fork below it, can point a repository at a sync script they +wrote and have that job request the credential, exactly as they can for a GitHub +App installation token. Storing it this way keeps it out of the variables API and +out of every fork's own storage; it is not a boundary against the admins of those +workspaces. + +A URL with the token written into it, in the resource or in a secret variable +the resource points at (`"url": "$var:..."`), is a plain git remote: it syncs on +deploy and by polling, and nothing else. Windmill does not know the token is +there, so it registers no webhook, opens no merge request, and neither reports +nor renews its expiry. Use the **GitLab** button to hand the token to Windmill +if you want any of that. + +## Expiry and renewal + +Windmill reads `expires_at` from the token it holds and shows it on the +repository in the workspace's git sync settings. Within three weeks of expiry it +rotates the token through GitLab's own +`POST /personal_access_tokens/self/rotate`, stores the replacement, and verifies +it. Only the token can rotate itself, so one without `api` (or `self_rotate`) is +a permanent warning rather than something Windmill can fix. + +Only the workspace that holds a credential renews it, so one renewal serves the +whole fork chain instead of each fork racing to renew the same token. A fork +reads the parent's without holding one, so it never renews; the parent does, and +every fork sees the replacement at once. + +Rotation is deliberately never retried. GitLab revokes the old token the instant +it issues the replacement, and presenting an already-rotated token to `/rotate` +again is treated as reuse: it revokes **the whole token family, including the +live replacement**. So a rotation that succeeded at GitLab but failed to persist +is surfaced as an error to act on, not retried. + +Non-expiring tokens are possible only for a **group service account PAT** on +self-managed, with `require_personal_access_token_expiry` turned off in the +instance's application settings. A group access token is always rejected without +an `expires_at`. + +## What each managed feature needs + +| Feature | Needs | +| --- | --- | +| Instant pull | A project hook Windmill creates, so Maintainer; and a Windmill base URL GitLab can reach | +| Merge requests on deploy | Developer, plus the `api` scope | +| Diff preview on a merge request | The project hook, plus permission to post merge request notes | + +Instant pull falls back to checking the tracked branch about every minute when +the hook cannot be created or delivered, so nothing silently stops syncing. + +## Self-managed differences + +**Webhooks to a private network are blocked by default.** GitLab refuses to +create a hook pointing at a private or local address until an administrator +enables *Allow requests to the local network from webhooks and integrations* +(Admin → Settings → Network → Outbound requests, +`allow_local_requests_from_web_hooks_and_services`). A Windmill instance on the +same private network as GitLab needs this; without it, hook creation fails with a +"blocked" error and the repository keeps polling. + +**A relative-URL install is not supported.** GitLab can be served under a path +prefix (`https://example.com/gitlab`), and that prefix cannot be told apart from +a group of the same name: `example.com/a/b/c.git` is either group `a/b` project +`c`, or prefix `a` with group `b` project `c`. Windmill reads it as the nested +group, so on a relative-URL install it derives the wrong API base and the managed +features stay unavailable. Such a repository still syncs through its token URL, +which needs no API base. + +Everything else is identical: Windmill talks to `/api/v4` and needs +no inbound access of its own beyond the hook deliveries. + +## The deploy preview is a note, not a pipeline status + +On GitHub the preview is a check run: its own object, advisory unless the +repository makes it required. GitLab has no equivalent. Its only comparable +primitive is a commit status, and posting one has side effects Windmill will not +impose on a project: + +- GitLab files the status **as a job inside whatever pipeline already covers that + commit**, so a failed Windmill status fails the project's own pipeline, and its + reviewers see their test suite as failed. +- `allow_failure` is ignored on the commit-status endpoint, so the status cannot + be made advisory. +- On a commit with no pipeline it creates an `external` pipeline instead, which + then gates merging under *Pipelines must succeed*, including while it is still + running. + +So on GitLab the preview lives entirely in a **merge request note** that Windmill +keeps up to date: it carries the workspace, the status line, the commit, a link +to the job, and the full list of changes merging would deploy. A note cannot +block a merge or change what the project's own CI reports. + +The note is upserted rather than appended, so a merge request accumulates one +Windmill comment however many times it is pushed to. diff --git a/frontend/src/lib/components/ApiConnectForm.svelte b/frontend/src/lib/components/ApiConnectForm.svelte index e4ee3ca7b9..dedeee90e2 100644 --- a/frontend/src/lib/components/ApiConnectForm.svelte +++ b/frontend/src/lib/components/ApiConnectForm.svelte @@ -12,6 +12,7 @@ import { Loader2 } from 'lucide-svelte' import { untrack } from 'svelte' import GitHubAppIntegration from './GitHubAppIntegration.svelte' + import GitLabIntegration from './GitLabIntegration.svelte' import BedrockCredentialsCheck from './BedrockCredentialsCheck.svelte' import { isCloudHosted } from '$lib/cloud' import ResourceGen from './copilot/ResourceGen.svelte' @@ -28,6 +29,13 @@ isValid?: boolean linkedSecretCandidates?: string[] | undefined description?: string | undefined + /** Workspace the resource is being saved into, which is not always the one + * being navigated. The GitLab picker has to store the credential where the + * resource will look for it. */ + workspace?: string + /** Fired once the GitLab picker has stored the picked project's token, so a + * form that would otherwise file the URL as a secret knows it holds none. */ + onCredentialStored?: () => void onSynced?: () => void } @@ -39,6 +47,8 @@ isValid = $bindable(true), linkedSecretCandidates = undefined, description = $bindable(undefined), + workspace = undefined, + onCredentialStored, onSynced = undefined }: Props = $props() @@ -249,6 +259,19 @@ }} onDescriptionUpdate={(newDescription) => (description = newDescription)} /> + + { + args = newArgs + rawCode = JSON.stringify(args, null, 2) + rawCodeEditor?.setCode(rawCode) + }} + /> {#if resourceType?.includes('bedrock') && !isCloudHosted()} diff --git a/frontend/src/lib/components/AppConnectInner.svelte b/frontend/src/lib/components/AppConnectInner.svelte index 7f2b646f6b..b0655590c2 100644 --- a/frontend/src/lib/components/AppConnectInner.svelte +++ b/frontend/src/lib/components/AppConnectInner.svelte @@ -1373,6 +1373,15 @@ {linkedSecretCandidates} {resourceType} {resourceTypeInfo} + workspace={effectiveWorkspace} + onCredentialStored={() => { + // `forceSecretValue` files a git_repository's `url` in a secret + // variable, for the URLs that carry a token in them. The picker's + // does not: the token is stored separately, so that variable would + // hold nothing secret and add a second place to keep in step with + // the resource. + linkedSecrets = linkedSecrets.filter((f) => f !== 'url') + }} bind:args bind:isValid onSynced={getResourceTypeInfo} diff --git a/frontend/src/lib/components/GitHubAppIntegration.svelte b/frontend/src/lib/components/GitHubAppIntegration.svelte index 11844d7d9c..89a80de6b0 100644 --- a/frontend/src/lib/components/GitHubAppIntegration.svelte +++ b/frontend/src/lib/components/GitHubAppIntegration.svelte @@ -180,7 +180,7 @@ {#if !githubState.loadingGithubInstallations} + {/snippet} + {#snippet content({ close })} +
+
+
+
GitLab instance
+ +
+
+
Project access token
+
+ Create it on the project you are syncing, with the api scope and the + Maintainer role. +
+ +
+ Windmill stores it and renews it before it expires. Use one token per project: a group + token covers the group, but renewal replaces it for one project at a time. +
+
+
+ +
+ {#if listError} + {listError} + {/if} + {#if staleListing} +
+ The token changed. List the projects again to choose one it can reach. +
+ {/if} + {#if projects.length > 0 && !staleListing} +
+
Project
+