mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-09 08:03:50 +00:00
feat: bring gitlab repositories to parity for git sync (#10938)
* feat: track and rotate gitlab git-sync repository tokens * chore: point ee-repo-ref at the gitlab credential branch * fix: strip server-owned credential status and correct expiry copy * fix: gate credential maintenance on enterprise and alert on stalled renewal * fix: alert on an auto-renewed token only once it has actually expired * feat: receive gitlab push webhooks for instant git sync pull * feat: open gitlab merge requests and post diff previews on them Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: keep gitlab merge request previews out of the project's own pipeline Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: bound the credential maintenance pass and gate the gitlab picker on a license Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: create the gitlab picker's variable in the edited workspace Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: make the gitlab picker's variable path collision-resistant Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: state the gitlab scope and rotation facts the code relies on Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: resolve the check marker's repository from its path, not a stored url Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: refuse to finish a check whose repository has been repointed Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: trust a check marker's captured url when it carries no identity Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: recreate a missing webhook from credential maintenance Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: state that relative-url gitlab installs are out of scope Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: keep credential status out of exports and clear stale webhook warnings Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: refuse an unprovable check and guard the picker on the stored repository Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: re-check the picker's target path at the moment it is written Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: snapshot the picker's inputs before it starts writing Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: recommend a project access token per repository Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * [ee] feat: keep the git-sync credential in workspace settings * [ee] fix: drop a removed repo's credential and honor the workspace override * [ee] fix: resolve a fork's git credential from its whole ancestry * [ee] refactor: reuse fork_ancestor_chain instead of a second ancestry walk * [ee] fix: resolve an app installation from the whole ancestry, not the parent * [ee] revert: keep the app installation fallback at one level * fix: store the git credential only once the resource is saved * fix: keep a repository's credential when it leaves git sync settings * docs: cut the gitlab picker's token guidance down to what it needs * feat: mark a repository whose credential windmill holds * fix: ignore the managed-credential marker when the url carries a token * docs: drop the picker's setup alert for a line by the token field * feat: replace a repository's stored token from its resource * fix: store a picked credential for its own workspace, before the resource * refactor: key a stored git credential by its repository, not its resource Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: refresh the sqlx cache for the repository-keyed credential queries Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: gate the credential pass budget on the features that use it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: decide credential rotation ownership by repository, not resource path Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * refactor: renew only the credentials windmill holds, not tokens in a repo url Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: authenticate the fork-branch poll and correct the renewal guidance Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: do not claim a managed credential for a url the client cannot resolve Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: define the credential facade for private builds without enterprise Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: pin the listed token before the await and name the real renewal blocker Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: pin the token the replace flow checked, and derive the scope test once Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: classify the renewal state once so the card cannot contradict itself Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * refactor: ask only whether the token gets renewed, not why it does not Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * refactor: replace the managed-credential marker with a server answer Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: read renewal from the credential and its origin, not a removed field Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: read the provider for url-token repos, await the origin before defaulting, and visit unchecked repos last The maintenance pass sorted repositories with no recorded check first on the premise that they cost nothing, but a token-in-URL remote on a host that is not GitLab is probed every pass and never records a check, so it held the head of the list ahead of the tokens that expire. Such repositories now sort last. The card decided its delivery defaults before the origin lookup landed, so a freshly picked GitLab repository never got webhook delivery; the two lookups are awaited together. The resource editor offers to replace a token only where it is held, not in a fork that borrows it, and the replace flow refuses a URL it cannot parse instead of keying the token to it. Attaching a stored credential to a commit-hash probe now requires admin, matching the installation credential beside it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: describe the gitlab listing token the way the picker and the setup guide do Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * refactor: a token in the repository url is a plain remote, not a tracked credential Drops the status fingerprint that told one URL token from another, the docs' promise that such a token's expiry is reported, and the test's expectation that a URL-token repository declares a host. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: the card reads the credential origin for managed controls and honours the licence for a borrowed token Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: bump the ee ref Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: hide a repository's credential line once nothing is held for the repository it names Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: describe the exported credential status as it is Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * refactor: run the credential maintenance pass as its own task, without a budget The pass ran inside the monitor's join, whose deadline cancels every future in it, and a rotation cancelled between GitLab issuing a token and Windmill storing it loses the token family. A wall-clock budget with a least-recently-checked ordering kept it under the deadline. Spawning the pass instead makes the deadline irrelevant, so the budget, the ordering and the counter go; the advisory lock keeps a slow pass from overlapping the next, as it already did. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: say what detaching the maintenance pass buys, and what it does not Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: run git sync on the hub script version that reads a stored credential Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: run the deploy push and the connection test on the hub versions that read a stored credential Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: keep App repositories and plain remotes out of the stored-credential paths Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: host-neutral deploy preview wording, drop the project filter from the GitLab picker Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: bump ee ref, rotation no longer retains a second connection per repository Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: bump ee ref, the rotation write-back holds a single connection Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: hold the credential maintenance lock in a transaction so a dead sweep releases it Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * docs: describe the credential-stored callback as it fires Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * fix: keep the credential maintenance lock past the pool's idle-in-transaction timeout Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C1xHmkxuxYb1GYvth1BS75 * chore: update ee-repo-ref to e092518ee60e33160fee9ae91a4d109566f7b0ee This commit updates the EE repository reference after PR #771 was merged in windmill-ee-private. Previous ee-repo-ref: 74481f7cc345757aebb2a8b04d3a22978328c348 New ee-repo-ref: e092518ee60e33160fee9ae91a4d109566f7b0ee Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
windmill-internal-app[bot]
parent
6860521b4e
commit
9444049d60
@@ -0,0 +1,170 @@
|
||||
# Git sync with GitLab
|
||||
|
||||
GitLab has no equivalent of a GitHub App, so there is nothing to install and no
|
||||
consent screen. What Windmill needs instead is one credential you create in
|
||||
GitLab and paste once. With it, a GitLab repository gets the same managed
|
||||
features an app-backed GitHub repository has: instant pull over a webhook, merge
|
||||
requests opened on deploy, and a diff preview posted onto the merge request.
|
||||
|
||||
## The credential
|
||||
|
||||
Create a **project access token** on the project you are syncing (Settings →
|
||||
Access tokens). It is a bot identity that outlives the person who created it,
|
||||
which is what you want for a credential the instance uses unattended, and it
|
||||
reaches exactly the one project.
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Scope | `api` |
|
||||
| Role | Developer to push deploy branches; **Maintainer** to also manage the webhook and open merge requests |
|
||||
| Expiry | Required. A group service account PAT can be non-expiring on self-managed (see below); an access token cannot |
|
||||
|
||||
**Use a separate token per repository.** A group access token works too and
|
||||
reaches every project in the group, which is convenient for a lot of
|
||||
repositories — but Windmill stores the credential per repository, and renewal
|
||||
rewrites the repository it renewed for. Any other repository holding that same
|
||||
token keeps the revoked one and stops syncing until you paste a new token there.
|
||||
Each stranded repository says so on its card, so it is visible rather than
|
||||
silent, but a token per repository avoids it entirely.
|
||||
|
||||
`api` is a superset: it authorizes Git over HTTPS as well, so no separate
|
||||
`write_repository` is needed to clone and push, and it is also what makes the
|
||||
token renewable. A `write_repository`-only token can still push, but Windmill
|
||||
cannot inspect or renew it and reports that in the workspace's git sync settings.
|
||||
|
||||
### The identity Windmill acts as
|
||||
|
||||
GitLab issues an access token to a bot user it creates for it — `project_<id>_bot_…`
|
||||
for a project token, `group_<id>_bot_…` for a group one — and the bot's display
|
||||
name is **the name you gave the token**. That name is the byline on everything
|
||||
Windmill does: the author of deploy commits, of the merge requests it opens, and
|
||||
of the preview notes it writes. Name it for what it is, `windmill-sync` or
|
||||
similar, rather than something only you will recognise.
|
||||
|
||||
Each token you create adds another bot member to the project or group. Renewal
|
||||
does not — it keeps the same bot — so a repository accumulates one bot, not one
|
||||
per year.
|
||||
|
||||
Renewal goes through GitLab's own self-rotation endpoint. Both kinds of access
|
||||
token are held as their bot user's personal access token, so the token rotates
|
||||
itself and Windmill never needs a credential with rights over the project or
|
||||
group.
|
||||
|
||||
## Connecting a repository
|
||||
|
||||
In the resource form for a `git_repository` resource, use the **GitLab** button:
|
||||
paste the instance URL and the token, pick a project from the list, and Windmill
|
||||
keeps the token for you. The resource itself gets the plain remote URL
|
||||
(`"url": "https://gitlab.com/group/project.git"`), with no credential in it.
|
||||
|
||||
The token is stored encrypted on the workspace, keyed by the repository it was
|
||||
issued for rather than by the resource naming it, the same way a GitHub App
|
||||
installation is held against the account it covers. Nothing reads it back out
|
||||
over the API: the server attaches it when it talks to GitLab, and a sync job
|
||||
receives it only against its own job token. Repointing a resource's `url` asks
|
||||
for a different repository's token and finds none, so the edit carries nothing
|
||||
with it; a repository that genuinely moved needs its token entered again.
|
||||
|
||||
Because the repository is the key, the token is stored the moment you pick the
|
||||
project, before the resource is saved. Renaming the resource later keeps it, and
|
||||
cancelling the edit leaves a stored token that nothing uses until some resource
|
||||
points at that repository again.
|
||||
|
||||
Forks of the workspace read this one copy rather than getting their own, so
|
||||
renewal reaches all of them at once and the token is not duplicated into every
|
||||
descendant workspace.
|
||||
|
||||
Treat workspace admin as equivalent to holding the token. An admin of the
|
||||
workspace, or of any fork below it, can point a repository at a sync script they
|
||||
wrote and have that job request the credential, exactly as they can for a GitHub
|
||||
App installation token. Storing it this way keeps it out of the variables API and
|
||||
out of every fork's own storage; it is not a boundary against the admins of those
|
||||
workspaces.
|
||||
|
||||
A URL with the token written into it, in the resource or in a secret variable
|
||||
the resource points at (`"url": "$var:..."`), is a plain git remote: it syncs on
|
||||
deploy and by polling, and nothing else. Windmill does not know the token is
|
||||
there, so it registers no webhook, opens no merge request, and neither reports
|
||||
nor renews its expiry. Use the **GitLab** button to hand the token to Windmill
|
||||
if you want any of that.
|
||||
|
||||
## Expiry and renewal
|
||||
|
||||
Windmill reads `expires_at` from the token it holds and shows it on the
|
||||
repository in the workspace's git sync settings. Within three weeks of expiry it
|
||||
rotates the token through GitLab's own
|
||||
`POST /personal_access_tokens/self/rotate`, stores the replacement, and verifies
|
||||
it. Only the token can rotate itself, so one without `api` (or `self_rotate`) is
|
||||
a permanent warning rather than something Windmill can fix.
|
||||
|
||||
Only the workspace that holds a credential renews it, so one renewal serves the
|
||||
whole fork chain instead of each fork racing to renew the same token. A fork
|
||||
reads the parent's without holding one, so it never renews; the parent does, and
|
||||
every fork sees the replacement at once.
|
||||
|
||||
Rotation is deliberately never retried. GitLab revokes the old token the instant
|
||||
it issues the replacement, and presenting an already-rotated token to `/rotate`
|
||||
again is treated as reuse: it revokes **the whole token family, including the
|
||||
live replacement**. So a rotation that succeeded at GitLab but failed to persist
|
||||
is surfaced as an error to act on, not retried.
|
||||
|
||||
Non-expiring tokens are possible only for a **group service account PAT** on
|
||||
self-managed, with `require_personal_access_token_expiry` turned off in the
|
||||
instance's application settings. A group access token is always rejected without
|
||||
an `expires_at`.
|
||||
|
||||
## What each managed feature needs
|
||||
|
||||
| Feature | Needs |
|
||||
| --- | --- |
|
||||
| Instant pull | A project hook Windmill creates, so Maintainer; and a Windmill base URL GitLab can reach |
|
||||
| Merge requests on deploy | Developer, plus the `api` scope |
|
||||
| Diff preview on a merge request | The project hook, plus permission to post merge request notes |
|
||||
|
||||
Instant pull falls back to checking the tracked branch about every minute when
|
||||
the hook cannot be created or delivered, so nothing silently stops syncing.
|
||||
|
||||
## Self-managed differences
|
||||
|
||||
**Webhooks to a private network are blocked by default.** GitLab refuses to
|
||||
create a hook pointing at a private or local address until an administrator
|
||||
enables *Allow requests to the local network from webhooks and integrations*
|
||||
(Admin → Settings → Network → Outbound requests,
|
||||
`allow_local_requests_from_web_hooks_and_services`). A Windmill instance on the
|
||||
same private network as GitLab needs this; without it, hook creation fails with a
|
||||
"blocked" error and the repository keeps polling.
|
||||
|
||||
**A relative-URL install is not supported.** GitLab can be served under a path
|
||||
prefix (`https://example.com/gitlab`), and that prefix cannot be told apart from
|
||||
a group of the same name: `example.com/a/b/c.git` is either group `a/b` project
|
||||
`c`, or prefix `a` with group `b` project `c`. Windmill reads it as the nested
|
||||
group, so on a relative-URL install it derives the wrong API base and the managed
|
||||
features stay unavailable. Such a repository still syncs through its token URL,
|
||||
which needs no API base.
|
||||
|
||||
Everything else is identical: Windmill talks to `<your-gitlab>/api/v4` and needs
|
||||
no inbound access of its own beyond the hook deliveries.
|
||||
|
||||
## The deploy preview is a note, not a pipeline status
|
||||
|
||||
On GitHub the preview is a check run: its own object, advisory unless the
|
||||
repository makes it required. GitLab has no equivalent. Its only comparable
|
||||
primitive is a commit status, and posting one has side effects Windmill will not
|
||||
impose on a project:
|
||||
|
||||
- GitLab files the status **as a job inside whatever pipeline already covers that
|
||||
commit**, so a failed Windmill status fails the project's own pipeline, and its
|
||||
reviewers see their test suite as failed.
|
||||
- `allow_failure` is ignored on the commit-status endpoint, so the status cannot
|
||||
be made advisory.
|
||||
- On a commit with no pipeline it creates an `external` pipeline instead, which
|
||||
then gates merging under *Pipelines must succeed*, including while it is still
|
||||
running.
|
||||
|
||||
So on GitLab the preview lives entirely in a **merge request note** that Windmill
|
||||
keeps up to date: it carries the workspace, the status line, the commit, a link
|
||||
to the job, and the full list of changes merging would deploy. A note cannot
|
||||
block a merge or change what the project's own CI reports.
|
||||
|
||||
The note is upserted rather than appended, so a merge request accumulates one
|
||||
Windmill comment however many times it is pushed to.
|
||||
Reference in New Issue
Block a user