* docs: add design doc for automatic git-to-windmill pull sync
* docs: add migration plan and implementation phases to git-sync pull design
* feat(git-sync): add auto_pull settings schema and pull enqueue primitive
Adds AutoPullSettings/AutoPullMode/AutoPullStatus on GitRepositorySettings
(workspace_settings.git_sync JSONB), the GIT_SYNC_PULL_SCRIPT_PATH constant,
and should_pull/effective_poll_interval_s helpers with unit tests. Exports the
EE enqueue_git_pull_job primitive. Foundation for repo→Windmill auto-pull.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): poll repos and auto-pull new commits into the workspace
Phase 1 of automatic repo → Windmill sync. A monitor task (EE-licensed,
single-replica via advisory lock) git ls-remotes each auto-pull-enabled
repository ~every minute and enqueues a pull when the tracked branch moves,
reusing the {workspace_id}:git_sync concurrency key so pulls serialize with
in-flight push commits.
- windmill-store: background (no-authed) resolver get_git_repo_head_for_autopull
that resolves the repo resource (incl. $var: refs) and ls-remotes; GitHub-App
repos are skipped here and will sync via webhooks (phase 2).
- monitor.rs: poll/reconcile/persist with optimistic sha advance and failure
status; targeted jsonb update so concurrent settings edits aren't clobbered.
- edit_git_sync_repository: preserve server-owned auto_pull state on UI save.
- openapi: AutoPullSettings/AutoPullMode/AutoPullStatus + auto_pull field.
- frontend: per-repo "Automatically deploy changes from Git" toggle with last
sync status; demote the GitHub Actions link to an advanced CI option.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): wire webhook lifecycle + receiver; share reconcile logic
OSS side of phase 2 auto-pull webhooks:
- edit_git_sync_repository creates/removes the repo webhook on save (EE-gated,
best-effort → falls back to polling).
- monitor poller now delegates to the shared windmill_git_sync reconcile/persist
helpers (also used by the webhook receiver), removing duplicated logic.
- export the shared reconcile/persist/failure helpers; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref for phase 3 in-app PR creation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): show webhook vs polling status on the auto-pull toggle
When a repo has an active webhook (auto_pull.webhook_id set), the status line
reads "instant via webhook"; otherwise it reads the ~1-minute polling cadence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): post PR diff check on dry-run completion (phase 4)
Worker completion hook in process_completed_job: when a DeploymentCallback job
carrying the __git_sync_pr_check marker finishes, parse the dry-run SyncResponse
and patch the GitHub check run with the diff summary (success/neutral/failure).
Export enqueue_git_pull_dry_run; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref (drop unused GHES webhook_secret)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* revert(git-sync): defer phase 4 PR diff checks (OSS side)
Remove the worker completion hook that posted the PR check run, drop the
enqueue_git_pull_dry_run re-export and the orphaned sqlx cache, bump EE ref.
Phases 1-3 (polling, webhooks, in-app PR creation) are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "revert(git-sync): defer phase 4 PR diff checks (OSS side)"
This reverts commit 0137d3ca48.
* chore(git-sync): point EE ref at restored phase 4 commit
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref for clone_ref dry-run
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump init-repository hub script to v28784
Picks up the clone_ref param (windmill-integrations#158) so the phase 4 PR-check
dry-run can clone the PR head. Backward compatible; manual pull/push and the
automated pull/poller/webhook all move to the same published version.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref for auto-pull admin-permissioning fix
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref for superadmin pull fallback
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(git-sync): refresh auto-pull tooltip; bump EE ref for webhook secret encryption
The auto-pull toggle tooltip claimed GitHub App repos would sync via
webhooks "in a future update"; webhook delivery now works, so describe
the webhook-vs-polling behavior accurately. Bump the EE ref to pick up
encrypting the webhook HMAC secret at rest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(git-sync): poll app-backed repos in auto/polling mode
The auto-pull poller skipped app-backed repos (the ls-remote head check
can't authenticate a tokenless URL), so auto- and polling-mode app repos
never synced when their webhook wasn't live. Wire the poller to fetch the
head via the GitHub API for app repos and reconcile. Bump the EE ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(git-sync): auto-pull UI — direction split, delivery mode, fallback notice
Reorganize the repository card into two clearly labeled directions:
"Push to Git on deploy (Windmill → Git)" and "Pull from Git (Git →
Windmill)". In the pull section:
- new connections default to auto-pull enabled (webhook with polling
fallback); existing repos load with auto-pull off and are unchanged
- a Delivery selector chooses "Webhook with polling fallback" or
"Polling only (air-gapped)"
- a notice surfaces webhook_error when delivery falls back to polling
- a reminder to remove any pre-existing GitHub Action that pushed into
Windmill, to avoid conflicting double-syncs
Adds the webhook_error field to AutoPullSettings (+ openapi) and bumps
the EE ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(git-sync): clearer push indicator + gate webhook delivery to app repos
- Push-on-deploy is shown with a check icon + concise line (via the
shared GitSyncModeDisplay, restyled from the oversized "Sync:" text);
the setup wizard reuses it without the check (pre-save preview).
- The delivery-mode selector only shows for GitHub App-backed repos;
token-based repos show a "webhooks require the GitHub App (managed or
GHES)" note with a docs link and poll instead. Bumps the EE ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(git-sync): fork auto-sync (phase 5) + live deploy check (phase 6)
Phase 5 — fork auto-sync configured at the parent (replaces the *-to-forks
GitHub Actions):
- Add fork_open_prs + fork_pull_sync to GitRepositorySettings (openapi + UI).
- UI: two "Forks of this workspace" toggles in the repo card, gated on
app-backed and not-a-fork; serialize the flags on save.
- On fork creation, strip the inherited auto_pull block (and fork_* flags) from
the copied git_sync repo: a fork must not carry the parent's webhook id (it
would delete the parent's hook on disable) or self-poll on top of the parent's
fan-out. Push-direction config + installation are still inherited unchanged.
Phase 6 — live deploy status check on the commit (Cloudflare-style): an
in-progress "Windmill" check on the head commit that flips to "Deployed N
changes"; completion handled by the generalized git-sync check hook.
Bump EE ref for the phase 5-6 EE implementation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* chore(git-sync): bump EE ref for PAT auto-pull mode normalization
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): address PR review findings
- webhook_secret: redact from the settings API response and Debug output (still
persisted encrypted); it's a server-only HMAC key the UI never needs.
- poller: honor each repo's effective poll interval (relaxed ~10 min when a
webhook is live) instead of probing every ~60s tick.
- settings save: roll back a just-created webhook if the settings transaction
doesn't commit, so a failed save can't orphan a hook.
- auto-pull head check: fail SSH remotes with an actionable message (background
polling has no SSH identity) instead of a confusing ls-remote error.
- deploy/PR check summary: a pull result carrying neither changes nor a settings
diff now falls back to the unsummarized path instead of a false "in sync".
- UI: reset isGithubApp on resource change / failed fetch so webhook + fork
controls can't show for the wrong repo.
- tests: cover parse_git_sync_changes and format_change_list edge cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): correct feature gating for OSS builds
- monitor.rs: keep the AUTO_PULL_LAST_POLL static, slack const, and
poll_git_auto_pull_inner all behind #[cfg(feature = "private")] (an inserted
static had split the cfg off the function, ungating it in OSS builds).
- edit_git_sync_repository: the webhook create/rollback block references
windmill_common::git_sync_ee (private module), so gate it on
all(enterprise, private) instead of enterprise only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* chore(sqlx): cache workspace_diff query pulled in from origin/main
Re-merged origin/main (advanced past the earlier merge); regenerate the offline
sqlx entry for the new workspace_comparison test query so SQLX_OFFLINE builds
(cargo_test) pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): address Codex review findings (webhook cleanup on delete)
- Deleting a git-sync repository now tears down its managed GitHub webhook
(deletion bypassed the sync_repo_webhook lifecycle, orphaning the hook so
GitHub kept delivering to the instance).
- Worker completion hook rolls back the optimistic auto-pull sha on job failure
(OSS side of the EE change) + caches the new marker query. Bump EE ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): delete repo webhook after the removal commits
Codex re-review nits:
- delete_git_sync_repository deleted the webhook before the settings transaction
committed; a failed save would then leave the repo pointing at a hook that no
longer exists (sync_repo_webhook treats a set webhook_id as live and won't
recreate it). Capture the hook id, commit the DB removal, then delete the hook.
- Reword a fork-copy comment to drop drafting-history wording per AGENTS.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): reconcile the edit-path webhook after the settings commit
Codex nit: edit_git_sync_repository ran sync_repo_webhook before the transaction
committed. The rollback only covered created hooks, but sync_repo_webhook also
deletes a hook on disable/switch-to-polling — a commit failure then left the DB
with a webhook_id whose hook was already gone (and it wouldn't be recreated).
Save + commit first, then reconcile the webhook against the durable config and
persist any hook id/secret change (best-effort). Bump EE ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): preserve webhook secret on whole-config save + default on visible add
Codex nits:
- edit_git_sync_config saved the client config verbatim, so the webhook_secret
redacted from the GET response would be dropped (breaking delivery). Preserve
server-owned auto-pull state (webhook id/secret, synced sha, last status) per
repo from the existing settings, matching edit_git_sync_repository.
- addSyncRepository (the visible add path) didn't set the auto_pull default, so
new sync repos added from the UI came up with auto-deploy off. Match
addRepository's default (webhook + polling fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* refactor(git-sync): drop fork_pull_sync (parent-level keep-forks-in-sync)
Removes the "Keep forks in sync with the tracked branch" toggle and its
fan-out. Pulling the tracked branch straight into every fork was the
inconsistent piece; the consistent model is per-fork branch sync (each
fork tracks its own wm-fork/** branch), which is a separate follow-up.
fork_open_prs is kept. Also tightens the fork toggle-section spacing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): detect dev workspaces in CLI fork branch derivation
isForkWorkspace / computeGitSyncDeployBranch keyed off the wm-fork- id
prefix. Dev workspaces are forks with a custom, prefix-less id, so their
wm-fork/** branch was never derived or created. Detect them via
parent_workspace_id too (which the backend already passes), mirroring the
backend's `parent.is_some() || wm-fork- prefix` rule.
Pairs with the hub-script clone-flag fix (windmill-integrations#163); both
take effect once the CLI is released and the pinned version is bumped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): reconcile webhooks on full-config save
edit_git_sync_config preserved server-owned webhook fields but never
created or deleted the managed GitHub webhook, so enabling auto-pull
through the whole-config endpoint only polled, and disabling or removing
a repo left an orphan hook still delivering. Mirror the per-repository
endpoint: after the commit is durable, reconcile every saved repo's
webhook (sync_repo_webhook) and delete the hooks of repos the save
removed, including the clear-whole-config case. Addresses the Codex nit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): address Codex nits (webhook orphan on cleared auto_pull, fork detection)
- edit_git_sync_config: also delete a repo's old webhook when the save drops
the repo OR clears its auto_pull. Webhook fields are only preserved onto a
Some auto_pull, so a save that present-but-clears a repo would otherwise
orphan its hook.
- GitSyncRepositoryCard: isFork now uses parent_workspace_id OR the wm-fork-
prefix (was AND), matching the backend/CLI rule, so prefix-less dev
workspaces are detected as forks and don't show the parent fork-PR toggle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* docs(git-sync): update design doc for the dropped fork_pull_sync
Phase 5 documented "Keep forks in sync with the tracked branch"
(fork_pull_sync) and its fan-out as implemented; that feature was removed.
Rewrite the section to reflect what ships (fork_open_prs), note the drop +
the per-fork-branch follow-up, and remove the stale fan-out mentions
elsewhere. Addresses the Codex nit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): redact webhook secrets from workspace export; fix doc endpoints
- Export (P1): strip the server-owned auto_pull state (webhook secret/id/error
+ synced sha + last pull status) from git_sync before it is written into an
export's settings.json for both settings formats. The HMAC webhook secret
must never leave the server (matching the GET-settings redaction), and a
re-imported workspace must not inherit another install's hook/sync state.
- Docs: the webhook receiver is a single per-workspace endpoint
/api/w/{workspace}/github_app/webhook (host-aware for managed + self-managed);
update the stale push_webhook/{id} and instance-global /api/github_app/webhook
references.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): skip deleted/archived workspaces in the auto-pull poller
The poller scanned workspace_settings directly, so an archived (soft-deleted)
or renamed-away workspace — whose settings row persists — kept polling and
could enqueue a pull into a dead workspace. Join workspace and require
NOT deleted. The EE webhook receiver gets the same filter (ee ref bumped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): never trust client-supplied server-owned auto-pull fields
Both write endpoints (edit_git_sync_repository, edit_git_sync_config)
persisted caller-supplied auto_pull.webhook_id / webhook_secret /
webhook_error / last_synced_sha / last_pull_status when adding a repo or
newly enabling auto-pull, letting a client inject a webhook id/secret or
fake sync state. Strip those server-owned fields from the request up front;
existing repos re-derive them from the DB (carried over), new ones start
clean and the server (re)creates the webhook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): merge webhook fields post-commit instead of clobbering the row
The post-commit webhook reconcile in edit_git_sync_repository and
edit_git_sync_config wrote the whole pre-reconcile git_sync snapshot back
after the main save committed. A concurrent git-sync edit or poller status
write that landed in the gap could then be dropped by the stale snapshot.
Re-read the current row and merge only the reconciled webhook id/secret/error
for the repos the reconcile actually changed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* feat(git-sync): parent-managed fork sync + PR-on-deploy toggles
Fork sync (push-on-merge-to-forks parity): a parent-level
auto_pull.sync_forks toggle routes changes on each fork's wm-fork/** branch
into that fork workspace, via the parent's existing webhook and one extra
fork-heads listing per poll tick (git ls-remote pattern for token repos,
git/matching-refs for app-backed). Fork state is a server-written
status-only auto_pull blob on the fork's own repo entry; the fork's card
shows a read-only "managed in the parent workspace" line with its branch
and last pull status. Dev workspaces (prefix-less ids) use the same branch
parsing (unit-tested in windmill-common).
PR-on-deploy: opening PRs for Windmill-pushed branches moves into the
deploy pipeline, per repo toggle (promotion_open_prs on the promotion
repo; parent-level fork_open_prs for fork deploys). The push job carries a
marker and the job-completion hook derives the pushed branch (helper
unit-tested against the CLI formula) and opens the PR outbound, so it
works without inbound webhooks; the webhook-side wm_deploy PR arm is
removed. The documented open-pr-* GitHub Actions remain valid alternatives
(PR creation is idempotent).
Fork guards: promotion mode, enabled auto-pull, and fork_open_prs are
rejected on fork workspaces (they are parent-managed; a fork's deploys
always target its wm-fork/** branch) and the promotion card is hidden in a
fork's settings. Enabling auto-pull now also requires EE, and the
post-commit webhook reconcile persists the normalized delivery mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* feat(git-sync): dev workspaces sync with their environment-label branch
A dev workspace's git branch is its environment label verbatim (dev/
staging, default dev) — a first-class env branch like the documented
push-on-merge-staging layout — instead of the wm-fork/** form. The label
rides the deploy job args (backend → hub script → CLI
--dev-workspace-label), the PR completion hook derives the same head, the
webhook/poller route label branches into the matching dev-workspace child
(poller lists them alongside wm-fork/* via extra ls-remote refs / per-label
API lookups), and manual pulls from the UI pass clone_ref accordingly. The
CLI refuses to deploy when the label branch equals the checked-out tracked
branch, which would otherwise commit fork content straight to it.
Because the branch is keyed on the label, the label is now immutable after
creation: set at create/attach only, the set_dev_workspace_label endpoint
is removed and the settings tab shows it read-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* feat(git-sync): nested fork routing + fork-of-dev branch rooting
A fork of a dev workspace now roots its wm-fork/** branch on the dev's
environment-label branch (the content it diverged from) and its PR merges
back into that branch: the backend passes parent_dev_workspace_label with
the deploy (parent row joined in both enqueue paths), the CLI gains
--parent-dev-workspace-label and checks it before the wm-fork- prefix
fallback when rooting a fork-of-a-fork branch, and the PR completion hook
uses it as the PR base.
Fork sync routing covers the whole live descendant chain of the
webhook/poller workspace (recursive, depth-capped) instead of direct
children only, and fork_open_prs is resolved at the root ancestor — only
the root can hold auto-pull config, so grandchild forks sync through it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* feat(git-sync): PR deploy-preview comment, clearer check copy, app-only hints
- The PR diff completion hook maintains one managed comment on the PR
(Cloudflare deploy-preview style: workspace, status, commit, collapsible
change list), upserted per synchronize via a hidden marker. The check run
stays for required-check gating.
- A settings difference in the diff summary is worded by cause: the PR
changes wmill.yaml, vs pre-existing drift between the repo's wmill.yaml
and the workspace, vs undetermined (neutral wording).
- Deploy-status check titles name the target workspace ("Deployed 2
change(s) to staging"), since GitHub shows a head commit's checks on any
PR containing it and a bare "Deployed" read as if the PR had deployed.
- Token-based repos see a hint pointing at the open-pr-on-commit /
open-pr-on-fork-commit workflows where the app-only PR toggles would be;
an API-set toggle on a non-app repo now logs a warning naming the
fallback; the design doc lists app-only features and their degradation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): EE-gate auto-pull UI, fork pull clone_ref, no-op push PR gate
- CE: the auto-pull and fork-PR toggles are disabled with an EE badge, and
new sync repos only default them on when licensed (basic git sync is
available on CE since #8493, but auto-pull is EE and the backend rejects it)
- The pull modal passes clone_ref for wm-fork- forks (wm-fork/<tracked>/<id>)
so a manual pull fetches the fork branch instead of the tracked branch head
- PR-on-deploy skips no-op pushes: when the push script reports pushed=false
(e.g. the deploy was caused by an auto-pull), the completion hook no longer
ensures a PR, so closed PRs aren't recreated by the sync loop
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* chore: refresh package-lock after main merge (windmill-utils-internal 1.8.2)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* test: auto-pull e2e integration tests; fix PR comment table formatting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): runtime license gate for auto-pull saves; user/group promotion-branch parity
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): explain in-sync PR verdicts with the repo's sync filter scope
A PR that only touches files outside the repository's include paths gets
"In sync", which reads as a wrong verdict; the check summary (and managed
comment) now name the filters, e.g. "Only files matching this repository's
sync filters deploy on merge: `f/**` (excluding `f/pat/**`)."
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* feat(git-sync): clearer card copy/structure; surface PR-creation failures
- Fork sync toggle renamed and kept in the pull section; the fork PR toggle
moves to the push section with a note that push settings apply to forks
- Fork/dev workspaces' push section names their actual branch instead of the
tracked-branch line; promotion repos hide the pull direction (promotion
pushes deploy branches on top of a sync-mode setup)
- Promotion mode line describes the wm_deploy/** branch + merge-to-promote
flow; workflow-fallback hints lead with the how-to and link to the docs;
test connection button demoted from accent per brand guidelines
- New server-owned open_pr_error on repo settings: the deploy completion hook
records why a PR couldn't be opened (e.g. app permission not yet approved)
and clears it on the next success; shown as a warning under the PR toggles
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix: cfg-gate scope-note helper (dead code on OSS builds)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): license-gate preserved auto-pull; attach strips parent-only settings
- edit_git_sync_repository re-checks the runtime Enterprise gate against the
EFFECTIVE repo state after preservation: the older-client arm copies the
existing auto_pull back, which the request-side check never saw
- attach_dev_workspace now mirrors the fork-creation copy on the attached
workspace's own git sync: promotion repos dropped, auto_pull/fork PRs/PR
error stripped, and any managed webhook deleted after commit (the attached
workspace is parent-managed and must not keep pulling its old tracked branch)
- integration test: attaching an auto-pull-enabled workspace strips it
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): detach clears standalone parent; reject label == tracked branch
- detach_dev_workspace clears parent_workspace_id for prefix-less (attached
standalone) workspaces so they stop classifying as forks and deploying to
wm-fork/** branches; wm-fork- re-designated forks keep their parent; cache
invalidations mirror attach
- dev-workspace create/attach reject an environment label that equals a
git-sync repository's tracked branch (prod's or the candidate's): deploys
would target the very branch the repo syncs from, and the CLI guard would
fail every push job after the fact
- CLI unit tests: prefix-less fork beats wm_deploy derivation; isForkWorkspace
parent-id argument
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* chore(git-sync): bump hub script pins (push 28786, pull 28785)
Published from windmill-integrations #163 with windmill-cli@1.753.1-gitsync.0:
dev-workspace label deploys, fork-of-dev rooting, fork checkout on the
existing remote branch, and the pushed-flag result.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): no parent-only defaults on fork repos; rename strips webhook state
- addSyncRepository skips the auto_pull/fork_open_prs defaults on fork/dev
workspaces where the backend rejects them (saving a new sync repo from an
EE fork 400'd deterministically)
- change_workspace_id strips webhook id/secret/error from the copied git_sync
and deletes the stale GitHub hooks post-commit: they deliver to the old
(archived) workspace URL, so the new workspace would report a live webhook
while polling at the relaxed interval; next save re-registers cleanly
- EE: PR diff checks for contributor-fork PRs clone the synthetic
pull/<n>/head ref (head.ref doesn't exist in the base repo)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* chore(git-sync): bump pull script pin to hub/28787 (synthetic PR ref support)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): targeted jsonb update for open_pr_error (no full-blob clobber)
The full read-modify-write raced the poller's concurrent last_synced_sha /
last_pull_status writes on the same column; mirror the EE status writer and
update only the matching repository element's open_pr_error key.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* style(git-sync): inline EE badge on gated toggles (matches settings nav)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* copy(git-sync): both directions in page/card descriptions; clearer promotion flow
- Page header and sync-card description mention the pull direction, not only
push-on-deploy
- Promotion description walks the actual flow (wm_deploy/** branch, merge to
promote, sync the target workspace) and points at the PR toggle / workflow;
the Git Promotion docs link now also shows on configured cards, not only in
the empty state
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): resolve branch-less resources' default branch for fork sync
A git resource without an explicit branch polled as the bare "HEAD" ref,
which the fork/dev-label fan-out cannot scope (wm-fork/<branch>/*), so fork
sync silently never ran on polling-only repos. Resolve the remote's default
branch name with `ls-remote --symref HEAD` (one call for name + head sha);
"HEAD" only remains when resolution fails. The polling e2e test now uses a
branch-less resource to cover this shape.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): runtime license gate for in-app PR creation
promotion_open_prs/fork_open_prs are rejected on save without an Enterprise
plan (like auto_pull), and the deploy completion hook re-checks the plan
before opening PRs so flags stored while licensed stop driving GitHub calls
after a lapse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* feat(git-sync): app-aware pull defaults, always webhook delivery, token-repo guidance
- Pull-from-Git defaults on only for app-backed repos (applied when the
selected resource resolves); polling is opt-in for token repositories,
with a warning alert recommending the GitHub App (instant pull + in-app
PRs) or the sync GitHub workflow
- App repos always use webhook delivery with polling fallback: the delivery
selector is gone and a stored polling mode is normalized back to auto
- Post-save modal reflects the auto-pull state instead of telling the user
to turn on a toggle that is already on
- Non-app PR hints recommend the GitHub App explicitly
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* copy(git-sync): single info box for token-repo pull guidance
Merges the instant-pull recommendation with the GitHub Action conflict note,
shown only for non-app repos; app repos need neither, and the redundant
'instant webhook sync requires' line is gone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* copy(git-sync): keep the GitHub Action conflict note on app repos
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* copy(git-sync): anchor docs links to their exact sections
GitHub App references point at integrations/git_repository#github-app, the
workflow hints at deploy_gh_gl#github-actions-setup, and the sync workflow
at git_sync#github-actions (all anchors verified against the live docs).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* copy(git-sync): fork workflow hint links to git_sync#github-actions
open-pr-on-fork-commit is documented on the git_sync page, not deploy_gh_gl.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* fix(git-sync): fork PRs are opt-in on new connections too
Only auto-pull and fork sync default on for new app-backed connections;
opening pull requests stays a deliberate per-repo decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* feat(git-sync): default the managed PR on for new app-backed promotion repos
A promotion deploy's wm_deploy/** branch exists to be merged; without a PR
it's an orphaned branch. Fork PRs stay opt-in. Also scope the sync-repo
auto-pull default to sync mode so promotion repos can't pick it up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* docs(git-sync): GHES self-managed app permission setup
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* docs(git-sync): frame permission update against GitHub Actions, not polling
Existing installations don't have polling; their git-to-Windmill direction
runs on GitHub Actions today, so the approval text describes the update as
replacing those workflows and notes every feature is opt-in.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* copy(workspaces): drop 'cosmetic' qualifier from dev-workspace label UI
* chore: update ee-repo-ref to 9b2a6375f838436cf68cff449cc9bc621cca5281
This commit updates the EE repository reference after PR #632 was merged in windmill-ee-private.
Previous ee-repo-ref: 99eef24e2f0402b9a997cde5f67be52ee5d54b0e
New ee-repo-ref: 9b2a6375f838436cf68cff449cc9bc621cca5281
Automated by sync-ee-ref workflow.
* fix(git-sync): reject '/' in fork and dev workspace ids
* fix(git-sync): bound auto-pull git probes with a per-command timeout
* fix(git-sync): persist webhook reconcile via targeted jsonb updates
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
36 KiB
Design: Automatic git → Windmill sync (pull-based)
Status: draft for review — exploration on branch explore-git-sync-improvements
1. Problem
Git sync is guided and automatic in one direction only. Windmill → repo is fully
managed: every deploy enqueues a deploymentcallback job that runs the hub sync
script and commits to the repo. The reverse direction (repo → Windmill) requires
each customer to install a GitHub Action that runs wmill sync push with a
long-lived Windmill token stored as a repo secret, against an instance URL that
GitHub-hosted runners must be able to reach.
This breaks down because each customer runs their own instance:
| Customer setup | GH runner → instance | GitHub webhook → instance | Instance → GitHub |
|---|---|---|---|
| Windmill cloud | ✅ | ✅ | ✅ |
| Self-hosted, public URL | ✅ | ✅ | ✅ |
| Self-hosted behind VPN/firewall | ❌ | ❌ | ✅ |
| GHES on the same private network | ❌ (github.com runners) | ✅ | ✅ |
Only instance → GitHub outbound works for everyone. The current GH Action design sits in the worst column; it also requires manual workflow-file installation, token provisioning, and ongoing maintenance per customer.
2. Current state (what already exists)
The building blocks are mostly shipped:
- Instance-side pull already works.
PullWorkspaceModal.svelteruns the hub init script (hubPaths.gitInitRepo) withpull: true,dry_runpreview, anduse_promotion_overridesas a worker job. It clones the repo with the repo credentials and applies the diff to the workspace. It is manual-only today. - Managed GitHub App.
windmill-sync-helper(github.com / *.ghe.com) is installed by the customer; installation tokens are minted by the customer portal (windmill-customer-service, route/github_sync/token) against a JWT the instance stores per installation (workspace_settings.git_app_installations, token logic inbackend/windmill-common/src/git_sync_ee.rs). The portal is stateless: it keeps no installation registry and receives no webhooks. - Self-managed app for GHES. Customers create their own GitHub App; app id +
private key live in instance settings (
GhesAppSettings.svelte,get_self_managed_installation_token). The current setup checklist tells the customer to leave the app webhook inactive. - Webhook machinery. The native GitHub trigger already creates/deletes repo
webhooks via the REST API (
windmill-native-triggers/src/github/external.rs) and GitHub HMAC (X-Hub-Signature-256) verification exists inwindmill-trigger-http/src/http_trigger_auth.rs. - Loop prevention convention. Windmill-authored commits carry a
[WM]prefix so CI can ignore them. - Promotion plumbing. The CLI implements
--promotion <branch>resolvingpromotionOverridesfromwmill.yaml; the pull job already acceptsuse_promotion_overrides. PR creation onwm_deploy/**branches is done by a documented GH Action (gh pr create), not by Windmill.
What is missing is only the trigger (push event → pull job), the routing (event ref → workspace), and PR creation moving instance-side for promotion/fork parity.
3. Goals / non-goals
Goals:
- Zero-CI automatic repo → Windmill deployment: install app, pick repo, done.
- Works for every connectivity profile, degrading gracefully from instant (webhook) to near-real-time (polling).
- Cover all documented setups: basic sync, multi-repo, promotion mode (single- and cross-instance), workspace forks, local-dev git entry.
- No new credentials handed to GitHub (no Windmill tokens as repo secrets).
- Coexist with customer CI: customers who keep the GH Action lose nothing.
Non-goals (this design):
- Replacing customer CI pipelines (tests, lint, custom gates).
- A central event-relay through the portal (delivering the managed app's webhook centrally and forwarding to instances). Considered and rejected for v1 in favor of programmatic repo webhooks; the portal stays a stateless token minter. See §16 for the comparison.
- GitLab/Bitbucket/Azure DevOps parity. The polling tier covers them credential-wise; their webhook tiers are follow-ups.
4. Design overview
flowchart LR
subgraph GitHub
R[Repo] -- "push event" --> W[repo webhook<br/>created via API]
end
subgraph Instance
W -- "HMAC-verified POST" --> E["/api/w/:ws/github_app/webhook"]
P[poller<br/>git ls-remote / schedule] --> REC
E --> REC[reconcile:<br/>ref match? sha moved?<br/>debounce]
REC --> J[pull job<br/>existing hub script, pull:true]
J --> WS[(workspace)]
end
One principle drives the security model: webhooks and polls are hints, the
pull is authoritative. A trigger never carries content; it only causes the
instance to compare the remote HEAD against last_synced_sha using its own
credentials and enqueue the existing pull job if the branch moved. A forged or
replayed trigger can only cause a cheap no-op reconcile.
5. Trigger tier 1 — webhooks
5.1 Managed app (github.com, GHE Cloud orgs)
A GitHub App's own webhook URL is fixed app-wide (it points at the portal), so
per-instance delivery uses repository webhooks created dynamically with the
installation token (POST /repos/{owner}/{repo}/hooks). Requires adding the
Repository webhooks: read & write permission to windmill-sync-helper
(see §9).
Flow when a repo is connected (or auto-pull is enabled on an existing repo):
- Instance generates a per-repo secret, stored in the repo's git-sync settings.
- Instance creates the webhook via API: events
["push"](later"pull_request", §11), URL{base_url}/api/w/{workspace}/github_app/webhook(host-aware, so managed and self-managed/GHES apps use the same per-workspace receiver), secret set. - GitHub immediately delivers a
pingevent. If the ping is not received within ~10 s, the instance deletes the hook and falls back to polling, surfacing "instance not reachable from GitHub — using polling (interval Xm)" in the UI. This doubles as an automatic reachability test; no guessing about firewalls. - Incoming deliveries are verified with
X-Hub-Signature-256against the stored secret (reuse the HMAC code fromhttp_trigger_auth.rs), then handed to the reconciler (§7).
Webhook lifecycle: deleted when the repo is disconnected or auto-pull disabled
(reuse the delete pattern from workspace_integrations.rs); recreated on
settings change; orphan hooks are detectable via GET /repos/.../hooks filtered
by our URL prefix.
5.2 Self-managed app (GHES, *.ghe.com data residency)
The customer owns the app, so the app-level webhook can point directly at the instance — no repo hooks needed, one webhook covers all installed repos:
GhesAppSettingsgains a "Webhook secret" field; the setup checklist changes from "Uncheck Active under Webhook" to "set webhook URL to{base_url}/api/w/{workspace}/github_app/webhook(the per-workspace receiver), subscribe to Push events, paste this generated secret".- That per-workspace endpoint verifies HMAC with the configured secret and
routes by repo full name + installation to matching workspaces (the routing
data is in
workspace_settings). - GHES typically shares a network with the instance, so this works air-gapped — the hardest github.com case is the easiest GHES case.
Polish (optional, recommended): replace the 8-step manual app-creation checklist with the GitHub App Manifest flow (supported on GHES), which pre-configures permissions, events, and webhook URL/secret in one click and eliminates checklist drift.
6. Trigger tier 2 — polling fallback
For private instances, plain token/SSH credentials (no app), and permission-not-yet-approved installs:
- Per auto-pull-enabled repo, on a configurable interval (default 60 s,
surfaced in settings), run
git ls-remote <repo> <tracked refs>and compare againstlast_synced_shaper ref.ls-remoteis a single cheap round-trip; no clone. - Implemented as an internal scheduled task keyed by repo settings id (not a user-visible schedule), running on workers like other background jobs.
- Polling is also the safety net under webhooks (missed deliveries, GitHub outages): when a webhook is active, the poll interval relaxes (e.g. 10 min) rather than turning off. This is the ArgoCD model: poll for correctness, webhook for latency.
7. Reconcile and pull semantics
Routing — an event/poll result is (repo, ref, head_sha, sender):
- Tracked branch (basic sync): ref equals the repo settings branch → pull into the owning workspace.
- Promotion target branch: ref equals a repo's promotion target → pull with
use_promotion_overrides: true(resolvespromotionOverridesfromwmill.yaml, mirroring CLI--promotion). - Fork branches
wm-fork/<parent-branch>/<fork-name>: parse the branch name, route to the fork workspace if it exists; ignore (log) if not. - Anything else: ignore.
- Matching keys on repo +
workspace_id— neverbase_url(avoids the known internal-vs-public URL mismatch that breaks CLI workspace matching). - One repo may match several workspaces (team partitioning with different path
filters): fan out, each workspace pulls with its own filters;
wmill.yamlin the repo stays authoritative for include/exclude.
Loop prevention (pull → deploys → deployment callback → commit → push event):
- Skip events whose sender is the app bot (
windmill-sync-helper[bot]/ the GHES app's bot) or whose head commit message carries the[WM]prefix. - Compare
head_shatolast_synced_shabefore enqueuing; the pull job records the synced sha on success. - Deploys applied by the pull job are tagged so the deployment callback can skip the no-op commit (belt and braces — the diff should be empty anyway).
Concurrency and debounce:
- At most one pull job per (repo, workspace) at a time; rapid pushes coalesce (same pattern as the 5 s deploy-callback batching, keyed on repo).
- Pulls and in-flight Windmill → repo commits on the same branch serialize on the same key to avoid races.
Execution: reuse the existing pull path of the hub init script for v1. Two known fragilities to fix as part of making this an unattended automation:
- The hub script pins a
windmill-cliversion; pin lag has caused 422s against newer backends. The automated path must pin the CLI to the instance's own version (or the pull logic moves into the backend natively as a v2). - Failures must be visible: pull jobs appear in the runs list like deployment
callbacks today (
/runs?job_kinds=deploymentcallbacksequivalent), plus a per-repo "last sync" status chip in git-sync settings and the workspace error handler firing on repeated failures.
8. Settings and schema
Per-repo (GitRepositorySettings, workspace_settings.git_sync):
auto_pull: Option<AutoPullSettings>
enabled: bool
mode: "webhook" | "polling" | "auto" (auto = try webhook, fall back)
poll_interval_s: Option<u32> (default 60; 600 when webhook active)
webhook_id: Option<i64> (GitHub hook id, managed app)
webhook_secret: Option<String> (encrypted; per-repo, managed app)
last_synced_sha: Option<map<ref, sha>> (per tracked ref)
last_pull_status: Option<...> (sha, time, job id, error)
Instance-level (GHES app config): webhook_secret alongside app id/private key.
No new tables; everything extends existing JSONB settings. A migration is only
needed if we decide last_synced_sha/status churn doesn't belong in
workspace_settings (alternative: small git_sync_pull_state table keyed by
workspace + repo path — decide at implementation review).
New instance endpoints (EE):
POST /api/w/{workspace}/github_app/webhook— per-workspace hook receiver for managed and self-managed apps (host-aware), HMAC-verified, returns 202.- Both are unauthenticated-but-verified endpoints; rate-limited; bodies are treated as hints only (§4).
9. GitHub App permission migration
New permissions needed, bundled into one update (each update re-prompts every existing installation's org admin):
| Permission | Used for | Phase |
|---|---|---|
| Repository webhooks: write | dynamic repo hook create/delete | 1 |
| Pull requests: write | instance-side PR creation (promotion/forks) | 2 |
| Checks: write | PR diff preview checks | 3 |
App-only features and their fallbacks: instant webhook sync (falls back to
polling), in-app PR creation for deploy branches (fall back to the
open-pr-on-commit / open-pr-on-fork-commit workflows; the toggles are
hidden for token repos and a set-but-inert toggle logs a warning), and the PR
diff comment/check + deploy status check (no fallback: they need the Checks
API and pull_request webhook deliveries). Token/PAT repositories keep the
full pull direction via polling.
Rollout behavior: until an org approves, webhook creation fails with a distinguishable error → the instance shows "approval pending" and stays on polling. Nothing breaks; latency is the only cost. The self-managed (GHES) checklist/manifest gains the same permissions — no central approval involved.
10. Setup UX
The git-sync wizard (DetectionFlow / repo card) gains a third guided
direction after the existing "test connection" and "initialize repo" steps:
- Toggle: "Automatically deploy changes from Git" (per repo).
- On enable: try webhook (ping self-test) → show resulting mode and latency ("instant via webhook" / "polling every 60s — instance not reachable from GitHub"), with a re-test button.
- The success modal's current "set up GitHub Actions" doc link becomes the advanced/CI path, not the default instruction.
- Promotion-mode repos additionally show "PRs will be opened by Windmill" once §11 lands.
11. Promotion & forks parity: PR creation moves instance-side
Both PR-producing branches are pushed by Windmill's own deploys — wm_deploy/**
(promotion) and wm-fork/** (fork deploys) — so opening the PR moves into the
deploy pipeline itself, per repo toggle: the push job carries a
__git_sync_open_pr marker and its completion hook opens (or reopens) the PR
via the installation token once the push has landed. Outbound-only, so it works
for every connectivity profile — no webhook required.
- Promotion:
promotion_open_prson the promotion repo ("Open a pull request for each deploy branch"). - Forks:
fork_open_prson the parent's sync repo ("Open a pull request when a fork deploys"), read by the fork's deploy callback — parent-owned likesync_forks, zero fork-side setup.
Both default on for newly configured repos and off in storage (upgrades don't
change behavior). ensure_pull_request treats an existing PR as success, so the
documented open-pr-on-commit / open-pr-on-fork-commit Actions can stay
installed for custom titles/CI without duplicate PRs. The merge side is already
covered by §7 routing (push event on the target branch).
12. Later: PR diff preview checks
With pull_request events (webhook tier) and checks: write: on PR
opened/synchronized, run the existing dry_run: true pull and post the diff
summary as a check run. This replicates the CI dry-run preview with zero
customer CI and completes the "Cloudflare Pages" experience: install app →
merges deploy, PRs show a Windmill diff. The commit-level "Deploying… →
Deployed" status (the other half of the Cloudflare feel) is Phase 6.
13. Coverage vs documented setups
| Documented setup | Covered by |
|---|---|
| Basic git sync (workspace ↔ branch) | §5/§6 trigger + existing pull job |
| Multi-repo primary/secondary | per-repo toggle; secondaries stay push-only |
| Promotion mode, single instance | §7 promotion routing + §11 PR creation |
| Promotion mode, cross-instance | each instance triggers independently — strictly better than CI (no cross-instance tokens/URLs) |
Workspace forks (wm-fork/**) |
§7 fork routing + Phase 5 (parent-level sync_forks) |
| PR dry-run preview | §12 (optional follow-up) |
| Local dev, git as entry point | ordinary push events; nothing special |
| Customers with real CI gates | unchanged; pull triggers are idempotent and coexist |
14. Migration of existing users
The defining advantage of the repo-webhook approach: existing git-sync users
already have the managed app installed. Migration is "grant one incremental
permission," not "reconnect." The windmill → repo direction uses the app's
Contents: write grant, which the new Repository webhooks: write permission
does not touch — so nothing existing breaks whether or not a user migrates.
Existing installs are enumerable from workspace_settings.git_app_installations
(each GitInstallation carries installation_id, account_id, and
github_base_url: None = managed github.com app, Some = self-managed/GHES).
That split is the migration cohort boundary.
Cohort A — managed app, already installed (the majority). Only gap is the
Repository webhooks: write grant. GitHub keeps the old grants working while the
new permission sits pending approval, so migration is lazy and never-blocking:
- Ship the app permission update +
auto_pullsettings defaulting off. Zero observable change until a user opts in. - Each managed-app repo gets an "Automatically deploy changes from Git" toggle in the existing settings UI.
- On enable, the instance attempts webhook creation:
- 403 (approval pending) → surface a deep link to the org's app installation page to approve the new permission, and start polling immediately so auto-pull works right now. The user is never blocked on a GitHub org admin.
- Success → ping self-test → webhook mode (or polling if unreachable).
- The instance does not need to hear the approval. The
installation/new_permissions_acceptedevent goes to the app webhook (the portal), not the instance — irrelevant here. The instance just retries webhook creation on its next poll cycle and silently upgrades polling → webhook once the grant lands. No portal state, no callback plumbing.
Cohort B — no app (plain git_repository resource, token/SSH). Nothing to
approve; flipping the toggle goes straight to polling with existing credentials.
Optional upsell: "install the Windmill GitHub App for instant sync."
Cohort C — self-managed / GHES (github_base_url: Some). Customer owns the
app, so there is no central approval. App-level webhooks need no extra
permission — migration is one documented step: paste the instance webhook URL +
generated secret into their app settings (the GhesAppSettings checklist flips
"leave webhook inactive" → "set this URL + secret"). Usually works air-gapped.
Cross-cutting:
- Opt-in, not auto-flipped. Do not silently enable pull on existing repos: some are backup/secondary push-only targets, or hold content the owner does not want deployed back. Surface a prominent "New: deploy automatically from Git" prompt instead.
- Existing CI coexists. A user already running
wmill sync pushvia GH Action keeps it; sha-idempotent triggers make double-firing harmless. Optional cleanup: detect the workflow file via the Contents API and offer one-click removal once webhook pull is confirmed. - The unavoidable cost. The permission bump nags every managed-app installation (even users who never enable auto-pull) with a "requesting updated permissions" prompt until approved or dismissed. No way around it for a single shared app. Mitigation is clear permission-purpose copy; the nag is cosmetic and does not break existing sync.
- Capability gating precedent.
GitRepositorySettings::is_script_meets_min_versionalready gates behavior on the pinned hub-script version — a "this install's app grant supports webhooks" capability flag fits the same pattern.
15. Implementation plan
Staged so each phase is independently shippable and reviewable. Phase 1 alone delivers automatic pull for every customer; webhooks are a latency upgrade.
Phase 1 — polling + reconcile + settings + UX (no app/permission changes)
Backend (EE):
GitRepositorySettings(backend/windmill-common/src/workspaces.rs): addauto_pull: Option<AutoPullSettings>(§8). Non-breaking JSONB addition.- Reconcile + enqueue: new function mirroring
push_git_sync_job(windmill-git-sync/src/git_sync_ee.rs:896) — given(repo, ref, head_sha), resolve matching workspace(s), compare againstlast_synced_sha, and enqueue the pull job with the same 5 s debounce machinery. The pull job for v1 is the existinggitInitRepohub script run withpull: true(mirror the payloadPullWorkspaceModal.sveltealready sends); record the synced sha on success. - Poller: register a periodic task in
backend/src/monitor.rs(alongside the othertokio::time::intervalloops) that, per auto-pull-enabled repo, runsgit ls-remotefor the tracked refs and feeds changes into the reconciler. - Pin the hub-script CLI to the instance version for the unattended path
(UI git-sync runs a version-pinned
windmill-cli; pin lag has caused 422s against newer backends).
Frontend:
GitSyncRepositoryCard.svelte: per-repo "Automatically deploy changes from Git" toggle + last-sync status chip; polling interval input.- Demote the
GitSyncSuccessModal.svelte"set up GitHub Actions" link to an advanced/CI option.
Validation: pull jobs visible in the runs list; error handler fires on repeated
failure. Tests: reconcile routing + sha-compare + loop-prevention (sender/[WM]).
Phase 2 — webhooks (managed app + GHES)
Ops (precedes code): update windmill-sync-helper to request
Repository webhooks: write (bundle Pull requests: write + Checks: write
now too, to avoid a second nag for phases 3–4). Update permission-purpose copy.
Backend (EE):
- Receiver endpoint (§8):
POST /api/w/{workspace}/github_app/webhook(per-workspace, managed and self-managed), HMAC-verified — reuseX-Hub-Signature-256validation fromwindmill-trigger-http/src/http_trigger_auth.rs; routes added togit_sync_ee.rsworkspaced_service/global_service. - Webhook create/delete via installation token — reuse the REST pattern from
windmill-native-triggers/src/github/external.rsand the delete pattern fromworkspace_integrations.rs. Ping self-test with reachability fallback (§5.1). - Lazy approval retry + 403 detection feeding the migration UX (§14 cohort A).
Frontend:
- Toggle now reports resulting mode/latency + re-test button; approval-pending deep link.
GhesAppSettings.svelte: webhook-secret field + updated checklist (and, optional, the App Manifest one-click flow, §5.2).
Phase 3 — PR creation instance-side (promotion + forks, toggled)
- The deploy's push job carries a
__git_sync_open_prmarker when the repo opted in (promotion_open_prson the promotion repo; parent-levelfork_open_prsfor fork deploys); the job-completion hook (maybe_open_git_sync_deploy_prinresult_processor.rs) derives the pushed branch (fork branch wins, else thewm_deploy/**formula) and callsensure_pull_request. Outbound with the installation token, so no webhook reachability is needed; app-backed repos only. - No-op pushes skip PR creation: the push script reports
pushed: falsewhen nothing was committed (e.g. the deploy was itself caused by an auto-pull, so the workspace already matches the repo), and the hook returns early — a PR the user closed isn't recreated by the sync loop. Results without the flag (older script pins) keep ensuring the PR. - Fork-branch routing edge cases (§7) hardened here.
Phase 4 — PR diff preview checks (optional)
- Subscribe
pull_requestevents; on open/synchronize run the existingdry_run: truepull and post the diff as a check run (checks: write). - The same completion hook maintains ONE managed comment on the PR
(Cloudflare deploy-preview style: workspace, status, commit, collapsible
change list), upserted per synchronize via a hidden
<!-- windmill-diff -->marker so reviewers see the current diff without opening the Checks tab.
Phase 5 — fork sync, configured at the parent (replaces the push-on-merge-to-forks GitHub Action) — implemented
Scope rule: Windmill absorbs automation for events it originates or
consumes; Actions remain for custom CI. Of the documented CI/CD workflows:
push-on-merge (repo → prod workspace) is phases 1–2 auto-pull;
push-on-merge-to-forks (fork branch → fork workspace) is this phase; and the
PR-opening workflows (open-pr-on-commit for wm_deploy/**,
open-pr-on-fork-commit for wm-fork/**) react to branches Windmill itself
pushes, so they move into the deploy pipeline as per-repo toggles (§11 / Phase
3) and the Actions become optional alternatives. (A fork_pull_sync "fan the
tracked branch out to every fork" draft was dropped: a fork syncs its own
wm-fork/** branch, never main directly, and blind fan-out would clobber a
dev workspace's local work.)
The premise. create_workspace_fork copies the parent's resources (so the
git_repository resource lands in the fork) and, via clone_workspace_data →
update_workspace_settings, the parent's git_app_installations and git_sync
(keeping the first sync-mode repo — WIN-1559). So a fork already pushes to its
wm-fork/<tracked>/<id> branch on deploy. What it must not inherit is the
auto_pull block: it carries the parent's webhook_id/webhook_secret (the
repo webhook is parent-owned), so update_workspace_settings strips it. The
server also rejects parent-only settings on a fork workspace (enabled
auto_pull, promotion mode, fork_open_prs) — a fork's deploys always target
its wm-fork/** branch, so none of them could take effect there, and fork sync
is exclusively parent-managed.
Design: one parent-level toggle, auto_pull.sync_forks (default on when
auto-pull is enabled). Matches the Action model (configured once at the repo,
fires for every wm-fork/** branch) and needs zero per-fork setup — no fork
webhook, resource, or config; applies to current and future forks.
How a fork branch change reaches the fork workspace (both delivery paths):
- Webhook: the push to a fork branch (
wm-fork/<base>/<suffix>, or a dev workspace's label branch) lands on the parent's webhook.handle_github_git_sync_eventrequires a[WM]-less head commit (a fork's own deploy must not pull itself back) andauto_pull.enabled && sync_forks, then callsreconcile_fork_branch_pull; branches that resolve to no live child no-op. - Polling: the parent's poll tick also lists every
wm-fork/<tracked>/*head plus its dev-workspace children's label branches in one extra call — agit ls-remotepattern (+ explicit refs) for token repos,git/matching-refs(+ per-label head lookups) for app-backed — and reconciles each (poll_git_fork_branchesinmonitor.rs).
Dev workspaces sync with their environment-label branch. A dev workspace's
branch is its label verbatim (dev/staging — the classic env-branch layout,
matching the documented push-on-merge-staging Action), not the namespaced
wm-fork/** form. The label is set at create/attach time and immutable
afterwards (the branch is keyed on it; the old set-label endpoint was removed),
defaulting to dev. The backend passes it with every deploy job
(dev_workspace_label arg → hub script → CLI --dev-workspace-label), the PR
completion hook derives the same branch, and the CLI refuses to deploy when the
label branch equals the checked-out tracked branch (which would otherwise
commit fork content straight to it).
reconcile_fork_branch_pull (windmill-git-sync EE) is the shared routing core:
resolve the branch to a live descendant of this workspace (recursive
parent_workspace_id walk + NOT deleted, so a crafted branch name can't
route a pull into an unrelated workspace) — the wm-fork/<base>/<suffix> form
via parse_fork_branch (windmill-common), or an environment-label branch via
a dev-workspace label lookup — then load the fork's own repo entry and run the
shared reconcile_and_enqueue_pull with the fork's per-ref dedup state and a
clone_ref override so the pull job clones the fork branch instead of the
resource's tracked branch. Descendants (not just direct children) because
forks of a dev workspace also sync through the root's webhook/poller —
only the root can hold auto-pull config. A fork-of-dev roots its wm-fork/**
branch on the dev's label branch and its PR merges back into it (the backend
passes parent_dev_workspace_label with the deploy; fork_open_prs is
resolved at the root ancestor).
State lives with the fork: its repo entry carries a server-written status-only
auto_pull blob (last_synced_sha keyed by the fork branch, last_pull_status;
enabled stays false). persist_auto_pull_state creates that blob when missing.
The fork's repo card shows a read-only "managed in the parent workspace" line
with the fork branch name plus the last pull status; the enable/disable control
exists only on the parent's card.
Perms: the toggle is parent-workspace admin (whoever edits the parent's
git_sync) — the same bar as "who set up the repo secret + workflow." No
per-fork authorization; matches the Action ergonomics.
Non-goals for v1: per-fork opt-out on the parent (default is all forks; add an
exclusion list later if asked); per-fork include/exclude filters (the pull
applies the repo's wmill.yaml like any pull).
Phase 6 — live deploy status check on the commit (Cloudflare-style) — implemented
Replicate the Cloudflare Pages deploy status: a check run that appears in the
commit/PR checks strip, starting in_progress ("Deploying…") and flipping to
completed/success ("Deployed"). This is not a GitHub Action — it's posted
via the Checks API, so it reuses the Phase 4 machinery
(create_check_run/update_check_run) and the checks: write grant already
requested. No new permission, no customer CI. The check lands on the head commit
of the tracked branch — exactly where Cloudflare's "Deployed to production" sits.
Today the deploy path posts nothing back: create_check_run
("status": "in_progress") and update_check_run
("status": "completed" + conclusion + output) already exist and are used for
the PR dry-run diff, but the real deploy pull (tracked-branch push/merge)
doesn't create one. Phase 6 runs that same two-step on the deploy path.
Flow:
- On a tracked-branch push/poll that triggers a deploy pull,
create_check_runon the head sha: name "Windmill" (vs "Windmill diff" for PR checks),in_progress, title "Deploying…", with adetails_urlto the Windmill run / workspace. Keep the returnedcheck_run_id. - Thread
check_run_id+repo_urlinto the pull job — same marker channel as the PR dry-run (add a__git_sync_deploy_checkmarker distinct from the PR-check marker so the completion hook knows which kind). - On completion (generalize
maybe_post_git_sync_pr_checkinresult_processor.rs),update_check_run→completed, conclusionsuccess("Deployed N changes to<workspace>" / "In sync, no changes") orfailurewith the error summary.
Backend (EE) touch points:
create_check_run: add a caller-supplied name + adetails_urlparam (small signature change; PR path keeps "Windmill diff").- Deploy trigger (tracked-branch case in
handle_github_git_sync_event, plus the poller reconcile): best-effort create thein_progresscheck and carry the id into the job payload. App-backed only; never block the deploy on it. - Completion hook: handle the deploy-check marker alongside the PR-check marker.
Implementation note. The
in_progresscheck is created insidereconcile_and_enqueue_pull(one place covers both the webhook and poller paths), best-effort and app-backed-only, then threaded to the pull job as a__git_sync_deploy_checkmarker.create_check_rungainedname+details_url+output_title(the PR path keeps"Windmill diff", no details/output). The completion hookmaybe_post_git_sync_pr_checkwas generalized tomaybe_post_git_sync_check, reading either marker. If the pull can't even be enqueued, the in-progress check is closed as failed so it doesn't hang.
Gating / edges: app-backed only (needs the installation token + checks: write;
PAT/polling repos skip silently); skip self-caused/no-op pulls ([WM]/bot,
sha unchanged) so it doesn't post a check for Windmill's own commits; one check
per (repo, head_sha, workspace) — when several workspaces pull the same commit,
name each with its workspace to disambiguate.
Optional richer variant — GitHub Deployments / Environments. Instead of (or
alongside) the check run, create a Deployment (POST /repos/.../deployments) +
status (POST /repos/.../deployments/{id}/statuses) so the deploy shows in the
repo's Environments timeline ("Production → Deployed"). Needs
deployments: write — a new grant and another approval nag — so keep it
opt-in / later. The check-run version is the cheap default and matches the visual
Cloudflare parity without a new permission.
16. Alternatives considered
Portal as webhook proxy (the rejected "option 2"). Subscribe the managed app
to push events — delivered centrally to stats.windmill.dev — and forward them
to instances. Its only genuine advantages are (a) zero-friction enablement for
existing installs, since app-level events need no new permission (no per-org
approval, unlike repo-webhook creation), and (b) central delivery observability.
Against that: it does not improve reachability (the portal forwards to the
same instance URL GitHub would hit, so private instances need polling either
way); app-level events fire for every push to every installed repo, org-wide, so
portal cost scales with customers' total push volume rather than synced repos;
the portal becomes stateful (installation→instance registry) and
availability-coupled, losing its current stateless-token-minter property; full
push payloads (commit messages, author emails) transit Windmill infrastructure;
and the instance cannot verify portal signatures, whereas repo webhooks get
per-repo HMAC. The one advantage that stings — the permission-bump nag — is
mitigated contextually (approval shown in settings on enable) and covered by
polling until approved. A narrow portal variant (portal stores events; private
instances poll /github_sync/events with their existing installation JWT) is
the only thing that would lower latency for unreachable instances; parked unless
demanded.
17. Open questions
- Does
last_synced_sha/pull-status churn stay inworkspace_settingsJSONB or move to a dedicated table? (Write frequency vs settings-blob contention.) - Exact debounce window for pull coalescing (reuse 5 s like deploy callbacks, or longer since clones are heavier?).
- Should phase 1 polling default to on for newly connected repos, or strictly opt-in? (Opt-in proposed; revisit after adoption data.)
- GHE Cloud orgs with IP allowlists: confirm hook deliveries to customer instances aren't filtered; the ping self-test catches it operationally either way.
- v2: move pull execution from the hub script into the backend natively (removes CLI pinning and hub round-trip entirely)?