mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-21 16:02:28 +00:00
e052d5ee4598332a6491b653d70fc82a08642d61
1052 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 hugocasa
|
2ce21c9ef8 |
feat(git-sync): enable per-item promotion mode on dev workspaces (#10205)
* feat(git-sync): enable per-item promotion mode on dev workspaces Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: keep unrelated git-sync Alert copy at its original wrapping Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): fall back to parent_path on empty deploy path + bump ee ref computeGitSyncDeployBranch used ?? so a backend-serialized empty path (rename out of the repo filter) skipped the deploy branch and could commit to the tracked base; use || to fall back to parent_path like the backend. Bumps ee-repo-ref for the single-object promotion_open_prs fix (windmill-ee-private#679). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): route dev-promotion non-branchable objects off the tracked base user/group objects (and any unresolvable ref) returned null in promotion mode, so a dev-workspace deploy pushed them straight to the parent's tracked branch. Fall back to the dev's env-label branch instead; the backend opens no PR for them (isolated, not promoted). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(git-sync): dev-workspace promotion via a toggle on the inherited repo A dev workspace reuses the single repo it inherited from prod: a 'Promote to prod via Git' toggle flips it between sync mode (deploys to the dev branch) and promotion mode (per-item wm_deploy/** PRs to prod), with a per-item/per-folder sub-toggle. Removes the redundant separate-promotion-repo setup for dev workspaces. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(git-sync): dev-promotion regression test + widen git_sync_e2e path filter Adds a CLI integration case covering dev-workspace promotion (script -> wm_deploy branch; user/group -> env-label branch, main never touched). Widens the git-sync-test.yml relevance filter to the deploy-branch derivation, git-sync guard, and CLI git-deploy files so the e2e suite runs on PRs like this one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): gate dev promotion toggle on EE, fix card mode + workflow path filters Codex review: (1) show the dev promotion toggle only under an active EE license and revert the optimistic save if the backend rejects it; (2) derive the dev card's display mode from use_individual_branch so promotion copy shows in promotion mode; (3) mirror the new relevance paths into the workflow's top-level push/pull_request filters so it actually triggers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): only use the single-card dev promotion UX when the dev has one repo Codex review: an attached dev workspace keeps its own repositories rather than inheriting prod's. Gating the single-card + toggle + hidden-secondaries UX on repositories.length <= 1 makes a multi-repo attached dev fall back to the normal layout, so no active repo is hidden and an unrelated repo isn't presented as prod's promotion target. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): runtime EE-plan gate for promotion mode, consistent with auto-pull/PR Codex review: promotion mode only had the CE compile rejection, while auto-pull and PR creation runtime-gate on the active plan (check_git_sync_ee_license). Add check_promotion_license and call it from both edit_git_sync_config and edit_git_sync_repository, plus the matching CE rejection on edit_git_sync_config so the two endpoints are symmetric. Promotion is now gated like every other git-sync EE setting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): dev promotion must reuse the parent workspace's repository Codex review: repository count doesn't prove a dev inherited prod's repo — an attached dev keeps its own. check_dev_promotion_targets_parent_repo resolves the promotion repo's URL and rejects enabling promotion unless it matches one the parent (prod) tracks, so branches/PRs can't target an unrelated repository. Called from both git-sync edit endpoints. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): dev promotion save-time check uses shared parent-repo matcher (url+branch) Delegates to windmill_common::git_sync_ee::dev_promotion_target_matches_parent so the settings gate and the deploy-time safety net share one url+branch identity check. Bumps ee-repo-ref for the EE deploy-time enforcement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for private resolve_repo_url_and_branch Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for promotion-target matcher authz doc Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(git-sync): bump hub scripts to gitsync-cli versions, fix promotion tooltips Point LATEST_GIT_SYNC_SCRIPT_PATH (28790 -> 28796) and GIT_SYNC_PULL_SCRIPT_PATH / gitInitRepo (28789 -> 28795) at the hub versions pinning windmill-cli@1.763.1-gitsync.0, which carries the dev-workspace promotion routing. Slugs unchanged, so the GitHub-App token check and hub script cache are unaffected. Tooltips: enabling promotion pushes a PR-ready wm_deploy/** branch; Windmill only opens the pull request itself when automatic pull requests are enabled. Reword both toggles to stop promising a PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): dev promotion mirrors to the env-label branch, PR toggles exclusive by branch type Bump ee-repo-ref for the dispatcher changes: a promotion dev's deploys now also push to its env-label branch (one extra mirror job per batch, users/groups mirror-only), and `fork_open_prs` no longer applies to a dev in promotion mode where `promotion_open_prs` governs. Frontend: the fork-PR toggle tooltip states its actual coverage (wm-fork/** and the dev branch of a dev workspace) and that a promotion dev's own pull request toggle takes over for wm_deploy/** branches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): reject dev promotion on pre-28796 pinned sync scripts An older pinned sync script bundles a CLI that force-disables per-item branches on every fork, so enabling promotion on a dev workspace with such a pin would silently keep deploying to the env-label branch. Both git-sync edit endpoints now reject the combination with an actionable error; the EE dispatchers (via ee-repo-ref bump) demote inherited configs to promotion-off semantics so markers, branch keys and the mirror match the branch the CLI actually pushes. Roots and auto-managed repositories are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): serialize dev promotion toggle saves The promotion and per-folder toggles persist immediately via whole-repo saves; leaving them interactive while one is pending lets rapid flips race, and the earlier save (enabling runs extra backend checks) can commit last, silently reversing the state the UI shows. Both toggles now disable while a save is in flight. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): lock auto-PR toggle during promotion save, rename-out branch routing Frontend: the automatic-PR toggle is revealed by the promotion toggle's in-flight save; an edit made mid-save was absorbed into the saved baseline without reaching the backend. It now disables during that save. EE (ee-repo-ref bump): dispatcher debounce/concurrency keys and PR markers follow the CLI's parent_path fallback for rename-out items, so their wm_deploy/** branches debounce per-branch and open their PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(git-sync): condense comments to durable constraints Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 8bf73f803158bcbf7b8d55a36f4a1ebfcc1bbcd9 This commit updates the EE repository reference after PR #679 was merged in windmill-ee-private. Previous ee-repo-ref: c2cd718cb53d234f909f485bd7cd43ed9605ffd1 New ee-repo-ref: 8bf73f803158bcbf7b8d55a36f4a1ebfcc1bbcd9 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
 Diego Imbert
|
7fb8a2e390 |
fix(parsers): keep s3 asset path suffix verbatim to preserve storage distinction (#10241)
* fix(parsers): keep s3 asset path suffix verbatim to preserve storage distinction Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01An2pTqSmqJd2XwnagvX4kM * package json * fix(pipelines): preserve named storage in generated TS/Python S3 URIs The TS/Python templates emitted `s3:///${s3Key(path)}`, stripping the leading slash and pinning the URI to default storage. For a named-storage asset path (`secondary/key`) that produced `s3:///secondary/key`, which resolves to the default storage with key `secondary/key`, dropping the named-storage dependency and reading/writing the wrong object. Emit the path verbatim after `s3://` (matching the DuckDB template) so a named-storage input/output keeps its storage; identical to the previous output for default-storage paths. Removes the now-unused `s3Key` helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(cli): align bun.lock parser versions with frontend The PR bumped windmill-parser-wasm-asset (1.749.0→1.753.0) and windmill-parser-wasm-regex (1.692.0→1.764.0) in package.json and the npm package-lock.json for both cli and frontend, but cli/bun.lock was left pinned to the old versions. Sync it so the CLI's wasm asset parser (used by localGraph inference) matches the frontend and deploy-time parser. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
 Ruben Fiszel
|
68debab877 |
feat(triggers): add AMQP (RabbitMQ) trigger via lapin (#10230)
* feat(triggers): add AMQP (RabbitMQ) trigger using the lapin library Fixes WIN-2214 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(triggers): defer AMQP cross-workspace deploy pending utils-internal publish Revert the amqp_trigger additions to the shared windmill-utils-internal TriggerDeployKind and the frontend cross-workspace deploy adapter: the frontend installs the published npm package, which lacks the new kind until a release is cut. AMQP create/edit/delete/list/sync/capture are unaffected (they use local types); only cross-workspace deploy/merge of AMQP triggers waits on the package bump. Also document the at-most-once ack in the consumer loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): address AMQP review — at-least-once ack, workspace cascade, contracts - ack AMQP deliveries only after successful dispatch; nack+requeue on failure - add ON DELETE CASCADE workspace FK so amqp_trigger rows are cleaned on workspace deletion (and the listener stops) - fix the /amqp_triggers/test OpenAPI body and add amqp_trigger to WorkspaceDiffRow.kind - register AMQP in the generated workspace trigger tool (create_trigger) - drop banned $bindable defaults on optional props in the config section - add build_uri unit tests (encoding, ports, vhost) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): stop AMQP poison-message loop and reconnect on transient drops Chaos testing against a live RabbitMQ broker showed the previous nack(requeue) + immediate re-poll spun a tight redelivery loop (~1000 critical-error reports/sec) on a poison message, and any connection blip permanently disabled the trigger (lapin has no built-in reconnect). - on dispatch failure: nack+requeue then stop consuming; the listener framework re-lists the trigger after its ping goes stale (~15s), backing redelivery off to that cadence instead of a tight loop (verified: rate dropped from ~1000/s to ~1 per ~26s, message preserved) - on connection/stream error: stop and let the framework reconnect instead of disabling; persistent failures are still disabled via get_consumer (verified: a forced connection close now auto-reconnects and resumes) - finish the AI create-trigger action wiring for AMQP: add amqp to CreatedResourceTriggerKind, the action-card registry, and the drawer registry so the result card renders and its "Open" action works Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP frontend registries and defer merge rows - add amqp to capturableTriggerTypes (so AmqpCapture mounts), the Runs jobTriggerKinds filter, and CLOUD_DISABLED_TRIGGER_TYPES - wire AMQP into global AI chat mode: TRIGGER_KINDS, the request union, writeTriggerSchema, triggerServices, and the draft adapter - stop emitting actionable AMQP fork-comparison rows (revert amqp_trigger from TRIGGER_OR_SCHEDULE_TABLES) since cross-workspace deploy is deferred until windmill-utils-internal is published — avoids a deploy that fails with "Unknown kind: amqp_trigger" - use design-system TextInput instead of raw <input> in the config section Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP session/draft registries and constrain prefetch - add amqp to the session-deploy, draft-compare, preview-router, and copilot workspace-item registries so AMQP drafts/deploys/nav/path resolution work - include amqp_count in the MoveDrawer attached-trigger rename warning - replace the raw prefetch <input> with a design-system TextInput bounded to an integer 1-65535 (backend u16) and block save on invalid values Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): make AMQP disconnect/reconnect consistent with the Kafka trigger lapin, like rdkafka, has no transparent reconnect, so the AMQP listener now mirrors the Kafka trigger's explicit reconnect loop instead of relying on the framework re-list (which disabled the trigger once get_consumer failed on a sustained outage): - get_consumer returns cheaply; consume owns a (re)connect loop that retries with a 30s backoff, reports a critical error every 10 failed attempts, and reports a recovered critical error once it reconnects — never disabling the trigger on a connectivity failure - a consumer/stream error breaks out to reconnect rather than disabling - dispatch failure still nacks+requeues (at-least-once) with a short backoff to avoid a tight poison-message loop, keeping the connection alive Verified against a live RabbitMQ broker: killing the broker keeps the trigger enabled and retrying (attempt N), and restarting it auto-reconnects (logs "reconnected after N attempts") and resumes dispatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP capture registries and constrain prefetch contract - add the 'amqp' case to triggerKindToTriggerType so opening the AMQP editor from a capture button no longer throws "Unknown TriggerKind: amqp" - register AmqpIcon in CaptureTable's icon map and add an AMQP entry to the script/flow CaptureButton menu - bound the OpenAPI prefetch_count to an integer 1-65535 (matches the Rust u16) and regenerate clients/prompts - require a non-empty exchange name when the exchange binding is enabled - build_uri: fall back to "/" on a blank vhost and bracket IPv6 hosts (+ tests) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(triggers): wire AMQP into pipeline graph, git-sync, and preprocessor types - asset_graph: discover attached amqp_trigger rows and emit an AMQP TriggerEdge so AMQP triggers render (and can be opened/deleted) on the data-pipeline canvas - frontend pipeline graph: add amqp to NativeTriggerKind, the add-trigger menu, node presentation, event-trigger set, annotation keywords, and the editor/service registrations - git-sync: add the amqp_trigger include pattern (+ test) so an AMQP git-sync deployment stages only its .amqp_trigger.* file, not an unrelated same-path object - preprocessor starters: add the AMQP event to the generated TS/Python/PHP trigger event types (kind/payload/exchange/routing_key/queue_name/redelivered/ delivery_tag) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): finish AMQP pipeline/parser wiring, prefetch validation, source lists - fix a stray edit that corrupted the pre-existing MqttTriggerEditor import ($lib/... path) in PipelineTriggerEditors.svelte - reject prefetch_count = 0 server-side in validate_config (RabbitMQ treats 0 as unlimited) and defensively skip basic_qos(0) in build_consumer (covers the capture path that bypasses CRUD validation) - recognize `// on amqp` in the canonical parser (TriggerSpec::Amqp) and add amqp to the CLI non-autorun/event-trigger sets so a pipeline cascade never runs an AMQP-only node as a manual root without an event - add amqp to the preprocessor intro lists and both pipeline AI instructions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): reject zero AMQP prefetch in all paths and finish guidance lists - extract a shared validate_amqp_options used by both CRUD validate_config and build_consumer, so capture configs (which bypass CRUD validation) also reject prefetch 0 instead of silently connecting with an unlimited buffer (+ unit tests for 0/1/65535/None) - add AMQP to the main script-writing preprocessor-sources prompt and the CLI triggers-skill guidance list Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(triggers): de-duplicate AMQP prefetch comment and fix GET response text - keep the zero-prefetch rationale only on the shared validate_amqp_options doc; drop the redundant call-site comments - correct the getAmqpTrigger OpenAPI 200 description ("deleted" -> "retrieved") Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to aaa6cb89b05b76139252c64f057e53b94d12ac60 This commit updates the EE repository reference after PR #680 was merged in windmill-ee-private. Previous ee-repo-ref: 5da5fd65aca9594b2611837a52e4677b544b0380 New ee-repo-ref: aaa6cb89b05b76139252c64f057e53b94d12ac60 Automated by sync-ee-ref workflow. * chore(migrations): consolidate the four AMQP migrations into one The table and the three enum ADD VALUE statements (trigger_kind, job_trigger_kind, draft_kind) are one atomic feature. ALTER TYPE ... ADD VALUE runs inside the migration transaction on PG >= 14 (Windmill's minimum) since the amqp_trigger table doesn't reference those enum types, so they can share a single migration instead of four. Verified applying cleanly in a single transaction on a fresh DB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
fd51d40f12 | feat(pipelines): catalog declared measures and dimensions (#10190) | ||
|
Ruben Fiszel
|
ddec2abbb3 |
feat(jobs): cap total queued jobs per workspace on cloud (#10218)
* feat(jobs): cap total queued jobs per workspace on cloud A workspace could flood the queue with an unbounded number of jobs across many concurrency keys and scripts (or keyless jobs), which the per-key cap from #10197 does not bound. Add a companion instance-wide ceiling on a workspace's total queued jobs. check_workspace_queue_cap rejects a push once the workspace has WORKSPACE_MAX_QUEUED_JOBS (default 20000, superadmin-configurable, 0 to disable) jobs queued, cloud-only and runtime-gated on CLOUD_HOSTED like the per-key cap. It runs on every push, so it applies even to premium workspaces and catches parallel for-loop floods. Jobs already queued still drain; only new pushes past the ceiling are rejected, so an in-flight flow only fails to push further work while at the ceiling. The setting loader self-gates on CLOUD_HOSTED so it is never loaded off cloud, from initial load or a settings-change reload. The depth count is bounded by the cap via LIMIT so a runaway backlog never costs an unbounded scan on the push path. * docs(jobs): note the workspace cap is a soft ceiling and the depth helper is count-only Records the two review points as constraints: the cap does not serialize admission (a soft ceiling by design, like the per-key cap), and workspace_queue_depth is pub only for the test, returns a count not job data, and leaves authorization to the caller. |
||
|
hugocasa
|
b070f56c5e |
feat: detect server-handled git-sync so CLI picks git push vs wmill sync push (#10201)
* feat: detect server-handled git-sync so CLI picks git push vs wmill sync push
Add a non-admin GET /w/{w}/workspaces/git_sync_deploy_mode endpoint returning
{configured, deploy_on_push}, so any workspace member (not just admins, who
alone can read get_settings) can tell whether pushing to the git remote deploys
via server-side auto-pull. Surface it through `wmill gitsync-settings status`
and align the deploy guidance/skills to prefer git push when the repo deploys on
push, falling back to `wmill sync push` otherwise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address review — clean JSON output, warn on git_sync parse failure
- gitsync-settings status --json-output now uses console.log so the JSON pipes
cleanly to jq (log.info wraps it in ANSI color codes)
- get_git_sync_deploy_mode logs a warning on git_sync deserialize failure instead
of silently reporting configured=false, and documents why it is not EE-gated
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address codex review — license-gate and branch-match deploy-on-push
- get_git_sync_deploy_mode now reports deploy_on_push only on Enterprise-licensed
instances (auto-pull can't run on CE/downgrade) and returns auto_pull_branches
so the client knows which tracked branches actually deploy on push
- gitsync-settings status matches the local git branch against auto_pull_branches
before recommending git push, so an untracked branch falls back to wmill sync push
- add an integration assertion for the endpoint's default (no git-sync) shape
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: point per-topic skills at the canonical Deploying section
The git-push-vs-wmill-sync-push decision lives in core.ts (AGENTS.wmill.md),
which is already in context. Have the per-topic skills reference the Deploying
section instead of re-encoding the detection, so there is one source of truth
and no drift (the compressed version also wrongly implied `gitsync-settings
status` detects the CI-workflow path, which only core.ts's filesystem check does).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: match remote+branch server-side for deploy-on-push detection
Codex flagged that a workspace-level auto-pull signal recommends `git push` even
when the local checkout is a different repo/branch than the one that auto-deploys.
Match precisely instead, without exposing anything sensitive:
- git_sync_deploy_mode takes optional remote+branch query params. The backend
normalizes each auto-pull repo's URL to host/path (dropping embedded
user:token credentials by rebuilding from parsed components, never scrubbing
the string) and compares to the caller's remote; deploy_on_push is true only on
a licensed instance where an auto-pull repo matches that remote and tracked
branch. The response is two booleans — no repo URLs or branches leave the server.
- Branchless (default-branch) and fork/sync_forks repos stay a safe fallback to
`wmill sync push` rather than a wrong git-push recommendation.
- CLI status sends `git remote get-url` + current branch (new getGitRemoteUrl
helper, --remote flag) and reports the matched result.
- Unit-test the URL normalization/credential-stripping directly, since a
regression there would be a token-handling bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address codex security findings in deploy-mode detection
- Strip credentials from the local remote client-side before sending, so a
token embedded in the URL never reaches the server's request-URI logs
- Fetch the remote via spawnSync arg array (not an interpolated shell string),
removing a command-injection path from a caller-supplied --remote value
- Use the remote's push URL (`git remote get-url --push`) and recommend the
qualified `git push <remote> <branch>`, so the pushed target matches the one
the server checked
- Keep the port in remote normalization so different services on the same host
don't collide into a false match
- Unit-test credential stripping (CLI) and port distinctness (backend)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: resolve $var repos and fork sync_forks in deploy-mode detection
- Interpolate $var:/$res: references in the repo url/branch the same way the
auto-pull poller does (system context, cached, only when a field is a
reference), so variable-backed git URLs match instead of falling through
- For a fork workspace, evaluate the root ancestor's git-sync settings and treat
its wm-fork/<base>/<id> branch as deploying when the root repo has
auto_pull.enabled && sync_forks and its base matches the tracked branch
- Read settings/resources on the plain pool (a fork member may not belong to the
root workspace); only booleans are returned
- Unit-test the fork/branch matching (base + sync_forks + workspace-id suffix)
A blank tracked branch (repo default) still needs a network ls-remote to resolve,
so it stays a safe fallback to `wmill sync push` rather than a wrong git push.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: share one git-repo resolver between poller and deploy-mode
The auto-pull poller and the deploy-mode endpoint both resolved a git-sync repo
resource (system context, $var:/$res: interpolation) with duplicated boilerplate.
Extract windmill_store::resources::resolve_git_repository_resource and have both
call it, so the interpolation lives in one place. Drops the endpoint's local
resolve_repo_url_branch helper and its raw SQL query (and cache entry).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address codex review — fork false-positives, shell-safety, auth contract
- Fork deploy detection now mirrors reconcile_fork_branch_pull: the wm-fork branch
must route to this workspace (first existing of the id candidates) and the repo
must be in the fork's own inherited settings, so a multi-repo root or an
ambiguous id can't produce a false deploy_on_push
- Recommended deploy command is shell-quoted (branch/remote names may contain
metacharacters and the output is agent-executed)
- Remote normalization folds only the host; repo paths stay case-sensitive
- Document the system/RLS-bypassing contract on the shared resolve helper and
restore the head-fetch doc; fix the overclaiming integration-test comment
- Dev-workspace label and default-branch cases remain documented safe fallbacks
Also restores 5 sqlx cache entries an earlier cleanup dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: require a runnable auto-pull delivery path for deploy_on_push
enabled auto-pull alone doesn't mean a push deploys: a webhook-only repo with no
active hook (failed registration), or a repo that only polling could serve on an
SSH URL (the poller rejects SSH), delivers nothing. Gate deploy_on_push on an
actual delivery path — active webhook, or a pollable non-app HTTPS repo — per the
repo's auto-pull mode. Unit-tested across modes/webhook/URL-scheme/app.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: read fresh repo config for on-demand deploy-mode detection
resolve_git_repository_resource took an implicit allow_cache=true (right for the
poller loop). An on-demand status could then match against a stale url/branch
cached by an earlier poll. Make allow_cache a parameter: poller keeps true, the
deploy-mode endpoint passes false so it reflects the current git-sync config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: defer the deploy fallback instead of assuming wmill sync push
When backend auto-pull doesn't match the checkout, `status` no longer flatly
recommends `wmill sync push` — a CI workflow may still deploy on push. It now
reports the backend signal and points at the Deploying guidance (check CI → git
push, else wmill sync push; record the choice as a `Deploy mode:` line in
AGENTS.md). deploy_command is null in JSON when undetermined. This resolves the
CI-backed false recommendation without the CLI re-implementing CI detection.
Also fix two review nits: restore the deploys_on_push_branch doc comment (it had
drifted onto has_runnable_delivery) and correct the app-repo comment (their
exclusion from the poll path is a conservative safe under-report, not "can't be
polled").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: drop the ambiguous-fork-id disambiguation from deploy-mode
The existence-query resolution guarded a very narrow case (a suffix owned by both
a coexisting wm-fork-<suffix> and <suffix> workspace, queried from the wrong one).
Not worth the per-fork query; keep the cheap candidate-family check plus the
inherited-repo membership test, which already close the real fork false-positive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: drop remote-URL matching; disambiguate deploy-mode by repo count
Matching the caller's git remote URL against each repo dragged in the whole
remote-URL surface (sending the URL, credential stripping, shell-safe remote
handling, fetch/push URL, port/case normalization) — and the risk that came with
it. Replace it with a simpler rule that fits the actual question:
- deploy_on_push is true only when exactly ONE licensed, deliverable auto-pull
repo tracks the pushed branch. With a single synced repo the local checkout is
unambiguously it; with several we can't tell which is the caller's, so we
return false and the CLI asks the user.
- The endpoint takes only `branch` (no `remote`); status no longer reads or
sends the git remote.
- On the fallback, status now tells the agent to ASK the user how the repo
deploys (CI git-push vs wmill sync push) and record it in AGENTS.md, instead of
assuming wmill sync push. Guidance updated to match.
Removes normalize_git_remote (+url dep), getGitRemoteUrl, stripGitRemoteCredentials,
shellQuote, the --remote flag, and their tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: share fork-branch routing between reconciler and deploy-mode
Deploy-mode detection was re-deriving fork/dev routing (root walk, wm-fork/dev
branch parsing, descendant resolution, inherited-repo check) that the auto-pull
reconciler already owns — the source of repeated edge-case bugs. Extract it into
windmill_common::workspaces::resolve_fork_branch_target and have both the endpoint
and reconcile_fork_branch_pull (EE) call it, so they can't drift and dev
workspaces are handled by construction.
Endpoint now resolves the root via the canonical cached fork_ancestor_chain
(dropping a duplicate CTE) and routes forks/dev workspaces through the shared
resolver. The .sqlx cache is unchanged (the moved queries already existed).
Bumps ee-repo-ref for windmill-labs/windmill-ee-private companion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: exclude archived roots and frame deploy_on_push:false as unconfirmed
- deploy_on_push now requires the root workspace to be live; polling and webhook
delivery both exclude deleted roots, so an archived root (or anything beneath
one) with retained git-sync no longer reports deployable
- status and the OpenAPI now describe false as "not confirmed" (it also covers
ambiguity and conservative false-negatives), not a definite no — the CLI asks
the user rather than asserting the push won't deploy
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref for EE branch merge of main
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
hugocasa
|
87be041c09 |
fix(git-sync): avoid percent-encoded colon in git-sync hub script path (#10213)
* fix(git-sync): avoid percent-encoded colon in git-sync hub script path The git-sync init/detection hub script slug contained a colon stored as `%3A` in the run-by-path URL. The generated API client re-encodes path params with encodeURI, turning `%3A` into `%253A` (double-encoding). Some hardened reverse proxies / WAFs reject double URL-encoding and return a bare 400 before the request reaches Windmill, breaking git-sync repository detection on those instances. The hub resolves scripts by numeric id and ignores the slug, so dropping the colon from the slug is behavior-neutral (same script, same id-keyed worker cache) while producing a colon-free run URL. Also force-cache GIT_SYNC_PULL_SCRIPT_PATH at build alongside LATEST_GIT_SYNC_SCRIPT_PATH so the backend-driven pull script is always baked into the image for airgapped workers, instead of relying on an incidental hubPaths.json overlap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for git-init slug match fix Pulls in windmill-ee-private#676 so the EE is_git_init_script check matches the colon-free git-init hub slug (GitHub App token grant for git-sync jobs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for git-init slug helper + test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to a3adea1ffb406e709cc480871df58fab6c51aca1 This commit updates the EE repository reference after PR #676 was merged in windmill-ee-private. Previous ee-repo-ref: cef4e008ef62dec434aa9bb3ec783db8aff6a1c1 New ee-repo-ref: a3adea1ffb406e709cc480871df58fab6c51aca1 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
71f2d47cb4 |
feat: cap queued jobs per concurrency key on cloud (#10197)
* feat: cap queued jobs per concurrency key on cloud * fix: close preprocessed-flow bypass and bound concurrency cap scan * fix: only cap concurrency keys with an active concurrent_limit * chore: only load concurrency key cap setting when cloud hosted * fix: reject queued-job import on cloud |
||
|
Ruben Fiszel
|
89948bb441 | chore(git-sync): bump hub scripts to windmill-cli@1.762.3 (#10189) | ||
|
Ruben Fiszel
|
2ff5a918d5 |
feat(worker-tags): add * fork marker to workspace-scoped custom tags (#10177)
* feat(worker-tags): add `*` fork marker to workspace-scoped custom tags Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(worker-tags): pin fork-marker tag admission through real lineage Also render WorkspaceMatcher in its authored form in the operator-facing "not in the allowed CUSTOM_TAGS" error, and correct the authorization note on workspace_with_fork_ancestors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(worker-tags): gate fork-lineage tag lookup on workspace membership exists_workers_with_tags is a global route whose workspace is a query param. Resolving its fork lineage for a caller who is not a member disclosed whether an arbitrary workspace descends from one named by a tag(parent*) rule. Require membership before the lineage lookup, and sharpen the workspace_with_fork_ancestors contract accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(sqlx): add offline cache for fork-marker test INSERT query The new test_fork_marker_tag_admission_through_lineage integration test inserts a fork workspace with a bind-parameterized query that had no cached entry, failing the SQLX_OFFLINE=true CI build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 Alexander Petric
|
396fb1c475 |
feat(otel-tracing-proxy): trust internal endpoints with untrusted CAs (#10139)
* [ee] feat(otel-tracing-proxy): trust internal endpoints with untrusted CAs Add `insecure_upstream_hosts` and `upstream_ca_certs` to the HTTP Request Tracing settings so the OTEL tracing proxy can reach internal endpoints with untrusted or private-CA certificates while keeping them traced. Wires the two settings through the worker config and live reload, adds the inputs to the instance settings UI, and pulls in the rustls upstream-client deps (hyper-rustls/tokio-rustls/rustls/ rustls-native-certs/rustls-pemfile; hyper-http-proxy switched to its rustls feature). The proxy-side implementation lives in the companion EE PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref to otel_ca companion commit Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(otel-tracing-proxy): expose new fields in declarative config; bump ee-ref Addresses code-review findings: - Add `insecure_upstream_hosts` and `upstream_ca_certs` to the declarative `OtelTracingProxySettings` in instance_config.rs so operator/GitOps-managed installs can set them and reconciliation no longer drops values saved via the UI. - Restore the trailing newline on ee-repo-ref.txt and bump it to the companion EE commit carrying the strict host-matching / port-ordering fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 51e50629f48dbc4f5520a787b4bdfb76f4cd38d3 This commit updates the EE repository reference after PR #665 was merged in windmill-ee-private. Previous ee-repo-ref: 49f458e4446395e98915c220baa757ab3b2ed2d8 New ee-repo-ref: 51e50629f48dbc4f5520a787b4bdfb76f4cd38d3 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
97f4477069 |
feat(forks): let a fork's creator manage developers on it without being an admin (#10166)
* feat(forks): let a fork's creator manage developers on it without being an admin Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(forks): bind the fork-creator grant to the member's parent username and lock the delete Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
51d8db6602 |
feat: automatic git-to-windmill sync (polling, webhooks, in-app PRs + checks) (#9552)
* docs: add design doc for automatic git-to-windmill pull sync
* docs: add migration plan and implementation phases to git-sync pull design
* feat(git-sync): add auto_pull settings schema and pull enqueue primitive
Adds AutoPullSettings/AutoPullMode/AutoPullStatus on GitRepositorySettings
(workspace_settings.git_sync JSONB), the GIT_SYNC_PULL_SCRIPT_PATH constant,
and should_pull/effective_poll_interval_s helpers with unit tests. Exports the
EE enqueue_git_pull_job primitive. Foundation for repo→Windmill auto-pull.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): poll repos and auto-pull new commits into the workspace
Phase 1 of automatic repo → Windmill sync. A monitor task (EE-licensed,
single-replica via advisory lock) git ls-remotes each auto-pull-enabled
repository ~every minute and enqueues a pull when the tracked branch moves,
reusing the {workspace_id}:git_sync concurrency key so pulls serialize with
in-flight push commits.
- windmill-store: background (no-authed) resolver get_git_repo_head_for_autopull
that resolves the repo resource (incl. $var: refs) and ls-remotes; GitHub-App
repos are skipped here and will sync via webhooks (phase 2).
- monitor.rs: poll/reconcile/persist with optimistic sha advance and failure
status; targeted jsonb update so concurrent settings edits aren't clobbered.
- edit_git_sync_repository: preserve server-owned auto_pull state on UI save.
- openapi: AutoPullSettings/AutoPullMode/AutoPullStatus + auto_pull field.
- frontend: per-repo "Automatically deploy changes from Git" toggle with last
sync status; demote the GitHub Actions link to an advanced CI option.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): wire webhook lifecycle + receiver; share reconcile logic
OSS side of phase 2 auto-pull webhooks:
- edit_git_sync_repository creates/removes the repo webhook on save (EE-gated,
best-effort → falls back to polling).
- monitor poller now delegates to the shared windmill_git_sync reconcile/persist
helpers (also used by the webhook receiver), removing duplicated logic.
- export the shared reconcile/persist/failure helpers; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref for phase 3 in-app PR creation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): show webhook vs polling status on the auto-pull toggle
When a repo has an active webhook (auto_pull.webhook_id set), the status line
reads "instant via webhook"; otherwise it reads the ~1-minute polling cadence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): post PR diff check on dry-run completion (phase 4)
Worker completion hook in process_completed_job: when a DeploymentCallback job
carrying the __git_sync_pr_check marker finishes, parse the dry-run SyncResponse
and patch the GitHub check run with the diff summary (success/neutral/failure).
Export enqueue_git_pull_dry_run; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref (drop unused GHES webhook_secret)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* revert(git-sync): defer phase 4 PR diff checks (OSS side)
Remove the worker completion hook that posted the PR check run, drop the
enqueue_git_pull_dry_run re-export and the orphaned sqlx cache, bump EE ref.
Phases 1-3 (polling, webhooks, in-app PR creation) are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "revert(git-sync): defer phase 4 PR diff checks (OSS side)"
This reverts commit
|
||
|
Ruben Fiszel
|
3bd9f05938 |
fix(alerts): identify server replica in low-disk alert + per-host dedup tag (#10143)
* fix(alerts): identify server replica in low-disk alert + per-host dedup tag The server-mode low-disk alert keyed its dedup tag on the mountpoint alone, so `simple_alert_helper` mapped every server replica onto a single alert row per mountpoint. With more than one replica that row flaps every monitor pass: a replica seeing low disk raises the alert while a replica seeing healthy disk recovers it. The alert text also could not say which replica tripped. The fix lives in windmill-ee-private (`low_disk_alerts` in windmill-common/src/ee.rs) and appends the hostname to both the message and the dedup tag, mirroring the worker branch. Also add a regression test pinning the server tag as per-host, and correct the monitor cadence comments: iterations are LISTEN_NEW_EVENTS_INTERVAL_SEC (10s by default), not 30s, so "~60s (2 iterations * 30s)" was wrong on both factors. * fix(alerts): widen healthchecks.check_type so per-host disk tags fit Alert tags embed a mountpoint and a hostname, both unbounded, but check_type was varchar(50). create_alert only logs the insert error while the notification still fires, so an overflowing tag re-alerts every monitor pass and never records recovery state. The server tag overflows for ordinary pod-length hostnames, and the existing worker tag already overflows for every tracked mount except "/". Widening the column fixes both; bounding the hostname would not, since the mountpoint alone can consume the budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 This commit updates the EE repository reference after PR #666 was merged in windmill-ee-private. Previous ee-repo-ref: ccd1e42cf6b2d051ca17074fbdf5b80a46cffe0f New ee-repo-ref: b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
4e0fd4db55 |
feat(alerts): include disk total and top consumers in low-disk alert (#10144)
* feat(alerts): include disk total and top consumers in low-disk alert Point ee-repo-ref at the companion windmill-ee-private commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(alerts): pass alert message lazily from the min-version check simple_alert_helper now takes the error message as a future so callers can put diagnostic work behind it. Update this call site and point ee-repo-ref at the companion windmill-ee-private commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: point ee-repo-ref at rebased EE branch Rebase onto EE main so the pin keeps the SAML metadata fixes (394ad23) that the previous ref carried, and pick up the mount-scoped consumer walk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for bind-mounted file exclusion Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 4d7aafdba33a879b3c60d390c960e57679da9e87 This commit updates the EE repository reference after PR #667 was merged in windmill-ee-private. Previous ee-repo-ref: 08d3aa4c5bf630d15a28289cca62a0f1da7b7386 New ee-repo-ref: 4d7aafdba33a879b3c60d390c960e57679da9e87 Automated by sync-ee-ref workflow. * chore: point ee-repo-ref at the merged EE work plus the test fix ee#667 squash-merged, so the previous pin was a branch commit no longer reachable from EE main. Point at ee#669, which branches from EE main and carries the /proc test-portability fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to d8a7ac6ae97642a7f4928e6be6846a32dabf4e26 This commit updates the EE repository reference after PR #669 was merged in windmill-ee-private. Previous ee-repo-ref: 5526aedd73654b9aa4086dae0441b9687ff6415d New ee-repo-ref: d8a7ac6ae97642a7f4928e6be6846a32dabf4e26 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasa
|
851e30914e |
feat(saml): add ALLOW_PRIVATE_SAML_METADATA_URLS SSRF bypass (#10077)
* feat(saml): add ALLOW_PRIVATE_SAML_METADATA_URLS SSRF bypass Introduce the ALLOW_PRIVATE_SAML_METADATA_URLS env var and its allow_private_saml_metadata_urls() helper, mirroring the existing ALLOW_PRIVATE_MCP_SERVER_URLS opt-out. This lets self-hosted deployments with internal SAML IdPs (private IPs, no public DNS) skip the metadata-URL SSRF check that otherwise blocks server startup. The companion EE change (saml_ee.rs) consumes the helper to gate the validate_url_for_ssrf() call and additionally treats a cleared (empty/whitespace-only) SAML_METADATA setting as no SAML configured. Fixes WIN-2169 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(saml): surface opt-in hint and record SSRF control in threat model Add saml_ssrf_error_message() so private-IdP metadata URL rejections point to ALLOW_PRIVATE_SAML_METADATA_URLS (mirroring the MCP helper), with a unit test. Record the new SSRF opt-in under T2 in THREAT_MODEL.md, and bump the EE ref for the companion saml_ee.rs change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(saml): add validate_saml_metadata_url with opt-in unit tests Factor the SAML metadata SSRF gating into validate_saml_metadata_url() (mirroring validate_mcp_server_url) so the private-URL opt-in branch is unit-tested at the ssrf layer: blocks private by default, allows on true/1, and keeps scheme/host syntax guards when the opt-in is on. Bump the EE ref for the companion saml_ee.rs change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 394ad23242de429aef4074cc1dc28867dac95870 This commit updates the EE repository reference after PR #659 was merged in windmill-ee-private. Previous ee-repo-ref: 86da208c5aef2570568e18c7ab98f4d58adeec18 New ee-repo-ref: 394ad23242de429aef4074cc1dc28867dac95870 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
ff774c46bf |
feat: add per-workspace job-retention override (#10050)
* feat: add per-workspace job-retention override (EE) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 2ba6a2a75b6fc97858b306b2c98ada481e363c10 This commit updates the EE repository reference after PR #658 was merged in windmill-ee-private. Previous ee-repo-ref: e7fb36acd813cd717bcf05f5aafbf81de271d618 New ee-repo-ref: 2ba6a2a75b6fc97858b306b2c98ada481e363c10 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
7efeae26d8 |
feat: add fork_parent_workspace claim to OIDC tokens for fork workspaces (#9987)
* feat: add parent_workspace claim to OIDC job tokens for fork workspaces Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: rename claim to fork_parent_workspace for clarity Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to e2df172596e00877068d4b0a98afaef62fe429d1 This commit updates the EE repository reference after PR #651 was merged in windmill-ee-private. Previous ee-repo-ref: f73001ac6c038694cfc2604233a59be1c0daa40b New ee-repo-ref: e2df172596e00877068d4b0a98afaef62fe429d1 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Diego Imbert
|
e47aedac0a |
feat: add SQL migrations for data tables (#9693)
* feat: add datatable_migrations table * feat: add route to run datatable migrations * feat: sync datatable migrations as .up.sql/.down.sql files * feat: add datatable migrate up/down commands and post-push run prompt * feat: add datatable migrate new command to scaffold migrations * feat: add datatable migrations management UI * feat: prompt to create migration on DDL in datatable SQL editors * feat: support running a single specific datatable migration * feat: view migration content, run single migration, fix stacked modal * feat: per-row revert button with out-of-order warning * fix: avoid migrations list flicker on refresh after an action * feat: generate initial datatable migration via pg_dump * fix: surface datatable migration API error details in toasts * fix: revert created migration if create-and-run fails to run * fix: include postgres error detail in migration run/rollback failures * feat: sync datatable migrations as files via the workspace export * refactor: move datatable migrations to migrations/datatable/ path * fix: drop redundant datatable_migration label in sync output * fix: exclude datatable migration sql files from script metadata generation * feat: run datatable migrations as user-permissioned labeled jobs * feat: reject invalid datatable migrations on sync push * feat: datatable migrate up/down default to all datatables, --datatable to target one * fix: surface postgres error detail when datatable migrations fail to run * chore: regenerate CLI docs for datatable migrate commands * feat: default new datatable migration to a BEGIN/END transaction template * fix: validate datatable migration name and datatable at the API boundary * fix: ensure detected DDL ends with semicolon when wrapped in transaction * fix: re-prompt instead of stripping DDL when new-migration modal is cancelled * feat: refresh datatable schema after running a migration from the SQL REPL * feat: record db manager DDL on data tables as migrations * feat: make datatable migrations opt-in per data table * fix: make migration view editor read-only so its code can scroll * fix: don't re-prompt DDL guard when creating a migration without running * feat: generate down migrations for db manager DDL (postgres) * fix: correct down migration for db manager alters (no double-wrap, serial) * feat: explain migrations purpose with a tooltip in the migrations modal * compare paeg * feat: add datatable_migration kind to workspace diff pipeline * chore: point ee-repo-ref at datatable_migration git-sync companion * fix: harden datatable migration version allocation and initial-migration bookkeeping, add tests * feat: deploy and run datatable migrations on workspace merge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Refactor + handle datatable setting delete/rename * refactor: move datatable migration rename/delete cascade into module Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(windmill-utils-internal): bump to 1.7.1 for datatable migration deploy provider methods Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(db-manager): add Migrations button to top bar, make Refresh icon-only Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * BEGIN/END placeholder in down migration * feat: autofocus migration name input and flag it red when empty Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(datatable-migrations): allow non-admins to create/run/revert migrations, gate only opt in/out Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * border nits * refresh db manager schema on migrations * BEGIN/END scaffold in CLI * feat(cli): push local datatable migrations before running on migrate up Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: flag invalid migration name with red border, not just empty Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: drop random slug from auto-generated migration names Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: offer revert-and-delete when deleting an installed migration Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: record fork merge as a migration when target datatable opts in * nit * clone migrations on fork * windmill-utils-internal * fix(datatable-migrations): serialize run/rollback with a per-db advisory lock Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-manager): fail closed when migrations-status check errors on DDL apply Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix generate_initial migration ordering comment to match code * chore(datatable-migrations): remove unused update_datatable_migrations endpoint Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: run DDL migration guard on the script editor Test button Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * split * ee-repo-ref * chore(frontend): sync package-lock with package.json (@emnapi deps) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(datatable-migrations): never resolve instance credentials into migration job args datatable_database_arg eagerly resolved instance data-table credentials (including the shared instance-wide Postgres password) and passed them as the migration job's plaintext `database` arg, landing in v2_job.args. Since the run route has no admin gate, a non-admin could run a migration and read args.database to recover the password, granting cross-workspace psql access to all instance data-table DBs. Pass a `datatable://<name>` reference for both resource-backed and instance data tables instead; the pg executor already resolves it to real credentials server-side at run time, so nothing sensitive is ever stored in the job args. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * fix: handle dollar-quoting and comments when splitting SQL statements * feat: deploy datatable migrations on merge with explicit opt-in error * fix(frontend): sync package-lock with npm 11 peer-dep resolution npm ci failed with 'Missing: @emnapi/core@1.11.2 / @emnapi/runtime@1.11.2 from lock file'. @napi-rs/wasm-runtime declares @emnapi/core|runtime ^1.7.1 as peerDependencies while @rolldown/binding-wasm32-wasi pins them to exactly 1.10.0. Newer npm (bundled with node 24 in CI) installs the peer deps at the highest match (1.11.2) alongside rolldown's nested 1.10.0, so the ideal tree needs both versions; the committed lock only had 1.10.0. Regenerate the lock with npm 11.18 so it carries both 1.11.2 (top-level, for the peer deps) and 1.10.0 (nested, for rolldown's pin). Verified npm ci passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit npm publish * fix: fail closed on migrations-status error in fork schema merge * nit CI emnapi/core version * prevent initial_datatable_migration if migrations already exist * fix(datatable-migrations): validate persisted data table names as path segments edit_datatable_config only validated rename segments, not the actual settings.datatables keys, so a data table could be saved directly under a name like '..' or one containing '/'. Since new tables default to migrations_enabled = true, generate_initial_datatable_migration would then insert a migration row and the sync export would build migrations/datatable/<name>/... paths from that name, producing malformed or directory-escaping export paths. Validate every persisted data table name in edit_datatable_config (alongside the existing rename checks) and add validate_datatable_path_segment to generate_initial_datatable_migration for defense in depth. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scope datatable _wm_migrations by data table and cascade renames/deletes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(system_prompts): resolve nested local command groups in CLI docs generator The CLI docs generator anchored on the first `new Command()` in a file and never resolved locally-defined command groups passed as `.command("name", localCmd)`. For datatable this flattened the nested `migrate` group: it emitted `datatable new/up/down` plus a bare `datatable migrate`, and mislabeled the datatable command with the migrate group's description. jobs was broken the same way (its description was pull's, and pull/push rendered empty). Anchor block extraction on the `export default`ed command, recurse into locally-defined `const x = new Command()` groups mounted as subcommands, and render nested sub-subcommands. Regenerated docs now show `datatable migrate new/up/down` and `jobs pull/push` with their real options. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: drop unreleased _wm_migrations legacy-upgrade handling Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: return datatable migration SQL from getItemValue for the diff drawer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(frontend): use windmill-utils-internal 1.8.2 for migration diff drawer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * nit * fix: handle datatable migration renames on push and dedupe timestamps * fix: reject rewriting an already-applied datatable migration on upsert Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): add missing @emnapi/core and @emnapi/runtime lockfile entries Resolves npm ci EUSAGE failure: the optional cpu:wasm32 @rolldown/binding-wasm32-wasi declares deps on @emnapi/core@1.11.2 and @emnapi/runtime@1.11.2 that had no resolved lockfile entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): datatable migrate up/down default to main datatable, not all Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: fail closed when applied status unreadable on datatable migration rewrite Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: surface full error detail in Database Manager DDL/query errors * "See migration" button in the toast * feat: add Enter shortcut to Create-a-migration in the DDL guard * fix(frontend): warn before running a newly-created datatable migration out of order The row-level Run action warns when earlier migrations are still pending, but the create-and-run paths ran a just-created migration with `only` directly, applying it ahead of older pending migrations without that confirmation. Reuse the same "Run migration out of order" confirmation across all create-and-run paths via a shared helper (datatableMigrationUtils): - NewDataTableMigrationModal "Create and run" (and the DDL guard path) - DatatableSchemaDiff fork→parent merge - dbOps schema ops (DB manager create/alter/drop) — the pure factory throws a MigrationRunCancelled sentinel on decline, which DBTableEditor treats as a silent cancel Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep renamed datatable migrations visible in compare view * fix: record per-migration deployment on datatable migrations disable * fix(cli): run deployed datatable migrations after workspace merge The merge command upserted datatable_migration definitions into the target workspace and reported the item as successfully deployed, but never ran the migrations. For forked datatables backed by separate databases, this left the target schema unchanged until someone manually ran `wmill datatable migrate up`, while the CLI reported a successful merge. Collect the datatable migrations deployed (not deleted) into the target and, after the deploy loop, offer to run them via the existing offerToRunNewMigrations helper — the same post-deploy run prompt the push/sync path uses (interactive only; `--yes`/non-TTY skip the mutating run, matching push behavior). Export parseDatatableMigrationDeployPath so the merge path can parse the deployed items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): serialize datatable migration edits/deletes with the run lock A migration run snapshots a migration's code_up from datatable_migrations and only records its version in the data table's _wm_migrations after the job succeeds. upsert_datatable_migration checked _wm_migrations before allowing an edit but took no lock, so a concurrent edit could read "not applied yet", rewrite code_up/code_down, and then the in-flight run would record the version for the old SQL — leaving _wm_migrations pointing at SQL that was never applied (migrate up then skips it; rollback runs a down that doesn't match). Serialize definition rewrites and deletes with the same per-database advisory lock the run/rollback paths use: - Factor the connect+advisory-lock into lock_datatable_migration_runs and the applied-versions read into read_applied_versions_on_client. - run_datatable_migrations now snapshots the definitions AFTER taking the lock, so code_up can't change between snapshot and version-record. - upsert (when changing an existing def) and delete take the lock across the applied-check and the write; delete now rejects deleting an already-applied migration (would orphan its _wm_migrations record), symmetric with upsert. Both fail closed if the data table database is unreachable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): stack the out-of-order migration confirm above the DB editor preview Creating a table on a migrations-enabled data table opened the DB table editor's "Confirm running the following" preview modal, whose confirm triggers applyDdl, which then asks for out-of-order confirmation. Both are ConfirmationModals with a hardcoded z-[9999]; the out-of-order one lives in DBManagerContent (mounted before the editor), so it rendered behind the still-open preview modal. Add an optional zIndexClass prop to ConfirmationModal (default z-[9999], backward-compatible) and give the DB-manager out-of-order confirm z-[10000] so it stacks on top. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 27672e37df5d9dfde94f19963d5ffcdf8dd5448c This commit updates the EE repository reference after PR #623 was merged in windmill-ee-private. Previous ee-repo-ref: 6c287041cd7edd4a77a4bc07ad0e156cec32cce4 New ee-repo-ref: 27672e37df5d9dfde94f19963d5ffcdf8dd5448c Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
3dcd3949a1 |
feat(pipelines): auto-derive cascade edges from ducklake/s3 reads (+ muted-read badge) (#9963)
* feat(pipelines): auto-derive cascade trigger edges from ducklake/s3 reads Within a `// pipeline`, a read of a ducklake table or s3 object now auto-wires its cascade trigger edge straight from the FROM clause, so `// on <asset>` is only needed for edges inference can't see (dynamic SQL) or to carry per-edge opts. Two opt-outs: `// mute <asset>` suppresses a single derived edge (a lookup / SCD input read every run but not cascaded on), and `// mute all` opts the script out of derivation entirely (back to explicit-`// on`-only). Explicit `// on` still wins the dedup. Scoped to ducklake + s3 reads; resource/datatable/volume stay explicit. Read-write (RW) and write inputs are excluded so a self-referential merge can't loop-trigger itself; ambiguous (None) access is skipped. - parser: `mute` / `mute_all` in PipelineAnnotations (Rust + TS mirror) - deploy: derive_pipeline_asset_trigger_refs → script_trigger rows - frontend: resolveGraph mirrors derivation for the live edit-mode canvas - tests: shared parity corpus + derive-helper units + resolveGraph overlays Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipelines): mark auto-derived cascade edges with a persisted derived flag + "auto" badge Persist script_trigger.derived (deploy: true for ducklake/s3-read derivation, false for explicit // on) and return it from the asset-graph endpoint so the canvas renders a Sparkles "auto" badge on auto-wired edges — the inference is now visible on both the deployed graph and the live edit canvas, not just implied. Dispatch (fetch_subscribers) ignores the flag, so a derived edge fires identically to an explicit // on. Also copy derived in the workspace-clone trigger copy, and backfill muteAssets/muteAll into two empty PipelineAnnotations literals the base commit left stale (check:fast). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): derive cascade edge from effective (alt-fallback) asset access derive_pipeline_asset_trigger_refs gated on the raw parser access_type, but the persisted asset.usage_access_type and the frontend canvas both use access_type.or(alt_access_type). An ambiguous parse with a manual read override was persisted/drawn as a read yet derived no edge, so the auto edge silently vanished on deploy. Gate on the effective access type for parity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipelines): badge muted reads instead of auto-derived edges Auto-derivation is the default now, so badging every derived cascade edge is noise. Drop the "auto" badge and the persisted `script_trigger.derived` flag (migration + insert param + graph field + clone copy) that only powered it, and instead badge the exception: a ducklake/s3 asset a script reads but does NOT cascade — `// mute <asset>` / `// mute all`. `computeMutedReadKeys` marks a read-only ('r') supported read with no cascade trigger and no self-write; the canvas renders a bell-off "muted" badge on that read edge. Also fixes two review parity nits: - TS `// on` parser now strips trailing `key=value` opts (e.g. `debounce=60s`) like the Rust `split_trailing_kv_opts`, so the ref dedups against inference. - A `// materialize` producer reading its own target is upgraded to `rw` (deploy) / excluded via the materialize write refs (canvas), so it neither self-cascades nor shows as a muted read. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): drop redundant // on for auto-derived reads; gate muted badge to pipeline scripts - Templates no longer scaffold `// on <asset>` for a ducklake/s3 input the body reads — the read auto-wires the cascade now that derivation is the default. Kept for datatable/resource (not auto-derived) and native triggers. The discoverability hint now mentions `// mute` (the newly relevant annotation). - computeMutedReadKeys only badges reads by `// pipeline` scripts. A plain script or flow reading a ducklake/s3 asset never had an auto trigger to suppress, so it must render as ordinary lineage, not "muted" (Codex review). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): only drop template // on when the body actually reads the input The redundant-`// on` removal assumed the generated body reads the ducklake/s3 input, but postgres/bash/generic bodies (and `data_upload`, which reads the picker file) ignore `input` — dropping `// on` there left the asset-created script with no cascade at all. Gate the drop on READS_INPUT_LANGS (bun/deno/python/duckdb) so non-reading templates keep the explicit trigger. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
91e1b087a2 |
feat(auth): add runtime NO_AUTH mode for authentication bypass (#9962)
* feat(auth): add runtime NO_AUTH mode for authentication bypass Adds a runtime `NO_AUTH` env flag that makes every request resolve as the `admin@windmill.dev` superadmin with no login required, so self-hosted deployments can front Windmill with their own authenticating gateway without building a dedicated `oss` (compile-time `no_auth`) binary. - `NO_AUTH` is honored in any build but is force-disabled when `CLOUD_HOSTED` is set, so the managed cloud always enforces real auth. - The existing compile-time `no_auth` feature keeps its always-on behavior (`cfg!(feature = "no_auth") || *NO_AUTH`), so `oss` builds are unchanged. - `Tokened` now yields a synthetic token in no-auth mode so handlers that require it (e.g. global_whoami, called by the frontend on load) resolve. - A loud startup banner warns when the mode is on; `HIDE_NO_AUTH_BANNER` silences it once the operator has deliberately deployed behind a gateway. Fixes WIN-2131 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(auth): dismissable NO_AUTH warning banner via global setting Replaces the HIDE_NO_AUTH_BANNER env flag with a UI warning banner that can be permanently dismissed for all users from within the running instance (not exposed in instance settings). - New `no_auth_banner_dismissed` global setting, only ever written by dismissing the banner itself. - `GET /api/settings/no_auth_banner` returns whether to show the banner (true only when NO_AUTH is active and it hasn't been dismissed). - NoAuthBanner.svelte renders a top-of-app warning in NO_AUTH mode; its dismiss button opens a confirmation modal, then writes the global setting via the existing setGlobal endpoint so it stays hidden for everyone. - The server still logs the startup NO_AUTH warning unconditionally. Fixes WIN-2131 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(auth): resolve NO_AUTH in AuthCache so all_runnables works Codex/Pi review flagged that `/api/users/all_runnables` still failed in NO_AUTH mode: `get_all_runnables` extracts `Tokened` and re-validates the request token per workspace via `AuthCache::get_authed`, which rejected the fabricated `"no_auth"` token (no matching DB row) with a 400. Short-circuit `AuthCache::get_opt_job_authed` (the resolver behind `get_authed`) to the admin superadmin in no-auth mode, so any direct cache caller resolves without a real token. Single-source the mode check and the synthetic identity via `is_no_auth()` / `no_auth_admin_authed()` and reuse them across the extractor, resolver, and login paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * revert(auth): drop the NO_AUTH dismissable UI banner The in-app banner added a GET /api/settings/no_auth_banner request to every instance load for little benefit. The startup log warning already surfaces that auth is bypassed to operators, so drop the banner, its endpoint, and the no_auth_banner_dismissed global setting entirely. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
b13113964a |
fix(pipelines): canonicalize S3 asset keys so SDK writes and DuckDB reads connect (#9939)
* fix(pipelines): canonicalize S3 asset keys so SDK writes and DuckDB reads connect
The SDK object forms — TS `writeS3File({s3:"exports/x"})` and Python
`write_s3_file(S3Object(s3="exports/x"))` — resolve to the URI `s3:///exports/x`
(empty default storage), whose parsed asset path was `/exports/x` (leading
slash). DuckDB `read_csv('s3://exports/x')` and the `// on s3://exports/x`
trigger form yielded the bare `exports/x`. The same object thus produced two
asset identities, so a DuckDB consumer never connected to a TS/Python producer
in the pipeline graph.
`parse_asset_syntax` (shared by the native backend parsers and the wasm parser
that drives `frontend/src/lib/infer.ts` and the CLI `localGraph`) now strips a
single leading slash from S3 paths, so `s3:///key`, `s3://storage/key`, DuckDB
`s3://…`, and `// on` all canonicalize to one key. Both deploy-time inference
and editor/CLI inference agree, and the producer's write edge and the
consumer's read/trigger edge share a node.
Only one leading slash is stripped, so `s3:///` triple-slash default-storage
keys collapse to the bare key while Hive-partition keys
(`s3://bucket/y=2024/f.parquet`) and explicit-storage `s3://storage/key` paths
are untouched. Non-S3 asset kinds (res://, ducklake://, …) keep their paths
verbatim.
Note: existing deployed pipelines that recorded `/key` paths need a redeploy to
pick up the canonical `key`; the fix is forward-consistent for anything parsed
after this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(pipelines): mark S3 asset-path normalization (item 6) resolved
The open-issues list still flagged the SDK-form leading-slash vs bare-URI
no-slash mismatch as "Still open", contradicting the fix in this PR. Mark it
resolved to match the updated Language-coverage prose.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs+test(pipelines): disclose S3 explicit-storage vs default-storage-nested-key aliasing
Collapsing to one canonical key means `s3://storage/key` (explicit storage) and
`s3:///storage/key` (default-storage nested key) now alias to the same node
`storage/key`, though they name different objects. Low-probability (needs a
storage config named to match a default-storage prefix) and inherent to a
best-effort lineage graph that doesn't split the first segment as a storage
name, but previously undisclosed. Document the tradeoff and pin the intended
aliasing with a test so it's intentional, not a latent surprise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): mirror S3 leading-slash strip in frontend live-preview parser
The pipeline graph live preview parses `// on` annotations client-side via the
hand-written `parsePipelineAnnotations.ts` (a TS mirror of the Rust annotation
scanner), NOT the wasm parser. Its `parseAssetSyntax` still returned the raw
suffix, so `// on s3:///exports/x` yielded `/exports/x` while the deploy-time
and wasm parsers now canonicalize to `exports/x`. `resolveGraph` synthesizes
trigger edges from that path, so the browser preview could still render
disconnected `/exports/x` and `exports/x` nodes for the exact triple-slash case
this PR fixes at deploy time.
Mirror the S3-only single-leading-slash strip in the TS parser and extend the
shared parity fixture corpus (run by both the Rust and TS parity suites) with
the triple-slash trigger case, so Rust/TS drift on this is now caught.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): seed slashless S3 template asset paths to match canonical key
`autoOutputAsset` seeded new S3 template outputs with a leading slash
(`/pipelines/…`), which the old parser required to match `s3:///key` writes.
This PR made `parse_asset_syntax` strip that slash, so the seeded draft asset
(stored as `outputAssets`, used by `resolveGraph` for inactive-draft node
identity) no longer matched the body-inferred identity `pipelines/…` — the live
preview could render a duplicate `/pipelines/…` node and a phantom post-deploy
drift warning.
Seed the canonical slashless key instead, and switch the DuckDB body's S3 URIs
from `s3://${path}` to `s3:///${path}` so the generated runtime URI stays the
triple-slash default-storage form byte-for-byte (the SDK sites already build
`s3:///` + bare key). Add a pure-logic parity test asserting, for every
language and S3 output kind, that the seeded asset path is slashless and that
every S3 URI the generated body emits is triple-slash and canonicalizes back to
that seeded path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): canonicalize S3 keys in CLI + frontend bounded-cascade resolvers
Two more hand-written S3-URI sites returned the raw suffix, so `s3:///exports/x`
stayed `/exports/x` while native/wasm parsers now canonicalize to `exports/x`:
- `cli/src/commands/pipeline/localGraph.ts` — the no-wasm fallback `// on`
scanner (go/bash/ruby). A fallback consumer's `// on s3:///x` would not
connect to a wasm-inferred `x` producer in `wmill pipeline show/run --local`.
- `boundedCascade.ts` `assetUriToNodeId` (duplicated in the CLI and the frontend
AssetGraph engines, kept in sync) — `--to s3:///exports/x` / a cascade bound
token would not resolve against the canonical graph node `s3object:exports/x`.
`resolveToken` delegates here, so it is covered too.
Mirror the S3-only single-leading-slash strip in all three, and add `s3:///`
tests to the CLI local-graph fallback suite and both bounded-cascade suites
(explicit-storage and Hive-partition keys asserted untouched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(pipelines): phrase S3 template test comment as a current invariant
Describe the slashless-seed requirement as the invariant it is, not as change
history, per the AGENTS.md "describe the code as it is" rule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): strip all leading slashes from S3 keys so trigger refs round-trip
`parse_asset_syntax` stripped only one leading slash, so `S3Object(s3="/x")` —
which resolves to the quad-slash URI `s3:////x` — parsed to path `/x`. But
`trigger_spec_to_row` rebuilds a stored trigger ref as `s3://<path>` =
`s3:///x`, which `parse_asset_trigger_ref` then parses back to `x`. The
producer recorded `/x` while its consumer trigger resolved to `x` → a broken
edge. The same asymmetry affects every `s3://`+path reconstruction site
(backend refs, frontend `assetUri`, page refs) whenever a path starts with `/`.
Strip ALL leading slashes so a canonical S3 path never starts with `/`; naive
`prefix + path` reconstruction then round-trips everywhere. Applied uniformly
across all six S3-URI sites (Rust `parse_asset_syntax`, the TS live-preview
parser, template `s3Key`, and the frontend+CLI `assetUriToNodeId` and CLI
fallback scanner). The pathological leading-slash key collapses to the bare key
— acceptable for a best-effort lineage graph that never split storage anyway.
Tests: a windmill-common round-trip test (parse → trigger_spec_to_row →
parse_asset_trigger_ref) over every URI form incl. the quad-slash case; a
`s3:////x` shared parity fixture (Rust + TS); and quad-slash assertions in the
Rust parser test and both bounded-cascade suites.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(pipelines): align S3 template parity helper with strip-all canonicalization
The template seed/body parity test's `canonicalS3Key` helper (and its comment)
still stripped a single leading slash, so it no longer mirrored the parser it
claims to pin. Strip all leading slashes to match `parse_asset_syntax` and the
frontend/CLI mirrors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Ruben Fiszel
|
55451db009 |
fix(datatable): self-teaching error for unresolved datatable:// references (#9941)
`ATTACH 'datatable://main'` (or any datatable schema/executor path) failed with a bare "datatable main not found", giving the user no way forward — the datatable substrate has no auto-provisioning like a DuckLake catalog, so the fix is always to create one in workspace settings, but nothing said so. `get_datatable_resource_from_db_unchecked` now returns a NotFound error that lists the workspace's configured data tables (to catch typos) and points at the "Data tables" settings tab, noting `main` is the default name used by `datatable://main`. The message bubbles up wherever the resolver is called (pipeline ATTACH, schema fetch, postgres executor, agent HTTP endpoint). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
39eb9de1bc |
feat(pipelines): fork data environments for ducklake materialization (dev data) (#9915)
* feat(pipelines): fork-scoped ducklake namespaces with read-defer to parent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork graph indicator + fork ducklake namespace cleanup endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork_views-keyed view transition, fork lineage clone, design doc Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): review hardening - fork DATA_PATH last-wins, registry cache TTL, defer tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): per-lake isolated/shared choice at fork creation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): chain-aware defer discovery + per-location fork namespace registry Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): lake-scoped fork schemas, catalog identity in registry, chain-aware graph chips Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): cleanup deletes fork data from the registered storage identity Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): collapse fork data-path segment to one component (slash-safe ids) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): per-catalog ancestor checks, ancestor extra_args passthrough, test compile fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): invalidate fork ancestor-chain cache on lineage mutations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): sweep descendant ancestor-chain caches on delete/reparent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): run fork ducklake cleanup inline in delete_workspace Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): resolve fork cleanup credentials pre-commit, destroy post-commit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): shared dev-workspace authz gate for namespace drop, invalidatable registration cache, segment-boundary delete filter - extract require_prod_admin_for_dev_workspace, used by both delete_workspace and drop_forked_ducklake_namespaces so the gates cannot drift - key FORK_DUCKLAKE_REGISTERED per workspace and invalidate it in cleanup_fork_ducklake_namespaces so a same-id fork recreated within the TTL re-registers its namespaces - filter listed object locations to the segment boundary before deletion Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): keep orphaned wm-fork-* workspaces ducklake-isolated parent_workspace_id is ON DELETE SET NULL, so a fork can outlive its parent with an empty ancestor chain while its cloned config still points at the shared lake. Key the isolation gate on the wm-fork- prefix as well as the chain (mirroring workspace_is_fork): orphaned forks get the write redirect, registration and cleanup with zero ancestors (no defer), and keep their 'fork' graph chips. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): attach orphaned wm-fork-* ancestors at their fork namespace Chain position alone classified the last ancestor as a root, but an orphaned wm-fork-* ancestor (its own parent deleted, SET NULL) ends the chain the same way while its data lives in its fork namespace — its descendants' defer views bound the dead root's lake instead. Key the root-vs-fork decision on the wm-fork- prefix too, matching the resolution gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): never inherit shared lake opt-out; durable cleanup ledger for failed fork deletions - fork creation strips cloned fork_behavior stamps before applying the request's shared_ducklakes list: sharing is a per-creation choice, a fork of a shared fork defaults back to isolated - fork_ducklake_namespace loses its ON DELETE CASCADE FK: rows are the durable cleanup ledger and outlive the workspace when physical cleanup fails post-commit; fork creation retries leftover rows for the reused id and refuses to create while a metadata schema still cannot be dropped (data-file leftovers alone are inert once the schema is gone and are swept by the next successful same-prefix cleanup) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): make orphaned-namespace cleanup retries independent of deleted fork resources - ledger rows gain a schema_dropped phase flag: set when the schema drop succeeded but data cleanup failed, so later retries skip the schema phase and need no catalog credentials at all; registration resets it on re-attach (ON CONFLICT DO UPDATE) since attaching recreates the schema - retry-path $res: resolution falls back to the workspace being forked (the deleted fork's resources were clones of a parent's); live paths (delete_workspace prepare, drop endpoint) pass no fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): fork tables from failed-after-commit runs stay fork-owned in defer and graph A failed materialization must not disguise a physically existing fork table as deferred: CREATE VIEW IF NOT EXISTS silently yields to the table, so reads hit fork data while the graph claims parent defer. - record_mat upsert preserves the last committed snapshot_id on failure - defer discovery and graph chips treat fork rows with a committed snapshot as fork-owned even when status is failed - inspect_fork_catalog also lists live fork tables (same round trip) and the defer list is filtered against them — covers rows recorded before this fix and tables created by raw SQL - drop stale FK-cascade wording in the design doc and sidebar comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork-mode ducklake settings — per-lake isolated/shared chips + banner, fork_behavior round-trip The workspace-settings ducklake editor had no fork awareness: no reminder of each lake's isolated/shared choice and no warning about what edits mean in a fork. It also rebuilt each lake explicitly on save, silently dropping fork_behavior — any settings save in a shared fork flipped the lake back to isolated. - fork detection mirrors the backend gate (parent link or wm-fork- prefix) - info banner explaining isolated vs shared semantics in a fork - per-lake chip (emerald 'isolated' / amber 'shared with parent') with tooltips, matching the pipeline graph chip colors - fork_behavior added to DucklakeSettingsType and preserved through convertDucklakeSettingsToBackend Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
33521505db |
feat(ducklake): scheduled lake maintenance (expiry, compaction, orphan cleanup) (#9916)
* feat(ducklake): scheduled lake maintenance (snapshot expiry, compaction, orphan cleanup) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ducklake): review fixes — starts_with not LIKE, CE license-lapse escape Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(ducklake): auth-contract docs + _unchecked rename per codex review Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(ducklake): move maintenance payload construction into EE module Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ducklake): fall through to script resolution for non-managed reserved-prefix schedules Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(ducklake): document accepted pre-existing-schedule limitation on the reserved prefix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ducklake): CE save-off clears the managed schedule row and queued occurrence Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 2fab310d4f50ed7c34857d69c9b854f4491bf217 This commit updates the EE repository reference after PR #645 was merged in windmill-ee-private. Previous ee-repo-ref: fff1fd830a36beba732486f05941ec243cf6b640 New ee-repo-ref: 2fab310d4f50ed7c34857d69c9b854f4491bf217 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
dce247c6d2 |
feat(pipeline): write-audit-publish for materialization data tests (#9911)
* feat(pipeline): write-audit-publish for materialization data tests (EE) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: EE worktree E0583 troubleshooting + duckdb feature check row Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: clarify EE symlink example (absolute target, EE repo layout) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipeline): move bootstrap DDL inside guarded WAP transaction Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pipeline): move WAP guard SQL builder into EE, OSS keeps placement only Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump ee-repo-ref to EE branch rebased on EE main Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style: reword test comment as current invariant per AGENTS.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump ee-repo-ref (EE module doc update) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pipeline): OSS emits typed materialize plan, EE owns WAP transform Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: make rewrite assertion build-aware; refresh oss module doc Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 7be0bad1a6d6b5c3a107c0a2cd4bf003c36ec34c This commit updates the EE repository reference after PR #644 was merged in windmill-ee-private. Previous ee-repo-ref: 63cabae75329429f647e01083936d70f8197dc9e New ee-repo-ref: 7be0bad1a6d6b5c3a107c0a2cd4bf003c36ec34c Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
42e11c6570 |
feat(pipelines): schema contracts — save-time consumer checks vs captured schemas (#9917)
* feat(pipelines): schema contracts — save-time consumer checks vs captured schemas Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: move schemaContractContext above schemaCanEvolve doc comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: emit scd2/on_schema_change in CLI local graph, address review notes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: gate editor _current ignore-suppression on scd2, matching backend Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
5d7fb6deca |
feat(pipelines): asset freshness — fresh/stale badge (CE) + watchdog (EE) (#9909)
* feat(pipelines): passive asset freshness tracking on the graph Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(pipelines): drop dead freshness-enforcement stub, document query ordering Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): freshness watchdog (EE) — auto re-run stale producers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): watchdog review fixes — archived workspaces, badge kind parity, scan index Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): CI review — no singlestepflow in freshness, +N parity, completion-time fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): CI review — history completedAt, freshness/asset trigger UI metadata Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 6f5fe0f7f56696fbef5a8349da38496c32e71666 This commit updates the EE repository reference after PR #643 was merged in windmill-ee-private. Previous ee-repo-ref: 1f13380354bf591ae25a2c20d36917534bcc5459 New ee-repo-ref: 6f5fe0f7f56696fbef5a8349da38496c32e71666 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
84141add1d |
feat(pipelines): workspace duckdb macro libraries (// macros / // use) (#9890)
* feat(pipelines): parse duckdb macro-library annotations (// macros, // use) * feat(pipelines): duckdb macro registry tables + deploy-path validation and writes * feat(pipelines): inject workspace duckdb macros into consumer jobs at run time * feat(pipelines): surface macro libraries and lib-consumer edges in asset graph api * feat(frontend): macro-library nodes, lib-consumer edges and scaffold in pipeline graph * docs: mark dbt gap #7 (packages/macros) shipped via workspace macro libraries * fix(pipelines): review fixes - char-safe parsing, local macros win, fork clone, trust-model docs * feat(frontend): duckdb macro autocomplete + workspace macro explorer drawer * fix(pipelines): address CI review - use-setup retention, splice past local defs, orphan filter, full consumer rescan, index-keyed strip * fix(pipelines): inject provider library setup for implicitly-called macros too * fix(pipelines): rls-gate macro listing + honor library-level // use transitively * fix(pipelines): weave injected macros around local definitions by bind order * fix(pipelines): injected library setup always runs before user blocks * perf(pipelines): cache macro registry per workspace with notify-event invalidation * perf(pipelines): disable macro registry cache on cloud |
||
|
hugocasa
|
7c7d7474cc |
feat: support workspace forks on cloud using parent workspace limits (#9864)
* feat: support workspace forks on cloud using parent workspace limits Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: clarify count_paid_seats approximates rather than mirrors billing seats Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: non-admin fork UI, attach cap, and fork-count for cloud forks Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: cloud fork billing cache on rename, usage display, attach cap edge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: fork count in cloud quotas + fork billing points to parent Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: invalidate billing/fork caches on fork deletion for id reuse Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: gate fork usage remap on CLOUD_HOSTED, not just the cloud feature Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: note cloud feature vs CLOUD_HOSTED gating in backend guide Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: reserve fork-cap slots for an attach candidate's whole subtree Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: invalidate team-plan cache on delete, raise fork depth cap Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: cap fork nesting depth (MAX_FORK_DEPTH, default 5) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: fork count/height robust to cycles and deleted intermediates Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): reset fork button loading state on creation error Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: invalidate billing cache for attached fork subtree; helper auth docs Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
d15033cde6 |
fix: invalidate bun bundle cache on transitive relative-import changes (#9891)
* fix: invalidate bun bundle cache on transitive relative-import changes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: do not memoize transient fetch errors in bundle-key import cache Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: use regular comment on lazy_static block (deny unused_doc_comments) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: align bundle-key import version selection with loader content endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
76a9523009 |
feat: use derived username instead of email for non-member superadmins (#9857)
* feat: use derived username instead of email for non-member superadmins Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address review - drop redundant username cache, guard whoami membership by email Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: use explicit non_member boolean instead of role string for superadmin banner Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve email from password table for non-member superadmin permissioned_as Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve non-member superadmin drafts via shared username->email resolver Adds resolve_username_to_email (usr, then super_admin password fallback for both derived-username and email modes) and uses it in get_email_from_permissioned_as and the drafts get/list endpoints, so a non-member superadmin's drafts resolve and no email leaks into the drafts payload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: superadmin-not-in-workspace schedule uses derived username as permissioned_as Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve non-member superadmin identity in draft owner-circles, username_to_email, and home filter Applies the password-fallback username resolution to the script/flow/app/draft owner-circle subqueries and the username_to_email endpoint (was an admins-workspace 'username == email' hack), and switches the home items-list user-folder filter to the non_member flag instead of the now-broken username-contains-@ heuristic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: backfill non-member superadmin favorites from email to derived username Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: propagate DB errors in username resolution instead of leaking email (CI review) Addresses cubic-dev-ai P2: get_instance_username_or_fallback_to_email now returns Result and only falls back to the email for a genuine 'no derived username'; a query error propagates so callers fail closed rather than leaking the raw email as the acting username. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: clarify non-member superadmin popover (username used + admin permissions) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep username_to_email endpoint member-only to not disclose non-member superadmin email (CI review) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: forbid disabling automate_username_creation once usernames assigned (CI review) Makes the setting effectively one-way once instance-wide usernames exist, so the global-uniqueness invariant that keeps stored u/<username> identities (schedules/triggers/drafts/superadmin ownership) unambiguous can never be dropped back to workspace-local uniqueness. Re-saving false on an already-disabled instance stays a no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
b52972d0de |
fix: validate workspace name length (max 50 chars) on create and fork (#9854)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
bf6be967fa |
fix: honor verify-ca/verify-full sslmode for postgres connections (#9835)
* fix: enforce tls verification for postgres verify-ca/verify-full sslmode Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: make PG_ACCEPT_INVALID_CERTS value-based and keep cache key well-formed Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: grandfather existing postgres resources via per-resource trust_cert flag Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scope pg trust_cert migration to resource data, drop schema patch Hub resource type sync (windmill cache-rt + startup SYNC_CACHED_RT) only touches the admins workspace and is opt-in, so the schema is left to the hub; the migration just grandfathers existing resource values so the upgrade is non-breaking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: replace pg trust_cert with verify-*-scoped accept_invalid_certs, drop migration Per-resource accept_invalid_certs (default false for new resources) replaces the trust_cert flag and grandfather migration. It only applies to verify-ca/verify-full; unset falls back to legacy behavior (verify only when a root cert is present) so existing and git-synced resources are not broken on upgrade. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: warn in job logs when a verify-* postgres resource skips cert verification Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
83f3d7f910 |
feat(licensing): enforce offline license seat cap (#9845)
* [ee] feat(licensing): enforce offline license seat cap Companion to windmill-ee-private. Aligns the offline-license seat count with the billing model and adds real-time enforcement when usage exceeds the cap. OSS side carries the ee_oss stubs, the reactivation cap-check call site, the regenerated SQLx cache, and the EE ref bump. - Exclude instance-disabled users (password.disabled) and service accounts from the seat count. Deactivating a user now frees a seat. - Service accounts no longer consume seats (no check at creation). - Hard-block reactivation when it would exceed the cap. - Invalidate the license (halting jobs) when seat usage exceeds the cap, mirroring CU-cap enforcement; recovers when usage drops back under or a higher-cap key is loaded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [ee] fix(licensing): bump EE ref for reactivation seat-check fixes Points to the EE companion commit that fixes reactivation double-counting and preserves the original seat alert tag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [ee] fix(licensing): reactivation seat delta includes pending invites Bumps the EE ref and drops the now-orphaned usr-only cache entry; the reactivation check reuses the existing usr ∪ workspace_invite query. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [ee] test(licensing): bump EE ref for offline seat-cap tests Adds #[sqlx::test] coverage for the offline seat counting and cap-check logic; EE-only (runtime queries, no cache change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: update ee-repo-ref to f814c3f75308c1ef1e4526d8d0eeb360ce16abe4 This commit updates the EE repository reference after PR #637 was merged in windmill-ee-private. Previous ee-repo-ref: b2622e3afc2fe1fe3e2ec978ca46cf9decf91b82 New ee-repo-ref: f814c3f75308c1ef1e4526d8d0eeb360ce16abe4 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
hugocasa
|
b4b0c6a93e |
feat: add dev workspaces paired with a lockable prod workspace (#9793)
* feat: add dev workspaces paired with a lockable prod workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate dev-workspace prod-lock on admin and prevent attach cycles Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: redirect locked-prod edits into the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make dev-workspace settings tab available on CE (was EE-gated) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: lock prod against forking too and funnel edits to the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: open dev item page on edit and tailor dev-workspace lock messages Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: prevent nested dev workspaces and hide dev option when one exists Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: drop the redundant already-has-dev hint on the fork form Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: badge dev workspaces and sort them ahead of forks in the tree/switcher Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: label dev workspaces as 'Dev workspace of X' instead of 'Fork of X' Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: label edit as 'Edit in <dev>', cover editor headers, auto-expand dev in tree Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: split prod lock into separate block-deploy and prevent-forking toggles Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: make resources/variables workspace-specific from compare page Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: steer AI-chat sessions to the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: refine session fork options and lock guidance for dev/prod Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: session picker reads prod's real rules, default to current ws Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: copy members into forks and clarify dev-workspace root labeling Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: place the workspace id field under the fork name Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address dev-workspace review findings and harden fork detection Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: regenerate sqlx offline cache Restores entries dropped during the origin/main merge and adds the dev-workspace queries (is_dev_workspace, ws_specific, has_parent). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address second-round dev-workspace review findings Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address Pi and Codex review findings on dev-workspace endpoints Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate locked-dev git-branch fork on admin and validate ws_specific path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: clear prod dev-lock when deleting an attached dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: consolidate dev-workspace migration and scope all-group join to attach Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: restore dev-workspace CHECK into consolidated migration and scope all-group join Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: drop copy_members from the dev-workspace attach path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: dev-workspace lifecycle/auth fixes from Codex review round Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: explicit create-in-other for workspace-specific items Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make create-in-other strictly create-only (never overwrite target) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: return 403 (not 401) for dev-workspace permission denials Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: allow attaching a same-family fork as a dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: emphasize the go-to-dev action in the no-direct-deploy alert Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: seed a resource's linked variables when creating it in the other workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: judge workspace deploy/fork locks against the user's identity in that workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: clarify create-in help text in workspace-specific panel Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: admin-gate dev-workspace creation and harden lock/seed edges Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: preserve a staged fork's source on picker create-mode re-entry Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: clear dev flag on archive and check dev existence server-side Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make create-in-other atomically create-only via direct create Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: create-only resource insert, ws-specific list scopes, archive lock guard Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: reserve the dev_workspace_lock protection-rule name from the public API Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: reattach create_protection_rule doc comment to its function Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: make dev-archive pairing teardown atomic with the archive Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: follow deploy_to on root rename; show dev pairing to non-member prod admins Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: copy creator metadata on fork; invalidate fork routing cache on rename Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: accept g/ paths in set_ws_specific; gate copy_members to dev workspaces Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
003a262a4e |
feat: column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) (#9814)
* feat: column-level lineage for ducklake pipelines via // column annotation Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: auto-derive column lineage from DuckDB SQL AST (annotation as override) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: clarify column-lineage inference is server-side; drafts use annotations Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): surface inferred column lineage in live pipeline drafts Threads the DuckDB SQL-AST column lineage (from the WASM asset parser) through ScriptEditor -> details pane -> page -> resolveGraph, merged with // column annotations (annotation wins) so the live preview matches the deployed graph. Takes effect once windmill-parser-wasm-asset is republished with the inference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(frontend): bump windmill-parser-wasm-asset to 1.740.0 for SQL column-lineage inference Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: column-lineage inference now runs live (WASM) too, merged with annotations Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): transitive column-lineage trace (impact analysis) Stitches every producer's column_lineage into a pipeline-wide column graph (columnLineageGraph.ts) and replaces the single-hop diagram with an interactive ColumnLineageTrace: select an asset to see its columns' full upstream/downstream lineage across scripts; click any column to highlight its complete transitive impact set (forward + backward) and dim the rest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address CI review on column lineage (parse-fallback, node-id, perf, leak) - backend: DuckDB SQL parse failure now falls back to `// column` annotation lineage instead of dropping it (Codex P1) - columnLineageGraph: collision-proof JSON node ids; deterministic first-write output anchoring when a producer has multiple ducklake writes (cubic P2 ×2) - pipeline page: gate buildColumnGraph to a ducklake-asset selection so it doesn't rebuild on every editor keystroke (cubic P2) - ScriptEditor: clear inferredColumnLineage on parse error so it can't leak across a script switch (cubic P2) - AssetGraphEdge: widen badge stacking offset 12px->18px to fully clear (cubic P3) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: resolve JOIN inputs + anchor column lineage to // materialize target Addresses the second Codex review pass (two P1s): - SQL inference now walks JOINed tables: build_from_maps maps every FROM entry AND its joins into the alias map, and single-table attribution requires no joins. `SELECT o.x, c.y FROM a o JOIN b c` now resolves c.y (was dropped). - The column graph anchors a producer's lineage to its declared // materialize target (surfaced on the runnable node) instead of guessing a ducklake write-edge, which is unordered for deployed graphs and ambiguous for multi-output scripts. Falls back to a write-edge when no materialize target. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate column-lineage badge to the // materialize target write-edge The canvas badge keyed on `e.asset_kind === 'ducklake'`, so a multi-output producer showed the same column mapping on every ducklake write-edge. Use the same materialize-target anchor as buildColumnGraph: the badge lands only on the declared output's edge, falling back to the ducklake write-edge when there's no materialize annotation. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: build column trace from displayGraph so View hides draft lineage The transitive column trace was built from graphWithDraft regardless of mode, so in View with drafts hidden it could surface draft `// column` lineage the deployed canvas doesn't show. Build it from `displayGraph` (the graph the canvas actually renders) so the trace matches: draft overlays in edit / show-drafts, deployed-only in plain View. (Codex P2) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: don't infer column lineage for local/temp staging CTAS A CTAS into a local/temp staging table isn't the materialized output, but its projection was inferred and (flat) column_lineage anchored to the script's // materialize target — so staging columns showed up as the final asset's. Gate inference to the actual output: a top-level managed-materialize SELECT, or a CTAS/CREATE VIEW whose target resolves to a real asset. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: scope inferred column lineage to one output asset Inference accumulated columns from every output-producing query into one flat list, all anchored (frontend) to the script's // materialize target — so an auxiliary CTAS into a different asset showed its columns on the materialized one. Tag each inferred entry with its output asset and, in parse_assets, scope the list to the // materialize target (keeping untagged top-level-SELECT entries); with no declared target, drop inference when entries span multiple output assets rather than attribute them to an arbitrary one. Parser-internal — no wire change. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: treat CREATE TEMP TABLE/VIEW as local even under an active USE A one-part temp name under `USE dl` resolved to an asset (ducklake://…/tmp) before being registered local, so a final SELECT reading it invented `final.total <- warehouse/tmp.amt` (a phantom DuckLake column) and recorded a phantom asset. track_table_definition now registers any temporary table/view as local up front, bypassing active-asset resolution; CreateTable/CreateView pass their `temporary` flag. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
ade74b297f |
feat: capture managed-materialize output schema as asset metadata (#2a) (#9812)
* feat: capture managed-materialize output schema as asset metadata (#2a) After a managed `// materialize` run, capture the producer's output schema via a DESCRIBE folded into the existing one-row summary read (no extra round-trip) and persist it in a new versioned `materialized_asset_schema` sidecar table. This is the producer-side capture that pipeline parity gap #2b (save-time consumer-ref contract enforcement) will read back. - materialized_asset_schema sidecar (asset-level grain), versioned: a new version row is inserted only when the captured column set changes. - output_schema column added to the materialize summary codegen. - worker extracts + records the schema on a successful materialize. - /assets/asset_schemas read endpoint exposing the evolution history. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address CI review on schema capture (partition col, order, status gate) - exclude the synthetic `_wm_partition` column from the captured schema for partitioned assets, so the recorded contract is the producer's logical output, not Windmill's storage detail (claude/cubic P1). - make the captured column list explicitly ordered (`row_number()` over the DESCRIBE + `list(... ORDER BY)`), so the `list()` aggregate can't reorder columns and spuriously bump the schema version (cubic P2). - gate the API `record_materialization` schema upsert on a `Materialized` status, so a failed/running write (or a client attaching a schema to one) can't advance the schema history (cubic P2). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address Codex review (manual-mode schema gate + auth contract docs) - gate output_schema extraction on the managed (`Some((Some(_), _))`) path so a `// materialize manual` run — whose result is the user's own query output — can't persist a caller-shaped `output_schema` into materialized_asset_schema (Codex P2). Verified e2e: a manual run returning a fabricated `output_schema:[{injected,EVIL}]` records the partition but writes no schema version, while the managed path still captures normally. - document the authorization contract on the new public `record_asset_schema` and `list_asset_schemas` helpers: they perform no access control (mirroring the materialized_partition siblings) and require callers to pass a workspace-authorized executor (Codex P1). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): schema-history tab on the ducklake asset node (#2a) Adds a "Schema" tab to DucklakeAssetPanel surfacing the captured output-schema versions persisted by the materialize run. Master-detail (mirrors the History tab): the version list (newest first, newest auto-selected) shows column count + snapshot + capture time; selecting a version renders its column/type table. Reads the GET /assets/asset_schemas endpoint via raw fetch, matching the sibling PartitionStatusGrid convention (these materialization endpoints are not in the generated client). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: schema tab is strategy-aware (history vs fixed schema) Only a whole-table `replace` producer (CREATE OR REPLACE) can change columns run-to-run; `append`/`merge`/partitioned writes INSERT into a fixed-schema table, so their schema is pinned at first materialize and the "history" framing is degenerate (always one version). - backend: surface the managed `materialize_strategy` (`replace`/`append`/ `merge`) on the asset-graph runnable node, alongside the existing `partition_kind` (same parse-from-annotation path). - frontend: the pipeline page derives `schemaCanEvolve` for the selected asset from its write-producer (`replace` && not partitioned) and threads it to the Schema tab. Evolvable → master-detail version history; fixed → a single current-schema table with a short "schema is fixed" note. Unknown defaults to evolvable so real history is never hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: schemaCanEvolve fails open on unknown producer strategy Previously a producer present but missing `materialize_strategy` (e.g. a draft-overlay runnable, synthesized without the field) fell through to canEvolve=false, hiding captured history behind the fixed-schema view — contradicting the "unknown defaults to evolvable" intent. Now the fixed view shows only when *every* producer is a known insert-style write (append/merge, or partitioned replace); any producer with unknown (missing) strategy is treated as evolvable, so real history is never hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
1c4bb8af14 |
test: de-flake asset-dispatch by bypassing cross-DB script-hash caches (#9820)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
3cda447621 |
fix(flows): reject corrupt step paths at deploy + atomic cache writes (#9751) (#9813)
* fix(flows): reject corrupt step paths at deploy + atomic cache writes (#9751) A flow step could execute an unrelated (and in the reported case, destructive) script at runtime even though every stored definition looked correct. A forensic dump traced it to two issues: - Deploy accepted absolute/local step paths. `wmill sync push` from a feature- branch checkout under /tmp baked an absolute path (`/tmp/.../ops/scripts/clean_device/...`) into a step's `value.path`. Persisted verbatim, it mis-resolved to an unrelated script at runtime. - The on-disk cache write was neither truncating nor atomic. `FsBackedCache::put` used `write+create`, so a shorter overwrite left stale trailing bytes and concurrent writers could interleave into a torn file — a corrupt cached blob that a worker then scheduled from. Fixes: - Reject non-workspace flow step paths (must be u/, f/, g/ or hub/) in `validate_flow_value` (covers create_flow + update_flow, recursively through loops/branches/AI-agent tools) and early in the CLI `pushFlow`. - Make `FsBackedCache::put` write a unique temp file (truncate + fsync) then atomically rename it over the target, cleaning up on error. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(flows): validate failure/preprocessor module paths + sub-flow paths in CLI Address PR review (cubic + claude): - Backend `validate_flow_value` is the authoritative guard but only walked `modules`; extend it to also validate `failure_module` and `preprocessor_module` (which can themselves be sub-flows/loops/branches), so an absolute path there can't be persisted. - CLI preflight only collected `type: "script"` paths; now collects sub-flow (`type: "flow"`) step paths too (recursively, incl. failure/preprocessor), so the comment's claim matches the behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cli): include AI-agent tool step paths in flow path preflight Address Codex review: collectStepPaths skipped aiagent tools, so a bad path in a tool fell through to the API error instead of the local fail-fast. The backend already validates these (traverse_modules walks AIAgent tools); this aligns the CLI early-error with it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(flows): make failure/preprocessor path test key explicit The test used `slot:` as a json! key. json! does interpolate an ident key to its variable's value (json!({slot:1}) with slot="failure_module" => {"failure_module":1}), so the test was correct and exercised the validation — but the behavior is subtle, so build the key explicitly via serde_json::Map to remove ambiguity (review nit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cache): use a UUID temp name for atomic put (shared-volume safe) Address Codex (P1): pid+counter temp names collide across container PID namespaces on a shared cache volume (same pid, PUT_SEQ resets to 0 per process), so two workers could truncate/clobber the same temp file before rename. Use a random UUID suffix (matching worker.rs's atomic-write helpers) — globally unique, so the cross-process temp-file hazard is closed. Also trims the comment to the AGENTS.md <=4-line limit (Pi nit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
0dbd9c1231 |
perf: eliminate dual-connection DB pool contention across worker, queue, and api (#9798)
* perf: eliminate dual-connection DB pool contention across worker, queue, and api Reuse the held transaction (or move pool reads before begin()) instead of checking out a second pool connection while a tx is open, extending the fix from #9789/#7861. Targets the per-worker pool (max 5) hot paths plus several server-pool API handlers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: pass owned pool to get_email_from_permissioned_as in http trigger handler The generified signature takes impl PgExecutor; the http trigger handler passed &db where db is already &DB, yielding &&Pool which does not impl PgExecutor (only surfaced under the full feature set in CI). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep RLS-exposed reads on the non-RLS pool and isolate flow-eval reads in a savepoint Addresses review of the dual-connection sweep: - worker_flow: wrap the stop_after_all_iters_if reads in a SAVEPOINT. The caller swallows the error and keeps using tx, so a DB read failure must not leave the outer transaction aborted (it would fail the later commit). Matches the previous pool-read semantics. - Revert reads that were moved onto an RLS (user_db) transaction back to the non-RLS pool, since RLS row-visibility/role context can change results: push_scheduled_job (email/tag/settings lookups; reachable with a user_db tx from api-schedule/api-flows), push_inner native-retry dedicated_worker routing (RLS isolation variants), resources.rs app-namespace folder auto-create (non-admins must not be blocked), and the script archive/delete UPDATEs. Non-RLS db.begin() reuse and move-before-begin are kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: failpoint proving the stop_after_all_iters_if savepoint isolates an aborted read Adds a worker-crate failpoints feature and a data-driven hook: when the stop_after_all_iters_if expr is the magic sentinel, the in-evaluation read runs SELECT 1/0 to abort its (savepoint) transaction. The test asserts the flow still completes (iteration marked failed) — which only holds if the savepoint keeps the outer status-update transaction committable. Without the savepoint the abort would poison the outer tx and the job would never complete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
aa098c70c0 |
perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes (#9786)
* perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: document delete_jobs auth contract and workspace-scope jobs_export purge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
12f92e3ab7 |
[ee] feat(backend): native script retry without one-step-flow wrapping (#9688)
* feat(backend): native script retry without one-step-flow wrapping Schedules and data pipelines that retry a single script previously wrapped it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a v2_job_status row, and UI projection complexity. This adds native retry on a plain JobKind::Script job. - RetrySettings: flatten Retry into a deduped retry_settings table, carried via the existing runnable_settings_handle (lazy, off the hot path). - push() materializes a bare-script-with-retry SingleStepFlow into a native Script job (gated on min-version + no handlers/retry_if). - add_completed_job re-pushes the next attempt on failure with backoff, tracking the attempt counter in v2_job_queue.extras and the chain via parent_job; schedule completion handlers fire only on the terminal attempt. - frontend: ScriptRetryChain shows the attempt chain on the run page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): native retry_if eval + per-occurrence schedule handlers Extends native script retry to the two cases that previously stayed on the one-step-flow path: - retry_if: evaluated natively on the failure path via a feature-gated windmill-jseval dep (quickjs) over the failure result + flow_input; push materializes such policies natively only when quickjs is available. - on_failure_times / on_recovery: apply_schedule_handlers now resolves each past scheduled occurrence's terminal status across its native-retry chain (root OR any parent_job=root child succeeded) and excludes the current occurrence, so the counting is per-occurrence rather than per-attempt. All scheduled-script retries now go native (schedule.rs gate removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): always materialize retry_if natively; unsupported without quickjs retry_if is evaluated by the worker (which always has quickjs), not the pusher, so gating materialization on the pusher's feature was wrong. The flow path was never a real fallback either — the flow runtime needs quickjs to evaluate retry_if too. retry_if now always goes native; on a worker without quickjs it is unsupported and fails closed (no retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): un-park asset-cascade (pipeline) retry Native retry resolves the blocker that parked pipeline retry: a retried subscriber is now a Script job (not a one-step flow / flow step), so it stays eligible for asset dispatch and can trigger its own downstream on recovery. - scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset edges (was dropped with a TODO warning). - asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job, so native-retry attempts dispatch on success while flow steps stay excluded. - tests: retry-bearing subscriber now dispatches as a native Script carrying the policy in runnable_settings_handle; native-retry attempt is eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): cap native retry interval, lazy result serialization, idempotent retry push Hardening from a self-review of the native retry path: - Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path (evaluate_retry); the exponential formula could otherwise schedule up to ~18h vs the flow path's 6h. - Serialize the failure result lazily (only when a retry_if policy needs it), so the common failure no longer pays the serialization on the failure path. - Push each retry with a deterministic id per (root, attempt). If a worker dies between enqueueing the retry and finalizing the current attempt, the reaper re-handles the attempt and lands here again — push rejects the duplicate id, so the retry is enqueued exactly once (no double-retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): defer schedule handlers idempotently on retry-push replay (review P1) Address local-review findings: - P1: retry_pending was derived from the retry push *result*, so on a worker crash + reaper replay the duplicate-id push returned Err → retry_pending flipped to false → apply_schedule_handlers fired for the non-terminal attempt (and the terminal attempt later fired them again). Pre-check whether the deterministic retry id already exists and report it as pending without re-pushing, so the handler-deferral invariant is crash-idempotent too. - P2: refresh the stale 'wrap the script in a one-step flow' comment in the asset-cascade retry push — it now materializes a native Script. - Add RetrySettings <-> Retry round-trip unit tests (clamping edges). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): native retry chain + per-occurrence status sqlx tests Close the two integration-test gaps flagged in local review: - chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts) and asserts crash-replay idempotency (the P1 fix: a replayed completion reports pending without double-enqueueing). - per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but- recovered occurrence counts as success, retries (parent_job set) are excluded from occurrence counting, and the current occurrence is excluded. - canceled_job_does_not_retry: cancellation wins over a pending retry. Runtime sqlx API (no .sqlx cache entry needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): exclude schedule handlers from the retry-attempt chain The retry chain listed all script children of the root by parent_job, but schedule completion handlers (on_failure/on_recovery/on_success) are also script children — when the occurrence has no retries, the handler's parent is the root itself, so a successful, never-retried job rendered a bogus 'Retries (1)' badge pointing at the handler. Filter children to re-runs of the same script (matching script_hash); real retries keep the root's hash, handlers run a different script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): surface schedule handlers on the run page Extend the run-page chain component with schedule completion handlers: - A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/ on_success runs that fired for that occurrence (found as children of the terminal attempt, identified by their synthetic created_by). - A handler's own run page now shows a 'Failure/Recovery/Success handler' label with a link back to the run it handled and its schedule. on_recovery and on_success share created_by, disambiguated by the recovery-only error_started_at arg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare An earlier `cargo sqlx prepare` on this branch ran before #8801's folder_default_permissioned_as test merged in, so it pruned the 3 query caches that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main. * fix(backend): only cascade assets from native retry attempts, not handlers (review P1) is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script child became asset-eligible — including schedule/error/recovery handlers (Script jobs with parent_job set and no flow_step_id). A handler that declares assets would then trigger a cascade the old parent_job IS NULL guard prevented. Gate parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as its chain parent (handlers run a different script). Runtime query, no sqlx cache. * fix(backend): cache the private-gated retry_setting asset-dispatch test query The same prepare-without-private that dropped the folder_default caches also pruned the cache for the retry_setting_dispatches_subscriber_as_native_script test query (asset_trigger_dispatch.rs:721). Regenerated with --features private. * fix(backend): exclude handler children from per-occurrence recovery (review) A scheduled occurrence's on_failure/on_success handler runs as a successful child (parent_job = occurrence), and the per-occurrence success EXISTS counted ANY successful child — so a failed occurrence whose error handler succeeded was marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in the merge) and on_failure_times counting. EE query now scopes the EXISTS to same-runnable children (only native retry attempts); regenerate sqlx cache + bump ee-repo-ref. native_retry_test gains a handler-child regression case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): scheduled-script retry is a native Script, not SingleStepFlow test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a native Script. Update both to assert kind='script' and that the retry policy is carried via runnable_settings_handle. * fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic) Address cubic CI review: - P1: the SingleStepFlow->native Script materialization dropped dedicated_worker, so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve it from the script row in push so the materialized Script keeps the dedicated tag. - P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the u32->i16 // retry count narrowing (scripts.rs) instead of wrapping. * fix(backend): use a retry-specific signal, not runnable equality (codex review) Address Codex CI review: - P1: is_native_retry_attempt treated any same-runnable parented Script child as a retry. WAC v2 inline children have that exact shape, so an inline child of an asset producer would cascade. Use a retry-specific signal instead: the job carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no flow_innermost_root_job. Apply the same flow_innermost guard to the EE per-occurrence EXISTS (WAC inline children must not count as a recovery). - P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate now resolves as 'retry pending' (re-check on the duplicate-id error) instead of flipping retry_pending to false and firing handlers early. - Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): explicit native_retry_attempt marker, drop heuristics Replace the per-site "is this a retry?" inference (parent_job + runnable match + flow_innermost / retry_settings) with one explicit marker: a sparse native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker also carries the attempt counter (previously in v2_job_queue.extras), so it's the single source of truth. - asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker. - EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost. - maybe_enqueue: reads/writes the marker (persistent) instead of queue extras. - Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap. - Eliminates handler / WAC-inline-child misclassification by construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex) The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but the periodic monitor.rs path deleted v2_job rows without it, orphaning markers. Add the same WHERE job_id = ANY(...) sweep there. * fix(backend): widen native_retry_attempt.attempt to integer (cubic) The smallint column was cast to/from u32 and could wrap a retry chain longer than i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32 attempt count, so no narrowing occurs on the maybe_enqueue read/write path. * feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT - Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi), computed from the native_retry_attempt marker. The run-page chain now filters retry attempts by is_retry instead of the script_hash heuristic, so WAC v2 inline children (same script, parent_job) no longer render as retries (codex). - Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep the plain catch-and-continue matching the other side-table deletes; the table is created by a startup migration so it always exists when cleanup runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but does not select is_retry, which errored with "no column found". Only the list endpoint populates the marker; #[sqlx(default)] lets every other query omit the column and default to None. * feat(backend): select is_retry in single-job GET too for consistency The list endpoint already exposes the marker; populate it on the single-job GET (both completed and queued variants) as well so a run loaded directly reflects its retry status. #[sqlx(default)] stays as a safety net for any other query. * feat(backend): reap orphaned native_retry_attempt markers via periodic sweep The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so direct job deletions (workspace/job delete, schedule clearing) would leave marker rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which must then be remembered for every future path), reap orphans in the periodic delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the job). The table is sparse so the anti-join drives off it and probes v2_job by PK — cheap. Retention still sweeps markers inline (keeps the table small so this stays cheap); a transient orphan is harmless (nothing reads is_retry for a gone job). * fix(frontend): include flow handlers in retry chain handler row (codex) Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...), whose handler job is a flow, not a script. The chain fetched children with jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts are still selected by is_retry and handlers by created_by, so both kinds surface. * fix(backend): carry concurrency/debouncing settings into native retries maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings ::default(), dropping the script/pipeline concurrency settings the failed job carried in its runnable_settings_handle. A retry of a concurrency-limited script then inserted no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and pass them in the payload, which push forwards to the materialized retry. Adds a regression test asserting the retry's handle resolves to the concurrency settings. * fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex) P1a (schedule.rs): the scheduled-retry materialization fetched the script's concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings ::default() into the SingleStepFlow payload, so the root attempt's handle held only the retry policy and the whole chain ran unbounded. Pass the fetched settings. Regression test asserts the root handle resolves to retry + concurrency. P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract — it is pub only for the integration test; the sole production caller is the worker completion path passing a DB-derived, already-authorized MiniCompletedJob. * docs(backend): attach native-retry auth contract to the function itself (codex) The doc block was merged with eval_retry_if's doc and bound to that function, leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if keeps its own doc; the native-retry + authorization contract now sits directly above maybe_enqueue_native_script_retry. * docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex) - Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since 1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the served spec couldn't see the field. Now current at 1.739.0. - schedule.rs: a retry_if gate is evaluated at failure time and fails closed without quickjs (no retry); it does not fall back to a flow path. - windmill-types jobs.rs: is_retry is selected by both the list and single-job GET endpoints (not list-only). * docs(backend): fix remaining stale retry_if/quickjs comments (codex) The retry_if block and the push materialization comments claimed push keeps retry_if on a flow path / the worker always has quickjs. The code always materializes native retry and the no-quickjs eval_retry_if path fails closed — correct the comments to that constraint. * docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex) - Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated and the job does not retry (no one-step-flow fallback). - jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence failure/recovery status across the retry chain, so the old 'restricted to schedules whose handlers don't need per-occurrence counting' claim is dropped. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
cd42c6ca18 |
fix: decrypt secret variables via external backend in common resolvers (#9784)
`get_variable_or_self`, `get_variable_or_self_as`, `get_secret_value_as_admin` (and `transform_json_unchecked`'s `$var:` branch) in windmill-common always ran the raw `variable.value` through `decrypt()`. With an external secret backend (HashiCorp Vault / Azure Key Vault / AWS Secrets Manager) configured, that column holds a `$vault:`/`$azure_kv:`/`$aws_sm:` marker rather than base64 ciphertext, so base64 decoding failed with `Invalid byte 36, offset 0` (the `$`). This broke GitHub App git sync (git_sync_ee.rs) and any other consumer of these resolvers when an external backend is active. Move backend resolution (`get_secret_backend`, `get_secret_value`, `is_*_stored_value`, caching) into `windmill-common::secret_backend::resolver` so the low-level variable resolvers can route external markers through the configured backend's `get_secret()` instead of `decrypt()`. The windmill-store and windmill-api `secret_backend_ext` modules now re-export these from windmill-common (single source of truth / single backend cache) and keep only their write-side helpers. No `_ee.rs` files change. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Diego Imbert
|
170cd79aaf |
fix: allow hyphens in postgresql database name validation (#9782)
* fix: allow hyphens in postgresql database name validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: cover hyphen acceptance in validate_dbname Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
f6998ec54c |
feat: data tests for ducklake pipeline materialization (#9708)
* feat: data tests for ducklake pipeline materialization Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): data_test count badge on pipeline graph nodes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: surface annotation badges (incl. data_test) on deployed pipeline nodes Backend graph endpoint now parses each pipeline member's deployed body and returns partition/freshness/tag/retry/data_test, so badges render on deployed nodes, not only live drafts. Aligns the TS DataTest.relationships fields to snake_case to match the Rust serde wire shape (the type is now populated from both the parser and the backend JSON). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): keep materialize output edge when editing the producer in the pipeline graph The live-edit overlay re-derived a selected/edited script's lineage from // on inputs + body-inferred assets only, so the // materialize <asset> output (an annotation, not body SQL) was judged stale and its write-edge dropped on select — leaving the materialized asset unlinked (and the node's annotation badges hidden). Include the parsed materialize target in liveRefKeys and the draft writeOuts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: run all data tests in one pass with a structured per-test result Replace the raise-on-first-violation probes with a single materialize summary that embeds every test's violating-row count in a data_tests column (computed in a CTE, since DuckDB rejects subqueries inside struct literals). The worker reads the breakdown and decides pass/fail: a clean run returns the per-test summary in the result; a failing run errors with the FULL list (every test, ✓/✗ + counts), not just the first failure. Verified live (EE) for built-ins + custom, pass and multi-failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): data-test pass/fail checklist in the job result DisplayResult renders a per-test checklist (✓/✗ + violation counts) above the raw result for managed materialize runs — from the structured data_tests on success, and parsed from the worker's breakdown message on failure. Shows in the script editor Test panel, the runs page, and the pipeline asset run pane. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): move data-test badge onto the producer→asset edge with run status The test badge now sits on the write-edge (the transformation link) rather than the producer node, since the tests assert on what the transformation produces. It's tinted by the producer's last-run status (green = passed, red = a test failed) and its hover title lists every declared test. Removes the now-redundant node badge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): render custom data-test scripts as their own clickable graph nodes A // data_test <script_path> custom test now appears as its own node below the asset it validates, joined by a dashed 'tests' edge. Clicking it opens the test script in the detail pane (dispatched like any runnable). Built-in tests stay folded into the edge badge; only script-backed tests become nodes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): type data-test edge field via AssetGraphResponse, not in-scope g BuiltEdge is declared at component scope, outside build(g), so referencing typeof g.runnables in its type failed CI's svelte-check (Cannot find name 'g'). Use the imported AssetGraphResponse type instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): anchor edge badge on routed path + a11y text on test icons Address review: the data-test edge badge anchored on the straight-line midpoint, floating off detoured edges — anchor it at detourX when the edge is routed through a gutter lane. Add sr-only pass/fail text so the checklist icons are distinguishable to screen readers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: close data-test enforcement bypass + gate badges to scripts + reject multi-stmt custom tests Address review (cubic) findings: - P1: managed materialize generates its own summary row carrying data_tests, and enforcement reads that column — but a // result_collection annotation (e.g. a scalar mode) could reshape the row and drop data_tests, silently bypassing a failing test. Force LastStatementAllRows for managed materialize runs so the summary row is always intact. - P2: asset-graph annotation badges were keyed by path only, so a flow sharing a path with a pipeline script inherited its badges. Gate the lookup on usage_kind == Script. - P2: a custom test body is embedded as a subquery, so a multi-statement body produced invalid SQL with an opaque DuckDB error. Validate single-statement up front with an actionable error; align docs/comments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: fail loud if fewer data-test outcomes recovered than declared Defense-in-depth from the fresh-context review: enforcement reads per-test outcomes off the materialize summary row, but if the data_tests column were ever dropped/reshaped at the FFI boundary, extract_data_tests would return fewer (or zero) outcomes and the run would silently pass unverified tests. Track the embedded test count on MaterializeExec and abort with a clear error when recovered < declared. Verified: normal run (4==4) unaffected; the scalar-result_collection bypass already fails. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: relationships data test same-lake reuse + schema-qualified target quoting Address Codex/Pi review (two P1s in the relationships codegen): - A relationship into the same ducklake as the materialize target minted a second ATTACH of that lake under _wm_ref_N while _wm_target already held it — DuckDB forbids attaching one database twice, so the test failed before it could run. Reuse _wm_target for same-lake references. - A schema-qualified target (ducklake://warehouse/main.dim_products.sku) emitted FROM _wm_ref_0."main.dim_products" — one quoted identifier with a literal dot — silently querying a nonexistent table. Quote each dotted segment so the dot stays a schema separator. Adds tests for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): refresh data_test badge on deployed-script drafts + scope to materialize target Address Codex review nits (both P2): - resolveGraph: the existing-runnable draft-overlay branch kept the deployed data_tests, so adding/removing // data_test lines on an already-deployed script left the badge stale until redeploy. Refresh it from the live parse like the new-runnable branch. - AssetGraphCanvas: data tests were attached to every write-edge from a producer. They assert on the // materialize target (always a ducklake asset in v1), so only the ducklake write-edge now carries the badge and custom-test nodes — a producer's other (S3/datatable) outputs no longer show them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
d131d754e1 |
feat: ducklake time-travel UX (snapshot history + AT VERSION reads) (#9709)
* feat: ducklake time-travel UX (snapshot history + AT VERSION reads) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: catalog-qualify ducklake time-travel FROM hints (lake. prefix) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: render ducklake snapshot_time (microseconds since epoch) correctly Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: merge ducklake History + Query into one master-detail tab Snapshot list (left) selects the version previewed in the read-only grid (right); newest auto-selected. Copy-clause moved to the preview's SQL line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scope ducklake snapshot history to the table's versions Catalog-wide snapshots predate a table's creation; previewing AT a version before the table existed errored ("Table ... does not exist at version N"). The DUCKLAKE_SNAPSHOTS marker now takes the table and lists only snapshots from its first creation onward. Also: narrower snapshot-list pane on large screens (target a fixed width, not a fixed fraction). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: load ducklake preview columns at the pinned version + reset on asset switch Addresses CI review (codex/pi P1, cubic P2): - Historical previews loaded current-schema columns, so an AT(VERSION) read enumerating a column added in a later snapshot failed. Now DESCRIBE-loads the column set at the pinned version; the read is gated on columns matching the current version to avoid a stale-colDefs race on version switch. - selectedVersion no longer sticks across assets: the panel is keyed on path (remounts per asset) and effectiveVersion falls back to newest when the pick isn't in the current list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: match History tab UI (master-detail, full-FROM copy) after merge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: handle catalog-only ducklake asset paths (no table segment) parseDbInputFromAssetSyntax threw on a catalog-only path like 'ducklake://main' (undefined.split('.')) — a real graph node (e.g. a consumer of the whole catalog). It now returns a table-less input instead of throwing, and DucklakeAssetPanel renders only the partition grid (no per-table history/time-travel) for table-less nodes. Adds parser unit tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: escape ducklake catalog name in client-built time-travel DESCRIBE fetchDucklakeColumnsAtVersion interpolated the catalog name into an ATTACH string literal without escaping; double single-quotes (mirrors backend escape_sql_literal) so a quote-containing catalog name can't break out. Also fixed the v1.x docs checklist line to match the shipped full-FROM copy affordance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
b5bd8245d8 |
fix: reject symlink traversal in job-dir path validation (#9713)
* fix: reject symlink traversal in job-dir path validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: cover dangling symlink in job-dir path validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: close symlink-traversal bypass via in-bounds `..` in path check Walk the normalized relative path instead of raw user components, so an in-bounds `..` (e.g. `foo/../evil/payload`) can no longer drift the walk past a planted symlink. Adds regression coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
ace7b68a28 |
fix: sanitize git credentials from ansible executor errors and logs (#9697)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
3ebf24359d | feat: ducklake materialization for data pipelines (#9689) |