mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-09 16:05:42 +00:00
e052d5ee4598332a6491b653d70fc82a08642d61
993
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
2ce21c9ef8 |
feat(git-sync): enable per-item promotion mode on dev workspaces (#10205)
* feat(git-sync): enable per-item promotion mode on dev workspaces Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: keep unrelated git-sync Alert copy at its original wrapping Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): fall back to parent_path on empty deploy path + bump ee ref computeGitSyncDeployBranch used ?? so a backend-serialized empty path (rename out of the repo filter) skipped the deploy branch and could commit to the tracked base; use || to fall back to parent_path like the backend. Bumps ee-repo-ref for the single-object promotion_open_prs fix (windmill-ee-private#679). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): route dev-promotion non-branchable objects off the tracked base user/group objects (and any unresolvable ref) returned null in promotion mode, so a dev-workspace deploy pushed them straight to the parent's tracked branch. Fall back to the dev's env-label branch instead; the backend opens no PR for them (isolated, not promoted). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(git-sync): dev-workspace promotion via a toggle on the inherited repo A dev workspace reuses the single repo it inherited from prod: a 'Promote to prod via Git' toggle flips it between sync mode (deploys to the dev branch) and promotion mode (per-item wm_deploy/** PRs to prod), with a per-item/per-folder sub-toggle. Removes the redundant separate-promotion-repo setup for dev workspaces. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(git-sync): dev-promotion regression test + widen git_sync_e2e path filter Adds a CLI integration case covering dev-workspace promotion (script -> wm_deploy branch; user/group -> env-label branch, main never touched). Widens the git-sync-test.yml relevance filter to the deploy-branch derivation, git-sync guard, and CLI git-deploy files so the e2e suite runs on PRs like this one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): gate dev promotion toggle on EE, fix card mode + workflow path filters Codex review: (1) show the dev promotion toggle only under an active EE license and revert the optimistic save if the backend rejects it; (2) derive the dev card's display mode from use_individual_branch so promotion copy shows in promotion mode; (3) mirror the new relevance paths into the workflow's top-level push/pull_request filters so it actually triggers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): only use the single-card dev promotion UX when the dev has one repo Codex review: an attached dev workspace keeps its own repositories rather than inheriting prod's. Gating the single-card + toggle + hidden-secondaries UX on repositories.length <= 1 makes a multi-repo attached dev fall back to the normal layout, so no active repo is hidden and an unrelated repo isn't presented as prod's promotion target. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): runtime EE-plan gate for promotion mode, consistent with auto-pull/PR Codex review: promotion mode only had the CE compile rejection, while auto-pull and PR creation runtime-gate on the active plan (check_git_sync_ee_license). Add check_promotion_license and call it from both edit_git_sync_config and edit_git_sync_repository, plus the matching CE rejection on edit_git_sync_config so the two endpoints are symmetric. Promotion is now gated like every other git-sync EE setting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): dev promotion must reuse the parent workspace's repository Codex review: repository count doesn't prove a dev inherited prod's repo — an attached dev keeps its own. check_dev_promotion_targets_parent_repo resolves the promotion repo's URL and rejects enabling promotion unless it matches one the parent (prod) tracks, so branches/PRs can't target an unrelated repository. Called from both git-sync edit endpoints. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): dev promotion save-time check uses shared parent-repo matcher (url+branch) Delegates to windmill_common::git_sync_ee::dev_promotion_target_matches_parent so the settings gate and the deploy-time safety net share one url+branch identity check. Bumps ee-repo-ref for the EE deploy-time enforcement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for private resolve_repo_url_and_branch Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref for promotion-target matcher authz doc Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(git-sync): bump hub scripts to gitsync-cli versions, fix promotion tooltips Point LATEST_GIT_SYNC_SCRIPT_PATH (28790 -> 28796) and GIT_SYNC_PULL_SCRIPT_PATH / gitInitRepo (28789 -> 28795) at the hub versions pinning windmill-cli@1.763.1-gitsync.0, which carries the dev-workspace promotion routing. Slugs unchanged, so the GitHub-App token check and hub script cache are unaffected. Tooltips: enabling promotion pushes a PR-ready wm_deploy/** branch; Windmill only opens the pull request itself when automatic pull requests are enabled. Reword both toggles to stop promising a PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): dev promotion mirrors to the env-label branch, PR toggles exclusive by branch type Bump ee-repo-ref for the dispatcher changes: a promotion dev's deploys now also push to its env-label branch (one extra mirror job per batch, users/groups mirror-only), and `fork_open_prs` no longer applies to a dev in promotion mode where `promotion_open_prs` governs. Frontend: the fork-PR toggle tooltip states its actual coverage (wm-fork/** and the dev branch of a dev workspace) and that a promotion dev's own pull request toggle takes over for wm_deploy/** branches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): reject dev promotion on pre-28796 pinned sync scripts An older pinned sync script bundles a CLI that force-disables per-item branches on every fork, so enabling promotion on a dev workspace with such a pin would silently keep deploying to the env-label branch. Both git-sync edit endpoints now reject the combination with an actionable error; the EE dispatchers (via ee-repo-ref bump) demote inherited configs to promotion-off semantics so markers, branch keys and the mirror match the branch the CLI actually pushes. Roots and auto-managed repositories are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): serialize dev promotion toggle saves The promotion and per-folder toggles persist immediately via whole-repo saves; leaving them interactive while one is pending lets rapid flips race, and the earlier save (enabling runs extra backend checks) can commit last, silently reversing the state the UI shows. Both toggles now disable while a save is in flight. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(git-sync): lock auto-PR toggle during promotion save, rename-out branch routing Frontend: the automatic-PR toggle is revealed by the promotion toggle's in-flight save; an edit made mid-save was absorbed into the saved baseline without reaching the backend. It now disables during that save. EE (ee-repo-ref bump): dispatcher debounce/concurrency keys and PR markers follow the CLI's parent_path fallback for rename-out items, so their wm_deploy/** branches debounce per-branch and open their PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(git-sync): condense comments to durable constraints Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 8bf73f803158bcbf7b8d55a36f4a1ebfcc1bbcd9 This commit updates the EE repository reference after PR #679 was merged in windmill-ee-private. Previous ee-repo-ref: c2cd718cb53d234f909f485bd7cd43ed9605ffd1 New ee-repo-ref: 8bf73f803158bcbf7b8d55a36f4a1ebfcc1bbcd9 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
68debab877 |
feat(triggers): add AMQP (RabbitMQ) trigger via lapin (#10230)
* feat(triggers): add AMQP (RabbitMQ) trigger using the lapin library Fixes WIN-2214 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(triggers): defer AMQP cross-workspace deploy pending utils-internal publish Revert the amqp_trigger additions to the shared windmill-utils-internal TriggerDeployKind and the frontend cross-workspace deploy adapter: the frontend installs the published npm package, which lacks the new kind until a release is cut. AMQP create/edit/delete/list/sync/capture are unaffected (they use local types); only cross-workspace deploy/merge of AMQP triggers waits on the package bump. Also document the at-most-once ack in the consumer loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): address AMQP review — at-least-once ack, workspace cascade, contracts - ack AMQP deliveries only after successful dispatch; nack+requeue on failure - add ON DELETE CASCADE workspace FK so amqp_trigger rows are cleaned on workspace deletion (and the listener stops) - fix the /amqp_triggers/test OpenAPI body and add amqp_trigger to WorkspaceDiffRow.kind - register AMQP in the generated workspace trigger tool (create_trigger) - drop banned $bindable defaults on optional props in the config section - add build_uri unit tests (encoding, ports, vhost) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): stop AMQP poison-message loop and reconnect on transient drops Chaos testing against a live RabbitMQ broker showed the previous nack(requeue) + immediate re-poll spun a tight redelivery loop (~1000 critical-error reports/sec) on a poison message, and any connection blip permanently disabled the trigger (lapin has no built-in reconnect). - on dispatch failure: nack+requeue then stop consuming; the listener framework re-lists the trigger after its ping goes stale (~15s), backing redelivery off to that cadence instead of a tight loop (verified: rate dropped from ~1000/s to ~1 per ~26s, message preserved) - on connection/stream error: stop and let the framework reconnect instead of disabling; persistent failures are still disabled via get_consumer (verified: a forced connection close now auto-reconnects and resumes) - finish the AI create-trigger action wiring for AMQP: add amqp to CreatedResourceTriggerKind, the action-card registry, and the drawer registry so the result card renders and its "Open" action works Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP frontend registries and defer merge rows - add amqp to capturableTriggerTypes (so AmqpCapture mounts), the Runs jobTriggerKinds filter, and CLOUD_DISABLED_TRIGGER_TYPES - wire AMQP into global AI chat mode: TRIGGER_KINDS, the request union, writeTriggerSchema, triggerServices, and the draft adapter - stop emitting actionable AMQP fork-comparison rows (revert amqp_trigger from TRIGGER_OR_SCHEDULE_TABLES) since cross-workspace deploy is deferred until windmill-utils-internal is published — avoids a deploy that fails with "Unknown kind: amqp_trigger" - use design-system TextInput instead of raw <input> in the config section Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP session/draft registries and constrain prefetch - add amqp to the session-deploy, draft-compare, preview-router, and copilot workspace-item registries so AMQP drafts/deploys/nav/path resolution work - include amqp_count in the MoveDrawer attached-trigger rename warning - replace the raw prefetch <input> with a design-system TextInput bounded to an integer 1-65535 (backend u16) and block save on invalid values Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): make AMQP disconnect/reconnect consistent with the Kafka trigger lapin, like rdkafka, has no transparent reconnect, so the AMQP listener now mirrors the Kafka trigger's explicit reconnect loop instead of relying on the framework re-list (which disabled the trigger once get_consumer failed on a sustained outage): - get_consumer returns cheaply; consume owns a (re)connect loop that retries with a 30s backoff, reports a critical error every 10 failed attempts, and reports a recovered critical error once it reconnects — never disabling the trigger on a connectivity failure - a consumer/stream error breaks out to reconnect rather than disabling - dispatch failure still nacks+requeues (at-least-once) with a short backoff to avoid a tight poison-message loop, keeping the connection alive Verified against a live RabbitMQ broker: killing the broker keeps the trigger enabled and retrying (attempt N), and restarting it auto-reconnects (logs "reconnected after N attempts") and resumes dispatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP capture registries and constrain prefetch contract - add the 'amqp' case to triggerKindToTriggerType so opening the AMQP editor from a capture button no longer throws "Unknown TriggerKind: amqp" - register AmqpIcon in CaptureTable's icon map and add an AMQP entry to the script/flow CaptureButton menu - bound the OpenAPI prefetch_count to an integer 1-65535 (matches the Rust u16) and regenerate clients/prompts - require a non-empty exchange name when the exchange binding is enabled - build_uri: fall back to "/" on a blank vhost and bracket IPv6 hosts (+ tests) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(triggers): wire AMQP into pipeline graph, git-sync, and preprocessor types - asset_graph: discover attached amqp_trigger rows and emit an AMQP TriggerEdge so AMQP triggers render (and can be opened/deleted) on the data-pipeline canvas - frontend pipeline graph: add amqp to NativeTriggerKind, the add-trigger menu, node presentation, event-trigger set, annotation keywords, and the editor/service registrations - git-sync: add the amqp_trigger include pattern (+ test) so an AMQP git-sync deployment stages only its .amqp_trigger.* file, not an unrelated same-path object - preprocessor starters: add the AMQP event to the generated TS/Python/PHP trigger event types (kind/payload/exchange/routing_key/queue_name/redelivered/ delivery_tag) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): finish AMQP pipeline/parser wiring, prefetch validation, source lists - fix a stray edit that corrupted the pre-existing MqttTriggerEditor import ($lib/... path) in PipelineTriggerEditors.svelte - reject prefetch_count = 0 server-side in validate_config (RabbitMQ treats 0 as unlimited) and defensively skip basic_qos(0) in build_consumer (covers the capture path that bypasses CRUD validation) - recognize `// on amqp` in the canonical parser (TriggerSpec::Amqp) and add amqp to the CLI non-autorun/event-trigger sets so a pipeline cascade never runs an AMQP-only node as a manual root without an event - add amqp to the preprocessor intro lists and both pipeline AI instructions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): reject zero AMQP prefetch in all paths and finish guidance lists - extract a shared validate_amqp_options used by both CRUD validate_config and build_consumer, so capture configs (which bypass CRUD validation) also reject prefetch 0 instead of silently connecting with an unlimited buffer (+ unit tests for 0/1/65535/None) - add AMQP to the main script-writing preprocessor-sources prompt and the CLI triggers-skill guidance list Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(triggers): de-duplicate AMQP prefetch comment and fix GET response text - keep the zero-prefetch rationale only on the shared validate_amqp_options doc; drop the redundant call-site comments - correct the getAmqpTrigger OpenAPI 200 description ("deleted" -> "retrieved") Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to aaa6cb89b05b76139252c64f057e53b94d12ac60 This commit updates the EE repository reference after PR #680 was merged in windmill-ee-private. Previous ee-repo-ref: 5da5fd65aca9594b2611837a52e4677b544b0380 New ee-repo-ref: aaa6cb89b05b76139252c64f057e53b94d12ac60 Automated by sync-ee-ref workflow. * chore(migrations): consolidate the four AMQP migrations into one The table and the three enum ADD VALUE statements (trigger_kind, job_trigger_kind, draft_kind) are one atomic feature. ALTER TYPE ... ADD VALUE runs inside the migration transaction on PG >= 14 (Windmill's minimum) since the amqp_trigger table doesn't reference those enum types, so they can share a single migration instead of four. Verified applying cleanly in a single transaction on a fresh DB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
fd51d40f12 | feat(pipelines): catalog declared measures and dimensions (#10190) | ||
|
Ruben Fiszel
|
ddec2abbb3 |
feat(jobs): cap total queued jobs per workspace on cloud (#10218)
* feat(jobs): cap total queued jobs per workspace on cloud A workspace could flood the queue with an unbounded number of jobs across many concurrency keys and scripts (or keyless jobs), which the per-key cap from #10197 does not bound. Add a companion instance-wide ceiling on a workspace's total queued jobs. check_workspace_queue_cap rejects a push once the workspace has WORKSPACE_MAX_QUEUED_JOBS (default 20000, superadmin-configurable, 0 to disable) jobs queued, cloud-only and runtime-gated on CLOUD_HOSTED like the per-key cap. It runs on every push, so it applies even to premium workspaces and catches parallel for-loop floods. Jobs already queued still drain; only new pushes past the ceiling are rejected, so an in-flight flow only fails to push further work while at the ceiling. The setting loader self-gates on CLOUD_HOSTED so it is never loaded off cloud, from initial load or a settings-change reload. The depth count is bounded by the cap via LIMIT so a runaway backlog never costs an unbounded scan on the push path. * docs(jobs): note the workspace cap is a soft ceiling and the depth helper is count-only Records the two review points as constraints: the cap does not serialize admission (a soft ceiling by design, like the per-key cap), and workspace_queue_depth is pub only for the test, returns a count not job data, and leaves authorization to the caller. |
||
|
|
11fda89b52 |
feat(telemetry): generic feature-usage telemetry with AI session metrics (#10200)
* feat(telemetry): add generic feature_usage table and batched logging endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(telemetry): log AI session usage events and document them in telemetry settings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): use escape sequence instead of literal NUL bytes in buffer key Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): validate dimensions, decouple retention, keepalive flush Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): allowlist feature-usage dimensions and index retention scans Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): pin tool-name allowlist and deploy session attribution Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(telemetry): route AI chat usage through feature_usage and drop ai_chat_usage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(telemetry): slim dimension validation to registered kinds plus key shape Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): backfill ai_chat_usage into feature_usage before dropping it Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): disclose provider and model identifiers in telemetry settings text Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(telemetry): issue all flush chunks before awaiting so pagehide keeps them Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 6306c072a50937ea9af44a5bcf42345543207486 This commit updates the EE repository reference after PR #672 was merged in windmill-ee-private. Previous ee-repo-ref: 964f242a0eb44db7f7d26636cc8d76aeabea2b73 New ee-repo-ref: 6306c072a50937ea9af44a5bcf42345543207486 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
Ruben Fiszel
|
71f2d47cb4 |
feat: cap queued jobs per concurrency key on cloud (#10197)
* feat: cap queued jobs per concurrency key on cloud * fix: close preprocessed-flow bypass and bound concurrency cap scan * fix: only cap concurrency keys with an active concurrent_limit * chore: only load concurrency key cap setting when cloud hosted * fix: reject queued-job import on cloud |
||
|
Ruben FiszelandClaude Opus 4.8
|
f32d7702bc |
fix(inputs): add ownership check to saved-input args read endpoint (#10194)
`get_args_from_history_or_saved_input` looked up saved-input rows by `id + workspace_id` only, without checking ownership or `is_public`. The `input` table has no RLS policy, so any authenticated workspace member could read another member's private saved-input args (which may hold credentials) just by knowing the UUID, even though `inputs/list` already scopes to `is_public OR created_by = caller`. Add `AND (is_public IS true OR created_by = $4)` to both `input`-table queries in the handler (the `?input=true` single-query branch and the default `v2_job UNION ALL input` branch), matching the predicate already used by `list_saved_inputs`. The `v2_job` branches are already covered by the table's RLS policies enforced via `SET LOCAL ROLE windmill_user` in `UserDB::begin()`, so they are left unchanged. Fixes WIN-2202 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
c82056cfde |
fix(schedules): stop disabling schedules on transient push errors (#10179)
* fix(schedules): stop disabling schedules on transient push errors A scheduled flow whose next-occurrence push failed after retry exhaustion used to be disabled, killing a healthy schedule over a transient DB blip (pool contention, statement timeout). Now that the unarmed-schedule reconciler exists (#10174), transient failures no longer disable: the current occurrence runs to completion and the reconciler re-arms the next occurrence once this run leaves the queue. In the flow schedule-push path after retry exhaustion we now branch on the error: QuotaExceeded/NotFound still disable (the schedule's own fault, and rearm_schedule would otherwise leave them enabled-yet-unarmed forever), while transient errors are only reported and the flow continues. The previous iteration returned a SchedulePushZombieError to force a zombie restart; that is removed, because zombie detection cancels (does not restart) same-worker flows, so it would have lost the current run of a same-worker scheduled flow. The now-obsolete SchedulePushZombieError type and its catch in worker.rs are deleted. Fixes WIN-2198 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(schedules): back off and surface repeated reconciler re-arm failures The unarmed-schedule reconciler retried a schedule that could not be re-armed on every pass, forever, logging only to the server. With the flow schedule-push path no longer disabling on non-transient errors, a persistently-broken push (bad stored cron/timezone/args, lapsed license key) now stays enabled and would spin in that loop silently. The reconciler now tracks consecutive re-arm failures per schedule: exponential back-off (2, 4, 8, … passes, capped) between retries so a broken schedule is not hammered, and after 3 consecutive failures it surfaces the cause once (records schedule.error + raises a critical alert) without disabling. Both reset the moment the schedule re-arms, which also clears the recorded error. Verified end-to-end on a running server: a flow schedule with a corrupted cron stays enabled, retries back off, the error is surfaced after the third failure, and it re-arms and clears the error once the cron is fixed. Fixes WIN-2198 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
2ff5a918d5 |
feat(worker-tags): add * fork marker to workspace-scoped custom tags (#10177)
* feat(worker-tags): add `*` fork marker to workspace-scoped custom tags Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(worker-tags): pin fork-marker tag admission through real lineage Also render WorkspaceMatcher in its authored form in the operator-facing "not in the allowed CUSTOM_TAGS" error, and correct the authorization note on workspace_with_fork_ancestors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(worker-tags): gate fork-lineage tag lookup on workspace membership exists_workers_with_tags is a global route whose workspace is a query param. Resolving its fork lineage for a caller who is not a member disclosed whether an arbitrary workspace descends from one named by a tag(parent*) rule. Require membership before the lineage lookup, and sharpen the workspace_with_fork_ancestors contract accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(sqlx): add offline cache for fork-marker test INSERT query The new test_fork_marker_tag_admission_through_lineage integration test inserts a fork workspace with a bind-parameterized query that had no cached entry, failing the SQLX_OFFLINE=true CI build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
9762089fcb |
fix(schedules): re-arm enabled schedules left with no queued occurrence (#10174)
* fix(schedules): re-arm enabled schedules left with no queued occurrence * fix(schedules): lock schedule row while re-arming and report outcome * fix(schedules): make reconcile lock cancellation-safe, re-check armed under lock Address review feedback on the schedule reconciler: - Use a transaction-scoped advisory lock (pg_try_advisory_xact_lock) instead of a session-scoped one. monitor_db runs under a 600s timeout; on cancellation a session lock on a pooled connection would be stranded, wedging reconciliation on every replica. An xact lock releases when its transaction is dropped. - rearm_schedule re-checks for a queued occurrence under the row lock and returns NoOp if already armed, closing the scan→lock window that could double-push across a cron boundary. Add a regression test. - Make reconcile_unarmed_schedules private (its only caller is in monitor.rs) and document its system-only contract. - Log the disable only after the guarded UPDATE actually disables the schedule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(schedules): never disable from reconciliation and cap re-arms per pass --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
97f4477069 |
feat(forks): let a fork's creator manage developers on it without being an admin (#10166)
* feat(forks): let a fork's creator manage developers on it without being an admin Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(forks): bind the fork-creator grant to the member's parent username and lock the delete Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
51d8db6602 |
feat: automatic git-to-windmill sync (polling, webhooks, in-app PRs + checks) (#9552)
* docs: add design doc for automatic git-to-windmill pull sync
* docs: add migration plan and implementation phases to git-sync pull design
* feat(git-sync): add auto_pull settings schema and pull enqueue primitive
Adds AutoPullSettings/AutoPullMode/AutoPullStatus on GitRepositorySettings
(workspace_settings.git_sync JSONB), the GIT_SYNC_PULL_SCRIPT_PATH constant,
and should_pull/effective_poll_interval_s helpers with unit tests. Exports the
EE enqueue_git_pull_job primitive. Foundation for repo→Windmill auto-pull.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): poll repos and auto-pull new commits into the workspace
Phase 1 of automatic repo → Windmill sync. A monitor task (EE-licensed,
single-replica via advisory lock) git ls-remotes each auto-pull-enabled
repository ~every minute and enqueues a pull when the tracked branch moves,
reusing the {workspace_id}:git_sync concurrency key so pulls serialize with
in-flight push commits.
- windmill-store: background (no-authed) resolver get_git_repo_head_for_autopull
that resolves the repo resource (incl. $var: refs) and ls-remotes; GitHub-App
repos are skipped here and will sync via webhooks (phase 2).
- monitor.rs: poll/reconcile/persist with optimistic sha advance and failure
status; targeted jsonb update so concurrent settings edits aren't clobbered.
- edit_git_sync_repository: preserve server-owned auto_pull state on UI save.
- openapi: AutoPullSettings/AutoPullMode/AutoPullStatus + auto_pull field.
- frontend: per-repo "Automatically deploy changes from Git" toggle with last
sync status; demote the GitHub Actions link to an advanced CI option.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): wire webhook lifecycle + receiver; share reconcile logic
OSS side of phase 2 auto-pull webhooks:
- edit_git_sync_repository creates/removes the repo webhook on save (EE-gated,
best-effort → falls back to polling).
- monitor poller now delegates to the shared windmill_git_sync reconcile/persist
helpers (also used by the webhook receiver), removing duplicated logic.
- export the shared reconcile/persist/failure helpers; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref for phase 3 in-app PR creation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): show webhook vs polling status on the auto-pull toggle
When a repo has an active webhook (auto_pull.webhook_id set), the status line
reads "instant via webhook"; otherwise it reads the ~1-minute polling cadence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): post PR diff check on dry-run completion (phase 4)
Worker completion hook in process_completed_job: when a DeploymentCallback job
carrying the __git_sync_pr_check marker finishes, parse the dry-run SyncResponse
and patch the GitHub check run with the diff summary (success/neutral/failure).
Export enqueue_git_pull_dry_run; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref (drop unused GHES webhook_secret)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* revert(git-sync): defer phase 4 PR diff checks (OSS side)
Remove the worker completion hook that posted the PR check run, drop the
enqueue_git_pull_dry_run re-export and the orphaned sqlx cache, bump EE ref.
Phases 1-3 (polling, webhooks, in-app PR creation) are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "revert(git-sync): defer phase 4 PR diff checks (OSS side)"
This reverts commit
|
||
|
|
3bd9f05938 |
fix(alerts): identify server replica in low-disk alert + per-host dedup tag (#10143)
* fix(alerts): identify server replica in low-disk alert + per-host dedup tag The server-mode low-disk alert keyed its dedup tag on the mountpoint alone, so `simple_alert_helper` mapped every server replica onto a single alert row per mountpoint. With more than one replica that row flaps every monitor pass: a replica seeing low disk raises the alert while a replica seeing healthy disk recovers it. The alert text also could not say which replica tripped. The fix lives in windmill-ee-private (`low_disk_alerts` in windmill-common/src/ee.rs) and appends the hostname to both the message and the dedup tag, mirroring the worker branch. Also add a regression test pinning the server tag as per-host, and correct the monitor cadence comments: iterations are LISTEN_NEW_EVENTS_INTERVAL_SEC (10s by default), not 30s, so "~60s (2 iterations * 30s)" was wrong on both factors. * fix(alerts): widen healthchecks.check_type so per-host disk tags fit Alert tags embed a mountpoint and a hostname, both unbounded, but check_type was varchar(50). create_alert only logs the insert error while the notification still fires, so an overflowing tag re-alerts every monitor pass and never records recovery state. The server tag overflows for ordinary pod-length hostnames, and the existing worker tag already overflows for every tracked mount except "/". Widening the column fixes both; bounding the hostname would not, since the mountpoint alone can consume the budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 This commit updates the EE repository reference after PR #666 was merged in windmill-ee-private. Previous ee-repo-ref: ccd1e42cf6b2d051ca17074fbdf5b80a46cffe0f New ee-repo-ref: b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasaandClaude Opus 4.8
|
7ebfad382a |
feat(ai-agent): give tools a real description instead of the tool name (#10083)
* feat(ai-agent): use a real tool description instead of the tool name Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): render tool-name error full width and hoist it above the description Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): make tool description field hug its content so a single line is vertically centered Add an optional minHeight param to the autosize action (default unchanged at 30px) and pass minHeight 0 for the tool description so an empty/one-line field no longer reserves the 30px floor and leaves dead space below the text. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(ai-agent): regenerate OpenFlow-derived prompts, CLI guidance, and copilot zod schema for tool description Fixes the check-freshness CI failure (system_prompts + skills.gen.ts) and makes the flow copilot's openFlow.json / openFlowZod.gen.ts aware of the new AgentTool.description field so AI-authored tools can set it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
710a13a59d |
fix(apps): cover script/flow component outputs in deployed-app S3 provenance gate (#10070)
* fix(apps): cover script/flow component outputs in deployed-app S3 provenance gate Deployed apps read S3 files on-behalf of the app author for logged-in viewers (#10048). A confused-deputy guard confines those reads to files the app "produced", but the recent-production check only matched inline `appscript`/ `preview` jobs nested under the app path. Files produced by the deployed script/flow components an app is wired to run (e.g. a SQL query persisted to S3) were therefore denied "File restricted" for every viewer, admins included. Expand the provenance check to also match completed `script`/`flow`/`flowscript`/ `flownode` jobs whose `runnable_path` is one of the app's declared triggerables, and accept the author identity via `permissioned_as = on_behalf_of` (not only `created_by = caller`) so files produced on-behalf of the author are covered. Reads outside the app's declared triggerables stay denied. Adds a regression test seeding a script-kind produced file that reproduces the "File restricted" denial before the fix and passes after. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): key S3 provenance on on-behalf identity + cover flow steps (review) Addresses the CI review on the S3 provenance gate: - P1 (confused deputy): the recent-production check keyed on `created_by = caller`, so a viewer who can run a declared script/flow directly (outside the app, with un-pinned inputs) could craft a result naming an author-only key and read it back through the app as the author. Key provenance instead on the producing job's `permissioned_as` matching the on-behalf identity the download reads as (the author in author-mode); a viewer's direct run has `permissioned_as = viewer` and no longer clears the gate. Drops `created_by` from both the appscript/preview and script/flow branches, closing the same latent hole in the pre-existing inline-script branch. - P2 (dead flow-step branch): `flowscript`/`flownode` jobs have `runnable_path = <flow_path>/<step_id>`, which exact `= ANY(...)` never matched. Split script vs flow triggerable paths; flow kinds now match the flow's own job (bare path) and its step jobs via a `<flow_path>/%` prefix, bounded to declared flows. - P2 (test realism): the regression test now uses the production component-prefixed triggerable key format (`<id>:script/...`), exercises a flow-step-produced key, and asserts a viewer's own direct run of a declared script stays denied (the P1 case). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): tie deployed-app S3 provenance to an app-origination marker (review) Second CI-review round flagged that `permissioned_as` still does not prove a job was app-launched: a runnable configured with its own `on_behalf_of` makes a direct `/jobs/run` resolve `permissioned_as` to that identity (the app author), so a viewer with run access could execute a declared runnable directly, craft an S3 result, and read it back through the app. The flow-path `LIKE fp || '/%'` match also let `_`/`%` in a declared path admit unrelated flows. Introduce a real app-origination marker instead of inferring provenance: - Add `JobTriggerKind::App`; `execute_component` stamps every app-launched job with `trigger_kind = 'app'` + `trigger = <app path>`. A direct `/jobs/run` cannot set this, so it is the authoritative signal that a file was produced *by the app*. - The provenance gate's recent-production check collapses to `trigger_kind = 'app' AND trigger = <this app path>` (+ the 3h window and result containment). This drops the forgeable `created_by`/`permissioned_as`/ `runnable_path`/kind logic entirely and removes the `LIKE` wildcard issue. - Provenance is scoped to THIS app's path, so another app's jobs (even same author) do not authorize this app's reads. Regression test rewritten to the marker model: an app-produced key clears for viewer and admin; a direct run whose `permissioned_as` resolves to the author stays denied (the forgery); another app's output stays denied. Adds `app` to the OpenAPI JobTriggerKind enum. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(apps): assert execute_component stamps trigger_kind='app' at runtime Adds an end-to-end test that runs a real script component through the app runtime (`apps_u/execute_component`) and asserts the enqueued job carries the app-origination marker `trigger_kind = 'app'` + `trigger = <app path>` (not the runnable path). The provenance-gate tests seed the marker directly; this proves the runtime actually produces the exact marker the gate depends on. execute_component commits the job row and returns its id, so the assertion reads the row directly — no worker needed to run the job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): reject trigger_kind=app for suspended-job reassignment (review) `JobTriggerKind::App` (added for the app-origination S3 marker) became a valid value for the resume/cancel suspended-trigger routes, whose handler derives the table name `<kind>_trigger`. There is no `app_trigger` table, so both endpoints would fail with a missing-relation database error (500). Reject `App` in `get_suspended_trigger` alongside webhook/schedule so it returns a clean 400. Adds a regression test asserting the reassignment route returns 400 (not 500) for trigger_kind=app. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): don't stamp app-origination marker on preview runs (review) The app-origination marker (trigger_kind='app') was stamped unconditionally, including preview mode. A preview lets a `jobs:run` caller supply arbitrary `raw_code` against ANY app path without that app's deployed policy (raw_code with no path/id skips all app authorization), so a preview returning `{"s3":"<author-only-key>"}` would forge the exact marker the S3 provenance gate trusts and read the victim app author's file. Gate the marker on `!is_preview`: only deployed, policy-checked executions are app-provenanced. Preview/editor S3 display does not rely on this marker (the editor routes reads through the force_viewer allowlist), so nothing legitimate regresses. Adds a regression test asserting a preview run's job is not stamped trigger_kind='app'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): editor-authorize preview marker + per-viewer S3 provenance isolation (review) Closes the codex P1 (preview forgery) without breaking editor preview downloads, and adds cross-viewer isolation to the provenance gate. - Preview marker now requires app write: `execute_component` stamps the app-origination marker on a preview only when the caller can EDIT that app (`require_is_writer`), instead of never stamping previews. An app editor already wields the app's author identity (they can deploy a component that reads the same file), so marking their own preview is no escalation and keeps preview-produced S3 results downloadable in the editor; a `jobs:run`-only caller who cannot edit the app still cannot forge the marker. Deployed runs are unchanged (always marked). - Per-viewer isolation: the provenance gate now also requires `j.created_by = <this caller>`. The security boundary stays the un-forgeable `trigger_kind='app'` marker; `created_by` is an additional filter ANDed under it, so it only narrows — a viewer can only download keys their OWN app runs produced, not another viewer's result. Restores the per-caller scoping #10048 had, now safe on top of the marker. Tests: preview marked iff caller can edit the app; cross-viewer isolation (another viewer's app-marked key denied, no admin bypass); direct-run and other-app keys still denied; deployed run still stamped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): require apps:write scope (not just writer ACL) to mark preview provenance (review) require_is_writer checks the user's underlying ACL but ignores token scopes, so a writer's token deliberately scoped to apps:run/apps:read/jobs:run but WITHOUT apps:write could still mark a preview and forge provenance — even though that token cannot deploy the app (update_app requires apps:write), breaking the "any marked caller can deploy equivalent code" rationale. Require BOTH apps:write:<path> scope (check_scopes) AND the writer ACL (require_is_writer) before stamping a preview's app-origination marker. Deployed runs unchanged. Adds a scope-restricted-writer token to the test (apps:run/read + jobs:run, no apps:write) and asserts its preview stays unmarked; retains the full-editor positive case and the non-editor negative case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): never app-provenance preview runs; read editor S3 as the caller (review) Simplifies the preview handling: a preview executes as the *caller* (Viewer mode), never as the author, so its results must be read back as the caller — never author-mode — and must never carry the app-origination marker. This removes the whole `require_is_writer` / `apps:write` / `can_preserve_on_behalf_of` reasoning (which was also unsound: a writer's token or session may not be able to deploy a component running as the app's on-behalf identity, so marking their preview could still escalate). - Backend: mark the app-origination marker for deployed runs only (`!is_preview`). - Frontend: `getS3File` (AppImage/AppPdf/AppDownload) now routes editor/preview reads through the viewer-scoped `job_helpers/download_s3_file` endpoint (reads as the caller), matching what DisplayResult/ParqetCsvTableRenderer already do; only a deployed app view uses the provenance-gated `apps_u` endpoint. This is the path that previously relied on marking previews, so nothing regresses. Test: a preview is never app-provenanced (owner's own preview and a non-editor's both stay unmarked). Cross-viewer isolation, deployed marking, and the reassignment guard are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(apps): app components run on-behalf of the app, not the referenced runnable (review) Root-causes codex's on-behalf-preview finding: `execute_component` was overriding the app's resolved on-behalf identity with the referenced script/flow's OWN `on_behalf_of` (its `on_behalf_of_email`). That is wrong in the app context — the app's execution mode should govern: - A Viewer-mode app could execute a component AS the referenced runnable's on_behalf identity (privilege confusion / escalation), instead of as the viewer. - A preview would run as that identity rather than as the caller, so its S3 output could not be read back as the caller — the download-identity mismatch codex flagged. Always use the app-resolved identity (author in author-mode, caller in viewer/preview); a referenced runnable's own `on_behalf_of` no longer leaks into app execution. Direct `/jobs/run` still honors a runnable's `on_behalf_of` (unchanged). With this, previews always run as the caller, so reading editor/preview S3 as the caller (viewer-scoped `job_helpers`) is unconditionally correct. - Test: the deployed-component e2e now seeds the script with a distinct on_behalf and asserts the component job's `permissioned_as` is the app identity, not the script's. - Also reword the getS3File `configuration` param comment to describe current state only (AGENTS.md comment rule). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(apps): surface 'app' trigger kind in Runs UI; condense provenance comments (review) Addresses codex review nits: - Add `app` to `jobTriggerKinds`, `triggerIconMap` (LayoutDashboard), and `triggerDisplayNamesMap` so app-component jobs (which now carry `trigger_kind = 'app'`) are filterable in Runs and render their trigger info. - Condense the app-origination marker, on-behalf-identity, and provenance-gate comments to state each invariant once in <=4 lines at its relevant site (AGENTS.md comment rule). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
92b7f375a9 |
fix: replicate all secrets on fork when external backend is configured (#10060)
* fix: replicate all secrets on fork with external backend (WIN-2161) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: add Azure KV fork secret-replication reproduction (WIN-2161) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: condense clone_variables invariant comment (WIN-2161) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: drive real create_fork handler in Azure KV repro (WIN-2161) Replace the windmill-common test that mirrored clone_variables' loop with an end-to-end test in windmill-api-integration-tests that exercises the real migration, create_fork and variable-read endpoints against a local Azure KV emulator. Verified it fails (404 "not found in Azure Key Vault") without the fix and passes with it; unique per-run ids keep it robust to the emulator's persistent state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
ff774c46bf |
feat: add per-workspace job-retention override (#10050)
* feat: add per-workspace job-retention override (EE) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 2ba6a2a75b6fc97858b306b2c98ada481e363c10 This commit updates the EE repository reference after PR #658 was merged in windmill-ee-private. Previous ee-repo-ref: e7fb36acd813cd717bcf05f5aafbf81de271d618 New ee-repo-ref: 2ba6a2a75b6fc97858b306b2c98ada481e363c10 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
8343203ec2 |
feat(mcp): add multi-workspace MCP tokens via the gateway endpoint (#10043)
* feat(mcp): add multi-workspace MCP tokens via the gateway endpoint A single MCP token with no bound workspace (workspace_id NULL + mcp scope) now works across every workspace the token owner can access, served through the existing /api/mcp/gateway endpoint. This avoids having to register one MCP server entry per workspace in clients like Claude/Cursor. In multi-workspace mode the runner exposes a synthetic `list_workspaces` tool plus the generic API endpoint tools, each workspace-scoped one gaining a required `workspace_id` argument (mirroring the proxy pattern users built externally). Per-workspace scripts/flows are not enumerated to avoid flooding the tool list — they are run via runScriptByPath/runFlowByPath with an explicit workspace_id. Auth is resolved per tool call: the gateway middleware detects a workspace-less mcp token and marks the request MultiWorkspaceMcp, and the runner resolves a per-workspace ApiAuthed from the raw token via the AuthCache (validating membership; superadmins may act in any workspace). Single-workspace tokens are unchanged. Frontend: the MCP token creation flow gains an "All workspaces" option that produces a workspace-less token and the gateway URL. Fixes WIN-2153 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(mcp): cover multi-workspace endpoint tool transformation Unit tests for endpoint_tool_to_mcp_tool_multi and list_workspaces_tool: workspace-scoped tools gain a required workspace_id arg, global tools are left unchanged, workspace_id is not duplicated, and list_workspaces takes no arguments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): forward script/flow args for runScriptByPath/runFlowByPath These endpoints have an additionalProperties body (no declared properties), so build_request_body previously returned an empty body and dropped every script/flow argument. This was latent for the per-path run endpoints and became load-bearing in multi-workspace mode, where scripts/flows can only be run via runScriptByPath/runFlowByPath — parameterized runs silently lost their arguments. build_request_body now forwards all arguments not consumed by a path/query parameter for pass-through (additionalProperties) bodies, keeping the strict declared-only behavior for endpoints with explicit properties. The runner strips the synthetic workspace_id argument before dispatch so it can't leak into the forwarded body. Reported by Codex review on #10043. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(mcp): note workspace_id requirement in multi-workspace tool descriptions Workspace-scoped tools already gain a required workspace_id parameter (with its own schema description) in multi-workspace mode, but the tool's prose description was unchanged. Append a note so models/clients that read the description text know to pass workspace_id (and to call list_workspaces first). Global tool descriptions are left untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(mcp): trim multi-workspace tool/arg descriptions The workspace_id note repeats across every workspace-scoped tool in each tools/list, so keep it terse: description suffix "Requires `workspace_id`." and arg description "Target workspace id (from list_workspaces)." to avoid spending tokens on repeated boilerplate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): enforce script/flow path scopes for multi-workspace run-by-path In multi-workspace mode runScriptByPath/runFlowByPath are the only way to run scripts/flows, but they were authorized against the endpoint scope only — never the caller's mcp:scripts:/mcp:flows: path scopes. A granular token could run items outside its allowed paths (e.g. mcp:scripts:f/team/* + mcp:endpoints:* running f/other/secret), and a mcp:endpoints:* token could run arbitrary scripts. Now these two endpoints are authorized by the script/flow scope of the requested path (matching single-workspace mode's per-item tools): exposed in list_tools only when the token grants some script/flow (McpScopeConfig::has_any), and at call time the path is checked via is_allowed("script"/"flow", path). Verified e2e: mcp:scripts:f/team/* runs f/team/* but is denied f/other/*; mcp:endpoints:* alone no longer exposes or runs run-by-path. Reported by Codex + Pi review on #10043. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): deny run-by-path for mcp:favorites multi-workspace tokens mcp:favorites sets granular=false, so the previous run-by-path scope check (gated on `granular`) was skipped entirely — a default "Favorites only" all-workspaces token could run any script/flow by naming its path, bypassing the favorites restriction. Favorites are an enumerated set reachable only through per-item tools, not by arbitrary path, so they grant nothing for run-by-path. has_any() now returns true only for mcp:all (not favorites), and the call-time check drops the `granular` gate and relies on is_allowed() directly (already false for favorites, true for mcp:all, pattern-matched for granular). Verified e2e: mcp:favorites no longer exposes or runs run-by-path; mcp:all still runs; granular script scopes still path-enforced. Reported by Codex review on #10043. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 4.8
|
6c521e9d87 |
fix(backend): propagate script timeout when restarting perpetual scripts (#10029)
Perpetual scripts (restart_unless_cancelled) re-pushed their restart job with custom_timeout = None, so every rerun ignored the script's configured timeout and fell back to the instance-level job_default_timeout. Only the first run honored the script timeout. Fetch the script timeout alongside restart_unless_cancelled (both cached by the immutable script hash) and pass it as custom_timeout when re-pushing the perpetual job. Fixes WIN-2149 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
f28ea9cb99 |
feat(db-health): add connection sizing guidance (#10014)
* feat(db-health): add connection sizing guidance The Database Connections panel showed current/max connections but gave no guidance on how to size max_connections for the deployment. Derive an estimate from the live worker fleet: each worker instance shares a pool sized DEFAULT_MAX_CONNECTIONS_WORKER + (workers - 1), and each server opens up to DEFAULT_MAX_CONNECTIONS_SERVER (both overridable via DATABASE_CONNECTIONS). The endpoint now returns live worker/instance counts, the default per-server and per-worker pool sizes, the estimated peak worker connections, the reserved superuser connections, and a recommended max_connections floor (workers + one server + 25% headroom). Servers do not ping worker_ping, so the recommendation assumes one server and exposes the per-server increment. The panel renders this as a sizing breakdown and warns when max_connections is below the recommended floor. Fixes WIN-2147 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(db-health): single source for pool-size constants + sizing tests Address review: db_connect.rs kept its own copies of DEFAULT_MAX_CONNECTIONS_* that duplicate the windmill_common constants the sizing guidance reads, so tuning the runtime pool size would silently leave the guidance stale. Re-export the windmill_common constants from db_connect.rs so there is one source of truth. Add unit tests for compute_connection_sizing covering the zero-fleet, single worker, multi-instance, and reserved-clamp cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(db-health): 20% headroom and 200-connection minimum floor Lower the sizing headroom from 25% to 20% and never recommend below 200 connections (postgres defaults to 100; cheap headroom for growth/bursts/psql). Update the guidance message and unit tests accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-health): honor DATABASE_CONNECTIONS in sizing recommendation Address Codex P1: the runtime caps every process's pool at DATABASE_CONNECTIONS when set (db_connect.rs), but the sizing guidance always used the default 50/5 pools. For a tuned deployment this under-estimated worker demand and could hide a genuine under-provisioning (e.g. DATABASE_CONNECTIONS=100 with 5 instances is 500 worker connections, not 25). compute_connection_sizing now takes the effective DATABASE_CONNECTIONS override (read the same way db_connect.rs reads it): when set, each worker instance and server pool is that value and the worker estimate is override * instances. The response exposes server_pool_size / worker_pool_size (effective) and database_connections_override; the panel renders both pool rows and labels them (default) vs (DATABASE_CONNECTIONS), and the message states which source is used. Adds a unit test for the override path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-health): exclude agent workers from connection sizing Agent workers reach the API over HTTP (MODE=agent, Connection::Http) and hold no postgres pool, but their pings still land in worker_ping (written server-side by /api/agent_workers/update_ping). Counting them inflated the connection estimate. Filter the fleet query by the worker-name prefixes: DB-connected workers use "wk-" (WORKER_NAME_PREFIX), agent workers use "ag-" (AGENT_WORKER_NAME_PREFIX). Only wk- workers/instances feed the estimate; ag- workers are counted separately and surfaced as context ("N agent workers excluded — they use HTTP, not postgres connections"). Adds a unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Fable 5
|
f65fe7bf58 |
fix: replicate external secret backend secrets when forking a workspace (#10007)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
e47aedac0a |
feat: add SQL migrations for data tables (#9693)
* feat: add datatable_migrations table * feat: add route to run datatable migrations * feat: sync datatable migrations as .up.sql/.down.sql files * feat: add datatable migrate up/down commands and post-push run prompt * feat: add datatable migrate new command to scaffold migrations * feat: add datatable migrations management UI * feat: prompt to create migration on DDL in datatable SQL editors * feat: support running a single specific datatable migration * feat: view migration content, run single migration, fix stacked modal * feat: per-row revert button with out-of-order warning * fix: avoid migrations list flicker on refresh after an action * feat: generate initial datatable migration via pg_dump * fix: surface datatable migration API error details in toasts * fix: revert created migration if create-and-run fails to run * fix: include postgres error detail in migration run/rollback failures * feat: sync datatable migrations as files via the workspace export * refactor: move datatable migrations to migrations/datatable/ path * fix: drop redundant datatable_migration label in sync output * fix: exclude datatable migration sql files from script metadata generation * feat: run datatable migrations as user-permissioned labeled jobs * feat: reject invalid datatable migrations on sync push * feat: datatable migrate up/down default to all datatables, --datatable to target one * fix: surface postgres error detail when datatable migrations fail to run * chore: regenerate CLI docs for datatable migrate commands * feat: default new datatable migration to a BEGIN/END transaction template * fix: validate datatable migration name and datatable at the API boundary * fix: ensure detected DDL ends with semicolon when wrapped in transaction * fix: re-prompt instead of stripping DDL when new-migration modal is cancelled * feat: refresh datatable schema after running a migration from the SQL REPL * feat: record db manager DDL on data tables as migrations * feat: make datatable migrations opt-in per data table * fix: make migration view editor read-only so its code can scroll * fix: don't re-prompt DDL guard when creating a migration without running * feat: generate down migrations for db manager DDL (postgres) * fix: correct down migration for db manager alters (no double-wrap, serial) * feat: explain migrations purpose with a tooltip in the migrations modal * compare paeg * feat: add datatable_migration kind to workspace diff pipeline * chore: point ee-repo-ref at datatable_migration git-sync companion * fix: harden datatable migration version allocation and initial-migration bookkeeping, add tests * feat: deploy and run datatable migrations on workspace merge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Refactor + handle datatable setting delete/rename * refactor: move datatable migration rename/delete cascade into module Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(windmill-utils-internal): bump to 1.7.1 for datatable migration deploy provider methods Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(db-manager): add Migrations button to top bar, make Refresh icon-only Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * BEGIN/END placeholder in down migration * feat: autofocus migration name input and flag it red when empty Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(datatable-migrations): allow non-admins to create/run/revert migrations, gate only opt in/out Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * border nits * refresh db manager schema on migrations * BEGIN/END scaffold in CLI * feat(cli): push local datatable migrations before running on migrate up Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: flag invalid migration name with red border, not just empty Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: drop random slug from auto-generated migration names Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: offer revert-and-delete when deleting an installed migration Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: record fork merge as a migration when target datatable opts in * nit * clone migrations on fork * windmill-utils-internal * fix(datatable-migrations): serialize run/rollback with a per-db advisory lock Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-manager): fail closed when migrations-status check errors on DDL apply Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix generate_initial migration ordering comment to match code * chore(datatable-migrations): remove unused update_datatable_migrations endpoint Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: run DDL migration guard on the script editor Test button Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * split * ee-repo-ref * chore(frontend): sync package-lock with package.json (@emnapi deps) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(datatable-migrations): never resolve instance credentials into migration job args datatable_database_arg eagerly resolved instance data-table credentials (including the shared instance-wide Postgres password) and passed them as the migration job's plaintext `database` arg, landing in v2_job.args. Since the run route has no admin gate, a non-admin could run a migration and read args.database to recover the password, granting cross-workspace psql access to all instance data-table DBs. Pass a `datatable://<name>` reference for both resource-backed and instance data tables instead; the pg executor already resolves it to real credentials server-side at run time, so nothing sensitive is ever stored in the job args. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * fix: handle dollar-quoting and comments when splitting SQL statements * feat: deploy datatable migrations on merge with explicit opt-in error * fix(frontend): sync package-lock with npm 11 peer-dep resolution npm ci failed with 'Missing: @emnapi/core@1.11.2 / @emnapi/runtime@1.11.2 from lock file'. @napi-rs/wasm-runtime declares @emnapi/core|runtime ^1.7.1 as peerDependencies while @rolldown/binding-wasm32-wasi pins them to exactly 1.10.0. Newer npm (bundled with node 24 in CI) installs the peer deps at the highest match (1.11.2) alongside rolldown's nested 1.10.0, so the ideal tree needs both versions; the committed lock only had 1.10.0. Regenerate the lock with npm 11.18 so it carries both 1.11.2 (top-level, for the peer deps) and 1.10.0 (nested, for rolldown's pin). Verified npm ci passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit npm publish * fix: fail closed on migrations-status error in fork schema merge * nit CI emnapi/core version * prevent initial_datatable_migration if migrations already exist * fix(datatable-migrations): validate persisted data table names as path segments edit_datatable_config only validated rename segments, not the actual settings.datatables keys, so a data table could be saved directly under a name like '..' or one containing '/'. Since new tables default to migrations_enabled = true, generate_initial_datatable_migration would then insert a migration row and the sync export would build migrations/datatable/<name>/... paths from that name, producing malformed or directory-escaping export paths. Validate every persisted data table name in edit_datatable_config (alongside the existing rename checks) and add validate_datatable_path_segment to generate_initial_datatable_migration for defense in depth. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scope datatable _wm_migrations by data table and cascade renames/deletes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(system_prompts): resolve nested local command groups in CLI docs generator The CLI docs generator anchored on the first `new Command()` in a file and never resolved locally-defined command groups passed as `.command("name", localCmd)`. For datatable this flattened the nested `migrate` group: it emitted `datatable new/up/down` plus a bare `datatable migrate`, and mislabeled the datatable command with the migrate group's description. jobs was broken the same way (its description was pull's, and pull/push rendered empty). Anchor block extraction on the `export default`ed command, recurse into locally-defined `const x = new Command()` groups mounted as subcommands, and render nested sub-subcommands. Regenerated docs now show `datatable migrate new/up/down` and `jobs pull/push` with their real options. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: drop unreleased _wm_migrations legacy-upgrade handling Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: return datatable migration SQL from getItemValue for the diff drawer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(frontend): use windmill-utils-internal 1.8.2 for migration diff drawer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * nit * fix: handle datatable migration renames on push and dedupe timestamps * fix: reject rewriting an already-applied datatable migration on upsert Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): add missing @emnapi/core and @emnapi/runtime lockfile entries Resolves npm ci EUSAGE failure: the optional cpu:wasm32 @rolldown/binding-wasm32-wasi declares deps on @emnapi/core@1.11.2 and @emnapi/runtime@1.11.2 that had no resolved lockfile entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): datatable migrate up/down default to main datatable, not all Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: fail closed when applied status unreadable on datatable migration rewrite Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: surface full error detail in Database Manager DDL/query errors * "See migration" button in the toast * feat: add Enter shortcut to Create-a-migration in the DDL guard * fix(frontend): warn before running a newly-created datatable migration out of order The row-level Run action warns when earlier migrations are still pending, but the create-and-run paths ran a just-created migration with `only` directly, applying it ahead of older pending migrations without that confirmation. Reuse the same "Run migration out of order" confirmation across all create-and-run paths via a shared helper (datatableMigrationUtils): - NewDataTableMigrationModal "Create and run" (and the DDL guard path) - DatatableSchemaDiff fork→parent merge - dbOps schema ops (DB manager create/alter/drop) — the pure factory throws a MigrationRunCancelled sentinel on decline, which DBTableEditor treats as a silent cancel Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep renamed datatable migrations visible in compare view * fix: record per-migration deployment on datatable migrations disable * fix(cli): run deployed datatable migrations after workspace merge The merge command upserted datatable_migration definitions into the target workspace and reported the item as successfully deployed, but never ran the migrations. For forked datatables backed by separate databases, this left the target schema unchanged until someone manually ran `wmill datatable migrate up`, while the CLI reported a successful merge. Collect the datatable migrations deployed (not deleted) into the target and, after the deploy loop, offer to run them via the existing offerToRunNewMigrations helper — the same post-deploy run prompt the push/sync path uses (interactive only; `--yes`/non-TTY skip the mutating run, matching push behavior). Export parseDatatableMigrationDeployPath so the merge path can parse the deployed items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): serialize datatable migration edits/deletes with the run lock A migration run snapshots a migration's code_up from datatable_migrations and only records its version in the data table's _wm_migrations after the job succeeds. upsert_datatable_migration checked _wm_migrations before allowing an edit but took no lock, so a concurrent edit could read "not applied yet", rewrite code_up/code_down, and then the in-flight run would record the version for the old SQL — leaving _wm_migrations pointing at SQL that was never applied (migrate up then skips it; rollback runs a down that doesn't match). Serialize definition rewrites and deletes with the same per-database advisory lock the run/rollback paths use: - Factor the connect+advisory-lock into lock_datatable_migration_runs and the applied-versions read into read_applied_versions_on_client. - run_datatable_migrations now snapshots the definitions AFTER taking the lock, so code_up can't change between snapshot and version-record. - upsert (when changing an existing def) and delete take the lock across the applied-check and the write; delete now rejects deleting an already-applied migration (would orphan its _wm_migrations record), symmetric with upsert. Both fail closed if the data table database is unreachable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): stack the out-of-order migration confirm above the DB editor preview Creating a table on a migrations-enabled data table opened the DB table editor's "Confirm running the following" preview modal, whose confirm triggers applyDdl, which then asks for out-of-order confirmation. Both are ConfirmationModals with a hardcoded z-[9999]; the out-of-order one lives in DBManagerContent (mounted before the editor), so it rendered behind the still-open preview modal. Add an optional zIndexClass prop to ConfirmationModal (default z-[9999], backward-compatible) and give the DB-manager out-of-order confirm z-[10000] so it stacks on top. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 27672e37df5d9dfde94f19963d5ffcdf8dd5448c This commit updates the EE repository reference after PR #623 was merged in windmill-ee-private. Previous ee-repo-ref: 6c287041cd7edd4a77a4bc07ad0e156cec32cce4 New ee-repo-ref: 27672e37df5d9dfde94f19963d5ffcdf8dd5448c Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasaandClaude Opus 4.8
|
fd8e64d11f |
feat: add cosmetic dev/staging label for dev workspaces (#9959)
* feat: add cosmetic dev/staging label for dev workspaces Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: prefill dev fork name and use a link to switch its label Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: reword the dev/staging label link copy Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: preview the dev/staging label as a badge in the switch link Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: show the dev/staging badge in the session diff drawer header Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
55451db009 |
fix(datatable): self-teaching error for unresolved datatable:// references (#9941)
`ATTACH 'datatable://main'` (or any datatable schema/executor path) failed with a bare "datatable main not found", giving the user no way forward — the datatable substrate has no auto-provisioning like a DuckLake catalog, so the fix is always to create one in workspace settings, but nothing said so. `get_datatable_resource_from_db_unchecked` now returns a NotFound error that lists the workspace's configured data tables (to catch typos) and points at the "Data tables" settings tab, noting `main` is the default name used by `datatable://main`. The message bubbles up wherever the resolver is called (pipeline ATTACH, schema fetch, postgres executor, agent HTTP endpoint). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Fable 5
|
39eb9de1bc |
feat(pipelines): fork data environments for ducklake materialization (dev data) (#9915)
* feat(pipelines): fork-scoped ducklake namespaces with read-defer to parent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork graph indicator + fork ducklake namespace cleanup endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork_views-keyed view transition, fork lineage clone, design doc Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): review hardening - fork DATA_PATH last-wins, registry cache TTL, defer tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): per-lake isolated/shared choice at fork creation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): chain-aware defer discovery + per-location fork namespace registry Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): lake-scoped fork schemas, catalog identity in registry, chain-aware graph chips Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): cleanup deletes fork data from the registered storage identity Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): collapse fork data-path segment to one component (slash-safe ids) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): per-catalog ancestor checks, ancestor extra_args passthrough, test compile fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): invalidate fork ancestor-chain cache on lineage mutations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): sweep descendant ancestor-chain caches on delete/reparent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): run fork ducklake cleanup inline in delete_workspace Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): resolve fork cleanup credentials pre-commit, destroy post-commit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): shared dev-workspace authz gate for namespace drop, invalidatable registration cache, segment-boundary delete filter - extract require_prod_admin_for_dev_workspace, used by both delete_workspace and drop_forked_ducklake_namespaces so the gates cannot drift - key FORK_DUCKLAKE_REGISTERED per workspace and invalidate it in cleanup_fork_ducklake_namespaces so a same-id fork recreated within the TTL re-registers its namespaces - filter listed object locations to the segment boundary before deletion Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): keep orphaned wm-fork-* workspaces ducklake-isolated parent_workspace_id is ON DELETE SET NULL, so a fork can outlive its parent with an empty ancestor chain while its cloned config still points at the shared lake. Key the isolation gate on the wm-fork- prefix as well as the chain (mirroring workspace_is_fork): orphaned forks get the write redirect, registration and cleanup with zero ancestors (no defer), and keep their 'fork' graph chips. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): attach orphaned wm-fork-* ancestors at their fork namespace Chain position alone classified the last ancestor as a root, but an orphaned wm-fork-* ancestor (its own parent deleted, SET NULL) ends the chain the same way while its data lives in its fork namespace — its descendants' defer views bound the dead root's lake instead. Key the root-vs-fork decision on the wm-fork- prefix too, matching the resolution gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): never inherit shared lake opt-out; durable cleanup ledger for failed fork deletions - fork creation strips cloned fork_behavior stamps before applying the request's shared_ducklakes list: sharing is a per-creation choice, a fork of a shared fork defaults back to isolated - fork_ducklake_namespace loses its ON DELETE CASCADE FK: rows are the durable cleanup ledger and outlive the workspace when physical cleanup fails post-commit; fork creation retries leftover rows for the reused id and refuses to create while a metadata schema still cannot be dropped (data-file leftovers alone are inert once the schema is gone and are swept by the next successful same-prefix cleanup) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): make orphaned-namespace cleanup retries independent of deleted fork resources - ledger rows gain a schema_dropped phase flag: set when the schema drop succeeded but data cleanup failed, so later retries skip the schema phase and need no catalog credentials at all; registration resets it on re-attach (ON CONFLICT DO UPDATE) since attaching recreates the schema - retry-path $res: resolution falls back to the workspace being forked (the deleted fork's resources were clones of a parent's); live paths (delete_workspace prepare, drop endpoint) pass no fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): fork tables from failed-after-commit runs stay fork-owned in defer and graph A failed materialization must not disguise a physically existing fork table as deferred: CREATE VIEW IF NOT EXISTS silently yields to the table, so reads hit fork data while the graph claims parent defer. - record_mat upsert preserves the last committed snapshot_id on failure - defer discovery and graph chips treat fork rows with a committed snapshot as fork-owned even when status is failed - inspect_fork_catalog also lists live fork tables (same round trip) and the defer list is filtered against them — covers rows recorded before this fix and tables created by raw SQL - drop stale FK-cascade wording in the design doc and sidebar comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork-mode ducklake settings — per-lake isolated/shared chips + banner, fork_behavior round-trip The workspace-settings ducklake editor had no fork awareness: no reminder of each lake's isolated/shared choice and no warning about what edits mean in a fork. It also rebuilt each lake explicitly on save, silently dropping fork_behavior — any settings save in a shared fork flipped the lake back to isolated. - fork detection mirrors the backend gate (parent link or wm-fork- prefix) - info banner explaining isolated vs shared semantics in a fork - per-lake chip (emerald 'isolated' / amber 'shared with parent') with tooltips, matching the pipeline graph chip colors - fork_behavior added to DucklakeSettingsType and preserved through convertDucklakeSettingsToBackend Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
33521505db |
feat(ducklake): scheduled lake maintenance (expiry, compaction, orphan cleanup) (#9916)
* feat(ducklake): scheduled lake maintenance (snapshot expiry, compaction, orphan cleanup) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ducklake): review fixes — starts_with not LIKE, CE license-lapse escape Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(ducklake): auth-contract docs + _unchecked rename per codex review Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(ducklake): move maintenance payload construction into EE module Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ducklake): fall through to script resolution for non-managed reserved-prefix schedules Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(ducklake): document accepted pre-existing-schedule limitation on the reserved prefix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ducklake): CE save-off clears the managed schedule row and queued occurrence Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 2fab310d4f50ed7c34857d69c9b854f4491bf217 This commit updates the EE repository reference after PR #645 was merged in windmill-ee-private. Previous ee-repo-ref: fff1fd830a36beba732486f05941ec243cf6b640 New ee-repo-ref: 2fab310d4f50ed7c34857d69c9b854f4491bf217 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Fable 5
|
42e11c6570 |
feat(pipelines): schema contracts — save-time consumer checks vs captured schemas (#9917)
* feat(pipelines): schema contracts — save-time consumer checks vs captured schemas Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: move schemaContractContext above schemaCanEvolve doc comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: emit scd2/on_schema_change in CLI local graph, address review notes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: gate editor _current ignore-suppression on scd2, matching backend Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
5d7fb6deca |
feat(pipelines): asset freshness — fresh/stale badge (CE) + watchdog (EE) (#9909)
* feat(pipelines): passive asset freshness tracking on the graph Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(pipelines): drop dead freshness-enforcement stub, document query ordering Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): freshness watchdog (EE) — auto re-run stale producers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): watchdog review fixes — archived workspaces, badge kind parity, scan index Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): CI review — no singlestepflow in freshness, +N parity, completion-time fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): CI review — history completedAt, freshness/asset trigger UI metadata Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 6f5fe0f7f56696fbef5a8349da38496c32e71666 This commit updates the EE repository reference after PR #643 was merged in windmill-ee-private. Previous ee-repo-ref: 1f13380354bf591ae25a2c20d36917534bcc5459 New ee-repo-ref: 6f5fe0f7f56696fbef5a8349da38496c32e71666 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Fable 5
|
af36498432 |
feat(pipelines): record upstream snapshot ids on cascade-dispatched jobs (#9910)
* feat(pipelines): record upstream snapshot ids on cascade-dispatched jobs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: batch upstream-snapshot lookup and memoize per subscriber Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
84141add1d |
feat(pipelines): workspace duckdb macro libraries (// macros / // use) (#9890)
* feat(pipelines): parse duckdb macro-library annotations (// macros, // use) * feat(pipelines): duckdb macro registry tables + deploy-path validation and writes * feat(pipelines): inject workspace duckdb macros into consumer jobs at run time * feat(pipelines): surface macro libraries and lib-consumer edges in asset graph api * feat(frontend): macro-library nodes, lib-consumer edges and scaffold in pipeline graph * docs: mark dbt gap #7 (packages/macros) shipped via workspace macro libraries * fix(pipelines): review fixes - char-safe parsing, local macros win, fork clone, trust-model docs * feat(frontend): duckdb macro autocomplete + workspace macro explorer drawer * fix(pipelines): address CI review - use-setup retention, splice past local defs, orphan filter, full consumer rescan, index-keyed strip * fix(pipelines): inject provider library setup for implicitly-called macros too * fix(pipelines): rls-gate macro listing + honor library-level // use transitively * fix(pipelines): weave injected macros around local definitions by bind order * fix(pipelines): injected library setup always runs before user blocks * perf(pipelines): cache macro registry per workspace with notify-event invalidation * perf(pipelines): disable macro registry cache on cloud |
||
|
|
af01e90b5c |
feat(s3): replace CE 50MB upload cap with 10GiB workspace storage quota (#9874)
* fix(s3_proxy): enforce CE 50MB upload cap on multipart uploads Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(s3): replace CE 50MB upload cap with 10GiB workspace storage quota Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): gate CE quota OSS stubs to not(enterprise) to match callers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): delta-aware CE storage quota + guard usage-load retry loop Account for the overwritten object's size in the quota check so valid same-size overwrites near quota are not rejected (Codex review), and stop the storage-usage $effect from re-firing on persistent API errors (Pi review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): count chunked PUTs; revert overreaching volume quota copy Volumes write to workspace storage via a separate worker-side path with its own 50MB-per-file cap that this PR does not change, so revert the drawer copy that claimed they count toward the 10GiB quota (Codex review). Bump ee-repo-ref for the chunked-PUT accounting fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): add SQLx cache for CE usage-bump/quota queries; exclude volumes Regenerate the missing offline SQLx cache for the not(enterprise) bump and remaining-quota queries so the private CE offline build compiles, and bump ee-repo-ref for the volumes/-prefix exclusion from the counted quota (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): always HEAD for CE upload delta so overwrites don't inflate usage Bump ee-repo-ref for the fast-path overwrite-accounting fix (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): reserve volumes/ prefix on CE write surfaces to close quota bypass Reject direct writes to the reserved volume prefix on the app-upload surface and add the OSS stub; bump ee-repo-ref (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): refuse new multipart work when CE workspace is at quota Bump ee-repo-ref for the multipart-initiate/part quota gate (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(s3): reserve in-flight multipart bytes against CE storage quota Add workspace_multipart_inflight table + grants, SQLx cache for the reservation queries, and bump ee-repo-ref. Bounds abandoned multipart uploads that the list-based recount can't see (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): clear multipart reservation only after a successful complete Add exclude-upload arg to the OSS quota stub/caller and the SQLx cache for the updated remaining-quota query; bump ee-repo-ref (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(s3): per-part multipart reservation; commit only on part success Per-part workspace_multipart_inflight schema (upload_id, part_id) so retries replace rather than double-count; SQLx cache for the reworked queries; bump ee-repo-ref (Codex review). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(s3): HEAD the multipart overwrite target once per upload, not per part SQLx cache for the stored-credit lookup; bump ee-repo-ref. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to bea5a8b5120d6d69cab1ad4611ebe463559bd200 This commit updates the EE repository reference after PR #640 was merged in windmill-ee-private. Previous ee-repo-ref: 6e6ff86f1939cf74736b7d435bf6851416437523 New ee-repo-ref: bea5a8b5120d6d69cab1ad4611ebe463559bd200 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasaandClaude Opus 4.8
|
7c7d7474cc |
feat: support workspace forks on cloud using parent workspace limits (#9864)
* feat: support workspace forks on cloud using parent workspace limits Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: clarify count_paid_seats approximates rather than mirrors billing seats Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: non-admin fork UI, attach cap, and fork-count for cloud forks Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: cloud fork billing cache on rename, usage display, attach cap edge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: fork count in cloud quotas + fork billing points to parent Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: invalidate billing/fork caches on fork deletion for id reuse Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: gate fork usage remap on CLOUD_HOSTED, not just the cloud feature Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: note cloud feature vs CLOUD_HOSTED gating in backend guide Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: reserve fork-cap slots for an attach candidate's whole subtree Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: invalidate team-plan cache on delete, raise fork depth cap Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: cap fork nesting depth (MAX_FORK_DEPTH, default 5) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: fork count/height robust to cycles and deleted intermediates Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): reset fork button loading state on creation error Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: invalidate billing cache for attached fork subtree; helper auth docs Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Fable 5
|
d15033cde6 |
fix: invalidate bun bundle cache on transitive relative-import changes (#9891)
* fix: invalidate bun bundle cache on transitive relative-import changes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: do not memoize transient fetch errors in bundle-key import cache Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: use regular comment on lazy_static block (deny unused_doc_comments) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: align bundle-key import version selection with loader content endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszelandwindmill-internal-app[bot]
|
53bbb92953 |
feat(pipeline): backfill a range of partitions from the asset drawer (#9885)
* feat(pipeline): backfill a range of partitions from the asset drawer (ee) * feat(pipeline): cancel in-flight backfill job and show cancelling state * refactor(pipeline): move backfill range logic behind private feature * fix(pipeline): close backfill cancel-launch race and record dispatch intent * docs(openapi): producer_path also covers SDK write-edge producers * chore: update ee-repo-ref to c3852ecb36bd0be1a74c63169e513888f3347850 This commit updates the EE repository reference after PR #641 was merged in windmill-ee-private. Previous ee-repo-ref: 7c1450ef89fbc9e844a121b39cafe0d7235d704b New ee-repo-ref: c3852ecb36bd0be1a74c63169e513888f3347850 Automated by sync-ee-ref workflow. --------- Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
1a9debb689 |
fix(jobs): give flow dynselect a path and its worker tag, like scripts (#9867)
Fetching options for a `dynselect`/`dynmultiselect` input was inconsistent between deployed scripts and deployed flows: - scripts ran through `push_script_job_by_path_into_queue` — a `script` job with the script's path, tag, lock and codebase resolution; - flows ran their schema dyn-select code as an anonymous `preview` with no path and no tag (always the language default), and reported access failures as a raw `SqlErr: no rows`. Deployed scripts are left exactly as they were (that path already handles tag/lock/codebase/on-behalf-of correctly). The flow branch now: - carries the flow path on the preview job, - reads the flow's `tag` under RLS and routes the job to it (falling back to the language default when unset), matching the script's worker group, and - runs `check_tag_available_for_workspace` on that tag — the same gate a normal flow run and the script path apply — so a caller who can read the flow but is not allowed to use its (custom/scoped) worker tag is rejected consistently. The flow's tag read runs on every request, so it also serves as the per-request access check, replacing the raw error with a clean `NotAuthorized` / `NotFound`. Entrypoint-name validation now covers all branches (it is interpolated into the generated wrapper). Inline is unchanged: a `preview` with no path on the language default, blocked for operators. Fixes WIN-2118 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
7363d2c217 |
fix(forks): require admin of both sides for the compare visibility guard (#9869)
The blast-radius guard added in #9866 forced `all_ahead_items_visible` true for any fork/target admin. But `filter_visible_diffs` keeps a modified/conflict row (one that exists in the source AND the fork) only when the caller can see it on both sides, so an ahead diff can be dropped for a source-side visibility gap even when the caller is a fork admin. Forcing the flag on fork-admin alone then wrongly reported "all ahead items visible", letting the UI enable deployment from an incomplete comparison. Gate the guard on admin of BOTH the source and the fork (superadmin satisfies both), which is what actually guarantees full visibility of every item on every side. Adds a regression test where a fork admin who is only a plain member of the parent (no access to the item's folder) must still get `all_ahead_items_visible = false`, plus the superadmin sanity path. Also restores the SQLx offline cache entry for the phantom-trigger test INSERT that #9866 landed without (CI/`SQLX_OFFLINE=true` builds failed on it), and adds entries for the new test's all-literal queries. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
d7be355290 | sqlx nits | ||
|
Ruben FiszelandClaude Opus 4.8
|
76a9523009 |
feat: use derived username instead of email for non-member superadmins (#9857)
* feat: use derived username instead of email for non-member superadmins Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address review - drop redundant username cache, guard whoami membership by email Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: use explicit non_member boolean instead of role string for superadmin banner Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve email from password table for non-member superadmin permissioned_as Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve non-member superadmin drafts via shared username->email resolver Adds resolve_username_to_email (usr, then super_admin password fallback for both derived-username and email modes) and uses it in get_email_from_permissioned_as and the drafts get/list endpoints, so a non-member superadmin's drafts resolve and no email leaks into the drafts payload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: superadmin-not-in-workspace schedule uses derived username as permissioned_as Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve non-member superadmin identity in draft owner-circles, username_to_email, and home filter Applies the password-fallback username resolution to the script/flow/app/draft owner-circle subqueries and the username_to_email endpoint (was an admins-workspace 'username == email' hack), and switches the home items-list user-folder filter to the non_member flag instead of the now-broken username-contains-@ heuristic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: backfill non-member superadmin favorites from email to derived username Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: propagate DB errors in username resolution instead of leaking email (CI review) Addresses cubic-dev-ai P2: get_instance_username_or_fallback_to_email now returns Result and only falls back to the email for a genuine 'no derived username'; a query error propagates so callers fail closed rather than leaking the raw email as the acting username. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: clarify non-member superadmin popover (username used + admin permissions) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep username_to_email endpoint member-only to not disclose non-member superadmin email (CI review) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: forbid disabling automate_username_creation once usernames assigned (CI review) Makes the setting effectively one-way once instance-wide usernames exist, so the global-uniqueness invariant that keeps stored u/<username> identities (schedules/triggers/drafts/superadmin ownership) unambiguous can never be dropped back to workspace-local uniqueness. Re-saving false on an already-disabled instance stays a no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
83f3d7f910 |
feat(licensing): enforce offline license seat cap (#9845)
* [ee] feat(licensing): enforce offline license seat cap Companion to windmill-ee-private. Aligns the offline-license seat count with the billing model and adds real-time enforcement when usage exceeds the cap. OSS side carries the ee_oss stubs, the reactivation cap-check call site, the regenerated SQLx cache, and the EE ref bump. - Exclude instance-disabled users (password.disabled) and service accounts from the seat count. Deactivating a user now frees a seat. - Service accounts no longer consume seats (no check at creation). - Hard-block reactivation when it would exceed the cap. - Invalidate the license (halting jobs) when seat usage exceeds the cap, mirroring CU-cap enforcement; recovers when usage drops back under or a higher-cap key is loaded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [ee] fix(licensing): bump EE ref for reactivation seat-check fixes Points to the EE companion commit that fixes reactivation double-counting and preserves the original seat alert tag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [ee] fix(licensing): reactivation seat delta includes pending invites Bumps the EE ref and drops the now-orphaned usr-only cache entry; the reactivation check reuses the existing usr ∪ workspace_invite query. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [ee] test(licensing): bump EE ref for offline seat-cap tests Adds #[sqlx::test] coverage for the offline seat counting and cap-check logic; EE-only (runtime queries, no cache change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: update ee-repo-ref to f814c3f75308c1ef1e4526d8d0eeb360ce16abe4 This commit updates the EE repository reference after PR #637 was merged in windmill-ee-private. Previous ee-repo-ref: b2622e3afc2fe1fe3e2ec978ca46cf9decf91b82 New ee-repo-ref: f814c3f75308c1ef1e4526d8d0eeb360ce16abe4 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
hugocasaandClaude Opus 4.8
|
b4b0c6a93e |
feat: add dev workspaces paired with a lockable prod workspace (#9793)
* feat: add dev workspaces paired with a lockable prod workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate dev-workspace prod-lock on admin and prevent attach cycles Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: redirect locked-prod edits into the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make dev-workspace settings tab available on CE (was EE-gated) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: lock prod against forking too and funnel edits to the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: open dev item page on edit and tailor dev-workspace lock messages Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: prevent nested dev workspaces and hide dev option when one exists Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: drop the redundant already-has-dev hint on the fork form Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: badge dev workspaces and sort them ahead of forks in the tree/switcher Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: label dev workspaces as 'Dev workspace of X' instead of 'Fork of X' Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: label edit as 'Edit in <dev>', cover editor headers, auto-expand dev in tree Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: split prod lock into separate block-deploy and prevent-forking toggles Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: make resources/variables workspace-specific from compare page Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: steer AI-chat sessions to the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: refine session fork options and lock guidance for dev/prod Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: session picker reads prod's real rules, default to current ws Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: copy members into forks and clarify dev-workspace root labeling Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: place the workspace id field under the fork name Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address dev-workspace review findings and harden fork detection Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: regenerate sqlx offline cache Restores entries dropped during the origin/main merge and adds the dev-workspace queries (is_dev_workspace, ws_specific, has_parent). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address second-round dev-workspace review findings Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address Pi and Codex review findings on dev-workspace endpoints Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate locked-dev git-branch fork on admin and validate ws_specific path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: clear prod dev-lock when deleting an attached dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: consolidate dev-workspace migration and scope all-group join to attach Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: restore dev-workspace CHECK into consolidated migration and scope all-group join Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: drop copy_members from the dev-workspace attach path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: dev-workspace lifecycle/auth fixes from Codex review round Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: explicit create-in-other for workspace-specific items Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make create-in-other strictly create-only (never overwrite target) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: return 403 (not 401) for dev-workspace permission denials Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: allow attaching a same-family fork as a dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: emphasize the go-to-dev action in the no-direct-deploy alert Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: seed a resource's linked variables when creating it in the other workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: judge workspace deploy/fork locks against the user's identity in that workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: clarify create-in help text in workspace-specific panel Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: admin-gate dev-workspace creation and harden lock/seed edges Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: preserve a staged fork's source on picker create-mode re-entry Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: clear dev flag on archive and check dev existence server-side Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make create-in-other atomically create-only via direct create Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: create-only resource insert, ws-specific list scopes, archive lock guard Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: reserve the dev_workspace_lock protection-rule name from the public API Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: reattach create_protection_rule doc comment to its function Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: make dev-archive pairing teardown atomic with the archive Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: follow deploy_to on root rename; show dev pairing to non-member prod admins Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: copy creator metadata on fork; invalidate fork routing cache on rename Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: accept g/ paths in set_ws_specific; gate copy_members to dev workspaces Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
75ba81b2d2 |
fix(audit): don't read pg_authid from an elevated context in S3 export migration (#9832)
* fix(audit): don't read pg_authid from an elevated context in S3 export migration Migration 20260626132251 aborted instance startup on managed Postgres (e.g. Cloud SQL) with "Modifying pg_authid or pg_auth_members is not allowed in elevated context": the audit S3 export "oldest in-flight xact_start" floor probe calls pg_has_role(...), which reads pg_authid, and managed providers forbid that read from an elevated context. The migration ran the probe inline in its UPDATE, so the whole migration — and the instance boot — failed. Extract the probe into a shared SQL function audit_logs_s3_oldest_inflight_ts() that returns the oldest in-flight xact_start (when cluster-wide stats are visible) or NULL otherwise. The pg_has_role read is wrapped in a plpgsql BEGIN/EXCEPTION subtransaction, so a pg_authid failure returns NULL (callers fall back to a conservative 7-day window / reject) instead of aborting. is_superuser (a GUC, no catalog read) is checked first to short-circuit. The migration's trigger and UPDATE, the OSS backfill try_start, and the EE exporter/startup anchor (companion windmill-ee-private PR) all route through it. Because 20260626132251 already shipped, it is added to the potentially_stale list in windmill-api/src/db.rs: on startup the stale _sqlx_migrations row (checksum mismatch) is deleted and the fixed, idempotent migration re-applies, so already-migrated instances upgrade without a checksum-mismatch boot failure. Fixes WIN-2108 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 95352c13c4c82247d8cfd80936f9203aeb079802 This commit updates the EE repository reference after PR #635 was merged in windmill-ee-private. Previous ee-repo-ref: 136f49a52af922868acac33abf8198913a9e835c New ee-repo-ref: 95352c13c4c82247d8cfd80936f9203aeb079802 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
003a262a4e |
feat: column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) (#9814)
* feat: column-level lineage for ducklake pipelines via // column annotation Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: auto-derive column lineage from DuckDB SQL AST (annotation as override) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: clarify column-lineage inference is server-side; drafts use annotations Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): surface inferred column lineage in live pipeline drafts Threads the DuckDB SQL-AST column lineage (from the WASM asset parser) through ScriptEditor -> details pane -> page -> resolveGraph, merged with // column annotations (annotation wins) so the live preview matches the deployed graph. Takes effect once windmill-parser-wasm-asset is republished with the inference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(frontend): bump windmill-parser-wasm-asset to 1.740.0 for SQL column-lineage inference Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: column-lineage inference now runs live (WASM) too, merged with annotations Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): transitive column-lineage trace (impact analysis) Stitches every producer's column_lineage into a pipeline-wide column graph (columnLineageGraph.ts) and replaces the single-hop diagram with an interactive ColumnLineageTrace: select an asset to see its columns' full upstream/downstream lineage across scripts; click any column to highlight its complete transitive impact set (forward + backward) and dim the rest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address CI review on column lineage (parse-fallback, node-id, perf, leak) - backend: DuckDB SQL parse failure now falls back to `// column` annotation lineage instead of dropping it (Codex P1) - columnLineageGraph: collision-proof JSON node ids; deterministic first-write output anchoring when a producer has multiple ducklake writes (cubic P2 ×2) - pipeline page: gate buildColumnGraph to a ducklake-asset selection so it doesn't rebuild on every editor keystroke (cubic P2) - ScriptEditor: clear inferredColumnLineage on parse error so it can't leak across a script switch (cubic P2) - AssetGraphEdge: widen badge stacking offset 12px->18px to fully clear (cubic P3) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: resolve JOIN inputs + anchor column lineage to // materialize target Addresses the second Codex review pass (two P1s): - SQL inference now walks JOINed tables: build_from_maps maps every FROM entry AND its joins into the alias map, and single-table attribution requires no joins. `SELECT o.x, c.y FROM a o JOIN b c` now resolves c.y (was dropped). - The column graph anchors a producer's lineage to its declared // materialize target (surfaced on the runnable node) instead of guessing a ducklake write-edge, which is unordered for deployed graphs and ambiguous for multi-output scripts. Falls back to a write-edge when no materialize target. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate column-lineage badge to the // materialize target write-edge The canvas badge keyed on `e.asset_kind === 'ducklake'`, so a multi-output producer showed the same column mapping on every ducklake write-edge. Use the same materialize-target anchor as buildColumnGraph: the badge lands only on the declared output's edge, falling back to the ducklake write-edge when there's no materialize annotation. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: build column trace from displayGraph so View hides draft lineage The transitive column trace was built from graphWithDraft regardless of mode, so in View with drafts hidden it could surface draft `// column` lineage the deployed canvas doesn't show. Build it from `displayGraph` (the graph the canvas actually renders) so the trace matches: draft overlays in edit / show-drafts, deployed-only in plain View. (Codex P2) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: don't infer column lineage for local/temp staging CTAS A CTAS into a local/temp staging table isn't the materialized output, but its projection was inferred and (flat) column_lineage anchored to the script's // materialize target — so staging columns showed up as the final asset's. Gate inference to the actual output: a top-level managed-materialize SELECT, or a CTAS/CREATE VIEW whose target resolves to a real asset. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: scope inferred column lineage to one output asset Inference accumulated columns from every output-producing query into one flat list, all anchored (frontend) to the script's // materialize target — so an auxiliary CTAS into a different asset showed its columns on the materialized one. Tag each inferred entry with its output asset and, in parse_assets, scope the list to the // materialize target (keeping untagged top-level-SELECT entries); with no declared target, drop inference when entries span multiple output assets rather than attribute them to an arbitrary one. Parser-internal — no wire change. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: treat CREATE TEMP TABLE/VIEW as local even under an active USE A one-part temp name under `USE dl` resolved to an asset (ducklake://…/tmp) before being registered local, so a final SELECT reading it invented `final.total <- warehouse/tmp.amt` (a phantom DuckLake column) and recorded a phantom asset. track_table_definition now registers any temporary table/view as local up front, bypassing active-asset resolution; CreateTable/CreateView pass their `temporary` flag. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
577ceeee86 |
perf(audit): re-anchor S3 audit export on enable + opt-in backfill (#9818)
* [ee] perf(audit): re-anchor S3 audit export on enable + opt-in backfill
The S3/GCS audit-log export's steady-state query filters by `age(xmin)`
(unindexable), so the only scan bound is the timestamp floor. On a fresh
enable the floor was epoch, and on a re-enable the cursor resumed from its
pre-disable position — either way the first run scanned the whole
`audit_partitioned` table. Under a `statement_timeout` (e.g. Aiven) that scan
never completes: the cursor never advances, nothing is exported, and the
repeated full scans saturate the database.
Re-anchor on enable (EE companion, windmill-ee-private#634):
- New trigger migration records a recent timestamp floor instead of the epoch
sentinel and `DO UPDATE`s the cursor to the current snapshot xmin on
re-enable, so the export always resumes from ~now and never rescans history.
Includes a one-time fixup for legacy epoch-sentinel checkpoints on upgrade.
Opt-in historical backfill (new `audit_logs_s3_backfill` module + endpoints):
- Exports a chosen `[from, to)` window on demand, scanning strictly by
`timestamp` (the partition key) in bounded keyset pages — each query is an
index scan capped at one page (verified via EXPLAIN: later partitions
`never executed`, ~11ms/page), so it stays well under any statement timeout
regardless of window size. Writes alongside the steady-state objects under
logs/audit/, without touching the xmin cursor.
- POST /settings/audit_logs_s3_backfill {from,to} (super-admin + Enterprise),
GET /settings/audit_logs_s3_backfill_status.
Also repurposes the status endpoint's `bootstrapping` flag to mean "draining a
backlog" (the cursor is capped and catching up), and updates the setting
description to point operators at the backfill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): heartbeat backfill lease per object; bump EE ref
Address review (cubic): persist progress (refreshing the lease heartbeat) after
every object PUT in the backfill page loop, not only once per page, so the gap
between heartbeats stays well under STALE_HEARTBEAT_SECS even on slow uploads
and another replica can't re-claim mid-page and run a concurrent backfill.
Bumps ee-repo-ref.txt to pull in the EE test-race fix (folding the backlog-drain
regression into the single audit e2e test).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject unstable backfill windows; bump EE ref
Address review (P1): the backfill keyset-pages over rows visible at scan time
and declares completion when the scan runs dry, but a row's `timestamp` is its
inserting transaction's `xact_start`. A window whose upper bound is recent or in
the future could silently omit a transaction that started inside `[from, to)`
but commits after the scan passed that timestamp. `try_start` now rejects any
`to` newer than the oldest in-flight `xact_start` (everything strictly older
than the oldest running transaction is committed and stable), using the same
trustworthy stats gating as the exporter's floor (restricted role / 2PC → a
7-day-old cutoff).
Bumps ee-repo-ref.txt for the EE monotonic-checkpoint fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): re-anchor legacy epoch checkpoints instead of synthetic floor
Address review (P1): the legacy-checkpoint fixup stamped last_oldest_inflight_ts
to now()-7d while leaving the old last_xmin in place. On an instance that
enabled export on the old code >7 days ago and got stuck before the first
successful batch, the next run would filter post-enable rows older than 7 days
out via `timestamp >= ts_floor` while still advancing last_xmin over the
interval — silently dropping them (the same floor-vs-cursor loss class fixed
elsewhere in this PR), and contradicting the "nothing committed after enabling
is skipped" guarantee.
A stuck epoch-sentinel checkpoint cannot be safely resumed (its backlog can be
arbitrarily old, so any recent floor prunes rows the cursor then skips, and an
epoch floor reintroduces the full scan). Re-anchor it to the migration's current
snapshot xmin instead — exactly like a fresh enable — so the export resumes
cleanly from ~now and the never-exported pre-upgrade window is recovered via the
opt-in backfill rather than silently dropped. Reword the setting description so
it no longer implies the disabled/legacy window is covered by the cursor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(audit): end-to-end integration tests for the object-store backfill
The backfill previously had only SQL-level/EXPLAIN validation. Add real
integration tests (in-memory object store, sqlx::test) exercising the public
path:
- backfill_exports_window_in_pages: with the page size forced to 2 rows, a
settled 3-day window is exported across multiple keyset pages; asserts every
in-window row lands exactly once, rows outside [from,to) are excluded, a day
that straddles a page boundary yields more than one object, progress counts
match, and a re-run is idempotent (deterministic keys overwritten, no dupes).
- backfill_rejects_unstable_window: a future/live `to` is rejected as unstable,
a window safely in the past is accepted.
Adds a test-only PAGE_ROWS override so multi-page behaviour is exercised with a
handful of rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(audit): note backfill scope is audit_partitioned only
Make explicit that, like the steady-state export, the backfill reads only
audit_partitioned; the pre-partitioning `audit` table is intentionally out of
scope (not a missed case).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject backfill windows before the partitioned boundary
Address review (Codex P1): the backfill reads only audit_partitioned, but
pre-partitioning history lives in the legacy `audit` table (still read by audit
list/get via UNION ALL, and retained for the configured period — 365 days by
default on EE). Since the setting text points operators at this API for
"pre-existing history", a window overlapping legacy rows would report completion
while silently omitting them.
Per the decision to not export the legacy table, reject instead of silently
omit: try_start now rejects a `from` earlier than the oldest audit_partitioned
timestamp (every legacy row predates the partition cutover, so a `from` at/after
that boundary can never overlap them). Reworded the setting text to scope the
backfill to the partitioned era. Added a regression test, plus an RAII guard
(cubic P2) so the test-only globals are restored even if an assertion panics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): backfill object keys per-window; require trustworthy settled cutoff
Address review (two P1s):
- Object-key overwrite loss: keys were `dt=<day>/audit_backfill_<min_id>.ndjson`.
A narrower, overlapping backfill can start a day's page at the same first row
(same min_id) but hold fewer rows, and `put` would overwrite a broader run's
object — silently dropping the rows only that object held. Include the
requested window in the key so different ranges write disjoint objects (same
window re-runs stay idempotent; consumers dedupe overlapping rows by id). New
regression test (verified red→green).
- Untrustworthy settled cutoff: when min(xact_start) isn't trustworthy (role
lacks pg_read_all_stats/superuser, or a prepared 2PC txn exists), the old
now()-7d fallback could still let an old transaction commit rows inside an
accepted window after the scan, so a "complete" backfill silently missed them.
Since a backfill asserts completeness, reject in those cases instead of
falling back. (The continuous exporter keeps its 7-day fallback — it only
claims bounded lag.)
Also makes the tests robust under the parallel runner: run_backfill takes the
store as a param, so tests pass a local in-memory store (no global
OBJECT_STORE_SETTINGS race) and serialize on the PAGE_ROWS override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject backfill overlapping legacy table; regen deref openapi; trim migration comment
Address review (1 P1 + 2 P2):
- Empty-partition backfill (P1): the min(audit_partitioned) guard no-ops when
audit_partitioned is empty, so an upgraded instance with legacy `audit` rows
but no partitioned rows yet would accept a window and complete with zero rows,
silently omitting the legacy rows. Check the legacy `audit` table directly:
reject any window that overlaps a legacy row (subsumes the boundary check and
covers the empty-partitioned case). Test updated accordingly.
- openapi-deref (P2): regenerate openapi-deref.yaml/json (served via include_str!)
so /openapi.{yaml,json} expose the new backfill endpoints.
- Migration comment (P2): trim the PR-history narration to the durable
constraints (why a recent floor and a monotonic cursor are required), per
AGENTS.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to b821fecccbcba2efed544890576bf2b84321d70d
This commit updates the EE repository reference after PR #634 was merged in windmill-ee-private.
Previous ee-repo-ref: 6b191b77aabcf77658ad4f9031576e0d7b66bf89
New ee-repo-ref: b821fecccbcba2efed544890576bf2b84321d70d
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
Ruben FiszelandClaude Opus 4.8
|
ade74b297f |
feat: capture managed-materialize output schema as asset metadata (#2a) (#9812)
* feat: capture managed-materialize output schema as asset metadata (#2a) After a managed `// materialize` run, capture the producer's output schema via a DESCRIBE folded into the existing one-row summary read (no extra round-trip) and persist it in a new versioned `materialized_asset_schema` sidecar table. This is the producer-side capture that pipeline parity gap #2b (save-time consumer-ref contract enforcement) will read back. - materialized_asset_schema sidecar (asset-level grain), versioned: a new version row is inserted only when the captured column set changes. - output_schema column added to the materialize summary codegen. - worker extracts + records the schema on a successful materialize. - /assets/asset_schemas read endpoint exposing the evolution history. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address CI review on schema capture (partition col, order, status gate) - exclude the synthetic `_wm_partition` column from the captured schema for partitioned assets, so the recorded contract is the producer's logical output, not Windmill's storage detail (claude/cubic P1). - make the captured column list explicitly ordered (`row_number()` over the DESCRIBE + `list(... ORDER BY)`), so the `list()` aggregate can't reorder columns and spuriously bump the schema version (cubic P2). - gate the API `record_materialization` schema upsert on a `Materialized` status, so a failed/running write (or a client attaching a schema to one) can't advance the schema history (cubic P2). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address Codex review (manual-mode schema gate + auth contract docs) - gate output_schema extraction on the managed (`Some((Some(_), _))`) path so a `// materialize manual` run — whose result is the user's own query output — can't persist a caller-shaped `output_schema` into materialized_asset_schema (Codex P2). Verified e2e: a manual run returning a fabricated `output_schema:[{injected,EVIL}]` records the partition but writes no schema version, while the managed path still captures normally. - document the authorization contract on the new public `record_asset_schema` and `list_asset_schemas` helpers: they perform no access control (mirroring the materialized_partition siblings) and require callers to pass a workspace-authorized executor (Codex P1). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): schema-history tab on the ducklake asset node (#2a) Adds a "Schema" tab to DucklakeAssetPanel surfacing the captured output-schema versions persisted by the materialize run. Master-detail (mirrors the History tab): the version list (newest first, newest auto-selected) shows column count + snapshot + capture time; selecting a version renders its column/type table. Reads the GET /assets/asset_schemas endpoint via raw fetch, matching the sibling PartitionStatusGrid convention (these materialization endpoints are not in the generated client). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: schema tab is strategy-aware (history vs fixed schema) Only a whole-table `replace` producer (CREATE OR REPLACE) can change columns run-to-run; `append`/`merge`/partitioned writes INSERT into a fixed-schema table, so their schema is pinned at first materialize and the "history" framing is degenerate (always one version). - backend: surface the managed `materialize_strategy` (`replace`/`append`/ `merge`) on the asset-graph runnable node, alongside the existing `partition_kind` (same parse-from-annotation path). - frontend: the pipeline page derives `schemaCanEvolve` for the selected asset from its write-producer (`replace` && not partitioned) and threads it to the Schema tab. Evolvable → master-detail version history; fixed → a single current-schema table with a short "schema is fixed" note. Unknown defaults to evolvable so real history is never hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: schemaCanEvolve fails open on unknown producer strategy Previously a producer present but missing `materialize_strategy` (e.g. a draft-overlay runnable, synthesized without the field) fell through to canEvolve=false, hiding captured history behind the fixed-schema view — contradicting the "unknown defaults to evolvable" intent. Now the fixed view shows only when *every* producer is a known insert-style write (append/merge, or partitioned replace); any producer with unknown (missing) strategy is treated as evolvable, so real history is never hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
5549bdc67a |
fix(debounce): never supersede a running debounce survivor (#9780)
* fix(debounce): never supersede a running debounce survivor
Companion to the windmill-ee-private change in upsert_debounce_key.
With debounce_args_to_accumulate + a concurrent_limit, a message arriving
while its debounce survivor is already running was marked completed/skipped
("Debounced Running by ...") and the running survivor deleted from the
queue, silently dropping accumulated elements. A slow step + concurrent
limit keeps the survivor running for a long window, so any arrival during
it was lost. The fix leaves a running survivor untouched and starts a fresh
debounce window for the late arrival.
Adds regression coverage in windmill-queue/tests/debounce_test.rs (push,
flow post-preprocessing, no-accumulation, committed-running, and
max-count-window cases) and refreshes the SQLx cache for the changed
upsert_debounce_key queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): add missing SQLx cache for test-only running-flag query
The cargo_test CI job compiles the test target with SQLX_OFFLINE=true; the
new regression tests use `UPDATE v2_job_queue SET running = true ...` which
was not in the offline cache (the library-only `cargo sqlx prepare` skipped
test targets). check_oss/check_ee passed because they don't build tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): harden running-survivor guard against concurrent arrivals
Companion to windmill-ee-private: switch the running-state check to a
correlated EXISTS on the post-conflict-lock holder so two late arrivals
racing after a survivor started running can't both spawn independent
windows (the row lock serializes them; the second debounces into the
first's fresh window).
Adds a concurrent regression test
(test_debounce_concurrent_arrivals_after_running_survivor) asserting
exactly one late arrival survives and the other is debounced, and refreshes
the SQLx cache for the updated upsert_debounce_key queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): serialize upsert per key (simpler, race-free)
Companion to windmill-ee-private: the running-survivor guard and batch
chaining are now protected by a per-key advisory lock instead of
snapshot-sensitive single-statement SQL. This closes a concurrent-arrival
data-loss race where a debounced late arrival's args could be dropped
because the batch lookup couldn't see the predecessor's just-committed
batch row.
Extends test_debounce_concurrent_arrivals_after_running_survivor to pull the
survivor and assert its accumulation includes BOTH racing late arrivals
(shared batch), and refreshes the SQLx cache for the rewritten queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): atomic upsert robust to concurrent pull-time key deletion
Companion to windmill-ee-private: keep upsert_debounce_key a single atomic
INSERT ... ON CONFLICT DO UPDATE so a chaining push cannot fail when the
worker pull path concurrently deletes the holder's debounce_key (the prior
read+UPDATE split could hit "no row updated"). Adds
test_debounce_push_races_key_deletion_by_pull (races a chaining push against
the key deletion 50x, asserts the push never errors) and refreshes the SQLx
cache.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(debounce): claim-based exactly-once batch consumption
Eliminates the rare duplicate/loss when two survivors land on one debounce
batch (a narrow push/pull race), without locking the worker pull hot path.
- migration: v2_job_debounce_batch gains consumed_at + consumed_by.
- pull side (maybe_apply_debouncing): instead of deleting the batch on consume,
a survivor atomically claims its own row + any unclaimed siblings (stamping
consumed_by = itself) and accumulates exactly the rows it claimed. A second
survivor of the same batch finds its row already consumed by another job and
runs empty (no duplicate); a re-pulled survivor recognizes its own prior claim
and keeps its accumulated args; a never-batched job (CE/legacy) keeps its own
args. Non-accumulate debounce paths still hard-delete their batch rows.
- complete_debounced_job (EE companion) never completes a running predecessor,
so its in-flight run is not killed (no loss); the claim then prevents the
duplicate the guard would otherwise allow.
- monitor: GC sweep deletes consumed batch rows past a 1h grace.
Together with the running-survivor guard this makes debounce accumulation
exactly-once. Adds tests: batch_consumed_exactly_once, repull_keeps_accumulated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): exhaustive edge cases + tighten consumed-batch GC grace
Tighten the consumed debounce-batch GC grace 1h -> 10min: per-op cost of the
claim is unchanged (an indexed mark is as cheap as the old delete), so the only
cost of retaining consumed rows is table growth, which a shorter grace bounds
under high-throughput debounce (a survivor that could still reference a row is
pulled long before 10min; GC is not correctness-critical since a re-pull whose
row was swept falls back to its persisted args).
Adds edge-case tests: never-batched keeps own args (CE fallback), concurrent
claim partitions a batch disjointly (exactly-once under real concurrency),
three survivors -> first takes all / rest run empty, non-accumulate debounce
hard-deletes its batch rows (no leak), and the GC sweep deletes only
past-grace consumed rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): port the #9781 regression case, flow-node guard, full-path bench
- Port the regression from #9781
(test_post_preprocessing_debounce_into_running_survivor_loses_message):
post-preprocessing survivor accumulates + runs, a later same-key message must
start a new batch (survive) not be folded into the running survivor. Exercises
the full EE path via jobs_ee::maybe_debounce_post_preprocessing.
- Add the third EE entry point's guard:
test_flow_node_debounce_running_survivor_not_superseded (maybe_debounce_flow_node).
- Add an #[ignore] full-source throughput bench (bench_debounce_full_path) driving
the real maybe_debounce + maybe_apply_debouncing end-to-end.
All debounce tests exercise the real jobs_ee implementation (run with
--features private,enterprise); none stub it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): scalar-arg accumulation + GC-then-repull no-loss
Close two accumulation edge gaps (both run on --features private,enterprise,
exercising the real jobs_ee path):
- accumulate bare-scalar values (the T | T[] union fallback): each scalar is
wrapped and accumulated into the survivor's list.
- GC reclaiming a survivor's consumed batch row before a re-pull must not lose
data: the re-pull finds no row and keeps its already-persisted accumulated
args (had_row=false fallback), rather than running empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): real-worker end-to-end accumulation test
Drives the full real path on --features enterprise,deno_core,private: push 3
same-key debounced flow jobs (real push() -> maybe_debounce collapses the
batch), a real worker pulls the survivor (real pull() -> maybe_apply_debouncing
claim+accumulate) and executes the deno flow, then asserts the executed result
is the full accumulated set [1,2,3] and the two superseded messages are skipped.
Complements the in-process unit tests with a genuine worker-execution run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): atomic claim+persist, GC only non-queued rows; reword comment
Address review findings:
- [P1] Claim and accumulated-args persist are now in one transaction. Before,
a crash between stamping batch rows consumed_by=self and the `UPDATE v2_job
SET args` could let a zombie re-pull see its own prior claim and keep only its
own args (dropping the siblings it had claimed). Wrapping claim + accumulate +
persist in a tx makes them commit together or roll back together (re-pull then
re-claims cleanly).
- [P1] GC of consumed batch rows now also requires the job to no longer be in
v2_job_queue. A consumed sibling can stay queued well past any time grace under
a concurrency limit / backlog; reclaiming its marker by age alone let its
eventual pull treat it as never-batched and re-run its item (a duplicate).
Keeping the row until the job leaves the queue preserves the "already consumed"
signal. Test extended with a still-queued consumed row that must survive GC.
- [P2] Drop "Customer" attribution from a test doc comment (AGENTS.md).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): emit accumulation log after committing the claim transaction
append_logs opened a second pool connection while the claim transaction (and its
batch row locks) were still held; under concurrent debounced pulls that risks
pool-exhaustion stalls/timeouts. Defer the log line until after tx.commit().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 6aabd7c5ce53b9153be05c3e7bc9a76eadb1a48a
This commit updates the EE repository reference after PR #631 was merged in windmill-ee-private.
Previous ee-repo-ref: 30d740e619fad219108ec4b4c6a9d67c1ab42d46
New ee-repo-ref: 6aabd7c5ce53b9153be05c3e7bc9a76eadb1a48a
Automated by sync-ee-ref workflow.
* fix(debounce): claim whole batch in one UPDATE (no deadlock); assert test setup
Both Codex (P1) and Claude (P2) flagged a deadlock: the claim used two writable
CTEs (claim_self then claim_rest), locking the self row before siblings, so two
survivors of the same batch pulled concurrently acquired row locks in opposite
order and PostgreSQL aborted one with deadlock_detected (a transient pull error
on exactly the two-survivors race this path handles).
Replace with a single `UPDATE ... WHERE debounce_batch = (...) AND consumed_at IS
NULL RETURNING id` that claims the whole batch: both transactions lock rows in
the same scan order, so one simply waits and re-evaluates under EvalPlanQual.
A `claimed_self` flag (EXISTS id = self in the claimed set) plus the `mine`
snapshot still distinguishes fresh-claim / consumed-by-other / own-re-pull.
Also assert add_survivor_to_batch_of actually inserts a row (rows_affected == 1)
so a mis-set-up test can't pass vacuously.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
Ruben FiszelandClaude Opus 4.8
|
aa098c70c0 |
perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes (#9786)
* perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: document delete_jobs auth contract and workspace-scope jobs_export purge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
12f92e3ab7 |
[ee] feat(backend): native script retry without one-step-flow wrapping (#9688)
* feat(backend): native script retry without one-step-flow wrapping Schedules and data pipelines that retry a single script previously wrapped it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a v2_job_status row, and UI projection complexity. This adds native retry on a plain JobKind::Script job. - RetrySettings: flatten Retry into a deduped retry_settings table, carried via the existing runnable_settings_handle (lazy, off the hot path). - push() materializes a bare-script-with-retry SingleStepFlow into a native Script job (gated on min-version + no handlers/retry_if). - add_completed_job re-pushes the next attempt on failure with backoff, tracking the attempt counter in v2_job_queue.extras and the chain via parent_job; schedule completion handlers fire only on the terminal attempt. - frontend: ScriptRetryChain shows the attempt chain on the run page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): native retry_if eval + per-occurrence schedule handlers Extends native script retry to the two cases that previously stayed on the one-step-flow path: - retry_if: evaluated natively on the failure path via a feature-gated windmill-jseval dep (quickjs) over the failure result + flow_input; push materializes such policies natively only when quickjs is available. - on_failure_times / on_recovery: apply_schedule_handlers now resolves each past scheduled occurrence's terminal status across its native-retry chain (root OR any parent_job=root child succeeded) and excludes the current occurrence, so the counting is per-occurrence rather than per-attempt. All scheduled-script retries now go native (schedule.rs gate removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): always materialize retry_if natively; unsupported without quickjs retry_if is evaluated by the worker (which always has quickjs), not the pusher, so gating materialization on the pusher's feature was wrong. The flow path was never a real fallback either — the flow runtime needs quickjs to evaluate retry_if too. retry_if now always goes native; on a worker without quickjs it is unsupported and fails closed (no retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): un-park asset-cascade (pipeline) retry Native retry resolves the blocker that parked pipeline retry: a retried subscriber is now a Script job (not a one-step flow / flow step), so it stays eligible for asset dispatch and can trigger its own downstream on recovery. - scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset edges (was dropped with a TODO warning). - asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job, so native-retry attempts dispatch on success while flow steps stay excluded. - tests: retry-bearing subscriber now dispatches as a native Script carrying the policy in runnable_settings_handle; native-retry attempt is eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): cap native retry interval, lazy result serialization, idempotent retry push Hardening from a self-review of the native retry path: - Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path (evaluate_retry); the exponential formula could otherwise schedule up to ~18h vs the flow path's 6h. - Serialize the failure result lazily (only when a retry_if policy needs it), so the common failure no longer pays the serialization on the failure path. - Push each retry with a deterministic id per (root, attempt). If a worker dies between enqueueing the retry and finalizing the current attempt, the reaper re-handles the attempt and lands here again — push rejects the duplicate id, so the retry is enqueued exactly once (no double-retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): defer schedule handlers idempotently on retry-push replay (review P1) Address local-review findings: - P1: retry_pending was derived from the retry push *result*, so on a worker crash + reaper replay the duplicate-id push returned Err → retry_pending flipped to false → apply_schedule_handlers fired for the non-terminal attempt (and the terminal attempt later fired them again). Pre-check whether the deterministic retry id already exists and report it as pending without re-pushing, so the handler-deferral invariant is crash-idempotent too. - P2: refresh the stale 'wrap the script in a one-step flow' comment in the asset-cascade retry push — it now materializes a native Script. - Add RetrySettings <-> Retry round-trip unit tests (clamping edges). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): native retry chain + per-occurrence status sqlx tests Close the two integration-test gaps flagged in local review: - chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts) and asserts crash-replay idempotency (the P1 fix: a replayed completion reports pending without double-enqueueing). - per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but- recovered occurrence counts as success, retries (parent_job set) are excluded from occurrence counting, and the current occurrence is excluded. - canceled_job_does_not_retry: cancellation wins over a pending retry. Runtime sqlx API (no .sqlx cache entry needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): exclude schedule handlers from the retry-attempt chain The retry chain listed all script children of the root by parent_job, but schedule completion handlers (on_failure/on_recovery/on_success) are also script children — when the occurrence has no retries, the handler's parent is the root itself, so a successful, never-retried job rendered a bogus 'Retries (1)' badge pointing at the handler. Filter children to re-runs of the same script (matching script_hash); real retries keep the root's hash, handlers run a different script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): surface schedule handlers on the run page Extend the run-page chain component with schedule completion handlers: - A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/ on_success runs that fired for that occurrence (found as children of the terminal attempt, identified by their synthetic created_by). - A handler's own run page now shows a 'Failure/Recovery/Success handler' label with a link back to the run it handled and its schedule. on_recovery and on_success share created_by, disambiguated by the recovery-only error_started_at arg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare An earlier `cargo sqlx prepare` on this branch ran before #8801's folder_default_permissioned_as test merged in, so it pruned the 3 query caches that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main. * fix(backend): only cascade assets from native retry attempts, not handlers (review P1) is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script child became asset-eligible — including schedule/error/recovery handlers (Script jobs with parent_job set and no flow_step_id). A handler that declares assets would then trigger a cascade the old parent_job IS NULL guard prevented. Gate parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as its chain parent (handlers run a different script). Runtime query, no sqlx cache. * fix(backend): cache the private-gated retry_setting asset-dispatch test query The same prepare-without-private that dropped the folder_default caches also pruned the cache for the retry_setting_dispatches_subscriber_as_native_script test query (asset_trigger_dispatch.rs:721). Regenerated with --features private. * fix(backend): exclude handler children from per-occurrence recovery (review) A scheduled occurrence's on_failure/on_success handler runs as a successful child (parent_job = occurrence), and the per-occurrence success EXISTS counted ANY successful child — so a failed occurrence whose error handler succeeded was marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in the merge) and on_failure_times counting. EE query now scopes the EXISTS to same-runnable children (only native retry attempts); regenerate sqlx cache + bump ee-repo-ref. native_retry_test gains a handler-child regression case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): scheduled-script retry is a native Script, not SingleStepFlow test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a native Script. Update both to assert kind='script' and that the retry policy is carried via runnable_settings_handle. * fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic) Address cubic CI review: - P1: the SingleStepFlow->native Script materialization dropped dedicated_worker, so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve it from the script row in push so the materialized Script keeps the dedicated tag. - P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the u32->i16 // retry count narrowing (scripts.rs) instead of wrapping. * fix(backend): use a retry-specific signal, not runnable equality (codex review) Address Codex CI review: - P1: is_native_retry_attempt treated any same-runnable parented Script child as a retry. WAC v2 inline children have that exact shape, so an inline child of an asset producer would cascade. Use a retry-specific signal instead: the job carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no flow_innermost_root_job. Apply the same flow_innermost guard to the EE per-occurrence EXISTS (WAC inline children must not count as a recovery). - P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate now resolves as 'retry pending' (re-check on the duplicate-id error) instead of flipping retry_pending to false and firing handlers early. - Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): explicit native_retry_attempt marker, drop heuristics Replace the per-site "is this a retry?" inference (parent_job + runnable match + flow_innermost / retry_settings) with one explicit marker: a sparse native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker also carries the attempt counter (previously in v2_job_queue.extras), so it's the single source of truth. - asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker. - EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost. - maybe_enqueue: reads/writes the marker (persistent) instead of queue extras. - Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap. - Eliminates handler / WAC-inline-child misclassification by construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex) The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but the periodic monitor.rs path deleted v2_job rows without it, orphaning markers. Add the same WHERE job_id = ANY(...) sweep there. * fix(backend): widen native_retry_attempt.attempt to integer (cubic) The smallint column was cast to/from u32 and could wrap a retry chain longer than i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32 attempt count, so no narrowing occurs on the maybe_enqueue read/write path. * feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT - Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi), computed from the native_retry_attempt marker. The run-page chain now filters retry attempts by is_retry instead of the script_hash heuristic, so WAC v2 inline children (same script, parent_job) no longer render as retries (codex). - Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep the plain catch-and-continue matching the other side-table deletes; the table is created by a startup migration so it always exists when cleanup runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but does not select is_retry, which errored with "no column found". Only the list endpoint populates the marker; #[sqlx(default)] lets every other query omit the column and default to None. * feat(backend): select is_retry in single-job GET too for consistency The list endpoint already exposes the marker; populate it on the single-job GET (both completed and queued variants) as well so a run loaded directly reflects its retry status. #[sqlx(default)] stays as a safety net for any other query. * feat(backend): reap orphaned native_retry_attempt markers via periodic sweep The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so direct job deletions (workspace/job delete, schedule clearing) would leave marker rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which must then be remembered for every future path), reap orphans in the periodic delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the job). The table is sparse so the anti-join drives off it and probes v2_job by PK — cheap. Retention still sweeps markers inline (keeps the table small so this stays cheap); a transient orphan is harmless (nothing reads is_retry for a gone job). * fix(frontend): include flow handlers in retry chain handler row (codex) Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...), whose handler job is a flow, not a script. The chain fetched children with jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts are still selected by is_retry and handlers by created_by, so both kinds surface. * fix(backend): carry concurrency/debouncing settings into native retries maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings ::default(), dropping the script/pipeline concurrency settings the failed job carried in its runnable_settings_handle. A retry of a concurrency-limited script then inserted no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and pass them in the payload, which push forwards to the materialized retry. Adds a regression test asserting the retry's handle resolves to the concurrency settings. * fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex) P1a (schedule.rs): the scheduled-retry materialization fetched the script's concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings ::default() into the SingleStepFlow payload, so the root attempt's handle held only the retry policy and the whole chain ran unbounded. Pass the fetched settings. Regression test asserts the root handle resolves to retry + concurrency. P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract — it is pub only for the integration test; the sole production caller is the worker completion path passing a DB-derived, already-authorized MiniCompletedJob. * docs(backend): attach native-retry auth contract to the function itself (codex) The doc block was merged with eval_retry_if's doc and bound to that function, leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if keeps its own doc; the native-retry + authorization contract now sits directly above maybe_enqueue_native_script_retry. * docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex) - Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since 1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the served spec couldn't see the field. Now current at 1.739.0. - schedule.rs: a retry_if gate is evaluated at failure time and fails closed without quickjs (no retry); it does not fall back to a flow path. - windmill-types jobs.rs: is_retry is selected by both the list and single-job GET endpoints (not list-only). * docs(backend): fix remaining stale retry_if/quickjs comments (codex) The retry_if block and the push materialization comments claimed push keeps retry_if on a flow path / the worker always has quickjs. The code always materializes native retry and the no-quickjs eval_retry_if path fails closed — correct the comments to that constraint. * docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex) - Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated and the job does not retry (no one-step-flow fallback). - jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence failure/recovery status across the retry chain, so the old 'restricted to schedules whose handlers don't need per-occurrence counting' claim is dropped. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
f6998ec54c |
feat: data tests for ducklake pipeline materialization (#9708)
* feat: data tests for ducklake pipeline materialization Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): data_test count badge on pipeline graph nodes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: surface annotation badges (incl. data_test) on deployed pipeline nodes Backend graph endpoint now parses each pipeline member's deployed body and returns partition/freshness/tag/retry/data_test, so badges render on deployed nodes, not only live drafts. Aligns the TS DataTest.relationships fields to snake_case to match the Rust serde wire shape (the type is now populated from both the parser and the backend JSON). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): keep materialize output edge when editing the producer in the pipeline graph The live-edit overlay re-derived a selected/edited script's lineage from // on inputs + body-inferred assets only, so the // materialize <asset> output (an annotation, not body SQL) was judged stale and its write-edge dropped on select — leaving the materialized asset unlinked (and the node's annotation badges hidden). Include the parsed materialize target in liveRefKeys and the draft writeOuts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: run all data tests in one pass with a structured per-test result Replace the raise-on-first-violation probes with a single materialize summary that embeds every test's violating-row count in a data_tests column (computed in a CTE, since DuckDB rejects subqueries inside struct literals). The worker reads the breakdown and decides pass/fail: a clean run returns the per-test summary in the result; a failing run errors with the FULL list (every test, ✓/✗ + counts), not just the first failure. Verified live (EE) for built-ins + custom, pass and multi-failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): data-test pass/fail checklist in the job result DisplayResult renders a per-test checklist (✓/✗ + violation counts) above the raw result for managed materialize runs — from the structured data_tests on success, and parsed from the worker's breakdown message on failure. Shows in the script editor Test panel, the runs page, and the pipeline asset run pane. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): move data-test badge onto the producer→asset edge with run status The test badge now sits on the write-edge (the transformation link) rather than the producer node, since the tests assert on what the transformation produces. It's tinted by the producer's last-run status (green = passed, red = a test failed) and its hover title lists every declared test. Removes the now-redundant node badge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): render custom data-test scripts as their own clickable graph nodes A // data_test <script_path> custom test now appears as its own node below the asset it validates, joined by a dashed 'tests' edge. Clicking it opens the test script in the detail pane (dispatched like any runnable). Built-in tests stay folded into the edge badge; only script-backed tests become nodes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): type data-test edge field via AssetGraphResponse, not in-scope g BuiltEdge is declared at component scope, outside build(g), so referencing typeof g.runnables in its type failed CI's svelte-check (Cannot find name 'g'). Use the imported AssetGraphResponse type instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): anchor edge badge on routed path + a11y text on test icons Address review: the data-test edge badge anchored on the straight-line midpoint, floating off detoured edges — anchor it at detourX when the edge is routed through a gutter lane. Add sr-only pass/fail text so the checklist icons are distinguishable to screen readers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: close data-test enforcement bypass + gate badges to scripts + reject multi-stmt custom tests Address review (cubic) findings: - P1: managed materialize generates its own summary row carrying data_tests, and enforcement reads that column — but a // result_collection annotation (e.g. a scalar mode) could reshape the row and drop data_tests, silently bypassing a failing test. Force LastStatementAllRows for managed materialize runs so the summary row is always intact. - P2: asset-graph annotation badges were keyed by path only, so a flow sharing a path with a pipeline script inherited its badges. Gate the lookup on usage_kind == Script. - P2: a custom test body is embedded as a subquery, so a multi-statement body produced invalid SQL with an opaque DuckDB error. Validate single-statement up front with an actionable error; align docs/comments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: fail loud if fewer data-test outcomes recovered than declared Defense-in-depth from the fresh-context review: enforcement reads per-test outcomes off the materialize summary row, but if the data_tests column were ever dropped/reshaped at the FFI boundary, extract_data_tests would return fewer (or zero) outcomes and the run would silently pass unverified tests. Track the embedded test count on MaterializeExec and abort with a clear error when recovered < declared. Verified: normal run (4==4) unaffected; the scalar-result_collection bypass already fails. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: relationships data test same-lake reuse + schema-qualified target quoting Address Codex/Pi review (two P1s in the relationships codegen): - A relationship into the same ducklake as the materialize target minted a second ATTACH of that lake under _wm_ref_N while _wm_target already held it — DuckDB forbids attaching one database twice, so the test failed before it could run. Reuse _wm_target for same-lake references. - A schema-qualified target (ducklake://warehouse/main.dim_products.sku) emitted FROM _wm_ref_0."main.dim_products" — one quoted identifier with a literal dot — silently querying a nonexistent table. Quote each dotted segment so the dot stays a schema separator. Adds tests for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): refresh data_test badge on deployed-script drafts + scope to materialize target Address Codex review nits (both P2): - resolveGraph: the existing-runnable draft-overlay branch kept the deployed data_tests, so adding/removing // data_test lines on an already-deployed script left the badge stale until redeploy. Refresh it from the live parse like the new-runnable branch. - AssetGraphCanvas: data tests were attached to every write-edge from a producer. They assert on the // materialize target (always a ducklake asset in v1), so only the ducklake write-edge now carries the badge and custom-test nodes — a producer's other (S3/datatable) outputs no longer show them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 GuilhemandClaude Opus 4.8
|
42c5e7a3fc |
feat: scope AI sessions per workspace root with lifecycle reconcile (#9734)
* feat: scope AI sessions per workspace family with lifecycle reconcile Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: centralize session reconcile trigger + extract pure lifecycle decision Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf: remove unused workspace family index * refactor: scope sessions by workspace root id, drop family_id column Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sessions): preserve user-archived sessions when archiving their workspace archiveSessionsForWorkspace tagged every session archivedByWorkspace, including ones the user had already archived by hand, so a later workspace unarchive auto-restored them. Skip already-archived sessions so only workspace-archived ones are tagged, matching decideSessionLifecycle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: archived-session banner with unarchive, suppress workspace-gone banner while archived Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: re-root sub-fork sessions on reconcile when an ancestor is deleted Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: group AI sessions by workspace family with show-all-workspaces filter Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: revert unrelated AIProviderPicker cosmetic changes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: hide per-session unarchive when workspace is gone, show move/discard instead Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: GC attached files on lifecycle delete + reconcile on sidebar fork delete Addresses Codex review: deleteSessionsForWorkspace/reconcile delete now GC linked files (deleteItemsForSession), matching deleteSession; sidebar deleteFork now reconciles so surviving child forks re-root off the deleted ancestor. Also de-flaked post-rehydrate reads in the IndexedDB tests via vi.waitFor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: don't strand user if post-delete reconcile throws; refresh stale warmSessions comment Addresses auto-review P2s: wrap reconcileAfterWorkspaceChange in deleteFork so the parent switch + navigation always runs even on reconcile failure; correct the warmSessions comment which no longer holds under 'Show all workspaces'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: don't fail/strand fork archive+delete when client session cleanup throws Addresses cubic P1/P2 on forks/compare: the workspace archive/delete is authoritative; wrap the best-effort session cleanup + reconcile so a local IndexedDB failure neither falsely reports failure nor blocks navigation away from the gone fork. Mirrors the SidebarContent fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop drafting-history aside from reconcileAfterWorkspaceChange comment Addresses auto-review P2: keep the refresh-before-reconcile invariant, drop the 'which they did inconsistently' narration per AGENTS.md (comments record constraints, not drafting history). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: clean up sessions on fork-id reuse + make all workspace-mutation cleanup best-effort Addresses Codex P1s: (1) CreateWorkspaceInner 'permanently delete existing fork' (id-reuse) now drops local sessions for that id so they don't resurface on the recreated fork; (2) workspace_settings archive/delete and SidebarContent child-delete loop + main delete now treat post-mutation session cleanup as best-effort, so a local IndexedDB failure can't strand the user or abort remaining deletes (matching the compare-page fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: make fork-reuse session cleanup fire-and-forget (non-blocking) Addresses cubic P2: don't await the best-effort cleanup so a slow IndexedDB op can't block the delete/reuse flow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: drop previous user's transient drafts on user change Addresses Pi P1: hydrateSessions preserved transient (unsent) drafts across user changes, so user A's draft + its pending fork/workspace state bled into user B's list and got reused by createSession. onUserChange now drops transients when the email changes; reconcile (intra-user) still preserves them. Regression test added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
55bed4abcf |
perf(audit): adaptive timestamp floor for S3 audit-log export (#9752)
* perf(audit): adaptive timestamp floor for S3 audit-log export (ee) EE change in windmill-ee-private (src/ee.rs); this OSS commit carries the regenerated sqlx cache for the new oldest-in-flight query and bumps ee-repo-ref.txt to the EE branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to ed89574be9117cda5e2d7d9de02cb5db066e93e3 This commit updates the EE repository reference after PR #628 was merged in windmill-ee-private. Previous ee-repo-ref: 8a7f645c0a194a284fe19dd20dbe79dd0733dfdb New ee-repo-ref: ed89574be9117cda5e2d7d9de02cb5db066e93e3 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |