Commit Graph
151 Commits
Author SHA1 Message Date
33521505db feat(ducklake): scheduled lake maintenance (expiry, compaction, orphan cleanup) (#9916)
* feat(ducklake): scheduled lake maintenance (snapshot expiry, compaction, orphan cleanup)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ducklake): review fixes — starts_with not LIKE, CE license-lapse escape

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(ducklake): auth-contract docs + _unchecked rename per codex review

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(ducklake): move maintenance payload construction into EE module

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ducklake): fall through to script resolution for non-managed reserved-prefix schedules

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(ducklake): document accepted pre-existing-schedule limitation on the reserved prefix

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ducklake): CE save-off clears the managed schedule row and queued occurrence

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: update ee-repo-ref to 2fab310d4f50ed7c34857d69c9b854f4491bf217

This commit updates the EE repository reference after PR #645 was merged in windmill-ee-private.

Previous ee-repo-ref: fff1fd830a36beba732486f05941ec243cf6b640

New ee-repo-ref: 2fab310d4f50ed7c34857d69c9b854f4491bf217

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-07-04 17:23:23 +02:00
Ruben Fiszelandwindmill-internal-app[bot] 53bbb92953 feat(pipeline): backfill a range of partitions from the asset drawer (#9885)
* feat(pipeline): backfill a range of partitions from the asset drawer (ee)

* feat(pipeline): cancel in-flight backfill job and show cancelling state

* refactor(pipeline): move backfill range logic behind private feature

* fix(pipeline): close backfill cancel-launch race and record dispatch intent

* docs(openapi): producer_path also covers SDK write-edge producers

* chore: update ee-repo-ref to c3852ecb36bd0be1a74c63169e513888f3347850

This commit updates the EE repository reference after PR #641 was merged in windmill-ee-private.

Previous ee-repo-ref: 7c1450ef89fbc9e844a121b39cafe0d7235d704b

New ee-repo-ref: c3852ecb36bd0be1a74c63169e513888f3347850

Automated by sync-ee-ref workflow.

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-07-02 12:50:55 +02:00
Ruben Fiszel e77b7523a5 nit enterprise implies license feature 2026-07-02 06:01:22 +00:00
Ruben FiszelandClaude Opus 4.8 12f92e3ab7 [ee] feat(backend): native script retry without one-step-flow wrapping (#9688)
* feat(backend): native script retry without one-step-flow wrapping

Schedules and data pipelines that retry a single script previously wrapped
it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a
v2_job_status row, and UI projection complexity. This adds native retry on a
plain JobKind::Script job.

- RetrySettings: flatten Retry into a deduped retry_settings table, carried
  via the existing runnable_settings_handle (lazy, off the hot path).
- push() materializes a bare-script-with-retry SingleStepFlow into a native
  Script job (gated on min-version + no handlers/retry_if).
- add_completed_job re-pushes the next attempt on failure with backoff,
  tracking the attempt counter in v2_job_queue.extras and the chain via
  parent_job; schedule completion handlers fire only on the terminal attempt.
- frontend: ScriptRetryChain shows the attempt chain on the run page.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(backend): native retry_if eval + per-occurrence schedule handlers

Extends native script retry to the two cases that previously stayed on the
one-step-flow path:

- retry_if: evaluated natively on the failure path via a feature-gated
  windmill-jseval dep (quickjs) over the failure result + flow_input; push
  materializes such policies natively only when quickjs is available.
- on_failure_times / on_recovery: apply_schedule_handlers now resolves each
  past scheduled occurrence's terminal status across its native-retry chain
  (root OR any parent_job=root child succeeded) and excludes the current
  occurrence, so the counting is per-occurrence rather than per-attempt.

All scheduled-script retries now go native (schedule.rs gate removed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(backend): always materialize retry_if natively; unsupported without quickjs

retry_if is evaluated by the worker (which always has quickjs), not the
pusher, so gating materialization on the pusher's feature was wrong. The
flow path was never a real fallback either — the flow runtime needs quickjs
to evaluate retry_if too. retry_if now always goes native; on a worker
without quickjs it is unsupported and fails closed (no retry).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(backend): un-park asset-cascade (pipeline) retry

Native retry resolves the blocker that parked pipeline retry: a retried
subscriber is now a Script job (not a one-step flow / flow step), so it
stays eligible for asset dispatch and can trigger its own downstream on
recovery.

- scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset
  edges (was dropped with a TODO warning).
- asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job,
  so native-retry attempts dispatch on success while flow steps stay excluded.
- tests: retry-bearing subscriber now dispatches as a native Script carrying
  the policy in runnable_settings_handle; native-retry attempt is eligible.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(backend): cap native retry interval, lazy result serialization, idempotent retry push

Hardening from a self-review of the native retry path:
- Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path
  (evaluate_retry); the exponential formula could otherwise schedule up to
  ~18h vs the flow path's 6h.
- Serialize the failure result lazily (only when a retry_if policy needs it),
  so the common failure no longer pays the serialization on the failure path.
- Push each retry with a deterministic id per (root, attempt). If a worker
  dies between enqueueing the retry and finalizing the current attempt, the
  reaper re-handles the attempt and lands here again — push rejects the
  duplicate id, so the retry is enqueued exactly once (no double-retry).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(backend): defer schedule handlers idempotently on retry-push replay (review P1)

Address local-review findings:
- P1: retry_pending was derived from the retry push *result*, so on a worker
  crash + reaper replay the duplicate-id push returned Err → retry_pending
  flipped to false → apply_schedule_handlers fired for the non-terminal
  attempt (and the terminal attempt later fired them again). Pre-check whether
  the deterministic retry id already exists and report it as pending without
  re-pushing, so the handler-deferral invariant is crash-idempotent too.
- P2: refresh the stale 'wrap the script in a one-step flow' comment in the
  asset-cascade retry push — it now materializes a native Script.
- Add RetrySettings <-> Retry round-trip unit tests (clamping edges).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(backend): native retry chain + per-occurrence status sqlx tests

Close the two integration-test gaps flagged in local review:
- chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry
  through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts)
  and asserts crash-replay idempotency (the P1 fix: a replayed completion reports
  pending without double-enqueueing).
- per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence
  terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but-
  recovered occurrence counts as success, retries (parent_job set) are excluded
  from occurrence counting, and the current occurrence is excluded.
- canceled_job_does_not_retry: cancellation wins over a pending retry.

Runtime sqlx API (no .sqlx cache entry needed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): exclude schedule handlers from the retry-attempt chain

The retry chain listed all script children of the root by parent_job, but
schedule completion handlers (on_failure/on_recovery/on_success) are also
script children — when the occurrence has no retries, the handler's parent is
the root itself, so a successful, never-retried job rendered a bogus
'Retries (1)' badge pointing at the handler. Filter children to re-runs of the
same script (matching script_hash); real retries keep the root's hash, handlers
run a different script.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(frontend): surface schedule handlers on the run page

Extend the run-page chain component with schedule completion handlers:
- A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/
  on_success runs that fired for that occurrence (found as children of the
  terminal attempt, identified by their synthetic created_by).
- A handler's own run page now shows a 'Failure/Recovery/Success handler'
  label with a link back to the run it handled and its schedule. on_recovery
  and on_success share created_by, disambiguated by the recovery-only
  error_started_at arg.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare

An earlier `cargo sqlx prepare` on this branch ran before #8801's
folder_default_permissioned_as test merged in, so it pruned the 3 query caches
that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main.

* fix(backend): only cascade assets from native retry attempts, not handlers (review P1)

is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script
child became asset-eligible — including schedule/error/recovery handlers (Script
jobs with parent_job set and no flow_step_id). A handler that declares assets
would then trigger a cascade the old parent_job IS NULL guard prevented. Gate
parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as
its chain parent (handlers run a different script). Runtime query, no sqlx cache.

* fix(backend): cache the private-gated retry_setting asset-dispatch test query

The same prepare-without-private that dropped the folder_default caches also
pruned the cache for the retry_setting_dispatches_subscriber_as_native_script
test query (asset_trigger_dispatch.rs:721). Regenerated with --features private.

* fix(backend): exclude handler children from per-occurrence recovery (review)

A scheduled occurrence's on_failure/on_success handler runs as a successful
child (parent_job = occurrence), and the per-occurrence success EXISTS counted
ANY successful child — so a failed occurrence whose error handler succeeded was
marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in
the merge) and on_failure_times counting. EE query now scopes the EXISTS to
same-runnable children (only native retry attempts); regenerate sqlx cache + bump
ee-repo-ref. native_retry_test gains a handler-child regression case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(backend): scheduled-script retry is a native Script, not SingleStepFlow

test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted
the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a
native Script. Update both to assert kind='script' and that the retry policy is
carried via runnable_settings_handle.

* fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic)

Address cubic CI review:
- P1: the SingleStepFlow->native Script materialization dropped dedicated_worker,
  so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve
  it from the script row in push so the materialized Script keeps the dedicated tag.
- P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the
  u32->i16 // retry count narrowing (scripts.rs) instead of wrapping.

* fix(backend): use a retry-specific signal, not runnable equality (codex review)

Address Codex CI review:
- P1: is_native_retry_attempt treated any same-runnable parented Script child as
  a retry. WAC v2 inline children have that exact shape, so an inline child of an
  asset producer would cascade. Use a retry-specific signal instead: the job
  carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no
  flow_innermost_root_job. Apply the same flow_innermost guard to the EE
  per-occurrence EXISTS (WAC inline children must not count as a recovery).
- P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate
  now resolves as 'retry pending' (re-check on the duplicate-id error) instead of
  flipping retry_pending to false and firing handlers early.
- Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(backend): explicit native_retry_attempt marker, drop heuristics

Replace the per-site "is this a retry?" inference (parent_job + runnable match +
flow_innermost / retry_settings) with one explicit marker: a sparse
native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker
also carries the attempt counter (previously in v2_job_queue.extras), so it's the
single source of truth.

- asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker.
- EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost.
- maybe_enqueue: reads/writes the marker (persistent) instead of queue extras.
- Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap.
- Eliminates handler / WAC-inline-child misclassification by construction.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex)

The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but
the periodic monitor.rs path deleted v2_job rows without it, orphaning markers.
Add the same WHERE job_id = ANY(...) sweep there.

* fix(backend): widen native_retry_attempt.attempt to integer (cubic)

The smallint column was cast to/from u32 and could wrap a retry chain longer than
i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32
attempt count, so no narrowing occurs on the maybe_enqueue read/write path.

* feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT

- Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi),
  computed from the native_retry_attempt marker. The run-page chain now filters
  retry attempts by is_retry instead of the script_hash heuristic, so WAC v2
  inline children (same script, parent_job) no longer render as retries (codex).
- Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep
  the plain catch-and-continue matching the other side-table deletes; the table is
  created by a startup migration so it always exists when cleanup runs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it

The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but
does not select is_retry, which errored with "no column found". Only the list
endpoint populates the marker; #[sqlx(default)] lets every other query omit the
column and default to None.

* feat(backend): select is_retry in single-job GET too for consistency

The list endpoint already exposes the marker; populate it on the single-job GET
(both completed and queued variants) as well so a run loaded directly reflects
its retry status. #[sqlx(default)] stays as a safety net for any other query.

* feat(backend): reap orphaned native_retry_attempt markers via periodic sweep

The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so
direct job deletions (workspace/job delete, schedule clearing) would leave marker
rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which
must then be remembered for every future path), reap orphans in the periodic
delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the
job). The table is sparse so the anti-join drives off it and probes v2_job by PK —
cheap. Retention still sweeps markers inline (keeps the table small so this stays
cheap); a transient orphan is harmless (nothing reads is_retry for a gone job).

* fix(frontend): include flow handlers in retry chain handler row (codex)

Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...),
whose handler job is a flow, not a script. The chain fetched children with
jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts
are still selected by is_retry and handlers by created_by, so both kinds surface.

* fix(backend): carry concurrency/debouncing settings into native retries

maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings
::default(), dropping the script/pipeline concurrency settings the failed job carried
in its runnable_settings_handle. A retry of a concurrency-limited script then inserted
no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and
pass them in the payload, which push forwards to the materialized retry. Adds a
regression test asserting the retry's handle resolves to the concurrency settings.

* fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex)

P1a (schedule.rs): the scheduled-retry materialization fetched the script's
concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings
::default() into the SingleStepFlow payload, so the root attempt's handle held only
the retry policy and the whole chain ran unbounded. Pass the fetched settings.
Regression test asserts the root handle resolves to retry + concurrency.

P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract
— it is pub only for the integration test; the sole production caller is the worker
completion path passing a DB-derived, already-authorized MiniCompletedJob.

* docs(backend): attach native-retry auth contract to the function itself (codex)

The doc block was merged with eval_retry_if's doc and bound to that function,
leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if
keeps its own doc; the native-retry + authorization contract now sits directly
above maybe_enqueue_native_script_retry.

* docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex)

- Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since
  1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the
  served spec couldn't see the field. Now current at 1.739.0.
- schedule.rs: a retry_if gate is evaluated at failure time and fails closed without
  quickjs (no retry); it does not fall back to a flow path.
- windmill-types jobs.rs: is_retry is selected by both the list and single-job GET
  endpoints (not list-only).

* docs(backend): fix remaining stale retry_if/quickjs comments (codex)

The retry_if block and the push materialization comments claimed push keeps
retry_if on a flow path / the worker always has quickjs. The code always
materializes native retry and the no-quickjs eval_retry_if path fails closed —
correct the comments to that constraint.

* docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex)

- Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated
  and the job does not retry (no one-step-flow fallback).
- jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence
  failure/recovery status across the retry chain, so the old 'restricted to
  schedules whose handlers don't need per-occurrence counting' claim is dropped.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 13:22:26 +00:00
Ruben FiszelandClaude Opus 4.8 f595787409 fix: invalidate relative-import cache when imported script changes (#9443)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 10:09:06 +00:00
centdix c96be0e903 refactor: move bedrock proxy handling to windmill-ai (#9309)
* refactor: move bedrock proxy handling to windmill-ai

* docs: track ai refactor follow-ups
2026-05-25 14:22:10 +00:00
d6c642b170 feat: add Azure Event Grid triggers (#8888)
* feat: add Azure Event Grid triggers (EE)

Introduces a new enterprise trigger kind `azure` that supports three
modes via a single unified trigger type:
- basic_push: Azure Event Grid basic — custom topics, system topics
  (Storage, Resource Manager, Key Vault, etc.), domains (push only)
- namespace_push: Event Grid Namespace topics (CloudEvents over HTTP push)
- namespace_pull: Event Grid Namespace topics (HTTP pull with lock-token
  ack/reject for dead-lettering)

Auth uses a Service Principal resource (tenant_id, client_id,
client_secret, subscription_id). Subscriptions are created in
CloudEvents 1.0 schema so the push webhook handler and the pull listener
share one payload parser.

Backend
- New crate `windmill-trigger-azure` (OSS stubs + EE impl symlinked from
  windmill-ee-private)
- Migration `azure_trigger` table with CHECK constraints enforcing
  mode/columns coherence
- `TriggerKind::Azure`, `JobTriggerKind::Azure`,
  `DeployedObject::AzureTrigger` variants
- Push route `/api/azure/w/{workspace}/*path` handles classic
  Event Grid SubscriptionValidation handshake and CloudEvents 1.0
  abuse-protection OPTIONS handshake
- Optional inbound JWT validation (audience check only for v1)
- Feature flag `azure_trigger` propagated through windmill-api,
  windmill-store (resource helper), and added to ee_core

Frontend
- `triggers/azure/` editor with mode toggle (basic/namespace-push/
  namespace-pull) and per-mode config (topic ARM id / namespace +
  topic name / subscription / filters / push auth / pull options)
- Registered in icon map, display names, save functions, badge,
  wrapper, editor, add-trigger menu

OpenAPI
- `AzureTrigger`, `AzureTriggerData`, `AzureMode`,
  `AzureSubscriptionMode`, `AzureDeliveryConfig`, `TestAzureConnection`
  schemas; `/azure_triggers/*` endpoints; client regenerated

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to eaa7c3a9cb37a9ccc93f10a2535d929365acd2d8

This commit updates the EE repository reference after PR #541 was merged in windmill-ee-private.

Previous ee-repo-ref: 9689014e8c12c36c1059fd8fa5758d550b8b8bc9

New ee-repo-ref: eaa7c3a9cb37a9ccc93f10a2535d929365acd2d8

Automated by sync-ee-ref workflow.

* feat(azure-trigger): secret-auth push, ARM discovery, capture isolation, CLI + parity

Frontend:
- Split mode selector into Namespace/Basic + Pull/Push
- ARM resource dropdowns (namespaces, Basic topics, namespace topics)
  populated from the service principal; cascade with stale-selection
  reset on SP / edition change
- Remove stale authenticate toggle + audience input (server-managed
  push_auth_config has replaced them)
- Azure listing page: "Create from template" button; "Also delete Azure
  subscription" toggle in the delete modal; simplified trigger label
  falling back to path
- AzureCapture.svelte: "Test subscription name" with -wm-capture suffix
- CompareWorkspaces.svelte: wire Azure for fork/compare
- Drop Trigger-deployed/event-loss warning (capture subscription is
  isolated with -wm-capture)

Backend:
- Shared-secret push auth (see EE crate for detail)
- JSONB push_auth_config column (renamed from delivery_config), #[serde(skip)]
  so clients/CLI/exports never see it
- Drop redundant enabled column; mode supersedes
- Azure capture infra: AzureTriggerConfig + set_azure_trigger_config +
  azure_payload route + TriggerKind::Azure arm; PT15M queue TTL on
  capture subscriptions so they bound storage after tab close
- Granular ACLs, users offboarding, trash, git-sync deployed-object:
  all include azure_trigger

CLI:
- Add azure to TRIGGER_TYPES, pushObj dispatch, getTypeStrFromPath,
  trigger commands (get/update/create/list/template), sync delete
  switch + regex; e2e test for `trigger new --kind azure`
- system_prompts: SCHEMA_MAPPINGS + schema_names include AzureTrigger;
  auto-generated/* regenerated

Skill:
- .claude/skills/adding-a-trigger/ checklist covering every file that
  needs editing when wiring a new trigger type (learned from this PR)

ee-repo-ref bumped to b0e490cbf3724b7b64c6a5b010e3bdf24acd873c.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(azure-trigger): ci — ShareModal Kind + regenerated system_prompts

- frontend/src/lib/components/ShareModal.svelte: add 'azure_trigger'
  to the Kind type so the listing page's "Permissions" action compiles
  (ts2345 — caught by npm_check on CI, missed by fast-check locally).
- system_prompts/auto-generated/: regenerate to drop the stale
  delivery_config / AzureDeliveryConfig fields from the Azure schema
  (check-freshness on CI).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(azure-trigger): use workspace constant_time_eq crate

Drop hand-rolled constant-time compare in favour of the workspace
constant_time_eq crate (same one used by http_trigger_auth).

ee-repo-ref bumped to 9659382d47286e7f7f66d01b6f5dd8d4ed34848b.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(azure-trigger): pass placeholder + disabled via inputProps

`TextInput`'s `placeholder` and `disabled` go through its `inputProps`
prop — CI's `npm run check` caught the stale top-level passing that
`npm run check:fast` missed. Align with the DefaultEmailConfigSection
pattern.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(azure-trigger): correct LATEST_GIT_SYNC_SCRIPT_PATH version to 28213

The hub deploy of the azure-aware sync-script is version 28213, not
28214. Backend was pinning a non-existent hub script, which broke the
git_sync_e2e suite (every deploy's sync step 404'd).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(azure-trigger): add azure_triggers to token scope selector + skill

- windmill-api/src/token.rs: `build_trigger_scope_domains` was missing
  `("azure_triggers", "Azure Event Grid")`, so the CreateToken UI's scope
  selector didn't surface azure_triggers:read/write. Backend already had
  `ScopeDomain::AzureTriggers` wired (scopes.rs), this just exposes it.
- .claude/skills/adding-a-trigger/SKILL.md: capture both scope-related
  files under the hardcoded-arrays section so future triggers don't miss
  the UI surface.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(adding-a-trigger-skill): clarify token.rs scope effect

Not a regression — nothing was working before. Skipping TRIGGER_DOMAINS
just means the scope works via API/CLI but has no UI checkbox.

* docs(adding-a-trigger-skill): trim token.rs bullet

* fix(azure-trigger): regen openapi-deref + swap textarea for TextInput

- Run build_openapi.sh to regenerate openapi-deref.{yaml,json} with the
  12 azure_triggers paths + schemas. These files are served by the
  runtime (include_str! in windmill-api/src/lib.rs) to external SDK
  consumers; without this regen the new endpoints wouldn't be advertised.
- Replace the raw <textarea> for event type filters with the
  design-system TextInput in textarea mode (frontend/CLAUDE.md bans raw
  HTML elements).

Addresses cubic + claude PR review items.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-04-23 16:30:18 +00:00
centdixandClaude Opus 4.6 26a6d1e4ce refactor: create windmill-ai crate (part 1 — types, traits, base modules) (#8530)
* refactor: create windmill-ai crate and move base AI types from windmill-common

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: move worker AI types to windmill-ai crate

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: move QueryBuilder trait and StreamEventSink abstraction to windmill-ai

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add base64 dependency to windmill-ai for bedrock PDF support

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add windmill-ai refactor plan

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: address PR review — remove dead bedrock feature, add boxed_sink helper, move plan to docs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 11:20:27 +00:00
wendrul 34e3115bcb fix: raw apps bundle not found during deployment error (#8515) 2026-03-25 10:59:48 +00:00
Ruben FiszelandClaude Opus 4.6 50ef9e79fc fix: propagate enterprise feature to windmill-api-schedule (#8391)
The enterprise feature was not being forwarded to windmill-api-schedule
in windmill-api's Cargo.toml, causing the #[cfg(not(feature = "enterprise"))]
guards in create_schedule to remain active even in EE builds. This made
on_recovery, on_success, and on_failure_times>1 incorrectly rejected
with "only available in enterprise version" for enterprise customers.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 11:59:15 +00:00
Diego Imbert 577484d06a Separate asset parsers (#8321)
* Refactor asset_parser

* package update

* package lock
2026-03-11 16:54:05 +00:00
4f29e05e3a feat: add git sync support for workspace dependencies (#8144)
* feat: add git sync support for workspace dependencies

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: implement git sync for workspace dependencies

Signed-off-by: pyranota <pyra@duck.com>

* remove deno.lock

Signed-off-by: pyranota <pyra@duck.com>

* update ee

Signed-off-by: pyranota <pyra@duck.com>

* add tests to cli

Signed-off-by: pyranota <pyra@duck.com>

* sqlx

* chore: update ee-repo-ref to 09dfb247f6f59c61b7f2431932c4557fb26c22d8

This commit updates the EE repository reference after PR #446 was merged in windmill-ee-private.

Previous ee-repo-ref: 8a8832ae5d7efab85b3a57a740308ececa0e2aac

New ee-repo-ref: 09dfb247f6f59c61b7f2431932c4557fb26c22d8

Automated by sync-ee-ref workflow.

* fix test

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Pyra <92104930+pyranota@users.noreply.github.com>
Co-authored-by: pyranota <pyra@duck.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-03-10 11:29:11 +00:00
825df2161e refactor: extract google ai logic to windmill-common and use native gemini api in chat proxy (#8115)
* refactor: extract google ai logic to windmill-common and use native gemini api in chat proxy

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: use x-goog-api-key header for google ai non-chat requests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: transform gemini models response to openai format and use correct auth header

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: skip thought parts from gemini thinking models in sse stream

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Revert "fix: skip thought parts from gemini thinking models in sse stream"

This reverts commit dfa01d282c.

* fix: handle tool calls and sanitize schemas in gemini chat proxy

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: move Gemini→OpenAI response conversion to windmill-common

Extract streaming and non-streaming Gemini response conversion into
shared functions in ai_google so the API proxy and worker use the same
logic instead of duplicating format translation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: review fixes for google ai refactor

- Remove duplicate parse_data_url from worker utils, use shared version
  from windmill_common::ai_google in both google_ai and anthropic providers
- Improve error diagnostics in google.rs by including HTTP status code
  in error messages from Gemini API responses
- Change GeminiToolCallEvent::into_extra_content to instance method
  to_extra_content using &self

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: deduplicate worker Gemini message conversion using pre-flight pattern

Replace the worker's `convert_messages_to_gemini` and
`convert_content_to_parts_with_s3` (~130 lines) with the existing
pre-flight pattern: `prepare_messages_for_api` converts S3 objects to
data URLs, then the shared `openai_messages_to_gemini` handles the rest.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: hugocasa <hugo@casademont.ch>
2026-03-09 15:15:37 +00:00
Ruben Fiszel 5f0ef936d1 feat: add sandbox annotations, volume mounts, for AI sandbox starting with claude (#8058) 2026-03-05 06:19:51 +00:00
7d9d16a6a3 feat: runScript inline for path and hash (#8019)
* runScript inline for path and hash

* Update backend/windmill-api/src/jobs.rs

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* refactor: unify inline script param structs and deduplicate closures

- Replace RunInlineScriptByPathFnParams and RunInlineScriptByHashFnParams
  with a single RunInlineScriptFnParams using InlineScriptTarget enum
- Collapse two nearly-identical closures in worker.rs into one
- Merge duplicate InlineByPath/InlineByHash into InlineScriptArgs
- Extract shared run_inline_script_inner helper in API handler
- Add missing check_scopes to run_inline_script_by_hash endpoint
- Fix duplicate lines from prior commit in run_inline_script_by_path
- Change tag from "inline_preview" to "inline" for deployed scripts

Co-authored-by: Diego Imbert <diegoimbert@users.noreply.github.com>

* Integration tests

* rm

* rename feature to run_inline

* Run inline integration tests

* Fix tests

* check path scope

* openapi fix

* nits

* remove register_potential_assets_on_inline_execution

* unused variable

* refactor

* Pass user_db to check script permission

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Diego Imbert <diegoimbert@users.noreply.github.com>
2026-02-27 13:59:14 +01:00
Ruben FiszelandClaude Opus 4.6 6bf544f507 refactor: extract object store into dedicated crate with filesystem backend (#7996)
* refactor: extract object store code into windmill-object-store crate with filesystem backend

Consolidate all object_store-dependent code from windmill-common into a new
windmill-object-store crate. Add a filesystem-backed object store implementation
using LocalFileSystem for dev/testing without cloud credentials. Includes 30
comprehensive tests covering render_endpoint, lfs_to_object_store_resource,
duckdb_connection_settings, error mapping, and filesystem-backed integration tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

* all

* all

* all

* fix: fix raw_app hardcoded path, add missing ObjectStoreResource import, and add tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: move S3ModeFormat to windmill-types, make windmill-parser-sql optional, restore debug logs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-18 12:24:45 +00:00
Ruben Fiszel b9e0ccaefd chore: split api-scripts and api-flows 2026-02-11 13:27:00 +00:00
Ruben FiszelandClaude Opus 4.6 e26e437dd1 refactor: extract 12 leaf crates from windmill-api (#7899)
* feat(backend): extract 12 leaf crates from windmill-api to improve incremental compilation

Extract independent modules from windmill-api (90k LOC monolith) into
separate leaf crates to reduce incremental compilation times. Modules
extracted: assets, configs, debug, flow-conversations, inputs,
npm-proxy, openapi, schedule, settings, workers, agent-workers, and
alerting (from windmill-common).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add windmill-api-settings dep to root crate, make ee_oss public

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: add agent_workers integration tests

8 tests covering agent worker lifecycle:
- Simple script execution (bun)
- Script with arguments
- Script with logs (verified in job_logs table)
- Script failure handling
- Complex result (nested objects/arrays)
- Agent token creation via API
- Token creation + Initial/MainLoop ping cycle
- Multiple sequential job execution

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-11 10:31:03 +00:00
Ruben FiszelandClaude Opus 4.6 623407de6a chore: make rquickjs, kube, otel proxy deps optional behind feature flags
- rquickjs: gated behind `quickjs` feature in windmill-jseval, propagated through windmill-worker/windmill-api, added to oss_core
- windmill-autoscaling: made optional in windmill-api (was unconditional), enabled via enterprise feature
- opentelemetry-proto, prost, hudsucker, rcgen, hyper-http-proxy, hyper-tls, hyper-util: made optional in windmill-worker, enabled via enterprise feature

This significantly reduces compilation time for vanilla `cargo check` without features.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 16:13:16 +00:00
Ruben Fiszel bb57f8cd29 remove unecessary deps for vanilla cargo run 2026-02-08 14:08:36 +00:00
Ruben FiszelandClaude Opus 4.6 c580572252 refactor: extract windmill-api-scripts and windmill-api-users subcrates (#7850)
* refactor: extract windmill-api-scripts and windmill-api-users subcrates

Split the monolithic windmill-api crate by extracting scripts.rs, flows.rs,
users.rs, and users_oss.rs into dedicated subcrates. This reduces incremental
rebuild times when editing these modules.

Changes:
- Create windmill-api-scripts crate (scripts.rs + flows.rs, ~4.3K lines)
- Create windmill-api-users crate (users.rs + users_oss.rs, ~2.4K lines)
- Move clear_schedule to windmill-queue (shared by scripts, flows, workspaces)
- Move username utilities (VALID_USERNAME, INVALID_USERNAME_CHARS,
  generate_instance_wide_unique_username) to windmill-common/src/usernames.rs
- Move COOKIE_DOMAIN, IS_SECURE, WithStarredInfoQuery, BulkDeleteRequest,
  WebhookShared to windmill-common for cross-crate access
- Original files in windmill-api become thin stubs with pub use re-exports
- EE-dependent route handlers remain in windmill-api (create_user, rename_user,
  set_password, reset_password, etc.)
- Feature forwarding for enterprise, private, parquet, no_auth

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: extract windmill-api-workspaces subcrate (Step 3)

Move workspaces.rs, workspaces_extra.rs, workspaces_oss.rs, and
workspaces_ee.rs into a new windmill-api-workspaces crate (~7K lines).

Routes that depend on windmill-api internals (AI copilot, teams,
tarball export, critical alerts, stripe) remain in the windmill-api
stub. The subcrate handles all other workspace management routes.

Also moved send_email_if_possible to windmill-common/email_oss.rs
to make it available across subcrates without circular deps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

* refactor: extract windmill-api-groups subcrate (groups.rs + folders.rs)

Extract groups.rs (1,093 lines) and folders.rs (833 lines) into a new
windmill-api-groups subcrate. Both modules had clean dependencies on
already-extracted crates (windmill-api-auth, windmill-common,
windmill-api-workspaces). Also removes unused re-exports of
get_instance_username_or_create_pending and INVALID_USERNAME_CHARS
from windmill-api/src/utils.rs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: add granular_acls.rs and folder_history.rs to windmill-api-groups

Extract granular_acls.rs (395 lines) and folder_history.rs (68 lines) into
the windmill-api-groups subcrate. Both modules only depend on already-extracted
crates and belong to the same access-control domain as groups and folders.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: remove unused imports and dead code from subcrate extraction

- Remove unused BASE_URL import from lib.rs
- Remove workspaces_extra.rs and workspaces_oss.rs re-export stubs (no consumers in windmill-api)
- Remove dead send_email_if_possible OSS stub (callers moved to windmill-api-users)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

* chore: bust CI cargo cache for subcrate split

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: re-export BASE_URL for EE files that use crate::BASE_URL

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: forward no_auth feature to windmill-api-users, remove dead code

- Add "windmill-api-users/no_auth" to windmill-api's no_auth feature
  so the login bypass in users.rs:1600 activates correctly
- Remove dead send_email_if_possible from windmill-api-users/users_oss.rs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: re-enable cargo cache for backend tests

Cache was disabled to bust stale entries from before subcrate split.
Now that a clean build has run, re-enable for faster CI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: install mold+clang in CI workflows

The .cargo/config.toml uses mold linker for x86_64-linux.
Build scripts require linking even during cargo check.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: increase cargo test timeout to 30 min

Exit code 143 (SIGTERM) means the 20-min timeout was hit during
compilation without cache. Bump to 30 min as safety net.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: limit cargo build jobs to 4 to prevent OOM in CI

Exit code 143 (SIGTERM) after 8 min = OOM kill during compilation.
8 parallel LLVM codegen jobs exhaust memory on ubicloud-standard-8.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 12:12:03 +00:00
Ruben FiszelandClaude Opus 4.6 3f68e3a58e refactor: isolate deno_core into windmill-runtime-nativets subcrate (#7848)
* refactor: isolate deno_core into windmill-runtime-nativets subcrate

Remove deno_core from flow eval and isolate nativets V8 runtime into a
dedicated subcrate so deno_core compilation no longer blocks
windmill-worker or windmill-api.

- Create windmill-jseval crate: QuickJS-based JS eval for flow
  expressions and batch rerun, extracted from windmill-worker
- Create windmill-runtime-nativets crate: all deno_core/V8 deps and
  nativets script execution, with build.rs snapshot generation
- Simplify windmill-worker: remove all deno_* direct deps, empty
  build.rs, gate nativets behind optional dep
- Update windmill-api: use windmill-jseval for batch rerun instead of
  deno_core, remove deno_core feature entirely
- Add nativets integration tests (nativets_jobs.rs) and parallel
  stress test (nativets_stress.rs, 8 workers x 200 jobs)
- Remove dead code: deno flow eval path, USE_QUICKJS env var,
  parity tests (replaced with 63 standalone expected-value tests)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address PR review feedback for deno_core isolation

- Deduplicate unsafe_raw() into windmill-common/src/utils.rs (single source)
- Delete orphaned runtime.js and windmill-client.js from windmill-worker/src/
- Fix operator precedence in windmill-jseval with explicit parentheses
- Remove unnecessary return keyword in heap limit callback
- Remove redundant as usize casts
- Remove ~150 lines of commented-out code from runtime.js
- Remove commented-out #[cfg] in build.rs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* otel ee

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 09:16:37 +00:00
Ruben FiszelandClaude Opus 4.6 e1a815f6a0 refactor: extract windmill-dep-map crate for parallel api/worker compilation (#7846)
* refactor: extract windmill-dep-map crate for parallel api/worker compilation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: resolve WebhookShared type mismatch and missing enterprise propagation

- Make windmill-api webhook_util re-export from windmill-common instead of
  duplicating types, fixing Extension<WebhookShared> mismatch between
  windmill-store and windmill-api
- Add windmill-api-jobs/enterprise to windmill-trigger enterprise feature
  so check_license_key_valid is available when trigger subcrates enable
  enterprise on windmill-trigger

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: stop trigger features from unconditionally enabling enterprise

Move enterprise propagation for all trigger subcrates from individual
trigger feature definitions to the enterprise feature itself, so
enterprise is only enabled when explicitly requested.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: remove unused pub use re-exports and disable CI cargo cache

- Remove unused re-exports from windmill-worker/src/lib.rs:
  trigger_dependents_to_recompute_dependencies, handle_job_error,
  and unused bun/otel items
- Fix callers to use direct module paths instead
- Add windmill-dep-map as dev-dependency for tests
- Disable cargo cache in backend-check CI (faster from-scratch builds)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: restore bun re-exports used by tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

* chore: re-enable cargo cache for check_ee_full CI job

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 00:39:56 +00:00
Ruben FiszelandClaude Opus 4.6 9ff8a85af6 refactor: extract windmill-api into subcrates for parallel compilation (#7845)
* refactor: extract windmill-api into 4 subcrates (api-auth, store, api-sse, api-jobs)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: eliminate refresh_token OnceLock bridge in windmill-store

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: eliminate FromRequestParts OnceLock bridge in windmill-api-auth

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: wire subcrates into workspace and clean up unused re-exports

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: resolve cargo check --all-features errors in subcrate wiring

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* sqlx

* all

* chore: update ee-repo-ref for warning fixes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: extract windmill-trigger crate and expand windmill-api-jobs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-trigger-kafka crate from windmill-api

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-trigger-postgres crate from windmill-api

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-trigger-websocket and windmill-trigger-mqtt crates

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-trigger-nats, sqs, gcp, and email crates

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-trigger-http crate from windmill-api

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: move token creation and permission helpers to windmill-api-auth

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: extract windmill-native-triggers crate from windmill-api

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* sqlx

* all

* refactor: extract windmill-api-embeddings crate and fix CI warnings

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: resolve type mismatch in oauth2_oss and remaining warnings

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use correct HTTP_CLIENT config in embeddings crate (30s timeout, cert override)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* all

* fix: gate oauth_refresh_ee on oauth2 feature to fix warnings

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* all

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-07 22:12:55 +00:00
Ruben Fiszel c1534ccabc reorganize meta features and introduce an oss meta-feature with no_auth 2026-02-06 09:13:32 +00:00
795e2bebe6 feat: kafka trigger kerberos/gssapi support (#7815)
* feat: kafka trigger kerberos/gssapi support

* chore: update ee-repo-ref to bb32d562120dc34bbd8d659d92a0d4b8824b8c4c

This commit updates the EE repository reference after PR #405 was merged in windmill-ee-private.

Previous ee-repo-ref: 128c6549d4557895a362fb720f56afa54d6f566b

New ee-repo-ref: bb32d562120dc34bbd8d659d92a0d4b8824b8c4c

Automated by sync-ee-ref workflow.

* adding kafka-gssapi to all_sqlx_features

* ee ref

* ee ref

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2026-02-06 06:23:24 +00:00
Ruben Fiszel 309de58134 Revert "feat: restriction rulesets for workspaces (#7791)" (#7820)
This reverts commit a1cd02d7f8.
2026-02-05 18:16:40 +00:00
wendrulandwindmill-internal-app[bot] a1cd02d7f8 feat: restriction rulesets for workspaces (#7791)
* Add frontend for the workspace proctection rules

* api to add and update workspace protection rules

* Fix bypasser selection

* Fix Select bug on refocus

* Change rulesets to use bitflags

* Messages for protection rules errors

* claude remove ui for rules

* Hide edit buttons when rule

* No edit refactor pt1

* Update edit buttons to be disabled when rule is active

* Merge deploy ui and rulsets in one tab

* Remove not cleaned line in migration

* multiple fixes

* Remove old protection rule logic

* Add prrotection rule for deploying through Merge UI

* Add Alert on legacy Deploy UI

* Add backend enforcing of workspace rules

* Finish backend blocking on rulsets

* Last changes to api ruleset blocks

* Prepare sqlx

* Remove unused import and argument

* Update SQLx metadata

* fix npm run check

* Re trigger CI

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-02-05 18:02:41 +00:00
hugocasaandRuben Fiszel 63f9d85bf6 feat: public app rate limiting + fork hub raw apps + raw apps publish to hub button (#7789)
* feat: public app rate limiting + fork hub raw apps + raw apps publish to hub button

* sqlx

* missing sqlx file

* cache rate limiting

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2026-02-04 17:53:55 +00:00
635a24f82c feat: runtime assets (#7656)
* Runtime assets

* Nits

* Revert "Nits"

This reverts commit 3031a2ddd1.

* detection_kinds

* don't delete runtime assets

* Show latest executions

* conditional unique idx

* nit status

* refactor

* nit refactor

* prepare sql

* Detect assets in complex JSON input objects

* false positive prevent

* nit

* redundant idx

* Update frontend/src/lib/components/assets/AssetsUsageDrawer.svelte

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* Update backend/migrations/20260122134517_runtime_assets.up.sql

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

* runtime assets are inserted in a loop

* nit

* nit fix

* Don't use lazy static

* fix compilation

* nits

* missing on conflict do nothing

* add index

* Fix max n logic

* created at

* nits

* remove pagination

* sqlx prepare

* Only detect resource assets in input

* get_runtime_asset_sender()

* use global get_runtime_asset_sender to avoid prop drilling

* nit refactor : register_runtime_asset

* get job_id from token

* job as a usage kind

* fixes

* ee

* nit refactor

* merge access types when same job uses same asset multiple times

* Refactor to support wmill s3 API

* nit

* parse_wmill_sdk_sql_assets refactor

* Detect datatable and ducklake usage

* nit order by

* Join with v2_job

* better UI

* add sequential id for cursor pagination

* useInfiniteQuery

* useScrollToBottom

* sql index

* claude code stash

* migration fixes

* Infinite scroll UI

* nit

* style nit

* runtime asset created at

* Asset filters

* fix usage kind filter

* also check runnable_path for jobs when filtering

* better filters

* avoid flickering

* debounced filters

* nit

* tooltips

* fix: update AssetUsage type to match new ListAssetsResponse structure

The ListAssetsResponse changed from an array to an object with an 'assets' property.
Updated the type extraction accordingly.

Co-authored-by: Diego Imbert <diegoimbert@users.noreply.github.com>

* sqlx prepare

* Delete .claude/hooks/.symlink-manifest

* unnecessary dep

* nit refactor

* nit comment

* nit naming

* CI fix attempt 1

* ee ref

* nit remove alerts

* nit

* chore: update ee-repo-ref to 138a4f5f868f3bded5bb7cb77b222b532c07e4af

This commit updates the EE repository reference after PR #395 was merged in windmill-ee-private.

Previous ee-repo-ref: 7d3a21d53066726e97dfea9f117373299bc9318c

New ee-repo-ref: 138a4f5f868f3bded5bb7cb77b222b532c07e4af

Automated by sync-ee-ref workflow.

---------

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Diego Imbert <diegoimbert@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-02-02 10:09:40 +00:00
dieriba 6418c4bcc6 feat: nextcloud native triggers (#6797) 2026-01-26 16:49:52 +00:00
Ruben Fiszelandrubenfiszel ac56ffe052 chore(main): release 1.615.0 (#7674)
* chore(main): release 1.615.0

* Apply automatic changes

* pin rdkafka

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-01-26 11:41:33 +00:00
2553d987d9 feat(ai): native bedrock compatibility (#7668)
* check endpoint

* use flag for cred check on worker

* use sdk for auth chat side

* cleaning

* cleaning

* also handle GET

* Add session token support in BedrockClient::from_credentials

Users with temporary STS credentials need session tokens for AWS
authentication. This adds the optional session_token parameter to
support these use cases.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Cache AWS SDK config loading to avoid repeated I/O

aws_config::load_defaults() performs environment variable lookups and
potentially file I/O on each call. Cache the result in a static OnceCell
to improve performance on high-frequency Bedrock requests.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add debouncing to Bedrock credential check button

Prevent rapid clicks from spawning multiple concurrent flow preview
jobs for the worker credential check.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Remove orphaned SigV4 signing comment

The manual SigV4 signing code was removed in favor of the AWS SDK,
but this documentation comment was left behind.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Remove verbose tracing info and change-related comments

- Remove per-request auth method info logs (too verbose for production)
- Simplify from_env log to single debug-level message
- Remove comments describing future changes rather than current behavior

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* cleaning

* Fix Bedrock feature flag compilation warnings

Gate AWS-specific imports, struct fields, and methods behind the bedrock
feature flag to eliminate dead code warnings when building without the
bedrock feature enabled.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* bedrock flag

* cleaning

* Move BedrockCredentialsCheck from edit drawer to add drawer

The component was incorrectly shown when editing resources instead of
when adding new ones. Moved it from ResourceEditor.svelte to
ApiConnectForm.svelte.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* cleaning

* cleaning

* make aws-config optional

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2026-01-26 11:04:14 +00:00
Ruben Fiszel 1b9d1c56c7 feat: add HashiCorp Vault secret storage integration (#7599)
* feat: add HashiCorp Vault secret storage integration

- Create SecretBackend trait abstraction for secret storage
- Add VaultBackend implementation with CRUD operations
- Integrate secret backend into variable CRUD operations
- Add migration functions (DB → Vault and Vault → DB)
- Add frontend configuration UI for secret backend
- Add test connection and migration endpoints
2026-01-18 23:08:29 +00:00
Ruben Fiszel b3cb41efa4 feat: add private npm registry proxy support for ATA in webide (#7597) 2026-01-17 10:44:30 +00:00
centdix 5c08abe141 feat(aiagent): handle oauth for mcp tools (#7564)
* feat(aiagent): handle oauth for mcp tools

* cleaning

* cleaning

* move oauth2

* cleaning

* cimd

* wrap path

* fix frontend

* fix

* refresh resources

* cleaning

* update ee ref

* update sqlx

* cleaning

* update

* cleaning

* update ref

* cleaning

* cleaning

* update ref
2026-01-14 12:06:40 +00:00
HugoCasa f72c16456f Revert "feat(aiagent): handle oauth for mcp tools [merge-ee-first] (#7544)"
This reverts commit e823c953d1.
2026-01-14 12:59:14 +01:00
centdix e823c953d1 feat(aiagent): handle oauth for mcp tools [merge-ee-first] (#7544)
* feat(aiagent): handle oauth for mcp tools

* cleaning

* cleaning

* move oauth2

* cleaning

* cimd

* wrap path

* fix frontend

* fix

* refresh resources

* cleaning

* update ee ref

* update sqlx

* cleaning

* update

* cleaning

* update ref

* cleaning

* cleaning
2026-01-14 11:57:00 +00:00
Ruben Fiszel 4451a37999 feat: debuggers for python and bun v0 (#7546) 2026-01-13 15:20:06 +00:00
Ruben Fiszel 83fea45d48 split oauth into separate crate 2026-01-09 07:03:29 +00:00
Ruben Fiszel d432cdbec1 split out windmill-mcp + pin tokio-utils 2026-01-09 06:16:26 +00:00
centdix a8e34de58d upgrade rmcp crate (#7518) 2026-01-07 16:02:51 +00:00
d544da342c fix: update to astral-tokio-tar for CVE (#7423)
* fix: Migrate to astral-tokio-tar to address CVE-2025-62518

Migrates from the abandoned tokio-tar library to the actively maintained astral-tokio-tar to address CVE-2025-62518 (TARmageddon), a high-severity RCE vulnerability (CVSS 8.1). The vulnerability involves a boundary-parsing bug in PAX/ustar header handling that enables file smuggling attacks via malicious TAR archives.

This is a drop-in replacement requiring only dependency updates in Cargo.toml files. The astral-tokio-tar library uses the same tokio_tar module name, so no source code changes are needed. All references to the vulnerable tokio-tar v0.3.1 have been removed from Cargo.lock and replaced with the patched astral-tokio-tar v0.5.6.

Related to previous PR #6943 which was closed due to CLA issues. Security disclosure available at https://edera.dev/stories/tarmageddon and patch details at https://github.com/astral-sh/tokio-tar/releases/tag/v0.5.6

* update

---------

Co-authored-by: Devdatta Talele <devtalele0@gmail.com>
Co-authored-by: Devdatta Talele <50290838+devdattatalele@users.noreply.github.com>
2025-12-20 09:04:36 +00:00
centdix 8c55f61bba feat(ai): support IAM auth for bedrock provider (#7379)
* support iam for bedrock ai

* lock

* cleaning
2025-12-16 22:00:14 +00:00
Diego ImbertandRuben Fiszel cf36fe3bb1 fix: use proper TLS connector for DuckLake instance catalog setup (#7138)
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-11-14 13:01:43 +01:00
centdixandClaude 97ac1be036 feat(aiagent): allow mcp as tools (#6790)
* draft mcp client

* testing

* fix

* cleaning

* mcp resource in inputtransforms

* cleaning

* big cleaning

* cleaning

* no arc

* add utils file

* refactor tools

* add mcp actions

* draft frontend

* send arguments from backend

* better frontend

* cleaning

* use token for auth

* add logo

* rm

* fix

* fix

* chore: refactor mcp for ai agents (#6829)

* Add Tool enum for AIAgent with backward compatibility

- Created Tool enum that can be either Windmill (FlowModule) or Mcp (resource reference)
- Created McpToolRef struct to hold MCP resource path
- Implemented custom Deserialize for Tool with backward compatibility:
  - New format: {type: 'windmill'|'mcp', ...}
  - Old format: FlowModule objects (automatically wrapped in Tool::Windmill)
- Updated AIAgent to use Vec<Tool> instead of Vec<FlowModule>
- Updated FlowValue::traverse_leafs to handle Tool enum
- Backward compatible: old flows with Vec<FlowModule> will deserialize correctly

* Refactor AI executor to process Tool enum instead of extracting MCP from input_transforms

- Separate Windmill tools and MCP resource paths from tools list
- Process Windmill FlowModules into Tool definitions
- Load MCP tools from resource paths in Tool::Mcp variants
- Remove old logic that extracted mcp_resources from input_transforms
- Import FlowModule, remove unused InputTransform
- Fix type issues: use .as_str() for path and handle Option<bool> properly

* handle in args

* mcp as flowmodule

* frontend

* config for mcp

* simplify logic

* fix ai executor logic

* cleaning

* clean frontend

* fix

* better resource picker

* fix and styling

* add endpoint to fetch tools

* apply tool filtering

* fix name validation

* better ui

* use cache

* fix

* fix merge

* refactor: Separate MCP tools from FlowModule in AIAgent

- Add new AgentTool, ToolValue, and McpToolValue types
- Update AIAgent to use Vec<AgentTool> instead of Vec<FlowModule>
- Implement From traits for clean conversion between AgentTool and FlowModule
- Add backward compatibility via custom deserializer for AgentTool
- Simplify resolve_module logic by reusing existing resolve_modules function
- Update traverse_leafs to handle AgentTool structure

This refactoring separates MCP tools from FlowModule tools, making the
type system clearer and eliminating the need to treat MCP servers as
a special case of FlowModule.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Update ai_executor and worker_lockfiles for AgentTool

- Update ai_executor.rs to handle new AgentTool structure
  - Separate MCP tools from FlowModule tools using ToolValue enum
  - Convert AgentTool to FlowModule for backward compatibility
  - Add imports for AgentTool and ToolValue types

- Update worker_lockfiles.rs for lazy loading optimization
  - Convert AgentTool <-> FlowModule in insert_flow_modules
  - Preserve lazy loading for FlowModule tools via modules_node
  - Keep MCP tools inline (lightweight, no need for lazy loading)
  - Maintain backward compatibility with existing flows

This enables the lazy loading optimization for FlowModule tools while
keeping MCP tools inline, balancing performance and simplicity.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* cleaning

* adapt frontend

* cleaning

* cleaning

* type fix

* cleaning

* fix back comp

* move mcp button position

* nit

* cleaning

* fix nested removal

* cleaning

* opti

* fix chat markdown display

* fix chat messages layout

* fix back comp frontend

* fix deserializer

* nit

* simpler serializer

* use if else

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-10-21 15:28:52 +00:00
centdix 389b692523 fix(mcp): use stateless mode for openai sdk compatibility (#6656)
* update crate

* use non stateful mode

* fix

* fix
2025-09-23 10:16:26 +00:00
Ruben Fiszel 0647202a4f fix: check perms more tightly for running jobs (#6532)
* all

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* all

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* masterpiece

* nits
2025-09-05 06:07:45 +00:00
Diego Imbert 420be5a1f8 feat: use S3 Proxy for duckdb instead of direct connection (#6505)
* s3 proxy works with get (no auth yet)

* nit

* support s3:// syntax

* Support s3:// syntax and fix vite api proxy normalizing double slashes in URI

* s3 checks authed

* nit

* PUT works

* delete file works

* Derive the JWT signature from the backend

* Authorize s3 correctly (JWT signature is never sent in cleartext)

* convert object store error to wmill error for correct status code

* stash

* fix

* POST first request proxy works

* s3 put for duckdb

* factor out direct proxy code

* Fix Issue with backend proxy and wrong signature due to Host header mismatch

* Add _default_ syntax to solve URI normalization issues with signing

* restricted to user paths toggle

* user path restriction works !

* change restriction to allow

* fix

* factor out code

* better permissions UX in object storage settings

* Revert to restrict_to_user_paths

* check permissions in old s3 api

* DuckDB now uses S3 Proxy and no longer needs LFS query

* implement todo

* fix hardcoded w_id

* s3 proxy size limit

* s3_proxy is ee

* nit

* add Google Cloud Storage as option to secondary storage

* GCS secret in duckdb

* fix toolchain compile

* Remove user permissions for v0

* fix ci 2

* fix CI OSS

* fix missing feature flag

* fix unused warning

* integration test fails bc rustc 1.85.0

* ee ref

* fix ci ...

* update ee ref
2025-09-03 15:03:44 +00:00
dieriba f3fd1e90b0 feat: refactor trigger crud (#6472)
* base

* base crate

* websocket updated

* crud refactored

* fix and nits

* fix compiler warning, nits and update repo ref

* fix oss

* fix compilation

* update ref

* fix

* update feature

* listener base

* add listener

* refactor logic done and implemented for postgres

* fix capture

* websocket

* implem for all triggers

* update sqlx,repo ref and fix

* fix oss

* unify struct fix websocket

* nits and fix oss runtime axum error

* perf cache query

* update .sqlx

* update .sqlx

* fix

* fix unused

* fix

* fix

* nits and fix http handler update endpoint

* update .sqlx

* update repo ref

* nits

* fix

* update ref

* fix

* update .sqlx
2025-09-02 11:28:06 +00:00