Compare commits

...
Author SHA1 Message Date
centdix 00392ba548 fix 2026-06-23 16:23:41 +02:00
hugocasaandClaude Opus 4.8 ba4b368706 fix: prevent variable push from corrupting is_secret variables (#9705)
* fix: prevent variable push from corrupting is_secret variables

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(cli): unit-test looksLikeWorkspaceCiphertext shape detection

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): scope is_secret downgrade to single-file push, not sync push

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): warn when variable push stores a secret value as already-encrypted

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(cli): route workspace-resolution and auth diagnostics to stderr

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(cli): rephrase comments to describe current behavior, not history

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 12:45:36 +02:00
Ruben Fiszelandrubenfiszel 723a65920f chore(main): release 1.737.0 (#9728)
* chore(main): release 1.737.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-06-23 12:10:15 +02:00
c644311eca fix(ext-jwt): reject external JWT auth for non-existent workspaces (#9723)
* fix(ext-jwt): reject external JWT auth for non-existent workspaces

External JWTs are validated (not generated) on our side and never revoked
by us. The usage-tracking upsert into unique_ext_jwt_token ran
unconditionally, so a token carrying a workspace_id whose workspace no
longer exists kept refreshing its row on every presentation — surfacing
as a "new token" in the superadmin external-JWT view.

Gate jwt_ext_auth on the requested workspace existing (EE companion). When
it does not, auth fails (token is unusable) and no usage row is written.
The check is existence-only and intentionally ignores the soft-delete
flag, so deleted-then-restored workspaces keep working.

Bumps ee-repo-ref.txt to the EE companion commit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(ext-jwt): cache workspace-existence lookups in jwt_ext_auth

Bumps ee-repo-ref.txt to the EE companion commit that caches the
workspace-existence check added in the previous commit, so a token aimed
at a missing workspace no longer hits the DB on every request (auth
failures aren't cached upstream).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to ac1f6f666f36141cb6ba6f8eaa614821a90464ad

This commit updates the EE repository reference after PR #626 was merged in windmill-ee-private.

Previous ee-repo-ref: e23fa03ec16909c127e8ecf0855595911c29512d

New ee-repo-ref: ac1f6f666f36141cb6ba6f8eaa614821a90464ad

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-06-23 11:25:23 +02:00
Ruben FiszelandClaude Opus 4.8 31d9215e5a fix: bound orphan-cleanup drain rate with capped multi-batch loop (#9730)
Follow-up to #9727. The orphan cleanups (cleanup_job_perms_orphaned and
cleanup_job_result_stream_orphaned_jobs) deleted at most one 100k batch per
monitor iteration. Each statement stays short and lock-light, but a single
batch per ~30s cycle caps the drain rate at ~100k/30s, so a large one-time
backlog (tens of millions of rows) takes ~hours to clear.

Loop the batched delete up to ORPHAN_CLEANUP_MAX_BATCHES (10) times per cycle,
stopping early once a batch deletes fewer than ORPHAN_CLEANUP_BATCH_SIZE rows.
Each DELETE remains bounded (≤100k, short locks, no long single statement),
while per-cycle throughput rises to ~1M rows so backlogs drain ~10x faster.
The per-cycle cap keeps monitor_db responsive.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 11:24:36 +02:00
Ruben FiszelandClaude Opus 4.8 75bafabeee perf(monitor): hash active-root exclusion in retention delete (WIN-2088) (#9732)
* perf(monitor): hash active-root exclusion in retention delete

The expired-job retention delete (delete_expired_jobs_batch) excluded jobs
belonging to still-active root flows with
`COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)`. That
ScalarArrayOp is evaluated per candidate row as a linear scan of $3, so cost
grows with the number of active root jobs.

Express the exclusion as `NOT IN (SELECT u FROM unnest($3) u WHERE u IS NOT
NULL)` instead. The subquery form lets Postgres build a one-time hashed SubPlan
and apply it as a filter on the ordered index scan, giving O(1) membership per
candidate while preserving the `ORDER BY completed_at ASC LIMIT` early
termination. The `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics
($3 holds non-null PK ids).

Measured on a 2M-row synthetic v2_job_completed (batch LIMIT 20000, 5-run min):

  active roots | != ALL (before) | NOT IN hashed (after)
  -------------|-----------------|----------------------
  100          | 108 ms          | 104 ms
  1000         | 168 ms          | 105 ms
  10000        | 719 ms          | 131 ms

Both forms return identical row sets (verified via EXCEPT, 0 diff). Neutral at
small active-root counts, ~5.5x faster when many flows are active.

Relates to WIN-2088

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(monitor): apply hashed active-root exclusion to log_cleanup mirror

windmill-api-settings/log_cleanup.rs::delete_expired_jobs_batch carries a
byte-identical copy of the retention delete and shared its prepared-query
cache. Updating only monitor.rs removed that shared cache entry and broke the
SQLX_OFFLINE build of the mirror. Apply the same NOT IN (hashed SubPlan)
rewrite so both copies converge on one cached query and the mirror gets the
same speedup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 11:23:50 +02:00
Ruben FiszelandClaude Opus 4.8 fa3596885b fix: allow SQL args in managed // materialize scripts (#9733)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 11:13:04 +02:00
Ruben FiszelandClaude Opus 4.8 6d94865109 fix: optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries (#9727)
The job_perms/job_result_stream_v2 orphan cleanups in monitor_db used
`NOT IN` anti-joins, `RETURNING job_id` + `fetch_all` (loading every deleted
UUID into memory) and no batch limit. On high-throughput instances these
tables can accumulate tens of millions of orphaned rows, so a single execution
ran for ~298s; because monitor_db awaits each iteration, the cleanup ran
effectively continuously, saturating DB I/O and starving audit partition
creation.

Rewrite both deletes as bounded `NOT EXISTS` anti-joins selecting `ctid` with
a LIMIT 100000, executed via `.execute()` (using rows_affected instead of
fetch_all). Each run is now fast and bounded, while the 30s monitor cadence is
preserved so the tables keep draining promptly.

Fixes WIN-2088

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 10:14:40 +02:00
Ruben FiszelandClaude Opus 4.8 8dea38383f fix: prevent silent audit-partition outage via monitor watchdog + alert (#9729)
The monitor loop runs ~25 periodic tasks under a single join!, so any one
stuck on a non-DB await (statement_timeout only bounds DB statements) freezes
the whole loop indefinitely — silently halting audit-partition creation. Once
the missing partition's date is reached, audit inserts fail; because login
writes its audit row in the same transaction, that poisons the login tx and
locks every user out.

- Wrap monitor_db in a 600s timeout (> statement_timeout) so a stuck task can
  no longer freeze the loop; report a critical error and continue on timeout.
- After creating partitions, verify the lookahead window is actually covered
  and raise a critical alert naming any missing partitions, turning a silent
  latent outage into an early page.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 10:13:29 +02:00
2879cbb65a feat(apps): opt-in sandbox isolation for published & raw apps (alpha) (#9420)
* feat(apps): sandbox published & raw apps with a scoped embed token

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: point ee-repo-ref at embed-token EE commit

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(apps): allow top-navigation from the sandboxed app iframe

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(apps): share app localStorage across apps via the embedder

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(apps): publisher disable-sandbox option with per-version viewer consent

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(sqlx): cache for disable-sandbox queries

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: bump ee-repo-ref to disable-sandbox EE commit

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): always sandbox the served raw-app wrapper + viewer fixes

The raw-app wrapper served by get_raw_app_data now always carries
`CSP: sandbox`. The publisher "disable sandbox isolation" opt-out is applied
entirely on the viewer side, which (after per-version consent) builds its own
same-origin blob wrapper — so the backend-served document stays isolated
regardless of how it is reached, never via a relaxed real-origin URL.

Also:
- CORS on the global /apps_u mount so the opaque viewer can load custom-path
  public apps cross-origin.
- Reject runnable-bridge messages unconditionally until the iframe is bound.
- Relay the viewer's in-app hash up to the embedder address bar so deep links
  stay shareable (hash only; embedder keeps its own pathname).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(apps): render public raw apps single-iframe (drop embed token)

Public raw apps now render directly on the real origin with a single
opaque bundle iframe and the page credential, instead of the opaque
viewer + scoped-token indirection. The author bundle stays isolated in
its own opaque iframe (CSP-sandboxed); low-code apps, whose code runs in
the viewer frame, keep the opaque viewer + scoped token.

embed_token now reports raw_app and skips minting a token for raw apps;
the access check still gates visibility.

Also set disable_sandbox: None in the remaining Policy constructors so
the full feature build (all_sqlx_features, enterprise, license) compiles.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: bump ee-repo-ref to single-iframe raw-app EE commit

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(apps): grandfather existing apps as legacy-unsandboxed + authed-only consent

Existing apps are stamped by migration as `legacy_unsandboxed` so they keep
running same-origin on upgrade — no breakage and no consent prompt. New apps are
sandboxed by default; re-deploying an app clears the flag.

The publisher `disable_sandbox` consent prompt is now shown only to authenticated
viewers — an anonymous viewer has no session to expose, so the prompt was
meaningless friction.

embed_token reports `legacy_unsandboxed` and `authed`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: bump ee-repo-ref to legacy-unsandboxed EE commit

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(apps): deploy-time migration prompt for legacy-unsandboxed apps

On the first re-deploy of a grandfathered (legacy-unsandboxed) app, the
publisher must explicitly choose: enable sandbox isolation (the flag is
cleared → the app becomes sandboxed) or keep running without isolation
(→ disable_sandbox, with per-version viewer consent). updatePolicy() no
longer carries the legacy flag through a deploy, so the choice is what
sticks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(apps): disable the sandbox-isolation toggle until the app is deployed

The Deploy-drawer "Disable sandbox isolation" toggle called setPublishState()
— which updates the app by path — even before the app was first deployed, when
the path is empty, throwing an error. Guard it with disabled={!savedApp},
matching the adjacent visibility toggle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(apps): sandbox the in-workspace low-code app viewer in an opaque iframe

Extend the opaque-origin iframe isolation to the logged-in /apps/get viewer.
/apps/get becomes an embedder that keeps the workspace chrome + Edit button and
renders the app inside a cookieless, chrome-less /app_embed viewer route, handed
a scoped embed token minted from the member's session. The app frame runs in an
opaque origin (no allow-same-origin), so it cannot reach the member's session
cookie or window.parent.

- apps.rs: get_app_embed_token_for_path (authed, by-path, scope + RLS gated);
  mint_app_embed_token grants a path-scoped apps:read:{path} so the viewer can
  load its own app definition and no other
- lib.rs: CORS on /apps (bearer-token only, no cookies) for the opaque viewer's
  by-path reads
- new /app_embed/[workspace]/[...path] viewer route (private analog of /public)
- PublicAppFrame: viewerUrl prop to point the opaque iframe at the viewer route

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(apps): unify in-workspace app viewers on the shared sandboxed path

Route every in-workspace app display (low-code and raw) through the same
PublicAppFrame -> PublicApp machinery as the public viewer, so the sandbox /
legacy-unsandboxed / disable-sandbox-consent behavior is identical on every page.

- new InWorkspaceAppViewer renders both app types via PublicAppFrame; /apps/get
  and /apps_raw/get become thin wrappers over it
- /apps_raw/get previously rendered RawAppPreview directly (always isolated, with
  no legacy-grandfathering or consent handling); now consistent with the rest
- retire the legacy same-origin raw viewer /apps/get_raw/[version] and re-point the
  apps-list row to /apps_raw/get; remove the dead /apps_raw/[ws]/[version] route
- load the raw bundle secret in the shared viewer (getAppByPath doesn't return it)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): address PR review feedback (scope + policy hardening, nits)

- require handler-level apps:read on list_apps / list_search_apps so a scoped
  embed token cannot read app definitions through the list endpoints. The route
  layer treats apps:run as satisfying read; the handler check (which does not)
  closes the gap.
- treat legacy_unsandboxed as backend-owned: strip any client-provided value in
  create/update so it can only be set by the grandfather migration, not the API.
- document mint_app_embed_token's caller-verifies-access contract.
- use Button's declared onClick prop for the consent action (was onclick, which
  fell into the rest-spread and bypassed the component's click handling).
- test: lock that the embed scopes cannot satisfy domain-level apps:read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(apps): document embed-token endpoints in openapi + fix doc nit

Second-round review nits:
- add the three app embed-token endpoints (apps/embed_token/p/{path},
  apps_u/embed_token/{secret}, and the EE apps_u/embed_token_by_custom_path) plus
  the EmbedTokenResponse schema to openapi.yaml; note .html on get_data
- mint_app_embed_token doc: "Both" -> "All" (it lists three call sites)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(apps): bound embed-token scopes to the caller's own

The embed-token mint now enforces ensure_scopes_within_caller, so the
minted scope set is always within the calling credential's own scopes
(a no-op for regular unscoped sessions). Adds a unit test locking the
boundary and documents the contract on mint_app_embed_token.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): raw-app ctx in external embeds + page credential in direct render

- RawAppPreview: engage the storage relay only in opaque frames (probe Web
  Storage instead of just window.parent), so a public raw app embedded in an
  external iframe hydrates ctx/storage directly; add a relay-timeout fallback
  so an unresponsive parent can never stall the ctx handshake.
- PublicAppFrame: in direct render, expose the page's own bearer credential
  through the AuthToken context (JWT public URLs), matching the previous
  route behavior; opaque-viewer mode keeps the embed token.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(apps): sandbox isolation UI polish + COI embed support for raw apps

- Deploy drawer: move the sandbox toggle out of "Public URL" into its own
  "Sandbox isolation" section (the setting applies to every viewing surface,
  not just the public URL), with positive phrasing, visible helper text, and
  state-aware alerts (warning when disabled, info for pre-isolation apps).
  Toggling it now toasts its own message instead of the login-mode one.
- Extract the deploy-time migration prompt into a shared
  LegacySandboxMigrationModal built on the common Modal component, and wire
  it into the raw app editor header too (it previously had no prompt, so
  re-deploying a pre-isolation raw app silently changed behavior).
  updateRawAppPolicy now also drops the backend-owned legacy flag, matching
  the low-code updatePolicy.
- Viewer consent prompt: use the common ConfirmationModal and show the app
  path (new appPath prop) instead of the route pathname, falling back to
  "this app" when the path isn't known yet.
- COI embeds: propagate the wm_coep opt-in to the raw-app wrapper document
  and have the backend assert COEP require-corp on it when the flag is
  present — required for the bundle iframe to load when the public app page
  is embedded inside a cross-origin-isolated page. Previously this only
  worked in dev because the Vite proxy injects the header; the production
  response lacked it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(apps): app navigation parity across sandboxed and direct viewers

- Navbar component: same-app items relay query + hash to the embedder page
  (which mirrors them onto the root URL, keeping its own pathname and
  transport params), app items navigate the top page through a validated
  wm_embed_navigate relay instead of the cookieless viewer iframe, and
  external items keep opening a new tab. Selected-item detection now
  recognizes the /app_embed viewer route and ignores transport params.
- Frontend-script `goto` and button `onSuccess: gotoUrl`: same-window
  navigation goes through a shared appNavigateSameWindow helper that relays
  to the embedder inside the opaque viewer (same-origin paths SPA-navigate,
  http(s) URLs do a full load, other schemes rejected) and keeps plain
  window.location everywhere else.
- /apps/get and /apps_raw/get: key the viewer by workspace/path so in-route
  navigation fully remounts it — previously the URL changed but the app (and
  in sandbox mode its path-scoped token) did not follow.
- wm_embed/wm_embedder_origin added to the reserved query params so they no
  longer leak into the app's ctx.query.
- Raw apps: drop the sandbox attribute entirely for the unsandboxed
  (grandfathered/consented) blob path, matching the pre-isolation viewer
  exactly — the attribute added no isolation there and sandboxed popups
  (e.g. OAuth flows).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): preserve grandfathered policy across updates + in-workspace viewer parity

Round of compatibility hardening so pre-existing apps behave exactly as
before on every surface:

- `legacy_unsandboxed` is now preserved across app updates unless the payload
  explicitly clears it (`false`, sent by the editor's migration prompt and the
  sandbox toggle). Unrelated update paths — CLI / git-sync redeploys,
  publish-mode toggles, cross-workspace promotion — no longer silently drop
  the grandfathering. Clients still can never SET the flag.
- The embed-token endpoints (secret, path, EE custom-path) read only the
  sandbox-decision policy fields, leniently, and no longer mint a token for
  raw / legacy / disable_sandbox renders: the token is only consumed by the
  sandboxed low-code render, and minting for the others wrote a useless token
  row per view and could fail the render for scope-restricted callers.
- In-workspace viewer parity with the pre-sandbox `/apps/get`: new
  `inWorkspace` mode on PublicApp (no "Powered by Windmill" badge / user
  overlay, no HTML-result approval gate, column flex wrapper, `hideRefreshBar`
  honored again), and the page's query/hash are forwarded into the opaque
  viewer so `ctx.query` / `ctx.hash` reach the app.
- Raw apps: `window.ctx` is always `{ctx, workspace}` again (anonymous viewers
  of pre-existing bundles rely on `ctx.workspace`), and the runnable bridge's
  job-id scoping now applies only to sandboxed renders (`gateJobIds`) — an
  unsandboxed bundle holds the same credential as the bridge, so gating there
  only broke pre-existing apps polling persisted or runnable-returned job ids.
- Document `disable_sandbox` / `legacy_unsandboxed` in the openapi Policy
  schema; add a unit test for the lenient policy read.
- bump ee-repo-ref to the matching EE commit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): keep share-link viewer credentials out of the isolated app context

The JWT path segment of authenticated share URLs is an embedder-side
credential, consumed only to mint the scoped embed token. Two transport
channels still copied it into the isolated frame where app-authored code
runs:

- the opaque viewer iframe src defaulted to window.location.href — the
  public and custom-path routes now pass a sanitized viewerUrl (JWT segment
  stripped, query/hash preserved, captured once so the hash relay does not
  reload the iframe);
- document.referrer on the same-origin iframe navigation carried the full
  embedder URL — both app iframes now set referrerpolicy="no-referrer"
  (sandboxed renders only for the raw bundle iframe, keeping exact legacy
  parity; nothing reads the referrer).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(frontend): drop unused import inherited from main merge

`slide` import in AssistantMessage.svelte (from #9539) turns `npm run check`
red on this branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): redirect the removed raw-app viewer path to the unified viewer

The old same-origin raw-app viewer route (/apps/get_raw/{version}/{path}) was
removed in favor of the sandboxed unified viewer. Re-add a thin client route at
the old path that redirects stale bookmarks to /apps_raw/get/{path}, preserving
query + hash (the pinned version is dropped — the unified viewer shows latest).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): narrow embed-token scopes and base consent on browser session

- Embed token: resource access is metadata-only (list/type/exists) via a
  `resources:run` marker — resource values (get/get_value/get_value_interpolated/
  list_search) are no longer reachable. Job reads are by-id only: an `app_embed`
  sentinel blocks the workspace-wide job enumeration/export routes (jobs/list,
  list_filtered_uuids, queue/list, completed/list, queue/export) while by-id
  result polling keeps working.
- disable_sandbox consent now gates on whether the browser holds any Windmill
  session (cookie-only whoami) rather than workspace-scoped auth, so a viewer
  logged into a different workspace is still prompted before a same-origin render.
- db-explorer: resolve the MySQL database name server-side (the metadata query
  already falls back to DATABASE()) instead of reading the resource value
  client-side; getTablesByResource derives the default db from the schema.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(apps): trim embed-scope and consent comments

Reduce duplication — state the resource/job route exclusions and the
workspace-session-vs-cookie rationale once at their source and reference them
elsewhere; drop contrast/justification phrasing. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(apps): make app sandbox isolation opt-in (alpha)

Replace the disable_sandbox + legacy_unsandboxed policy pair and the
per-version viewer consent with a single positive `sandbox` opt-in flag.
Apps are unsandboxed by default (same-origin, full session — the
pre-isolation behavior), so existing apps are unchanged and no migration
is needed. Publishers opt an app into isolation from the deploy drawer,
flagged alpha.

- Policy.sandbox: Option<bool>; EmbedTokenResponse -> {token, expiration,
  raw_app, sandbox}; mint an embed token only for sandboxed low-code apps.
- Drop the legacy-unsandboxed migration and the deploy-time migration
  prompt; remove the consent modal and the browser-session probe.
- Deploy drawer: a single "Sandbox isolation" toggle (alpha), off by
  default, shared by the low-code and raw editors.
- Bump ee-repo-ref to the companion EE commit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): confine embed token to its intended user/folder/job routes

The embed token's broad read scopes spanned whole domains while the
matching routers are CORS-enabled for the opaque app iframe:

- users:read / folders:read were domain-wide, so the token could reach
  users/list, users/list_usage, users/username_to_email/*, folders/list,
  etc. Restrict to an app_embed-sentinel allowlist: only users/whoami and
  folders/listnames; deny the rest of those domains.
- jobs:read allowed jobs/completed/export, missed by the job denylist.
  Add it alongside jobs/queue/export.

Extend the embed-scope allow/deny test matrix to cover all of these.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(apps): align sandbox comments with the opt-in model

The consent prompt, deploy-time migration, and legacy-unsandboxed
grandfathering were removed when sandbox isolation became an opt-in
policy flag; update the comments that still described them so they
match the two-state (default-unsandboxed / opt-in-sandboxed) reality.
Comments only, no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): confine embed-token job reads to runs the app launched

App component jobs are stamped `created_by = the viewer`, so an embed token
reads its own runs via the launched-by-viewer fast path. The token then also
inherited the viewer's broader job access (share links, folder ACLs, admin
RLS), letting user-authored app JS reuse it to read unrelated jobs by id. Stop
embed tokens at the fast path: only jobs the viewer launched, never those
merely visible to them. Return NotFound so the untrusted app can't probe
existence.

Regression test: an embed token reads its own launched job but is denied the
foreign job (result/logs/getupdate) an admin viewer's normal token can read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): allowlist embed-token apps/jobs routes + scope run to the app

The embed token's apps:run/jobs:read reached more than a running app needs.
Replace the job denylist with strict per-domain allowlists on the app_embed
sentinel:

- Apps: only the app's own definition (apps/get/p/<path>) and the public
  app-serving endpoints (apps_u/*). Denies workspace app inventory
  (exists, custom_path_exists, list, list_paths*).
- Jobs: only the by-id poll routes the frontend JobLoader uses. Denies job
  counts and the job_signature/resume_urls capability-minting routes (the
  by-id reads remain confined to the app's own runs).

Drop unqualified apps:run from APP_EMBED_SCOPES; mint apps:run:<path> instead
and authorize apps:run:<requested path> first in execute_component, so the
token can only run its own app's components, not another app's.

Extend the embed-scope route matrix and add a path-scoped run unit test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(apps): clarify the sandbox toggle vs the on-behalf-of model

The deploy-drawer sandbox copy leaned on "session" in a way that collided
with the on-behalf-of permissioning right above it. Reword it to say the
toggle governs what the app's browser-side code can reach in the viewer's
browser — distinct from who its runnables execute as — and rename the label
to "Isolate the app from the viewer's browser session".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): path-scope embed-token S3 download to its own app

The apps_u/* allowlist also admitted apps_u/download_s3_file/<path>, whose
handler authorized any authenticated caller — so an embed token minted for app
A could download app B's S3 files via B's on-behalf policy. Add the same
path-scoped guard execute_component uses: download_s3_file_from_app now checks
apps:read:<path> first, confining the token to its own app. Other path-taking
apps_u routes are already covered (writes lack apps:write; embed_token/p
path-checks; public_resource is type-constrained).

Extend the path-scoping unit test to cover apps:read (download) alongside
apps:run (execute).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): path-scope public-app-by-secret read to the embed token's app

The apps_u/* allowlist admitted apps_u/public_app/<secret>, whose handler only
checked the viewer's read access — so an embed token minted for app A could read
app B's definition by secret (confused deputy via the viewer's identity).
get_public_app_by_secret now binds a scoped caller to the resolved app with
check_scopes(apps:read:<path>), confining it to its own app; unscoped sessions
and anonymous access are unchanged.

get_raw_app_data needs no binding (pure secret capability, no caller identity).
Document the full set of app-resolving handlers the path-scoped read covers.

Bump ee-repo-ref for the companion custom-path fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): preserve pre-sandbox behavior for db-explorer, edit link, jwt

Three behavior-parity fixes for non-sandboxed (existing) apps that the
sandbox-isolation refactor changed incidentally:

- DB-explorer MySQL table picker: when the connection can see multiple
  non-system schemas, label the default db's tables unprefixed again. The
  resource-value read was removed globally, so identify the default db from
  the introspection script's `DATABASE() AS default_db_name` (carried on
  SQLSchema.defaultDb) instead of guessing "the single schema key". Equivalent
  to the prior resource.database match; editor-only (table picker).
- In-workspace Edit button: restore `?nodraft=true` on both /apps/get and
  /apps_raw/get, so opening the editor from the viewer loads the deployed
  version, not a draft.
- Custom-path (/a) viewer: restore the "could not authenticate user with jwt
  token" toast when a path JWT fails to resolve a user, instead of silently
  falling through.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): confine embed-token S3 downloads to the app's own keys/outputs

download_s3_file_from_app authorized any authenticated caller for any S3 key
(opt_authed.is_some() bypass). A sandboxed app's embed token carries the
viewer's identity, so app-authored JS could fetch arbitrary S3 keys readable by
the on-behalf identity, beyond the app's own declared keys or outputs.

Route app embed tokens through the same allowlist as anonymous viewers — the
app's declared allowed_s3_keys, or files produced by this app's own component
runs — instead of the authed bypass. The produced-files check is parameterized
by created_by (the embed viewer for a token, else anonymous) so a sandboxed
app's own S3 outputs still render while arbitrary keys are denied.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): let embed tokens cancel their own jobs; gate cancel to launcher

A sandboxed low-code app supersedes an in-flight component run on re-run by
canceling it, but the embed token only had jobs:read, so cancellation silently
failed and prior jobs ran to completion.

- Permit the by-id jobs_u/queue/cancel POST for app_embed tokens at the route
  layer (the only write reachable through the existing by-id allowlist).
- Gate cancel_job_api: an app_embed token may cancel ONLY jobs it launched
  (created_by == viewer). cancel_job_api had no other per-job ownership check,
  so this also confines the token instead of letting it cancel any job by id.
- /app_embed now sets workspaceStore so cancellation targets the right
  workspace instead of an empty/stale one in the cookieless iframe.

Add a shared has_app_embed_sentinel helper; cover cancel in the route matrix
and the jobs_read_auth integration test (own job cancelable, foreign denied).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): drop get_root_job_id from the embed-token job allowlist

Audit of the embed token's reachable job routes: get_root_job (jobs_u/
get_root_job_id) has no access check in its handler at all — it returns any
job's root-job id by id — and the app runtime never calls it. Remove it from
the by-id allowlist so the embed token can't probe a foreign job's flow lineage;
add a denied-route assertion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(apps): scope sandboxed-app localStorage per app

Sandboxed apps shared one localStorage store (one key on the real origin), so an
app could read or clobber another app's keys — and, with job ids stashed there,
reuse its embed token to read another app's job. Scope the backing store per app.

The embed-token endpoints now return the resolved app_path (EmbedTokenResponse;
not a new disclosure — the viewer already receives the path when it loads the
app). PublicAppFrame (low-code) and RawAppPreview (raw) key their backing store
by it: wm_apps_localstorage:<app_path>. Same app shares one store across its
public and in-workspace surfaces; different apps are isolated. Unsandboxed apps
are unaffected (real same-origin localStorage, as before).

Bump ee-repo-ref for the companion custom-path change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): scope embed access checks to embed tokens + key app storage by workspace

- Apply the path-scoped read/run checks on the public-by-secret read and the
  component run path only when the caller is an app embed token, so other
  caller types keep their prior access.
- Key the sandboxed app's backing client storage by workspace + path instead
  of path alone, and return the resolved workspace from the embed-token
  endpoints so the custom-path viewer can derive it.
- Show a clear message instead of an indefinite loader when the viewer route
  is opened outside its embedder.

Bumps ee-repo-ref to 5b8476b.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(apps): mint embed tokens only from the trusted embedder caller

An app embed token must not reach the embed-token mint endpoints; refresh
minting stays with the embedder session/JWT. Enforced at the scope route
layer and at the mint chokepoint, with a route-matrix regression test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(apps): support S3 upload and frontend-script S3 download in sandboxed apps

Sandboxed apps run with a scoped embed token (no cookie). Let the app's
S3 file-input upload and the frontend-script download({s3}) helper work in
that context: upload is reachable with apps:run and re-checked per-app at the
handler; the script download routes through the app-scoped apps_u endpoint
with the embed token instead of the cookie-authed job_helpers path. Default
(unsandboxed) apps are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: update ee-repo-ref to b0cb761bf9852974e571b2978032d310cc998517

This commit updates the EE repository reference after PR #600 was merged in windmill-ee-private.

Previous ee-repo-ref: e673c714a4618fdb72353a475f49c748e6016642

New ee-repo-ref: b0cb761bf9852974e571b2978032d310cc998517

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2026-06-23 10:05:00 +02:00
Ruben Fiszelandrubenfiszel e82a6a6830 chore(main): release 1.736.0 (#9720)
* chore(main): release 1.736.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-06-23 09:44:35 +02:00
Ruben FiszelandClaude Opus 4.8 e16061df06 fix(health): detect read-only replica via pg_is_in_recovery() (#9722)
The /api/health/status database check used `SELECT 1`, which succeeds
even on a read-only standby. After a PostgreSQL failover where the
primary becomes a secondary, the health check kept reporting healthy
while all writes failed with "cannot execute INSERT in a read-only
transaction", so Kubernetes liveness probes never restarted the pod.

Use `SELECT NOT pg_is_in_recovery()` instead: it returns true on a
primary and false on a standby, so a read-only replica is now reported
unhealthy. Result handling checks the returned bool (Ok(Some(true)))
rather than just query success.

Fixes WIN-2085

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 09:37:29 +02:00
Ruben Fiszel 3bf5b72afa fix(drafts): stop mis-filing workspace-blind legacy drafts on migration (#9725) 2026-06-23 09:27:02 +02:00
centdixandClaude Opus 4.8 6f4017d694 feat(ai-chat): workspace AI chat skills (SKILL.md upload + read_skill tool) (#9648)
* feat(ai-chat): workspace ai_skill table + CRUD API

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ai-chat): AI Skills workspace settings tab with SKILL.md upload

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ai-chat): advertise skills in global system prompt + read_skill tool

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ai-chat): move custom skills into AI settings (paste or folder)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ai-chat): cap folder import (depth<=3, max 50 skills, confirm dialog)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* style(ai-chat): give import folder its own labeled subsection

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ai-chat): resolve svelte-check never-narrowing in skills preview

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: address ai skills review issues

* fix: validate ai skills and reload workspace list

* fix(ai-chat): spec-align skill validation and cap skills per workspace

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ai-chat): reject duplicate skill uploads, audit skill names

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ai-chat): sync deref openapi specs with skill validation rules

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 00:16:13 +02:00
Ruben FiszelandClaude Opus 4.8 d5cb944cf9 fix(frontend): ensure type:object in test_run_flow tool schema for Anthropic (#9721)
Flows with no defined inputs can produce a sparse schema (e.g. { order: [] })
that lacks the "type": "object" field. buildSchemaForTool spread this schema
into the tool parameters as-is, so the Anthropic API rejected the tool
definition with `400 invalid_request_error:
tools.N.custom.input_schema.type: Field required`. The existing fallback in
anthropic.ts only triggers when parameters is falsy, but the sparse schema is
truthy.

Default type:object before spreading the schema in buildSchemaForTool, and
backfill type/properties/required in FlowAIChat's getFlowInputsSchema as a
defense-in-depth measure.

Fixes WIN-2087

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 00:08:13 +02:00
hugocasaandClaude Opus 4.8 e19594df2a fix: re-enforce scoped API token boundaries across handlers (#9712)
* fix: re-enforce per-path token scope on store rename, delete and interpolation

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: enforce token scope on workspace export and resume-url minting

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: enforce per-item and runnable scope on trigger create paths

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: enforce app write scope before persistence and on rename

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: enforce scope containment on mcp oauth approval

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: scope mcp endpoint-proxy jwt to the proxied route

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: treat resource-linked variables and resources as covered by resource scope

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: only require variables:read for plaintext-secret workspace export

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: handle singlestepflow resume, reject empty mcp grant, scope var-skipped tarball

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:55:19 +02:00
Ruben FiszelandClaude Opus 4.8 6e96f90065 fix(frontend): destroy old WebsocketProvider on workspace switch in MultiplayerMenu (#9719)
Switching workspaces created a new WebsocketProvider without destroying
the old one. The leaked provider kept reconnecting, causing alternating
websocket traffic between old and new workspace rooms and flickering in
the Live Activity sidebar.

Add a disconnectWorkspace() cleanup that destroys the provider and resets
connected/awareness state, call it at the start of connectWorkspace()
before creating a new provider (matching ScriptEditor.svelte), and run it
from an onDestroy hook so the provider is torn down on unmount.

Fixes WIN-2086

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 23:20:08 +02:00
Ruben Fiszelandrubenfiszel 83ec0dd07a chore(main): release 1.735.0 (#9700)
* chore(main): release 1.735.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-06-22 14:07:47 +02:00
GuilhemandClaude Opus 4.8 e20a27745a fix(frontend): strip raw-app post-deploy diff noise (raw_app/lock/data) (#9706)
* fix(frontend): strip raw-app post-deploy diff noise (raw_app/lock/data)

The raw-app editor's Diff drawer showed a spurious deployed-vs-current
diff immediately after deploy, even with no edits: `raw_app: true`, a
server-recomputed inline-script `lock`, and an empty `data` mismatch.

These come from comparing the deployed app row (from getAppByPath) against
the editor's current value, which differ on server-managed fields the
editor never carries, on inline-script locks (recomputed at deploy, cleared
on edit), and on `data` (the deployed row omits an empty `data` while the
editor always carries the default `{tables: []}`).

Add `stripRawAppDiffNoise` (strip server columns, null inline locks,
canonicalize data) and apply it symmetrically to both diff sides in the
editor header. For the session/compare draft diff, the draft is stored flat
(files/runnables/data top-level) while the deployed row nests under `value`,
so add `canonicalRawAppDiffValue` (= appSourceToDraftValue + stripRawAppDiffNoise)
and route both sides through it in getDraftDiffValues. Both diff surfaces now
share the same normalizer.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): use canonicalized current value in deploy-drawer raw-app diff

The Deploy drawer's "Diff" action still built the current side inline from
raw editor state, bypassing stripRawAppDiffNoise — so inline-script `lock`
and data-shape noise could resurface via Deploy → Diff even though the
top-level Diff button was already fixed. Route it through `currentDiffValue`
(and strip the savedApp fallback) so both entry points behave identically.

Addresses Codex review finding on PR #9706.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 13:15:14 +02:00
hugocasaandClaude Opus 4.8 e403f92d7e fix: enforce job_dir containment when writing module files (#9703)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 13:14:14 +02:00
Ruben FiszelandClaude Opus 4.8 8a0b0abead fix: ignore NotFound errors when deleting log files from object store (#9707)
* fix: ignore NotFound errors when deleting log files from object store

Periodic and manual log cleanup delete log files from instance object
storage. S3's DeleteObjects silently ignores missing keys, but GCS
returns a 404 for each individual delete, which the object_store crate's
default delete_stream surfaces as Error::NotFound. This produced noisy
error/warning logs on every cleanup cycle even though the cleanup
succeeded (DB records are removed regardless).

Treat a NotFound delete as a successful no-op in both delete handlers:
- monitor.rs: skip logging NotFound errors
- log_cleanup.rs: count NotFound as deleted instead of an error

Fixes WIN-2081

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: report 404 (already-absent) count in object store log cleanup

Track delete calls that returned 404 (object already absent) separately
from real deletes so operators can see how many of the attempted deletes
were no-ops, instead of those numbers silently folding into s3_deleted.

- monitor.rs: emit a final info summary per cleanup cycle:
  "N deleted, M already absent (404), K failed" (only when work occurred)
- log_cleanup.rs: add s3_not_found to LogCleanupProgress (serde default for
  backward-compatible deserialization of in-flight rows), thread it through
  s3_bulk_delete and all call sites, and log a final summary on release
- openapi.yaml + generated client + ObjectStoreConfigSettings.svelte:
  surface the 404 count in the manual cleanup status UI

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: import ObjectStoreError directly from object_store_reexports

The object_store_reexports module already re-exports object_store::Error
under the name ObjectStoreError, so `Error as ObjectStoreError` failed to
resolve (no `Error` in that module). This compiles only behind the
parquet feature, which the local dev `cargo watch` doesn't enable, so it
was caught by CI's full-feature check rather than locally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 13:12:29 +02:00
GuilhemandClaude Opus 4.8 ed016a5edb feat: clarify session draft bar tracks all workspace draft changes (#9714)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 13:11:45 +02:00
ef4962e52a fix(oauth): restore bring-your-own CC token URL override (#9711)
* fix(oauth): restore bring-your-own CC token URL override

Re-add the optional resource-level token URL field for client-credentials
connections, sent only with the caller's own client_id/secret. Updates the
connect/create_account request schemas and bumps the EE ref.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(oauth): keep openapi-deref unchanged from main

The dereferenced specs are not regenerated per-PR (already stale on main,
CI only lint-validates them). Revert the incidental full regen so the PR
diff stays focused on openapi.yaml.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(oauth): host-pin CC token URL override server-side

Add is_instance_templated_cc so the EE handlers can reject a bring-your-own
token URL override for {instance}-templated providers (defense in depth for
direct API callers). Bump the EE ref.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(oauth): serve cc_token_url in deref specs, enforce CC grant gate

Add cc_token_url to the dereferenced OpenAPI artifacts served at /openapi.yaml
and /openapi.json so generated clients see the new field (kept to a focused add
rather than a full regen). Bump the EE ref for the grant-gate enforcement.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to de49fda2320504ad9e7d2d31c7033d71dbf6ca43

This commit updates the EE repository reference after PR #625 was merged in windmill-ee-private.

Previous ee-repo-ref: a939228d0314c21937687d43c8ef354bdc87c40e

New ee-repo-ref: de49fda2320504ad9e7d2d31c7033d71dbf6ca43

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-06-22 13:05:15 +02:00
centdixandClaude Opus 4.8 b0ddcf31e4 ci: add path-gated AI agent + ai_evals smoke workflows (#9640)
* ci: add path-gated AI agent integration tests workflow

Runs integration_tests/ai_agent_tests against real LLM providers
(Anthropic/OpenAI/Google) only when AI-agent backend code or the tests
change, since runs make paid LLM calls. Adds a conftest fixture that
skips provider-parametrized cases whose API keys are absent, so CI
exercises only the providers it has secrets for.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci: add path-gated ai_evals global-mode smoke workflow

Runs the global AI chat eval (global-test1) across one cheap model per
provider (Anthropic/OpenAI/Google/DeepSeek) only when the eval harness or
copilot chat code change, since runs make paid LLM calls. Builds Windmill
CE from source as the AI proxy; global tools/drafts run in the Vitest
bridge. Gates on the deterministic draft pipeline (run succeeded +
produced a draft + used write_script), not the variable LLM judge score.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci: run AI smokes on PR ready-for-review instead of every push

Switch the pull_request trigger from `synchronize` (every commit) to
`ready_for_review`, with a job guard skipping draft PRs, so the paid LLM
runs only fire when a PR is marked ready to merge (plus push-to-main and
manual dispatch).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ai_evals): lazily load cli mode so non-cli evals skip the cli toolchain

The entrypoint eagerly imported modes/cli, which pulls the wmill CLI
guidance modules and their JSR deps (@cliffy/*). Global/flow/script/app
runs then crashed with "Cannot find module '@cliffy/ansi/colors'" when
the cli workspace deps were not installed. Import createCliModeRunner
dynamically inside runCliBenchmark instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ai_agent): raise low max_completion_tokens to OpenAI's 16 minimum

OpenAI's /v1/responses rejects max_output_tokens < 16 with a 400, failing
test_low_max_tokens for openai. 16 still exercises a truncated response.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci: run ai_evals workflow on Node 22 for the frontend undici 8.x dep

The Vitest bridge loads frontend/node_modules/undici@8.x, which requires
Node >=22.19; Node 20 failed with "webidl.util.markAsUncloneable is not a
function" when loading vitest.config.ts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ai_evals): run frontend evals autonomously + give global-test1 more turns

Frontend evals (flow/script/app/global) ran the production chat prompt, which
assumes an interactive human — so cheaper models burned their turn budget
asking for confirmation, waiting for approval, or presenting a plan, sometimes
hitting maxTurns without producing a draft. Append a shared autonomy note in
baseEvalRunner (the path all frontend modes share, mirroring cli mode): act
directly on clear requests; only ask on genuinely ambiguous ones (preserving
the askUserQuestion cases). Also raise global-test1's maxTurns 8 -> 10 so a
model that over-explores still converges.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci(ai_evals): watch draft/prompt deps outside copilot/

The global eval runs production frontend code in-process, so the smoke's
behavior depends on files outside frontend/src/lib/components/copilot/**:
the draft model (userDraft.svelte.ts, userDraftDbSyncer.svelte.ts), script
inference (infer.ts), and the chat system prompts ($system_prompts ->
system_prompts/auto-generated). Add them to both push and PR path filters so
a change there actually triggers the smoke that gates on draft production.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: skip direct provider tests without credentials

* feat: add ai evals skip judge flag

* fix: simplify ai evals ci gate

* fix: simplify ai evals smoke gate

* fix: handle ai eval workflow triggers

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 12:41:31 +02:00
centdixandClaude Opus 4.8 74a2329d2e feat(copilot): improve global-mode path selection + add path-selection evals (#9698)
* test: add global-mode path-selection eval cases with seeded user

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(copilot): guide global-mode path selection with injected folder list

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(copilot): tailor global-mode folder guidance for workspace admins

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(copilot): type folders_read; isolate global-eval user from store

Addresses PR review:
- Add folders_read to the User/whoami openapi schema and UserExt; the global prompt builder and eval harness now read it typed instead of via inline casts (regen the client to pick it up).
- prepareGlobalSystemMessage takes an explicit user; the eval harness passes it rather than mutating the process-global userStore, removing the concurrency race (path cases no longer need --verbose).
- Rewrite the path-selection case comment as a current invariant.
- Add buildFolderGuidance unit tests in core.test.ts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 11:59:59 +02:00
Ruben FiszelandClaude Opus 4.8 ace7b68a28 fix: sanitize git credentials from ansible executor errors and logs (#9697)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 11:02:24 +02:00
6eb03d2590 round worker usage occupancy up to 5-minute chunks (#9702)
* feat: round worker usage occupancy up to 5-minute chunks

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to 8b12fa14ef7969e948169eadf1bf672d7928e5b1

This commit updates the EE repository reference after PR #624 was merged in windmill-ee-private.

Previous ee-repo-ref: 168498974150ff309658b2da43bce8444f13a765

New ee-repo-ref: 8b12fa14ef7969e948169eadf1bf672d7928e5b1

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-06-22 10:59:28 +02:00
GuilhemandClaude Opus 4.8 23bf6bf3da fix(frontend): deploy full script/flow draft from AI chat via shared module (#9642)
* fix(frontend): deploy full script/flow draft from AI chat via shared module

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): drop non-persisted priority/timeout from flow draft deploy

The flow branch of the shared deployDraft set `priority`/`timeout` on the
create/update body, but the backend does not persist those fields on flows
(a direct API write returns them as null). Remove the dead fields and the
unit-test assertions for them; the flow deploy still carries every config
field the backend actually stores (tag, dedicated_worker, …).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): chat deploy resolves draft storage path (honor chosen path)

The chat addresses drafts by their display/chosen path, but a draft_only item
created in the editor lives at a synthetic `u/{user}/draft_{uuid}` storage key
(chosen path held in the draft value). The shared deployer reads the draft via
getScriptByPath/getFlowByPath at the path passed, so passing the chosen path
404'd. Resolve to the storage path via getGlobalDraftStoragePath before
delegating; the deployer then deploys at the draft's own `path`. Regression from
the deploy-unification: the old builder read the already-resolved draft and
deployed at the chosen path directly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): chat raw-app deploy honors the draft's chosen path

The raw-app branch deployed at the path the chat was addressed by (args.path),
which for an editor-created draft_only raw app is the synthetic
`u/{user}/draft_{uuid}` storage key, not the chosen path. Resolve the storage
path and read the chosen path from the backend raw_app draft's `draft_path`
(confirmed shape: getAppByPath{getDraft,rawApp}.draft.draft_path), then create/
update there — mirroring the script/flow storage-path resolution. Content still
comes from the flat AppDraftValue, which the editor and chat both use.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): flush live draft before chat deploy; narrow raw-app catch

Addresses review feedback on the AI-chat deploy:

- Codex P1: script/flow deploy delegates to the shared deployer, which re-reads
  the persisted DB draft. An open editor's edit may still be parked in a
  debounced/disabled autosave, so the deploy could publish a stale draft and the
  post-deploy draft delete could drop the unsaved edit. Flush the draft's
  UserDraftDbSyncer key before delegating (always saves, like Ctrl/Cmd+S, since
  the user explicitly asked to deploy).
- Cubic P2: the raw-app draft_path lookup caught all errors and fell back to the
  storage path, masking real failures (network/5xx). Only fall back on 404;
  re-throw other errors so the deploy aborts instead of deploying to the wrong path.

Adds tests for both; updates the existing raw-app deploy tests to mock getAppByPath.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): flush raw-app draft before reading draft_path on chat deploy

Codex P1 follow-up: the raw-app branch derives the deploy targetPath by re-reading
draft_path from the persisted backend draft, but — unlike script/flow — didn't
flush first. An editor rename mirrored into draft_path can still be parked in a
debounced/disabled autosave, so an immediate chat deploy could read a stale
draft_path and deploy to the old path. Flush the raw_app draft key before the
getAppByPath read, mirroring the script/flow fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): abort chat deploy when pre-deploy draft flush conflicts/fails

Codex P1 follow-up: the pre-delegation UserDraftDbSyncer.flush() resolves even
when the save recorded a conflict (server has a newer version) or failed
(network/5xx) — it does not throw. The deploy would then re-read a stale or
conflicting persisted draft and publish it. Add flushDraftOrThrow(): after flush,
check getConflict() and getState().state === 'failed' and abort with a clear
message. Used by both the script/flow and raw-app deploy paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 10:55:15 +02:00
Diego ImbertandClaude Opus 4.8 4a8a724895 feat: scope default instance db name to workspace (dt_/dl_) (#9699)
* feat: default instance db name to dt_/dl_ workspace scope

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: cap instance db name at 63 chars and add unit tests

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 10:33:49 +02:00
hugocasaandClaude Fable 5 3a55800224 add whatsapp business icon (#9541)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-22 09:33:55 +02:00
GuilhemandClaude Opus 4.8 84cc043406 feat: link files & folders to the global AI chat (#9520)
* feat: add file attachments to the global AI chat

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat: add folder linking and file-type icons to chat attachments

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat: persist linked files, add @-menu file tree, and polish chat file UI

Persistence (survive reload, scoped to session.id):
- IndexedDB store (attachedFilesDB) holding Blob snapshots (every browser)
  and re-grantable File System Access directory handles (capable browsers)
- restore on session activation; re-grant locked handles on the next send;
  flush in-memory items when the session persists; GC on session delete
- capability via feature-detection (fsAccess), never UA sniffing
- folders auto-refresh (live re-enumerate + reconcile) on each send

@-mention file picker:
- Files branch in ChatContextPicker (new DrillPicker architecture); a linked
  folder's files render as a nested directory tree, picking inserts @filename
- attached-file mentions highlight in the input just like context mentions

UI polish:
- file/folder chips reuse the context-element chip style (icon -> X on hover)
- file + context badges sit above the fork/draft bar
- disabled dropdown items can surface an explanatory tooltip (DropdownV2Inner)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor: deepen the attached-files store — folders as first-class objects

Two seam fixes from an architecture pass, no behaviour change:

- addFolder(dirHandle) now enumerates internally (same junk-filtered walk
  used on restore/refresh), so callers never pre-enumerate. The dead
  drop-walkers (collectDroppedEntries, filterFolderPickerFiles) are deleted;
  isIgnoredPath/MAX_FOLDER_FILES move next to enumerateDir in fsAccess.

- The store exposes `folders` (name + aggregate status + children) and
  `standalone` as derived views, so the bar, the @-menu picker, the folder
  chip and the system-prompt roster stop re-grouping the flat row list and
  re-deriving folder status. Placeholder rows (isFolderRoot) become an
  implementation detail; the roster renders a locked folder as one line.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: drop the redundant context-badge row in the global chat

In GLOBAL mode selected context already appears as a highlighted @mention
in the input (deleting the mention deselects), so the hoisted badge row
above the chat duplicated it. File chips keep their row — attachments
aren't represented in the input.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: harden attachment edge cases found in review

- requestReadPermission/queryReadPermission never reject (the spec rejects
  with SecurityError when user activation is missing — now mapped to
  denied/prompt), and sendRequest wraps attachment upkeep in try/catch, so
  a permission hiccup can never silently swallow a Send.
- regrantLocked expands before dropping the locked placeholder: when the
  re-granted directory is gone from disk, the folder now shows
  "unavailable" instead of vanishing into a zombie that resurrects locked
  on the next reload.
- addFolder: re-picking a locked/unavailable folder relinks it (natural
  recovery gesture); a genuine second folder with the same basename gets a
  visible "already linked" rejection instead of a silent no-op.
- fileEngine: readFile clamps its byte slice to maxChars*4 before decoding
  and streamLines caps its per-line buffer, so newline-sparse files
  (minified JS, single-line JSONL) can't materialize unbounded strings;
  corrected the scan-cap comment's claim about catastrophic backtracking.

4 new unit tests (41 total).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: surface folder-picker failures instead of swallowing them

`pickDirectory` caught every `showDirectoryPicker` rejection and returned
undefined, so a real failure (an enterprise/browser policy blocking the File
System Access API, a lost user-activation, …) was indistinguishable from a
no-op — the picker just silently never opened. Now only `AbortError` (user
dismissed the dialog, or CDP intercepted it under automation) is treated as a
cancel; anything else is rethrown and `linkFolder` surfaces it as a toast.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: support folders in browsers without the File System Access API

Folders can now be added in every browser, not just Chromium. Where the File
System Access API is absent (Firefox/Safari), a dropped or picked folder's files
are snapshotted into the browser (via a webkitGetAsEntry drop-walk or a
`webkitdirectory` input) instead of linked as a live handle, and grouped/displayed
identically to a File System Access folder. The dropdown item reads "Link folder"
when a live link is possible and "Add folder" otherwise, with a tooltip pointing
to Chrome/Edge for a live link.

Snapshot folder children persist their `folder`/`relPath`, so they regroup into
the same folder chip on reload.

Removes the arbitrary file-count caps (500 per folder, 100 total) — only the
browser's memory / IndexedDB quota now bound a folder. Junk paths
(node_modules/.git/dist/dotfiles) are still skipped, folder-contents only, so an
explicitly attached standalone dotfile is kept.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: address review feedback — index-race guard + read_file line numbers

Both automated reviewers flagged two issues on the attached-files feature:

- (P1) Stale async indexing could corrupt a newer file. `#indexFile` applied
  its unawaited `buildLineIndex` result by display name, so if a row's file was
  swapped while indexing was in flight (remove + re-add a same-named file, or a
  folder refresh re-indexing an edited file) the stale result stamped the wrong
  lineIndex/lineCount — and `read_file` then sliced the new Blob with old
  offsets. Now patched via `#patchFile`, which applies the result only while the
  row still holds the exact file object that was indexed.

- (P2) `read_file` promised "line-numbered context" but returned raw text. It now
  prefixes each line with its absolute 1-based number (`<n>→<content>`), matching
  the tool contract; `numberLines` lives in fileEngine and is unit-tested.

Adds regression tests: a deterministic stale-index race test (controlled
buildLineIndex ordering) and numberLines coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: address re-review nits — read_file pagination + searchFiles regex state

- read_file: when the maxChars cap truncated a window short of its requested
  end line, the pagination note still reported the full range and gave no/wrong
  resume point, so the model couldn't reach the unread lines. The note now
  reports the last line actually returned and resumes at the next unread line
  (advancing past a single over-long line rather than re-truncating it forever).
- searchFiles: reset `regex.lastIndex` before each `.test()` — a caller-supplied
  `g`/`y` flag makes test() stateful and would silently drop matches. Not
  reachable from the current caller, but searchFiles is exported.

Adds regression tests for both.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: keep an emptied live folder linked and refreshing

A live (File System Access) folder carried its directory handle only on its child
file rows. When the folder was emptied on disk, refreshFolders/#reconcileFolder
removed the last child — dropping the only handle-bearing row — so the folder
vanished from the chip bar AND was never re-enumerated again (files added back on
disk weren't picked up until a reload). #expandFolder had the same gap on restore.

Now #ensureFolderRow leaves one handle-carrying placeholder row when a folder has
no readable children (keeps the chip visible and the live source alive), and drops
it once children return; refreshFolders collects sources from placeholder rows too,
and readyFiles never exposes a placeholder to the read/search tools. Adds a
regression test (empty → still visible → file returns → picked up).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: trim read_file char-cap output to match its pagination note

When the char cap cut partway into the line after some whole lines, readFile set
the note/endLine to the last complete line but still returned the partial next
line in `text` — so read_file showed (line-numbered) a line the note said would
come on the next read. Trim the returned text back to the last complete newline
so the body and the note agree. Test now asserts res.text for that case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: isolate search_files in a Worker (ReDoS) + path-aware folder dedup

- search_files runs a model-supplied regex, and a catastrophic-backtracking
  pattern (e.g. /^(a+)+$/) can't be interrupted mid-test, freezing the tab. Run
  the search in a Web Worker (searchFilesInWorker) and terminate it on a timeout,
  returning "pattern too expensive" instead of hanging. Degrades gracefully to a
  main-thread search where Workers are unavailable / fail to load.
- #isDuplicate keyed its content check on the file basename, so two distinct
  files sharing a basename under different folder subdirs (proj/a/index.ts vs
  proj/b/index.ts) were wrongly deduped and silently dropped from snapshotted
  folders. Key it on the relative path instead.

Adds tests: worker result/timeout-and-terminate, and same-basename-different-subdir.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: keep an initially-empty live folder linked (placeholder + persist)

addFolder only created rows / persisted the dir-handle when at least one text file
was found, so linking a folder that's empty (or all-binary) at pick time was a
silent no-op: no chip, nothing persisted, and refreshFolders had no source to
re-enumerate when files were added later. Now it always leaves a placeholder
(#ensureFolderRow) and persists the handle — matching the became-empty behavior —
so the folder stays visible, survives reload, and picks up files added afterward.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: keep empty-folder placeholders out of the real-file name space

The placeholder row for an empty live folder uses name = folder, which could
collide with a standalone file of the same name: addFiles deduped the file
against the placeholder, removeFile(name) dropped both rows, and #uniqueName
pushed the file to a "(2)" suffix. Placeholders are managed via removeFolder and
never read by the tools, so exclude isFolderRoot rows from #isDuplicate,
removeFile, get(), and #uniqueName. Adds a placeholder/standalone collision test.

(codex's other nit — @-mentions not highlighting filenames with spaces — left as
a known cosmetic limitation per the chosen scope.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: highlight @-mentions of filenames containing spaces

A file mention was inserted verbatim as `@my file.txt`, but the highlighter regex
`@[\w/.\-\[\]]+` stops at the space, so only `@my` was parsed/highlighted and the
mention didn't behave as advertised. Introduce a small shared `mention` module:
names with whitespace are inserted in a bracketed form `@[my file.txt]`, and the
shared regex + `mentionTitle` parse both bare and bracketed tokens. Both insertion
entry points (the inline `@` picker in ContextTextarea and the toolbar path in
AIChatInput) now use `formatMention`, so the full name highlights.

Verified in a real browser: `@[my file.txt]` renders as a single highlight span.
Unit tests cover format/parse/round-trip.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: search_files reports a requested file's real status, not "not attached"

search_files filtered the store down to readyFiles() before validating a
requested `file`, so searching an attached-but-not-ready file (indexing / errored
/ locked / unavailable) while another file was ready returned "No attached file
named X" — even though it is attached. Factor read_file's status reporting into a
shared notReadyMessage() and have search_files report the same accurate status
before searching the ready subset.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: clear attached files on new/loaded chat in the non-session global chat

saveAndClear() (the "New chat" button) and loadPastChat() left attachedFiles
intact. In an AI session that's intended — files are session-scoped and persist
across conversations. But the ephemeral global side-panel chat has no session, so
the next, unrelated conversation still got the previous file roster injected and
could read_file/search_files against it. Clear attachments on both transitions
when `!isSessionChat`; sessions keep them. Adds a lifecycle regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: keep an empty folder linked when regranting access after reload

regrantLocked() dropped the locked placeholder unconditionally after #expandFolder.
If the regranted folder was empty (or all-binary), #expandFolder's #ensureFolderRow
no-op'd (the locked placeholder still existed), so dropping it removed the only
handle-bearing row — unlinking the folder and stopping future refreshFolders from
ever seeing files added back. Re-ensure a ready placeholder after dropping the
locked one. Adds a regression test for the empty-regrant path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: round-trip @-mentions of filenames containing a closing bracket

The bracketed mention form `@[name]` broke when the name contained a `]`
(e.g. `notes ] draft.md`): the regex stopped at the first `]` and mentionTitle
resolved the wrong name, so it wouldn't highlight. Escape `\` and `]` when
bracketing, match escaped chars in MENTION_RE, and unescape in mentionTitle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: highlight @-mentions of filenames with HTML-sensitive / special chars

getHighlightedText() escapes the textarea value to HTML before parsing mentions,
then looked the parsed title up against raw attached names — so a file like
`R&D notes.md` (escaped to `R&amp;D notes.md`) never matched and wasn't highlighted.
Also, names with chars outside the bare set (`<`, `>`, `&`, parens, …) weren't
bracketed, so the bare regex truncated them. Now formatMention brackets any
non-bare-safe name, and the highlighter HTML-unescapes the parsed title before the
store lookup. Verified in a real browser with `R&D notes.md`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: report the real reason search_files has no readable targets

When attachments existed but readyFiles() was empty, search_files always told the
model "still being indexed, try again shortly". That's wrong for the placeholder
states this PR introduces: an empty or binary-only linked folder leaves only a
filtered-out `ready` placeholder, and a locked/unavailable restored folder exposes
no readable children. Now the message reflects the actual state — no searchable
text, restore access, or re-link — and only says "indexing" when something is.
Adds a focused fileTools test for the empty-ready states.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-22 09:21:14 +02:00
Ruben Fiszelandrubenfiszel 346cc30e2d chore(main): release 1.734.0 (#9691)
* chore(main): release 1.734.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-06-20 16:32:16 +02:00
Diego ImbertandClaude Opus 4.8 b0973c3023 hide primary storage row until added in workspace storage settings (#9692)
* feat(frontend): hide primary storage row until added in workspace storage settings

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(frontend): red border on empty storage resource picker

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(frontend): allow deleting primary storage when no secondary storages exist

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(frontend): disable storage save when a row is missing its resource

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 16:18:00 +02:00
Ruben Fiszel 3ebf24359d feat: ducklake materialization for data pipelines (#9689) 2026-06-20 15:42:03 +02:00
Diego ImbertandClaude Opus 4.8 09a80040ca fix(frontend): clear branch step state when switching outer loop iterations (#9650)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-20 12:22:15 +02:00
Ruben Fiszelandrubenfiszel d5388da953 chore(main): release 1.733.1 (#9685)
* chore(main): release 1.733.1

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-06-19 17:37:27 +02:00
Ruben FiszelandClaude Opus 4.8 119d94601d feat(jobs): auto-grant approvers a run-detail view link on the approval page (#9686)
* feat(jobs): auto-grant approvers a run-detail view link on the approval page

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(approval): show a clear run-details button for logged-in workspace members

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(approval): build run-details link from route params, not undefined job

getJob is fire-and-forget on the new approval page and is denied for an approver who lacks direct run read access — the exact case this link serves — leaving job undefined and producing /run/undefined. page.params.job is the flow id the view_token is minted for.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 17:29:19 +02:00
Ruben FiszelandClaude Opus 4.8 6aa9e515f7 de-flake asset-dispatch debounce test by reading settings from DB (#9687)
The debounce assertion resolved the dispatched job's debounce window through
`prefetch_cached_from_handle`, which goes through the process-global
runnable-settings cache (shared by every test running concurrently in the
binary) and its tempdir-backed file I/O. Read the persisted rows directly from
the test's isolated DB instead, removing that cross-test coupling and extra I/O
from the assertion path. Still validates the full wiring
(handle -> runnable_settings -> debouncing_settings).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 17:16:18 +02:00
960c55e1f3 preserve URL filter state on page refresh (#9680)
* fix(home): preserve URL filter state on page refresh

ListFilters.loadFilterFromUrl() runs synchronously at script init and
sets ownerFilter via binding. When $workspaceStore resolves asynchronously
after mount it triggered the $effect that resets ownerFilter, wiping the
URL-loaded filter before the user saw any results.

Skip the first $workspaceStore resolution using the same firstRun guard
pattern already used in this file (firstWorkspaceRun). Workspace switches
still correctly clear the filter.

Fixes #9624

* refactor(home): reset filters on workspace change instead of first-run guard

Track the previous workspace value and clear filters only when it actually
changes, rather than skipping the first $workspaceStore resolution. Encodes
the real invariant (reset on change) without depending on child/parent init
ordering, and preserves URL-loaded filters on initial mount by construction.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 17:12:34 +02:00
hugocasaandClaude Opus 4.8 c1f31c0e47 fix(backend): validate ansible vault_id entries before config generation (#9681)
* fix(backend): validate ansible vault_id entries before config generation

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(backend): cover ansible.cfg write-boundary vault_id validation

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 16:54:25 +02:00
hugocasaandClaude Opus 4.8 c39ee07c0b fix: validate websocket trigger urls and gate trigger test route (#9682)
* fix: validate websocket trigger urls and gate trigger test route

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: clarify validate_websocket_url_for_ssrf call sites

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 16:54:04 +02:00
hugocasaandClaude Opus 4.8 fb44fe7af2 fix: require super admin for object storage config test endpoint (#9683)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 16:53:49 +02:00
Ruben FiszelandClaude Opus 4.8 1be4df9acb fix(frontend): group live pipeline runs in the activity panel (#9684)
* fix(frontend): refetch pipeline dispatch edges on live runs so cascades group

The activity panel groups runs into cascades by connected components of the
dispatch-edge graph, but edges came only from the one-shot history preload
(refetched on workspace/folder/days change). dispatch_event rows are written
server-side when a producer completes, so a run launched live had its producer
and freshly-dispatched children appear as live poll events with no connecting
edge — they rendered as separate ungrouped rows instead of one cascade.

Add an edges-only refetch and trigger it whenever the live poll's event id-set
changes, so live cascades converge to grouped like historic/scheduled ones.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): address review nits on live dispatch-edge refetch

- Sequence same-scope edges-only refetches with a monotonic edgeSeq so a
  slower earlier response can't overwrite a newer one mid-cascade (the gen
  counter only guards scope changes).
- Condense the duplicated edge-refetch rationale: keep the canonical "why"
  in loadEdges, trim the page effect comment to its trigger/loop invariant.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 16:43:23 +02:00
Ruben Fiszelandrubenfiszel b2ce475fc3 chore(main): release 1.733.0 (#9677)
* chore(main): release 1.733.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-06-19 13:50:47 +00:00
Ruben FiszelandClaude Opus 4.8 5970510cf6 perf(backend): gate asset producer-change event on write-set changes (#9672)
* perf(backend): only emit asset producer-change event on write-set changes

Data Pipelines (#9193) made every script deploy emit a
`notify_asset_producer_change` event: `clear_static_asset_usage` inserted
into `notify_event` unconditionally on every clear, and the per-asset
insert path emitted nothing. So a plain script with no assets — the
overwhelming majority of deploys — wrote a `notify_event` row that made
every worker drop its `ASSET_PRODUCER_WRITES_CACHE` entry instance-wide,
needlessly thrashing the cache the feature added.

That cache only tracks script rows with write access (`usage_access_type
IN ('w','rw')`), so a deploy changes it only when the script gains or
loses a write producer. Gate the event on exactly that:

- `clear_static_asset_usage` / `clear_static_asset_usage_by_script_hash`
  emit only when the delete removed a 'w'/'rw' row (via `RETURNING`).
- `insert_static_asset_usage` emits only when it actually inserts a
  'w'/'rw' script row (no-op `ON CONFLICT`, read-only, and flow usage
  stay silent).

Plain, read-only, and flow deploys now emit nothing; producer-changing
deploys still invalidate, atomically and visible-only-on-commit as
before. Adds a test asserting the emit/no-emit matrix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(backend): dedup producer-change notify on write-asset redeploys

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(backend): derive replace write-set from persisted rows; document auth contract

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(backend): correct replace_static_asset_usage call-site comment

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 13:49:32 +00:00
6833a554ae fix: allow users to always discard their own drafts without write permission (#9659)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2026-06-19 13:34:38 +00:00
centdixandClaude Opus 4.8 4296a6ae1f feat(ai-chat): cap read_app_file + search_app grep tool to bound context in large raw apps (#9653)
* docs: add global AI chat context-optimization plan for raw apps

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ai-evals): add global raw-app debugging cases on a large fixture

Adds a ~20-file analytics_dashboard raw-app fixture (incl. a 5k-line data module
and a planted wrong-totals bug), two global cases (read-heavy debug + small-edit
baseline), app-seed support in the mock backend, directory-fixture loading, and a
decorateHelpers seam so read-dedupe is measurable. Records tokenUsage for before/
after comparison of the read-tool optimization.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ai-chat): cap and dedupe read_app_file to bound context in large apps

read_app_file now defaults to a head slice (1500 lines / 50k chars) with offset/
limit to page further, and skips resending a file whose earlier read is still in
context (per-conversation ledger keyed off the originating tool-call id, so it
self-heals after compaction). Bounds the file-content portion of global-chat
context when working in large raw apps.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ai-evals): add read-heavy raw-app debug case (large data module)

global-test31 induces the model to inspect the 5k-line seedData module, exercising
the read_app_file cap/offset path. Baseline ~262k tokens vs ~200k with the cap+dedupe
change (-24%).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: record A+B benchmark results and fixed-overhead finding

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ai-chat): clearer read_app_file past-EOF message + unit tests for cap/dedupe

Addresses local-review nits: out-of-range offset now reports 'offset N is past the
end of the file' instead of a backwards 'lines 11-10' label; adds unit coverage for
the slicing (line cap, offset/limit window, char budget, past-EOF) and re-read dedupe
(hit + miss-when-not-retained).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ai-chat): char-level paging + per-range dedupe for read_app_file

Adds char_offset/char_limit so minified/long-line files can be paged within a line
window, keys the re-read ledger by range (so reading different ranges no longer
collides), and dedupes on the full-file hash (a cached range stub is invalidated
when any byte of the file changes, not just the returned range). Tests updated for
the char-slice behavior plus single-line capping, char paging, and out-of-window
change detection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ai-chat): add read_app_file context micro-benchmark + re-read eval case

Adds a deterministic micro-benchmark (no LLM) that drives read_app_file through a
realistic big-project read pattern (large file, re-read, minified bundle, paging)
and asserts the cap+dedupe cut returned context >50% vs the old whole-file behavior
— isolating the feature's effect from model nondeterminism and guarding against
silent weakening. Adds global-test32, a cross-file consistency investigation that
revisits overlapping files so re-read dedupe is exercised in a real run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ai-evals): clarify test32 measures the read cap, not dedupe

Verified: sonnet and haiku both read each file once per conversation and retain
it, so test32 never triggers read_app_file re-read dedupe. Dedupe is measured
deterministically by the micro-benchmark instead. Comment corrected to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ai-chat): drop read_app_file re-read dedupe, ship the cap only

Benchmarking showed the per-conversation re-read dedupe never fires in practice:
across sonnet/opus/gpt-5.5/haiku, every model reads each file once per conversation
and keeps it in context (0 within-conversation re-reads). It was a correct but unused
guard, so this removes the ledger, full-file hash, retention predicate, the
AIChatManager wiring, and the eval decorateHelpers seam — keeping the read cap +
offset/limit/char paging (A), which is the lever that actually bounds context. The
micro-benchmark is now cap-only; test32 is kept as a multi-file read-load case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(ai-chat): add search_app grep tool for global raw-app chat (experimental)

Client-side grep over a raw app's frontend files and inline runnables (literal,
case-insensitive, optional file_glob/context_lines/max_matches, head-capped).
Completes the list -> search -> ranged-read triad. Includes the eval A/B gate
(WMILL_AI_EVAL_DISABLE_SEARCH_APP), unit tests + micro-benchmark, and a
find-all-usages eval case (global-test33).

Experimental: A/B benchmarking shows it is not an unconditional win — it helps
on find-all-usages but adds agentic iterations on navigable apps.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(ai-evals): accept search_app as a valid file-inspection tool in raw-app cases

Add requiredToolsAnyOf alternatives-group to ToolValidationSpec and switch
global-test29..32 to it so a model that locates files via search_app instead
of read_app_file no longer false-fails the tool assertion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: remove stale ai-chat context-optimization planning doc

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ai-chat): drop read_app_file char paging for a hard char cap

The char_offset/char_limit params guarded minified files (a single line over
the char budget) but were effectively unused in benchmarks. Remove them and the
in-window char paging; keep the hard 50k-char budget and, when a read hits it,
tell the model to narrow the line limit (or treat the file as unreadable if a
single line exceeds the budget). Proper long-line handling is left as a TODO.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ai-chat): bake search_app context to 1 line, clarify query is literal

Drop the context_lines param (models varied it to little effect) for a fixed
SEARCH_APP_CONTEXT_LINES=1, and cap on matching lines instead of pushed rows so
max_matches stays accurate with context always on. Sharpen the query description
to state it is a literal (non-regex) substring and to suggest the call form
(e.g. formatCurrency() to hit call sites and skip formatCurrencyPrecise.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(ai-chat): widen baked search_app context to 2 lines

Models that set the old context_lines param leaned to 2; match the lean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ai-chat): count every file with a match in search_app header

Move fileHadMatch ahead of the render cap so files whose matches fall past max_matches are still counted (with a regression test). Also swap the raw NUL globstar sentinel for a printable escape (the NUL bytes made core.ts read as binary to grep) and reword two comments to describe current constraints instead of drafting history.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ai-chat): drop redundant input echoes from app tool results

read_app_file and search_app no longer prefix results with the tool name or echo back the caller's own inputs (file path, query, file_glob) — the model already has them from the call args, and the unbounded query echo could push the search result past its output budget. Keeps the useful signals (line range, match/file counts, truncation) and the actionable advice. Also reword max_matches to 'matching lines' since it caps lines (each expands to context rows). Unit tests updated to the new format.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 13:33:54 +00:00
Ruben FiszelandClaude Opus 4.8 924f9c7e8d fix(backend): strip NUL bytes from draft values on write (#9673)
draft.value is a json column (not jsonb), so a client could store a U+0000
escape in it. Any later text extraction (`->>` / `to_jsonb`) on such a value
raises 22P05 "unsupported Unicode escape sequence" — one poisoned draft 500'd
the whole GET /drafts/list, silently hiding the home-page "This workspace has N
drafts" banner (and breaking the global drafts page).

Prevent it at the source: sanitize the value in update_draft (the only path that
writes client-supplied draft content) so a NUL never reaches the column.
strip_json_nul does a single backslash-parity-aware byte pass that removes real
NUL escapes (values and keys alike) while leaving a legitimate escaped backslash
intact — O(n) with no serde_json::Value tree to allocate, important because the
slow path is also hit by any value legitimately containing the text after a
backslash (e.g. script source). The clean path is a single substring check.

A SQL migration scrubs rows written before this, gated to genuinely-poisoned
rows (a real NUL makes value::jsonb raise, distinguishing it from a legitimately
escaped backslash). With the data clean, no read-side query needs to change.

Tests: unit tests for the strip helper (escaped-backslash no-op, real+literal
collision, odd-backslash-run parity, nested keys/values) and an integration test
that POSTs a NUL-bearing draft and asserts it is stored and listed NUL-free
(fails without the strip).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 13:33:36 +00:00
PyraandClaude Opus 4.7 ab3bc97cd9 feat(python, windows): enable S3 to cache wheels (#5199)
* add .minio-data to .gitignore

* feat(python): make S3 cache arch specific

Current schema:

S3-Bucket
├── python_311
│   ├── wheel==1.0
│   └── wheel2==1.0
└── python_312
    ├── wheel==1.0
    └── wheel2==1.0

New schema:

S3-Bucket
├── linux_aarch64
│   └── ...
└── linux_x86_64
    ├── python_311
    │   ├── wheel==1.0
    │   └── wheel2==1.0
    └── python_312
        ├── wheel==1.0
        └── wheel2==1.0

* remove .minio-data from .gitignore

* remove unneeded tracing::error

* feat(python, windows): enable S3 to cache wheels

* fix(python, windows): drop residual unix cfg gates on S3 cache items

Post-merge, PIPTAR_UPLOAD_CHANNEL and its call sites were already
cross-platform, but the types/functions they reference (PiptarUploadTask,
handle_piptar_uploads, pull_from_tar, OBJECT_STORE_SETTINGS) remained
unix-gated, breaking the Windows build.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-19 13:11:35 +00:00
Ruben Fiszel 496e770264 Revert "fix(backend): clean up unique_ext_jwt_token on workspace deletion (#9…" (#9678)
This reverts commit 9add719d93.
2026-06-19 13:01:15 +00:00
Ruben FiszelandClaude Opus 4.8 cafb473494 fix(python): split PIP_TRUSTED_HOST by whitespace to support multiple hosts (#9675)
* fix(python): split PIP_TRUSTED_HOST by whitespace for multiple hosts

When PIP_TRUSTED_HOST contains multiple space-separated hostnames, the
whole string was passed as a single --trusted-host argument
(--trusted-host "host1 host2") rather than one flag per host. This
matches pip's documented PIP_TRUSTED_HOST convention by emitting a
separate --trusted-host for each host, mirroring the existing
pip_extra_index_url handling.

Fixes WIN-2077

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(python): use shell word-splitting for nsjail trusted-host args

Address review feedback: the previous tr/sed pipeline only split on
single spaces, diverging from the Rust split_whitespace() paths. Repeated
or leading/trailing spaces produced empty --trusted-host flags and
tab-separated hosts were not split. Use an unquoted for-loop over
$TRUSTED_HOST so the shell's own IFS word-splitting handles arbitrary
whitespace and skips empty fields, matching the non-nsjail paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 12:59:08 +00:00
Ruben FiszelandClaude Opus 4.8 9add719d93 fix(backend): clean up unique_ext_jwt_token on workspace deletion (#9676)
The workspace_id column on unique_ext_jwt_token (migration 20260409145556)
has no FK constraint on the workspace table, and delete_workspace did not
remove its rows. Deleted workspaces left orphaned external JWT token records
that kept appearing in the superadmin External JWTs listing.

Add a DELETE FROM unique_ext_jwt_token WHERE workspace_id = $1 alongside the
other per-table cleanup statements in delete_workspace.

Fixes WIN-2078

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 12:48:48 +00:00
Ruben Fiszelandrubenfiszel b24616dc44 chore(main): release 1.732.0 (#9670)
* chore(main): release 1.732.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-06-19 11:34:59 +00:00
Ruben FiszelandClaude Opus 4.8 33617367d0 fix(backend): grant script_trigger access to windmill roles (#9674)
The script_trigger table (migration 20260423050000_script_trigger)
relied on ALTER DEFAULT PRIVILEGES to grant access to windmill_user and
windmill_admin. Those default privileges only apply to objects created
by the role that set them (migration 20250205131523), so deployments
whose migration runner is a different role leave script_trigger
ungranted.

Direct application writes run as the invoking role and fail with
"permission denied for table script_trigger" — notably
clear_script_triggers and insert_script_trigger in
windmill-common/src/assets.rs during every script save.

Add an explicit GRANT on script_trigger and its sequence, matching the
notify_event fix (#9665) and the asset table precedent.

Fixes WIN-2076

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 11:30:21 +00:00
Ruben FiszelandClaude Opus 4.8 3371265382 fix(frontend): ignore hash/assets in script diffs and drafts (WIN-2071) (#9664)
* fix(frontend): ignore hash/assets in script diffs and drafts

The script editor's draft value is seeded from the full `getScriptByPath`
DB row (since #9351), so it carries `hash` (the deployed version's
identity) and `assets` (re-derived from the script content by the
editor). Neither is editable draft content, yet both were persisted into
the draft row and surfaced as spurious changes in the workspace/fork
compare diff view.

- Add `hash`, `assets`, and the read-time-computed `inherited_labels` to
  `CLEANED_VALUE_KEYS` so the shared diff/unsaved-change strip ignores
  them everywhere (DiffDrawer + WorkspaceItemDiffViewer).
- Strip `hash` and `assets` from script drafts at the single persistence
  chokepoint (`UserDraftDbSyncer.save`) so every path — reactive
  autosave, Ctrl/Cmd+S flush, the pagehide keepalive — sends the same
  trimmed payload. On reload the deployed row re-supplies them.

Fixes WIN-2071

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(frontend): clarify draft sanitizer vs diff-strip relationship

The two field lists are intentionally not equal — only the hash/assets
overlap must stay consistent. Reword the comment so a future maintainer
doesn't add keys to one expecting parity with the other.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 11:21:24 +00:00
Ruben FiszelandClaude Opus 4.8 017c3d3343 feat(ansible): add AI chat and editor bar buttons for ansible (#9671)
Ansible scripts previously lacked the AI assistant and the contextual
variable helper that other scripting languages expose in the script editor.

- Add 'ansible' to SUPPORTED_CHAT_SCRIPT_LANGUAGES so the AI chat button
  shows in the editor toolbar and the AI chat opens in SCRIPT mode without
  the "language not supported" warning.
- Add an Ansible system prompt (system_prompts/languages/ansible.md) plus a
  LANGUAGE_METADATA entry, and regenerate the auto-generated prompts/skills
  so the AI has tailored Ansible context.
- Show the contextual variable picker for ansible and insert references as
  `{{ lookup('env', 'NAME') }}`, matching how Windmill exposes reserved
  variables as environment variables to the ansible-playbook process.

Fixes WIN-2072

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 11:12:33 +00:00
centdixandClaude Opus 4.8 0cc2257596 fix(ai): emit token usage in gemini proxy streaming translation (#9669)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 10:57:52 +00:00
Ruben Fiszelandrubenfiszel 887a3076b2 chore(main): release 1.731.0 (#9668)
* chore(main): release 1.731.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-06-19 10:16:35 +00:00
centdixandClaude Opus 4.8 c30bdecea7 fix(mcp): repair invalid type keywords in tool JSON schemas (#9667)
Drop empty/unknown `type` values (Windmill emits `type: ""` for untyped fields) and infer `type: array` for nodes carrying `items`, so generated MCP tool schemas validate against JSON Schema draft 2020-12. Anthropic's tool registration rejected the whole tool list otherwise.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 10:04:42 +00:00
Ruben FiszelandClaude Opus 4.8 7e4df02bd6 fix: trigger flow error handler on unrecoverable (OOM/zombie) step failures (#9662)
* fix: trigger flow error handler on unrecoverable (OOM/zombie) step failures

When a worker is OOM-killed mid-step, the zombie job handler fails the step
via handle_job_error with unrecoverable=true. update_flow_status_after_job_completion
had `false if unrecoverable => false`, which silently completed the flow with the
error and skipped the flow's failure module (error handler). It would also have
pinned the failure module to the dead worker via same_worker.

Unrecoverable failures now route to the failure module instead of being retried or
silently dropped, and the error-handler step is pushed as a regular queued job that
any live worker can pick up.

Fixes WIN-2070

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: skip retry on unrecoverable flow failures, add retry-skip regression test

Address review: the failure-module-on-unrecoverable change must also bypass the per-step retry policy in push_next_flow_job, otherwise an OOM/zombie-killed step with a retry config would be retried instead of routing to the error handler. Gate the retry evaluation on !unrecoverable and add a regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: add sqlx offline cache for new flow-step zombie test query

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: route unrecoverable continue_on_error step failures to the error handler

Addresses Codex/Pi review (P1): with continue_on_error on the failed step, the
step counter was advanced before the unrecoverable decision branch, so
push_next_flow_job pushed the next normal step instead of the failure module —
hiding the worker death and letting the flow complete successfully.

- Do not advance the step counter (inc) for an unrecoverable continue_on_error failure.
- Let the Failure arm in push_next_flow_job route to the failure step even on a
  continue_on_error module when unrecoverable.
- Add a regression test (a[continue_on_error] -> b + failure_module): asserts the
  failure module runs and step b does not.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 09:57:05 +00:00
hugocasaandClaude Opus 4.8 a425431e90 feat(backend): auto-reconnect postgres trigger listener with backoff (WIN-2073) (#9666)
* feat(backend): auto-reconnect postgres trigger listener with backoff

The Postgres trigger listener permanently disabled itself on any
connection error (stream close, receive error), so a transient network
interruption (e.g. a cloud provider maintenance window) permanently
killed the trigger.

Restructure the listener to match the Kafka trigger: the replication
connection is now established inside an outer reconnect loop in
`consume`. On a dropped stream or receive error it backs off 30s and
reconnects instead of disabling, reporting a critical error every 10
failed attempts and a recovery once it reconnects. Disabling is kept
only for unrecoverable misconfiguration (missing publication or
replication slot, unparsable replication message).

Fixes WIN-2073

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(backend): count postgres reconnects on stream drop and alert from first

Adopt the SQS listener's reconnection accounting in the Postgres trigger
listener. A stream close or receive error now counts toward the retry
counter and raises a throttled critical error (on the first occurrence,
then every 10 attempts), and the retry counter is reset / recovery is
reported only once the reconnected stream actually delivers a message.

Previously the inner-loop disconnect branches reset `tries` to 0 on every
successful (re)connect and never alerted, so a stream that connected and
then immediately dropped could ping-pong every 30s indefinitely without
ever raising an alert. Resetting on real progress rather than on a bare
connect closes that blind spot and matches the SQS pattern.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 09:56:36 +00:00
Ruben FiszelandClaude Opus 4.8 a682d02311 fix(backend): grant notify_event access to windmill roles (#9665)
The notify_event table (migration 20260203172950_polling_based_events)
relied on ALTER DEFAULT PRIVILEGES to grant access to windmill_user and
windmill_admin. Those default privileges only apply to objects created
by the role that set them (migration 20250205131523), so deployments
whose migration runner is a different role leave notify_event ungranted.

Trigger inserts were already worked around with SECURITY DEFINER
(migration 20260206060555), but direct application inserts that run as
the invoking role still failed with "permission denied for table
notify_event" — notably clear_static_asset_usage in assets.rs during
script save, and restart_worker_group in settings.

Add an explicit GRANT on notify_event and its sequence, matching the
existing explicit-grant pattern used for the asset table.

Fixes WIN-2074

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 09:55:18 +00:00
301 changed files with 26611 additions and 2276 deletions
+132
View File
@@ -0,0 +1,132 @@
name: AI Agent Integration Tests
# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against
# real LLM providers. Runs only when AI-agent backend code or the tests change,
# because each run makes real (paid) LLM calls. To avoid spending on every commit,
# the PR side triggers only when a PR is marked ready for review (out of draft) —
# not on `synchronize` — plus push to main and manual dispatch.
on:
workflow_dispatch:
push:
branches: [main]
paths:
- "integration_tests/ai_agent_tests/**"
- "backend/windmill-ai/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-worker/src/ai_executor.rs"
- "backend/windmill-worker/src/ai/**"
- "backend/windmill-worker/src/memory_common.rs"
- "backend/windmill-common/src/flow_conversations.rs"
- ".github/workflows/ai-agent-tests.yml"
pull_request:
types: [opened, reopened, ready_for_review]
paths:
- "integration_tests/ai_agent_tests/**"
- "backend/windmill-ai/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-worker/src/ai_executor.rs"
- "backend/windmill-worker/src/ai/**"
- "backend/windmill-worker/src/memory_common.rs"
- "backend/windmill-common/src/flow_conversations.rs"
- ".github/workflows/ai-agent-tests.yml"
concurrency:
group: ai-agent-tests-${{ github.ref }}
cancel-in-progress: true
jobs:
ai_agent_e2e:
# Skip draft PRs; the `opened`/`reopened` types would otherwise fire while
# still a draft. `ready_for_review` always arrives non-draft.
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
runs-on: ubicloud-standard-16
services:
postgres:
image: postgres:16
ports:
- 5432:5432
env:
POSTGRES_DB: windmill
POSTGRES_PASSWORD: changeme
options: >-
--health-cmd pg_isready --health-interval 10s --health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
cache-workspaces: backend
toolchain: 1.93.0
- uses: oven-sh/setup-bun@v2
with:
bun-version: 1.3.10
- uses: actions/setup-node@v4
with:
node-version: "20"
- uses: actions/setup-python@v5
with:
python-version: "3.11"
# CE build (no enterprise/license needed for AI agents). `quickjs` powers
# flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool
# test. Bun tool scripts run via the always-on worker (BUN_PATH).
- name: Build Windmill
working-directory: ./backend
env:
SQLX_OFFLINE: true
CARGO_BUILD_JOBS: 12
RUSTFLAGS: ""
run: cargo build --features quickjs,mcp
- name: Start Windmill
working-directory: ./backend
env:
DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
BUN_PATH: bun
NODE_BIN_PATH: node
RUST_LOG: info
run: |
mkdir -p ../integration_tests/logs
./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 &
echo "Waiting for Windmill to be ready..."
for i in $(seq 1 60); do
if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
echo "Windmill is ready"
break
fi
sleep 2
done
curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; }
- name: Run AI agent integration tests
timeout-minutes: 20
working-directory: ./integration_tests/ai_agent_tests
env:
WINDMILL_URL: http://localhost:8000
# Only the providers we have org secrets for. Other providers
# (Azure, Bedrock, OpenRouter) are skipped by conftest when their
# keys are absent — see skip_provider_without_credentials.
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
run: |
python -m venv .venv
.venv/bin/pip install -r requirements.txt
# The S3/vision-attachment tests need MinIO large-file storage and
# image-capable provider setup; out of scope for this cost-controlled
# smoke. Add MinIO secrets + a storage service to enable them.
.venv/bin/python -m pytest -v \
--ignore=test_user_attachments.py \
--ignore=test_user_images.py \
--ignore=test_image_output.py
- name: Archive Windmill logs
uses: actions/upload-artifact@v4
if: always()
with:
name: ai-agent-tests-windmill-logs
path: integration_tests/logs
+166
View File
@@ -0,0 +1,166 @@
name: AI Evals (global mode)
# Smoke-tests the production global AI chat proxy/frontend execution path via
# the ai_evals harness, one case across one cheap model per provider. Runs only
# when the eval harness or the global chat code change, since each run makes real
# (paid) LLM calls. The backend is built from source purely as the AI proxy the
# harness routes model calls through; the global tools/drafts run in-process in
# the Vitest bridge against production frontend code. To avoid spending on every
# commit, the PR side triggers only when a PR is marked ready for review (out of
# draft) — not on `synchronize` — plus push to main and manual dispatch.
on:
workflow_dispatch:
push:
branches: [main]
paths:
- "ai_evals/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-ai/**"
- "frontend/src/lib/components/copilot/**"
# The eval harness runs production frontend code in-process; these are the
# AI/draft-specific deps outside copilot/ that the global smoke exercises.
- "frontend/src/lib/userDraft.svelte.ts"
- "frontend/src/lib/userDraftDbSyncer.svelte.ts"
- "frontend/src/lib/infer.ts"
- ".github/workflows/ai-evals-test.yml"
pull_request:
types: [opened, reopened, ready_for_review]
paths:
- "ai_evals/**"
- "backend/windmill-api/src/ai.rs"
- "backend/windmill-ai/**"
- "frontend/src/lib/components/copilot/**"
# The eval harness runs production frontend code in-process; these are the
# AI/draft-specific deps outside copilot/ that the global smoke exercises.
- "frontend/src/lib/userDraft.svelte.ts"
- "frontend/src/lib/userDraftDbSyncer.svelte.ts"
- "frontend/src/lib/infer.ts"
- ".github/workflows/ai-evals-test.yml"
concurrency:
group: ai-evals-test-${{ github.ref }}
cancel-in-progress: true
jobs:
ai_evals_global:
# Provider secrets are unavailable to forked and Dependabot PRs.
if: >-
github.event_name != 'pull_request' ||
(
github.event.pull_request.draft == false &&
github.event.pull_request.head.repo.full_name == github.repository &&
github.event.pull_request.user.login != 'dependabot[bot]'
)
runs-on: ubicloud-standard-16
services:
postgres:
image: postgres:16
ports:
- 5432:5432
env:
POSTGRES_DB: windmill
POSTGRES_PASSWORD: changeme
options: >-
--health-cmd pg_isready --health-interval 10s --health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
cache-workspaces: backend
toolchain: 1.93.0
- uses: oven-sh/setup-bun@v2
with:
bun-version: 1.3.10
- uses: actions/setup-node@v4
with:
# Node 22.19+ is required by the frontend's undici 8.x, which the
# Vitest bridge loads; Node 20 fails with markAsUncloneable.
node-version: "22"
# CE build used only as the AI proxy (login, workspace, provider resource,
# /ai/proxy). No worker execution or MCP needed — global tools/drafts run
# in the Vitest bridge. quickjs matches the standard CE feature set.
- name: Build Windmill (AI proxy)
working-directory: ./backend
env:
SQLX_OFFLINE: true
CARGO_BUILD_JOBS: 12
RUSTFLAGS: ""
run: cargo build --features quickjs
- name: Start Windmill
working-directory: ./backend
env:
DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill
RUST_LOG: info
run: |
mkdir -p ../ai_evals/logs
./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 &
echo "Waiting for Windmill to be ready..."
for i in $(seq 1 60); do
if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then
echo "Windmill is ready"
break
fi
sleep 2
done
curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; }
- name: Install frontend deps + generate client
working-directory: ./frontend
run: |
npm ci
npm run generate-backend-client
- name: Run global AI evals
timeout-minutes: 20
working-directory: ./ai_evals
env:
WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000
WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests
# Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY.
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
run: |
bun install
mkdir -p results
# One cheap model per provider (anthropic/openai/googleai/deepseek).
fail=0
for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do
echo "::group::global-test1-script-create ($m)"
if ! bun run cli -- run global global-test1-script-create \
--model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then
echo "$m: harness/proxy errored"
fail=1
echo "::endgroup::"
continue
fi
# The CLI exits 0 when the harness records failed attempts, so gate
# on execution-only pass counts while ignoring model output quality.
if jq -e \
'.attemptCount > 0 and .passedAttempts == .attemptCount' \
"results/ci-$m.json" > /dev/null; then
echo "$m: OK — proxy/frontend execution completed"
else
echo "$m: FAILED proxy/frontend execution"
jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true
fail=1
fi
echo "::endgroup::"
done
[ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; }
- name: Archive logs and results
uses: actions/upload-artifact@v4
if: always()
with:
name: ai-evals-global-logs
path: |
ai_evals/logs
ai_evals/results
+123
View File
@@ -1,5 +1,128 @@
# Changelog
## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23)
### Features
* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904))
### Bug Fixes
* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911))
* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6))
* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a))
* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39))
* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd))
### Performance Improvements
* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011))
## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23)
### Features
* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd))
### Bug Fixes
* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72))
* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c))
* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76))
* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9))
* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6))
## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22)
### Features
* clarify session draft bar tracks all workspace draft changes ([#9714](https://github.com/windmill-labs/windmill/issues/9714)) ([ed016a5](https://github.com/windmill-labs/windmill/commit/ed016a5edb4527877bf7e6bf92feafc2710669c2))
* **copilot:** improve global-mode path selection + add path-selection evals ([#9698](https://github.com/windmill-labs/windmill/issues/9698)) ([74a2329](https://github.com/windmill-labs/windmill/commit/74a2329d2e8395141807c43acef07ef132490039))
* link files & folders to the global AI chat ([#9520](https://github.com/windmill-labs/windmill/issues/9520)) ([84cc043](https://github.com/windmill-labs/windmill/commit/84cc043406d63a1e1472165cf24ce8c09905fc5f))
* scope default instance db name to workspace (dt_/dl_) ([#9699](https://github.com/windmill-labs/windmill/issues/9699)) ([4a8a724](https://github.com/windmill-labs/windmill/commit/4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb))
### Bug Fixes
* enforce job_dir containment when writing module files ([#9703](https://github.com/windmill-labs/windmill/issues/9703)) ([e403f92](https://github.com/windmill-labs/windmill/commit/e403f92d7e84cebc78709dce1a0928048ba2506d))
* **frontend:** deploy full script/flow draft from AI chat via shared module ([#9642](https://github.com/windmill-labs/windmill/issues/9642)) ([23bf6bf](https://github.com/windmill-labs/windmill/commit/23bf6bf3da01d552d2dfab6dbcfd30f758ed34d2))
* **frontend:** strip raw-app post-deploy diff noise (raw_app/lock/data) ([#9706](https://github.com/windmill-labs/windmill/issues/9706)) ([e20a277](https://github.com/windmill-labs/windmill/commit/e20a27745a08d552e6d2c5a8bbaf08ccfe89c68f))
* ignore NotFound errors when deleting log files from object store ([#9707](https://github.com/windmill-labs/windmill/issues/9707)) ([8a0b0ab](https://github.com/windmill-labs/windmill/commit/8a0b0abead71320c4f69eb3007739a19f76d4126))
* **oauth:** restore bring-your-own CC token URL override ([#9711](https://github.com/windmill-labs/windmill/issues/9711)) ([ef4962e](https://github.com/windmill-labs/windmill/commit/ef4962e52aba0bc79bf72523de9101853c660654))
* sanitize git credentials from ansible executor errors and logs ([#9697](https://github.com/windmill-labs/windmill/issues/9697)) ([ace7b68](https://github.com/windmill-labs/windmill/commit/ace7b68a28b00d715298ffcb6ae907c1974a74b8))
## [1.734.0](https://github.com/windmill-labs/windmill/compare/v1.733.1...v1.734.0) (2026-06-20)
### Features
* ducklake materialization for data pipelines ([#9689](https://github.com/windmill-labs/windmill/issues/9689)) ([3ebf243](https://github.com/windmill-labs/windmill/commit/3ebf24359d66048d6361ce65cd879cdc04b737ed))
### Bug Fixes
* **frontend:** clear branch step state when switching outer loop iterations ([#9650](https://github.com/windmill-labs/windmill/issues/9650)) ([09a8004](https://github.com/windmill-labs/windmill/commit/09a80040ca268a4379d5302e3401435ba93247e0))
## [1.733.1](https://github.com/windmill-labs/windmill/compare/v1.733.0...v1.733.1) (2026-06-19)
### Bug Fixes
* **backend:** validate ansible vault_id entries before config generation ([#9681](https://github.com/windmill-labs/windmill/issues/9681)) ([c1f31c0](https://github.com/windmill-labs/windmill/commit/c1f31c0e4777bf0cfed0dd7f03249e9a61cd8cb9))
* **frontend:** group live pipeline runs in the activity panel ([#9684](https://github.com/windmill-labs/windmill/issues/9684)) ([1be4df9](https://github.com/windmill-labs/windmill/commit/1be4df9acb935250d4cc12e83cf67e366d870d5a))
* require super admin for object storage config test endpoint ([#9683](https://github.com/windmill-labs/windmill/issues/9683)) ([fb44fe7](https://github.com/windmill-labs/windmill/commit/fb44fe7af2b8ebe8ef64ffb0e5acce8580bf4200))
* validate websocket trigger urls and gate trigger test route ([#9682](https://github.com/windmill-labs/windmill/issues/9682)) ([c39ee07](https://github.com/windmill-labs/windmill/commit/c39ee07c0bcd2249dd19ffa5cd988125eefc6c9f))
## [1.733.0](https://github.com/windmill-labs/windmill/compare/v1.732.0...v1.733.0) (2026-06-19)
### Features
* **ai-chat:** cap read_app_file + search_app grep tool to bound context in large raw apps ([#9653](https://github.com/windmill-labs/windmill/issues/9653)) ([4296a6a](https://github.com/windmill-labs/windmill/commit/4296a6ae1f73564de4df54fe1df0a03c1df05dfd))
* **python, windows:** enable S3 to cache wheels ([#5199](https://github.com/windmill-labs/windmill/issues/5199)) ([ab3bc97](https://github.com/windmill-labs/windmill/commit/ab3bc97cd92b6480327029bcf018280442462af7))
### Bug Fixes
* allow users to always discard their own drafts without write permission ([#9659](https://github.com/windmill-labs/windmill/issues/9659)) ([6833a55](https://github.com/windmill-labs/windmill/commit/6833a554aeddb3e63173d3c3140b490c0bf2822b))
* **backend:** clean up unique_ext_jwt_token on workspace deletion ([#9676](https://github.com/windmill-labs/windmill/issues/9676)) ([9add719](https://github.com/windmill-labs/windmill/commit/9add719d936cdcfb2c4062629e3e1f792694dafe))
* **backend:** strip NUL bytes from draft values on write ([#9673](https://github.com/windmill-labs/windmill/issues/9673)) ([924f9c7](https://github.com/windmill-labs/windmill/commit/924f9c7e8d8863d9af40aee246a519b4be0e1ea2))
* **python:** split PIP_TRUSTED_HOST by whitespace to support multiple hosts ([#9675](https://github.com/windmill-labs/windmill/issues/9675)) ([cafb473](https://github.com/windmill-labs/windmill/commit/cafb473494d9cff3a8b2aeaf9f18b015f966e7b3))
## [1.732.0](https://github.com/windmill-labs/windmill/compare/v1.731.0...v1.732.0) (2026-06-19)
### Features
* **ansible:** add AI chat and editor bar buttons for ansible ([#9671](https://github.com/windmill-labs/windmill/issues/9671)) ([017c3d3](https://github.com/windmill-labs/windmill/commit/017c3d3343c2577501103be4b2dd8dac9727d80d))
### Bug Fixes
* **ai:** emit token usage in gemini proxy streaming translation ([#9669](https://github.com/windmill-labs/windmill/issues/9669)) ([0cc2257](https://github.com/windmill-labs/windmill/commit/0cc2257596a3965cf6db21a0090edcce6e1b8419))
* **backend:** grant script_trigger access to windmill roles ([#9674](https://github.com/windmill-labs/windmill/issues/9674)) ([3361736](https://github.com/windmill-labs/windmill/commit/33617367d09537667d2ab3f91135c736194b9e7e))
* **frontend:** ignore hash/assets in script diffs and drafts (WIN-2071) ([#9664](https://github.com/windmill-labs/windmill/issues/9664)) ([3371265](https://github.com/windmill-labs/windmill/commit/33712653821e83f2562dd5f271dbec0188d5d2f8))
## [1.731.0](https://github.com/windmill-labs/windmill/compare/v1.730.0...v1.731.0) (2026-06-19)
### Features
* **backend:** auto-reconnect postgres trigger listener with backoff (WIN-2073) ([#9666](https://github.com/windmill-labs/windmill/issues/9666)) ([a425431](https://github.com/windmill-labs/windmill/commit/a425431e9067bcf85474fdc7b7ef7f73e41b9071))
### Bug Fixes
* **backend:** grant notify_event access to windmill roles ([#9665](https://github.com/windmill-labs/windmill/issues/9665)) ([a682d02](https://github.com/windmill-labs/windmill/commit/a682d02311a2110bfc0d5e0a5b52e96147fe0dd7))
* **mcp:** repair invalid type keywords in tool JSON schemas ([#9667](https://github.com/windmill-labs/windmill/issues/9667)) ([c30bdec](https://github.com/windmill-labs/windmill/commit/c30bdecea77ff9b4d74d52961f3101201099b683))
* trigger flow error handler on unrecoverable (OOM/zombie) step failures ([#9662](https://github.com/windmill-labs/windmill/issues/9662)) ([7e4df02](https://github.com/windmill-labs/windmill/commit/7e4df02bd60c4d6ee8c92d3dfd19f4e587ff9632))
## [1.730.0](https://github.com/windmill-labs/windmill/compare/v1.729.0...v1.730.0) (2026-06-18)
+3 -1
View File
@@ -75,6 +75,8 @@ Public CLI surface:
- `--model <alias>`: choose the model under test
- `--models <a,b,c>`: run the same cases sequentially against several model aliases
- `--verbose`: stream assistant output for frontend runs
- `--skip-judge`: skip LLM judge scoring for the run
- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring
- `--record`: append a compact tracked summary line to `ai_evals/history/<mode>.jsonl` for full-suite runs only
- `--backend-validation <mode>`: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals
@@ -99,7 +101,7 @@ Notes:
- the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5`
- frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases
- `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there
- the judge model is separate and currently defaults to `claude-sonnet-4-6`
- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs
## Case Format
@@ -25,6 +25,12 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
);
const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1";
const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1";
const executionOnly =
process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1";
const judgeModel =
process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly
? null
: DEFAULT_JUDGE_MODEL;
const model = resolveEvalModel(
mode,
process.env.WMILL_FRONTEND_AI_EVAL_MODEL,
@@ -48,7 +54,8 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
cases: selectedCases,
runs,
runModel,
judgeModel: DEFAULT_JUDGE_MODEL,
judgeModel,
executionOnly,
concurrency: verbose ? 1 : undefined,
verbose,
onProgress: emitProgress
@@ -60,7 +67,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise<BenchmarkRunResult>
mode,
runs,
runModel,
judgeModel: DEFAULT_JUDGE_MODEL,
judgeModel,
caseResults,
});
}
@@ -32,6 +32,10 @@ const MUTATING_GLOBAL_TOOLS = new Set([
]);
const DISABLE_ACTIVE_EDITOR_CONTEXT_ENV =
"WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT";
// A/B gate for the search_app read tool: set to "1" to run the baseline arm
// (toolset without search_app) so its token cost can be compared against the arm
// that offers it.
const DISABLE_SEARCH_APP_ENV = "WMILL_AI_EVAL_DISABLE_SEARCH_APP";
const LIVE_EDITOR_ITEM_KINDS = {
script: "script",
@@ -46,6 +50,20 @@ export interface GlobalLiveEditorDraftFixture {
value?: unknown;
}
// Identity the global system prompt builds paths from. Production reads
// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs
// in, so without this the prompt sees an empty username (`u//...`) and no
// path-selection case is meaningful. Seeded per-case via the initial fixture and
// passed straight to `prepareGlobalSystemMessage` (no global-store mutation).
export interface GlobalUserFixture {
username: string;
is_admin?: boolean;
/** Folders the user can write to (the writable set whoami returns). */
folders?: string[];
/** Folders the user can read; read-only folders = folders_read \ folders. */
folders_read?: string[];
}
export interface GlobalEvalResult {
success: boolean;
state: GlobalDraftState;
@@ -61,6 +79,7 @@ export interface GlobalEvalResult {
export interface GlobalEvalOptions {
workspaceFixtures?: BenchmarkWorkspaceRunnables;
liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
user?: GlobalUserFixture;
model?: string;
maxIterations?: number;
provider?: AIProvider;
@@ -86,9 +105,11 @@ export async function runGlobalEval(
const model = options.model ?? "claude-haiku-4-5-20251001";
const injectActiveEditorContext =
process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1";
// Pass the seeded identity straight to the prompt builder rather than mutating
// the process-global `userStore`, so concurrent cases never race on it.
const rawResult = await runEval({
userPrompt,
systemMessage: prepareGlobalSystemMessage(),
systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }),
userMessage: prepareGlobalUserMessage(
userPrompt,
[],
@@ -193,25 +214,28 @@ function clearLiveEditorDrafts(
}
function getGlobalEvalTools(): ProductionTool<{}>[] {
return (globalTools as ProductionTool<{}>[]).map((tool) => {
if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
return tool;
}
const disableSearchApp = process.env[DISABLE_SEARCH_APP_ENV] === "1";
return (globalTools as ProductionTool<{}>[])
.filter((tool) => !(disableSearchApp && tool.def.function.name === "search_app"))
.map((tool) => {
if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) {
return tool;
}
return {
...tool,
requiresConfirmation: false,
validateBeforeConfirmation: undefined,
fn: async () =>
JSON.stringify(
{
success: false,
error:
"This mutating workspace tool is disabled during ai_evals global mode.",
},
null,
2,
),
};
});
return {
...tool,
requiresConfirmation: false,
validateBeforeConfirmation: undefined,
fn: async () =>
JSON.stringify(
{
success: false,
error:
"This mutating workspace tool is disabled during ai_evals global mode.",
},
null,
2,
),
};
});
}
+69 -1
View File
@@ -1,5 +1,12 @@
import { randomUUID } from 'node:crypto'
import type { CompletedJob, Flow, Job, Script } from '../../../frontend/src/lib/gen'
import type {
AppWithLastVersion,
CompletedJob,
Flow,
Job,
ListableApp,
Script
} from '../../../frontend/src/lib/gen'
import type {
DataTableTables,
DataTableTableSchema,
@@ -33,6 +40,18 @@ export interface BenchmarkWorkspaceFlow {
value: Flow['value']
}
export interface BenchmarkWorkspaceApp {
path: string
summary: string
value: {
files: Record<string, string>
runnables: Record<string, unknown>
data?: unknown
policy?: unknown
custom_path?: unknown
}
}
export interface BenchmarkWorkspaceJob {
/** Stable id so a case prompt can reference a specific run (e.g. for get_job_logs). */
id?: string
@@ -47,6 +66,7 @@ export interface BenchmarkWorkspaceJob {
export interface BenchmarkWorkspaceRunnables {
scripts?: BenchmarkWorkspaceScript[]
flows?: BenchmarkWorkspaceFlow[]
apps?: BenchmarkWorkspaceApp[]
datatables?: BenchmarkDatatableSeed[]
jobs?: BenchmarkWorkspaceJob[]
}
@@ -161,6 +181,22 @@ export function getBenchmarkFlowByPath(workspace: string, path: string): Flow |
return flow ? buildBenchmarkFlow(flow) : null
}
export function listBenchmarkApps(workspace: string): ListableApp[] | null {
const runnables = benchmarkWorkspaceRunnables.get(workspace)
if (!runnables) {
return null
}
return (runnables.apps ?? []).map(buildBenchmarkListableApp)
}
export function getBenchmarkAppByPath(workspace: string, path: string): AppWithLastVersion | null {
const app = benchmarkWorkspaceRunnables
.get(workspace)
?.apps?.find((entry) => entry.path === path)
return app ? buildBenchmarkApp(app) : null
}
export function createBenchmarkCompletedJob(input: {
workspace: string
jobKind: CompletedJob['job_kind']
@@ -604,3 +640,35 @@ function buildBenchmarkFlow(flow: BenchmarkWorkspaceFlow): Flow {
extra_perms: {}
} as Flow
}
function buildBenchmarkListableApp(app: BenchmarkWorkspaceApp): ListableApp {
return {
id: 0,
workspace_id: 'benchmark',
path: app.path,
summary: app.summary,
version: 1,
extra_perms: {},
edited_at: BENCHMARK_TIMESTAMP,
execution_mode: 'viewer',
raw_app: true
}
}
function buildBenchmarkApp(app: BenchmarkWorkspaceApp): AppWithLastVersion {
return {
id: 0,
workspace_id: 'benchmark',
path: app.path,
summary: app.summary,
versions: [1],
created_by: 'benchmark',
created_at: BENCHMARK_TIMESTAMP,
value: app.value,
policy: (app.value.policy ?? {}) as AppWithLastVersion['policy'],
execution_mode: 'viewer',
extra_perms: {},
custom_path: app.value.custom_path as string | undefined,
raw_app: true
}
}
+5
View File
@@ -24,6 +24,8 @@ export async function runFrontendBenchmarkAdapter(input: {
runs: number;
model?: string;
verbose?: boolean;
skipJudge?: boolean;
executionOnly?: boolean;
backendValidation?: string;
}): Promise<BenchmarkRunResult> {
const tempDir = await mkdtemp(
@@ -40,6 +42,9 @@ export async function runFrontendBenchmarkAdapter(input: {
WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "",
WMILL_FRONTEND_AI_EVAL_PROGRESS: "1",
WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0",
WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE:
input.skipJudge || input.executionOnly ? "1" : "0",
WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY: input.executionOnly ? "1" : "0",
WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "",
};
@@ -33,6 +33,7 @@ vi.mock('$lib/components/vscode', () => ({}))
vi.mock('$lib/gen', async () => {
const actual = await vi.importActual<any>('$lib/gen')
const {
getBenchmarkAppByPath,
getBenchmarkCompletedJob,
getBenchmarkCompletedJobResultMaybe,
getBenchmarkDatatableSchema,
@@ -42,6 +43,7 @@ vi.mock('$lib/gen', async () => {
getBenchmarkScriptByHash,
getBenchmarkScriptByPath,
hasBenchmarkWorkspace,
listBenchmarkApps,
listBenchmarkDatatables,
listBenchmarkDrafts,
listBenchmarkFlows,
@@ -299,12 +301,20 @@ vi.mock('$lib/gen', async () => {
}),
AppService: wrapService(actual.AppService, {
existsApp: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace) ? false : actual.AppService.existsApp(data),
hasBenchmarkWorkspace(data.workspace)
? Boolean(getBenchmarkAppByPath(data.workspace, data.path))
: actual.AppService.existsApp(data),
listApps: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace) ? [] : actual.AppService.listApps(data),
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkApps(data.workspace) ?? [])
: actual.AppService.listApps(data),
getAppByPath: async (data: { workspace: string; path: string }) => {
if (hasBenchmarkWorkspace(data.workspace)) {
throw new Error(`App "${data.path}" not found in benchmark workspace`)
const app = getBenchmarkAppByPath(data.workspace, data.path)
if (!app) {
throw new Error(`App "${data.path}" not found in benchmark workspace`)
}
return app
}
return actual.AppService.getAppByPath(data)
}
+279 -1
View File
@@ -4,7 +4,7 @@
It should take a string `name` input and return `Hello, ${name}!`.
Leave it as an AI draft only; do not deploy or save it.
runtime:
maxTurns: 8
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
@@ -943,3 +943,281 @@
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
# --- Raw app on a large project (context-usage benchmark) ---
# These cases run against the deliberately large `analytics_dashboard` raw-app
# fixture (~20 frontend files incl. a 5k-line data module, plus backend runnables).
# They exist to measure how much context the global chat consumes when working in a
# big raw app: test29 is a read-heavy debugging hunt, test30 is a small edit baseline.
# tokenUsage is recorded per run, so the same cases re-run after a read-tool change
# (the read_app_file cap + offset/limit paging) quantify the optimization. skipJudge:
# the judge only sees the drafts artifact and cannot run the app, so we validate
# deterministically.
- id: global-test29-raw-app-debug-large
prompt: |-
The analytics dashboard app at `f/evals/global/analytics_dashboard` has a bug:
the Revenue Summary tile shows a total that is lower than the per-order line
totals and the per-region breakdown. Track down what is computing revenue
incorrectly and fix it. Keep the change as an AI draft only; do not deploy or
save it.
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
runtime:
maxTurns: 20
validate:
draftCountExactly: 1
requiredDrafts:
- type: app
path: f/evals/global/analytics_dashboard
valueIncludes:
- "return order.unitPrice * order.quantity"
toolExpect:
requiredToolsAnyOf:
# Inspecting the app's files is satisfied by either reading them directly
# or grepping for the revenue calculation.
- [read_app_file, search_app]
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- inspects the dashboard app's files to locate the revenue calculation
- fixes the per-order revenue so it multiplies unit price by quantity
- leaves the result as an AI draft and does not deploy or save it
- id: global-test30-raw-app-small-edit-large
prompt: |-
In the dashboard app at `f/evals/global/analytics_dashboard`, change the main
page heading from "Operations Console" to "Revenue Overview". Leave everything
else unchanged. Keep it as an AI draft only; do not deploy or save it.
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
runtime:
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
- type: app
path: f/evals/global/analytics_dashboard
valueIncludes:
- "Revenue Overview"
toolExpect:
requiredToolsAnyOf:
# Inspecting the app's files is satisfied by reading them directly or
# grepping for the target with search_app.
- [read_app_file, search_app]
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- renames the main page heading to Revenue Overview
- does not change other dashboard behavior
- leaves the result as an AI draft only
- id: global-test31-raw-app-debug-inspect-data
prompt: |-
The raw app dashboard at `f/evals/global/analytics_dashboard` is reporting
revenue totals that look too low. Inspect the app's files — both the sample
order data module and the revenue calculation — to work out whether the bug is
in the data or in the calculation, then fix the actual cause. Keep the change as
an AI draft only; do not deploy or save it.
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
runtime:
maxTurns: 22
validate:
draftCountExactly: 1
requiredDrafts:
- type: app
path: f/evals/global/analytics_dashboard
valueIncludes:
- "return order.unitPrice * order.quantity"
toolExpect:
requiredToolsAnyOf:
# Inspecting the app's files is satisfied by reading them directly or
# grepping for the target with search_app.
- [read_app_file, search_app]
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- inspects both the sample order data module and the revenue aggregation logic
- identifies the per-order revenue bug and fixes it to multiply unit price by quantity
- leaves the result as an AI draft only
- id: global-test32-raw-app-cross-file-consistency
prompt: |-
The raw app dashboard at `f/evals/global/analytics_dashboard` shows revenue
totals that disagree between the Revenue Summary tile, the orders table, and the
regional breakdown. Investigate how each of those computes revenue, work out
which calculation is wrong, and fix it. Keep the change as an AI draft only; do
not deploy or save it.
# Cross-file investigation: forces the model through several overlapping files
# (the summary's aggregation helper, the orders table, the regional breakdown) —
# a realistic multi-file read load that exercises the read_app_file cap.
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
runtime:
maxTurns: 24
validate:
draftCountExactly: 1
requiredDrafts:
- type: app
path: f/evals/global/analytics_dashboard
valueIncludes:
- "return order.unitPrice * order.quantity"
toolExpect:
requiredToolsAnyOf:
# Inspecting the app's files is satisfied by reading them directly or
# grepping for the target with search_app.
- [read_app_file, search_app]
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- inspects the revenue calculation behind the summary tile, the orders table, and the regional breakdown
- identifies that the per-order revenue helper omits quantity and fixes it to multiply unit price by quantity
- leaves the result as an AI draft only
- id: global-test33-raw-app-rename-across-files
prompt: |-
In the dashboard app at `f/evals/global/analytics_dashboard`, rename the
`formatCurrency` helper to `formatMoney` everywhere it is defined, imported, and
called. Leave the separate `formatCurrencyPrecise` helper exactly as it is. Keep
the change as an AI draft only; do not deploy or save it.
# Find-all-usages rename: formatCurrency is defined once and called in 6 places
# spread across 4 component files (and imported in 4). Locating every usage is the
# exact task search_app is meant to make cheap — one grep returns all file:line
# rows instead of reading each component whole. valueExcludes "formatCurrency("
# asserts the definition and all call sites were renamed while tolerating the
# preserved formatCurrencyPrecise (which is never followed by "(").
initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard
runtime:
maxTurns: 22
validate:
draftCountExactly: 1
requiredDrafts:
- type: app
path: f/evals/global/analytics_dashboard
valueIncludes:
- "export function formatMoney"
- "formatMoney("
valueExcludes:
- "formatCurrency("
toolExpect:
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- renames the formatCurrency definition, imports, and all call sites to formatMoney
- leaves the unrelated formatCurrencyPrecise helper unchanged
- leaves the result as an AI draft only
# --- Path selection (u/<user> vs f/<folder>) ---
# These cases assert how the assistant picks a workspace path when the user gives
# none: a bare name defaults to the personal scope `u/<user>/`, an existing folder
# whose purpose matches is used, a non-admin targets a writable folder and never a
# read-only one, and shared intent with no matching folder asks rather than invents.
# Each depends on the seeded `user` fixture (username / is_admin / folders /
# folders_read) so the prompt's folder guidance and `u/{username}` are well-formed.
- id: global-path1-bare-name-defaults-to-personal
prompt: |-
Stage a quick draft helper that takes a string and returns it trimmed of
leading and trailing whitespace. Just keep it as a draft.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json
runtime:
maxTurns: 8
validate:
draftCountExactly: 1
requiredDrafts:
- type: script
pathStartsWith: u/admin/
toolExpect:
requiredToolsUsed:
- write_script
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- stages a single script draft for a trim helper
- defaults the path to the current user's personal scope (u/admin/...) since no path or folder was given
- does not invent an f/<folder> path
- leaves the result as a draft only
- id: global-path2-match-existing-folder
prompt: |-
Draft a flow for the marketing team's weekly campaign report.
It should take a week number and return a short summary string.
Keep it as a draft only.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
runtime:
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
- type: flow
pathStartsWith: f/marketing/
toolExpect:
requiredToolsUsed:
- write_flow
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- drafts a flow for the marketing campaign report
- places it in the existing marketing folder (f/marketing/...) rather than the personal scope or an invented folder
- leaves the result as a draft only
- id: global-path3-shared-intent-unknown-folder-asks
prompt: |-
Put together a draft onboarding checklist flow for the People Ops team to use
when a new hire joins. Keep it as a draft.
initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json
runtime:
maxTurns: 6
validate:
draftCountExactly: 0
toolExpect:
requiredToolsUsed:
- askUserQuestion
forbiddenToolsUsed:
- write_flow
- write_script
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- recognizes the request implies shared/team work but names no existing folder (none of marketing/data_engineering/shared_utils fit People Ops)
- asks which folder to use instead of guessing or inventing one
- does not create a draft until the folder is known
- id: global-path4-nonadmin-avoids-readonly-folder
prompt: |-
Draft a small flow that returns today's date as an ISO string, and stage it in
one of our shared team folders. Keep it as a draft.
initial: ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json
runtime:
maxTurns: 10
validate:
draftCountExactly: 1
requiredDrafts:
- type: flow
pathStartsWith: f/team_a/
forbiddenDrafts:
- type: flow
pathStartsWith: f/team_b/
toolExpect:
requiredToolsUsed:
- write_flow
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
skipJudge: true
judgeChecklist:
- drafts a flow that returns the current date as an ISO string
- places it in team_a (writable by this non-admin user) and not team_b (read-only)
- leaves the result as a draft only
+25 -3
View File
@@ -25,7 +25,9 @@ import {
import { runSuite } from "../core/runSuite";
import { EVAL_MODES, type EvalMode } from "../core/types";
import { DEFAULT_JUDGE_MODEL } from "../core/judge";
import { createCliModeRunner } from "../modes/cli";
// createCliModeRunner is imported lazily in runCliBenchmark so the non-cli modes
// (global/flow/script/app) don't pull in the wmill CLI toolchain and its JSR deps
// (e.g. @cliffy/*) just to load this entrypoint.
import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime";
import { resolveWindmillBackendSettings } from "../core/windmillBackendSettings";
import { assertWindmillBackendReachable } from "../adapters/frontend/windmillBackend";
@@ -97,6 +99,11 @@ async function main() {
"comma-separated model aliases to run sequentially",
)
.option("--verbose", "stream assistant output during frontend runs")
.option("--skip-judge", "skip LLM judge scoring for this run")
.option(
"--execution-only",
"only require the model/proxy/frontend loop to complete",
)
.option(
"--record",
"append a compact summary line to ai_evals/history/<mode>.jsonl",
@@ -115,6 +122,8 @@ async function main() {
model?: string;
models?: string;
verbose?: boolean;
skipJudge?: boolean;
executionOnly?: boolean;
record?: boolean;
backendValidation?: string;
},
@@ -127,6 +136,8 @@ async function main() {
model: options.model,
models: options.models,
verbose: options.verbose ?? false,
skipJudge: options.skipJudge ?? false,
executionOnly: options.executionOnly ?? false,
record: options.record ?? false,
backendValidation: options.backendValidation,
});
@@ -175,6 +186,8 @@ async function handleRun(input: {
model?: string;
models?: string;
verbose: boolean;
skipJudge: boolean;
executionOnly: boolean;
record: boolean;
backendValidation?: string;
}) {
@@ -230,6 +243,8 @@ async function handleRun(input: {
input.runs,
getCliEvalModel(model),
runModel,
input.skipJudge,
input.executionOnly,
)
: await runFrontendBenchmarkAdapter({
mode: input.mode,
@@ -237,6 +252,8 @@ async function handleRun(input: {
runs: input.runs,
model: model.id,
verbose: input.verbose,
skipJudge: input.skipJudge,
executionOnly: input.executionOnly,
backendValidation,
});
@@ -278,20 +295,25 @@ async function runCliBenchmark(
runs: number,
model: ReturnType<typeof getCliEvalModel>,
runModel: string,
skipJudge: boolean,
executionOnly: boolean,
) {
const { createCliModeRunner } = await import("../modes/cli");
const judgeModel = skipJudge || executionOnly ? null : DEFAULT_JUDGE_MODEL;
const caseResults = await runSuite({
modeRunner: createCliModeRunner(model),
cases,
runs,
runModel,
judgeModel: DEFAULT_JUDGE_MODEL,
judgeModel,
executionOnly,
});
return buildRunResult({
mode: "cli",
runs,
runModel,
judgeModel: DEFAULT_JUDGE_MODEL,
judgeModel,
caseResults,
});
}
+102
View File
@@ -0,0 +1,102 @@
import { describe, expect, it } from "bun:test";
import { runSuite } from "./runSuite";
import type { ModeRunner } from "./types";
const modeRunner: ModeRunner<undefined, undefined, { ok: boolean }> = {
mode: "global",
concurrency: 1,
loadInitial: async () => undefined,
loadExpected: async () => undefined,
run: async () => ({
success: true,
actual: { ok: true },
assistantMessageCount: 1,
toolCallCount: 0,
toolsUsed: [],
skillsInvoked: [],
tokenUsage: null,
}),
validate: () => [],
};
describe("runSuite", () => {
it("skips judge checks when the run disables judge scoring", async () => {
const [caseResult] = await runSuite({
modeRunner,
cases: [
{
id: "case-1",
prompt: "Create a draft script",
judgeChecklist: ["the output satisfies the prompt"],
},
],
runs: 1,
runModel: "model-under-test",
judgeModel: null,
});
const [attempt] = caseResult.attempts;
expect(attempt.passed).toBe(true);
expect(attempt.judgeScore).toBeNull();
expect(attempt.judgeSummary).toBeNull();
expect(attempt.checks.map((check) => check.name)).toEqual([
"run succeeded",
]);
});
it("only requires run success when execution-only is enabled", async () => {
let loadExpectedCalls = 0;
let validateCalls = 0;
let backendValidateCalls = 0;
const executionOnlyRunner: ModeRunner<
undefined,
undefined,
{ ok: boolean }
> = {
...modeRunner,
loadExpected: async () => {
loadExpectedCalls++;
return undefined;
},
validate: () => {
validateCalls++;
return [{ name: "validator failed", passed: false }];
},
backendValidate: async () => {
backendValidateCalls++;
return {
checks: [{ name: "backend validation failed", passed: false }],
};
},
};
const [caseResult] = await runSuite({
modeRunner: executionOnlyRunner,
cases: [
{
id: "case-1",
prompt: "Create a draft script",
expectedPath: "fixtures/expected.json",
toolExpect: { requiredToolsUsed: ["write_script"] },
judgeChecklist: ["the output satisfies the prompt"],
},
],
runs: 1,
runModel: "model-under-test",
judgeModel: "judge-model",
executionOnly: true,
});
const [attempt] = caseResult.attempts;
expect(attempt.passed).toBe(true);
expect(attempt.judgeScore).toBeNull();
expect(attempt.judgeSummary).toBeNull();
expect(attempt.checks.map((check) => check.name)).toEqual([
"run succeeded",
]);
expect(loadExpectedCalls).toBe(0);
expect(validateCalls).toBe(0);
expect(backendValidateCalls).toBe(0);
});
});
+36 -17
View File
@@ -15,11 +15,13 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
runs: number;
runModel: string | null;
judgeModel?: string | null;
executionOnly?: boolean;
concurrency?: number;
verbose?: boolean;
onProgress?: (event: FrontendBenchmarkProgressEvent) => void;
}): Promise<BenchmarkCaseResult[]> {
const judgeModel = input.judgeModel ?? DEFAULT_JUDGE_MODEL;
const judgeModel =
input.judgeModel === undefined ? DEFAULT_JUDGE_MODEL : input.judgeModel;
const concurrency = Math.max(1, input.concurrency ?? input.modeRunner.concurrency);
const results = new Array<BenchmarkCaseResult>(input.cases.length);
let cursor = 0;
@@ -52,6 +54,7 @@ export async function runSuite<TInitial, TExpected, TActual>(input: {
runs: input.runs,
judgeModel,
judgeThreshold: input.modeRunner.judgeThreshold ?? 80,
executionOnly: input.executionOnly ?? false,
modeRunner: input.modeRunner,
totalCases: input.cases.length,
verbose: input.verbose ?? false,
@@ -72,8 +75,9 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
caseIndex: number;
evalCase: EvalCase;
runs: number;
judgeModel: string;
judgeModel: string | null;
judgeThreshold: number;
executionOnly: boolean;
modeRunner: ModeRunner<TInitial, TExpected, TActual>;
totalCases: number;
verbose: boolean;
@@ -99,7 +103,9 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
try {
const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath);
const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath);
const expected = input.executionOnly
? undefined
: await input.modeRunner.loadExpected(input.evalCase.expectedPath);
const run = await input.modeRunner.run(input.evalCase.prompt, initial, {
evalCase: input.evalCase,
caseId: input.evalCase.id,
@@ -162,22 +168,30 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
});
const checks: BenchmarkCheck[] = [
buildCheck("run succeeded", run.success, run.error),
...input.modeRunner.validate({
evalCase: input.evalCase,
prompt: input.evalCase.prompt,
initial,
expected,
actual: run.actual,
run,
}),
...validateToolExpectations({
run,
toolExpect: input.evalCase.toolExpect,
}),
];
if (!input.executionOnly) {
checks.push(
...input.modeRunner.validate({
evalCase: input.evalCase,
prompt: input.evalCase.prompt,
initial,
expected,
actual: run.actual,
run,
}),
...validateToolExpectations({
run,
toolExpect: input.evalCase.toolExpect,
})
);
}
const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? [];
if (run.success && input.modeRunner.backendValidate) {
if (
run.success &&
!input.executionOnly &&
input.modeRunner.backendValidate
) {
try {
const backendValidation = await input.modeRunner.backendValidate({
evalCase: input.evalCase,
@@ -218,7 +232,12 @@ async function runCaseAttempts<TInitial, TExpected, TActual>(input: {
let judgeScore: number | null = null;
let judgeSummary: string | null = null;
if (run.success && !input.evalCase.skipJudge) {
if (
run.success &&
!input.executionOnly &&
input.judgeModel !== null &&
!input.evalCase.skipJudge
) {
const judge = await judgeOutput({
mode: input.modeRunner.mode,
prompt: input.evalCase.prompt,
+7
View File
@@ -168,6 +168,13 @@ export interface ToolCallArgumentRule {
export interface ToolValidationSpec {
requiredToolsUsed?: string[];
/**
* Each inner array is an alternatives group: the check passes when at least
* one tool in the group was used. Use when several tools satisfy the same
* intent so a model that picks any valid path passes — e.g. inspecting an
* app's files via either `read_app_file` or `search_app`.
*/
requiredToolsAnyOf?: string[][];
forbiddenToolsUsed?: string[];
toolCallArgs?: ToolCallArgumentRule[];
}
+43
View File
@@ -245,6 +245,49 @@ describe("validateToolExpectations", () => {
'accepted substrings: insert into, update; values: "DROP TABLE orders"',
});
});
it("passes requiredToolsAnyOf when any alternative in the group is used", () => {
const checks = validateToolExpectations({
run: {
success: true,
actual: {},
assistantMessageCount: 1,
toolCallCount: 1,
toolsUsed: ["search_app", "patch_app_file"],
skillsInvoked: [],
},
toolExpect: {
requiredToolsAnyOf: [["read_app_file", "search_app"]],
},
});
expect(checks).toContainEqual({
name: "uses one of read_app_file, search_app",
passed: true,
});
});
it("fails requiredToolsAnyOf when no alternative in the group is used", () => {
const checks = validateToolExpectations({
run: {
success: true,
actual: {},
assistantMessageCount: 1,
toolCallCount: 1,
toolsUsed: ["patch_app_file"],
skillsInvoked: [],
},
toolExpect: {
requiredToolsAnyOf: [["read_app_file", "search_app"]],
},
});
expect(checks).toContainEqual({
name: "uses one of read_app_file, search_app",
passed: false,
details: "tools used: patch_app_file",
});
});
});
describe("validateGlobalState", () => {
+10
View File
@@ -169,6 +169,16 @@ export function validateToolExpectations(input: {
);
}
for (const group of expect.requiredToolsAnyOf ?? []) {
checks.push(
check(
`uses one of ${group.join(", ")}`,
group.some((toolName) => input.run.toolsUsed.includes(toolName)),
`tools used: ${input.run.toolsUsed.join(", ") || "none"}`
)
);
}
for (const toolName of expect.forbiddenToolsUsed ?? []) {
checks.push(
check(
@@ -0,0 +1,68 @@
type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled'
interface Order {
id: string
region: string
quantity: number
unitPrice: number
status: OrderStatus
placedAt: string
}
// Server-side revenue rollup. Mirrors the client aggregation but is computed
// from the authoritative mocked order book so it can be used to cross-check
// the dashboard and to back the export.
const orders: Order[] = [
{ id: 'ORD-10001', region: 'North America', quantity: 3, unitPrice: 1195, status: 'delivered', placedAt: '2024-05-02' },
{ id: 'ORD-10002', region: 'EMEA', quantity: 5, unitPrice: 880, status: 'shipped', placedAt: '2024-05-03' },
{ id: 'ORD-10003', region: 'APAC', quantity: 2, unitPrice: 640, status: 'paid', placedAt: '2024-05-05' },
{ id: 'ORD-10004', region: 'LATAM', quantity: 7, unitPrice: 315, status: 'delivered', placedAt: '2024-05-07' },
{ id: 'ORD-10005', region: 'North America', quantity: 4, unitPrice: 150, status: 'refunded', placedAt: '2024-05-09' },
{ id: 'ORD-10006', region: 'EMEA', quantity: 6, unitPrice: 220, status: 'shipped', placedAt: '2024-05-12' },
{ id: 'ORD-10007', region: 'APAC', quantity: 1, unitPrice: 980, status: 'pending', placedAt: '2024-05-15' },
{ id: 'ORD-10008', region: 'North America', quantity: 8, unitPrice: 1100, status: 'delivered', placedAt: '2024-05-18' },
{ id: 'ORD-10009', region: 'EMEA', quantity: 2, unitPrice: 860, status: 'cancelled', placedAt: '2024-05-22' },
{ id: 'ORD-10010', region: 'LATAM', quantity: 9, unitPrice: 290, status: 'paid', placedAt: '2024-05-26' }
]
const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered']
export async function main({
from,
to,
region
}: {
from: string
to: string
region: string
}): Promise<{
totalRevenue: number
netRevenue: number
totalOrders: number
averageOrderValue: number
unitsSold: number
refundedRevenue: number
currency: string
}> {
let scoped = orders.filter((order) => order.placedAt >= from && order.placedAt <= to)
if (region && region !== 'all') {
scoped = scoped.filter((order) => order.region === region)
}
const booked = scoped.filter((order) => REVENUE_STATUSES.includes(order.status))
const totalRevenue = booked.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
const unitsSold = booked.reduce((acc, order) => acc + order.quantity, 0)
const refundedRevenue = scoped
.filter((order) => order.status === 'refunded')
.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
return {
totalRevenue,
netRevenue: totalRevenue - refundedRevenue,
totalOrders: booked.length,
averageOrderValue: booked.length === 0 ? 0 : Math.round(totalRevenue / booked.length),
unitsSold,
refundedRevenue,
currency: 'USD'
}
}
@@ -0,0 +1,4 @@
{
"name": "Compute Summary",
"language": "bun"
}
@@ -0,0 +1,51 @@
// Builds a downloadable report for the current dashboard view. Returns a data
// URL the browser can open directly so the export works without object storage.
export async function main({
from,
to,
region,
format
}: {
from: string
to: string
region: string
format: 'csv' | 'json'
}): Promise<{ url: string; rows: number; filename: string }> {
const summary = {
from,
to,
region: region || 'all',
generatedAt: new Date().toISOString(),
rows: [
{ region: 'North America', revenue: 211_400, orders: 168 },
{ region: 'EMEA', revenue: 142_900, orders: 121 },
{ region: 'APAC', revenue: 86_500, orders: 78 },
{ region: 'LATAM', revenue: 41_500, orders: 45 }
]
}
const scoped =
region && region !== 'all'
? summary.rows.filter((row) => row.region === region)
: summary.rows
let body: string
let mime: string
if (format === 'csv') {
const header = 'region,revenue,orders'
const lines = scoped.map((row) => `${row.region},${row.revenue},${row.orders}`)
body = [header, ...lines].join('\n')
mime = 'text/csv'
} else {
body = JSON.stringify({ ...summary, rows: scoped }, null, 2)
mime = 'application/json'
}
const encoded = Buffer.from(body, 'utf-8').toString('base64')
const filename = `revenue-report-${from}_${to}.${format}`
return {
url: `data:${mime};base64,${encoded}`,
rows: scoped.length,
filename
}
}
@@ -0,0 +1,4 @@
{
"name": "Export Report",
"language": "bun"
}
@@ -0,0 +1,40 @@
interface MetricCardData {
id: string
label: string
value: number
unit: 'currency' | 'count' | 'percent'
delta: number
hint: string
}
// Returns the headline metric cards for the selected range and region. Values
// are mocked but internally consistent (revenue / orders ≈ avg order value).
const baseByRegion: Record<string, { revenue: number; orders: number; units: number; refunds: number }> = {
all: { revenue: 482_300, orders: 412, units: 1840, refunds: 11_900 },
'North America': { revenue: 211_400, orders: 168, units: 770, refunds: 4_200 },
EMEA: { revenue: 142_900, orders: 121, units: 560, refunds: 3_500 },
APAC: { revenue: 86_500, orders: 78, units: 340, refunds: 2_600 },
LATAM: { revenue: 41_500, orders: 45, units: 170, refunds: 1_600 }
}
export async function main({
from,
to,
region
}: {
from: string
to: string
region: string
}): Promise<{ cards: MetricCardData[]; generatedAt: string }> {
const base = baseByRegion[region] ?? baseByRegion.all
const aov = base.orders === 0 ? 0 : Math.round(base.revenue / base.orders)
const cards: MetricCardData[] = [
{ id: 'revenue', label: 'Total Revenue', value: base.revenue, unit: 'currency', delta: 0.082, hint: `Booked revenue ${from} ${to}` },
{ id: 'orders', label: 'Orders', value: base.orders, unit: 'count', delta: 0.041, hint: 'Revenue-bearing orders in range' },
{ id: 'aov', label: 'Avg Order Value', value: aov, unit: 'currency', delta: -0.013, hint: 'Total revenue / order count' },
{ id: 'units', label: 'Units Sold', value: base.units, unit: 'count', delta: 0.067, hint: 'Total units in range' },
{ id: 'refunds', label: 'Refunded', value: base.refunds, unit: 'currency', delta: -0.021, hint: 'Revenue lost to refunds' },
{ id: 'conversion', label: 'Conversion', value: 0.187, unit: 'percent', delta: 0.009, hint: 'Sessions that became orders' }
]
return { cards, generatedAt: new Date().toISOString() }
}
@@ -0,0 +1,4 @@
{
"name": "Load Metrics",
"language": "bun"
}
@@ -0,0 +1,54 @@
type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled'
interface Order {
id: string
placedAt: string
customer: string
product: string
sku: string
region: string
channel: string
rep: string
quantity: number
unitPrice: number
status: OrderStatus
}
// Mocked order book. In a real deployment this would query the orders table;
// here it returns a representative slice so the table renders in preview.
const orders: Order[] = [
{ id: 'ORD-10001', placedAt: '2024-05-02T09:14:00Z', customer: 'Contoso Ltd', product: 'Aurora Analytics Suite', sku: 'ANL-100', region: 'North America', channel: 'direct', rep: 'Dana Wills', quantity: 3, unitPrice: 1195, status: 'delivered' },
{ id: 'ORD-10002', placedAt: '2024-05-03T11:42:00Z', customer: 'Fabrikam Inc', product: 'Borealis CRM', sku: 'CRM-210', region: 'EMEA', channel: 'partner', rep: 'Lena Fischer', quantity: 5, unitPrice: 880, status: 'shipped' },
{ id: 'ORD-10003', placedAt: '2024-05-05T15:03:00Z', customer: 'Tailspin Toys', product: 'Cascade Data Pipeline', sku: 'PIPE-330', region: 'APAC', channel: 'self-serve', rep: 'Sora Tanaka', quantity: 2, unitPrice: 640, status: 'paid' },
{ id: 'ORD-10004', placedAt: '2024-05-07T08:21:00Z', customer: 'Proseware Inc', product: 'Delta Insights', sku: 'INS-440', region: 'LATAM', channel: 'marketplace', rep: 'Diego Marin', quantity: 7, unitPrice: 315, status: 'delivered' },
{ id: 'ORD-10005', placedAt: '2024-05-09T13:58:00Z', customer: 'Litware Inc', product: 'Echo Monitoring', sku: 'MON-550', region: 'North America', channel: 'direct', rep: 'Owen Pratt', quantity: 4, unitPrice: 150, status: 'refunded' },
{ id: 'ORD-10006', placedAt: '2024-05-12T10:30:00Z', customer: 'Fourth Coffee', product: 'Helix Identity', sku: 'IDN-880', region: 'EMEA', channel: 'partner', rep: 'Aisha Khan', quantity: 6, unitPrice: 220, status: 'shipped' },
{ id: 'ORD-10007', placedAt: '2024-05-15T17:11:00Z', customer: 'Coho Vineyard', product: 'Kelvin Forecasting', sku: 'FCT-202', region: 'APAC', channel: 'direct', rep: 'Priya Nair', quantity: 1, unitPrice: 980, status: 'pending' },
{ id: 'ORD-10008', placedAt: '2024-05-18T12:05:00Z', customer: 'Alpine Ski House', product: 'Nimbus Compute', sku: 'CMP-505', region: 'North America', channel: 'self-serve', rep: 'Hugo Bernard', quantity: 8, unitPrice: 1100, status: 'delivered' },
{ id: 'ORD-10009', placedAt: '2024-05-22T14:47:00Z', customer: 'Trey Research', product: 'Onyx Security', sku: 'SEC-606', region: 'EMEA', channel: 'direct', rep: 'Sven Olsen', quantity: 2, unitPrice: 860, status: 'cancelled' },
{ id: 'ORD-10010', placedAt: '2024-05-26T16:39:00Z', customer: 'Blue Yonder Airlines', product: 'Polaris Reporting', sku: 'RPT-707', region: 'LATAM', channel: 'partner', rep: 'Mateo Russo', quantity: 9, unitPrice: 290, status: 'paid' }
]
export async function main({
from,
to,
region,
status
}: {
from: string
to: string
region: string
status: string
}): Promise<{ orders: Order[]; total: number }> {
let filtered = orders.filter((order) => {
const day = order.placedAt.slice(0, 10)
return day >= from && day <= to
})
if (region && region !== 'all') {
filtered = filtered.filter((order) => order.region === region)
}
if (status && status !== 'all') {
filtered = filtered.filter((order) => order.status === status)
}
return { orders: filtered, total: filtered.length }
}
@@ -0,0 +1,4 @@
{
"name": "Load Orders",
"language": "bun"
}
@@ -0,0 +1,45 @@
import React from 'react'
import type { DateRange } from '../lib/api'
import { rangeForPreset } from '../lib/api'
import { formatDateShort } from '../lib/format'
interface DateRangePickerProps {
preset: string
range: DateRange
onPresetChange: (preset: string, range: DateRange) => void
}
const PRESETS: { id: string; label: string }[] = [
{ id: '7d', label: 'Last 7 days' },
{ id: '14d', label: 'Last 14 days' },
{ id: '30d', label: 'Last 30 days' },
{ id: 'qtd', label: 'Quarter to date' }
]
export const DateRangePicker: React.FC<DateRangePickerProps> = ({
preset,
range,
onPresetChange
}) => {
return (
<div className="flex items-center gap-2">
<select
className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
value={preset}
onChange={(event) => {
const next = event.target.value
onPresetChange(next, rangeForPreset(next))
}}
>
{PRESETS.map((item) => (
<option key={item.id} value={item.id}>
{item.label}
</option>
))}
</select>
<span className="text-xs text-gray-400">
{formatDateShort(range.from)} {formatDateShort(range.to)}
</span>
</div>
)
}
@@ -0,0 +1,28 @@
import React from 'react'
interface EmptyStateProps {
title: string
description?: string
icon?: string
action?: React.ReactNode
}
export const EmptyState: React.FC<EmptyStateProps> = ({
title,
description,
icon = '📊',
action
}) => {
return (
<div className="flex flex-col items-center justify-center rounded-lg border border-dashed border-gray-300 bg-white py-12 text-center">
<div className="text-3xl" aria-hidden>
{icon}
</div>
<h3 className="mt-3 text-sm font-semibold text-gray-700">{title}</h3>
{description ? (
<p className="mt-1 max-w-sm text-sm text-gray-500">{description}</p>
) : null}
{action ? <div className="mt-4">{action}</div> : null}
</div>
)
}
@@ -0,0 +1,51 @@
import React, { useState } from 'react'
import { requestExport } from '../lib/api'
import type { DateRange } from '../lib/api'
interface ExportButtonProps {
range: DateRange
region: string
}
export const ExportButton: React.FC<ExportButtonProps> = ({ range, region }) => {
const [busy, setBusy] = useState(false)
const [error, setError] = useState<string | null>(null)
const handleExport = async (format: 'csv' | 'json') => {
setBusy(true)
setError(null)
try {
const result = await requestExport(range, region, format)
const anchor = document.createElement('a')
anchor.href = result.url
anchor.download = `revenue-report.${format}`
anchor.click()
} catch (err) {
setError(err instanceof Error ? err.message : 'Export failed')
} finally {
setBusy(false)
}
}
return (
<div className="flex items-center gap-2">
<button
type="button"
disabled={busy}
onClick={() => handleExport('csv')}
className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm font-medium text-gray-700 hover:bg-gray-50 disabled:opacity-50"
>
{busy ? 'Exporting…' : 'Export CSV'}
</button>
<button
type="button"
disabled={busy}
onClick={() => handleExport('json')}
className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm font-medium text-gray-700 hover:bg-gray-50 disabled:opacity-50"
>
Export JSON
</button>
{error ? <span className="text-xs text-rose-600">{error}</span> : null}
</div>
)
}
@@ -0,0 +1,59 @@
import React from 'react'
import type { DateRange } from '../lib/api'
import type { OrderStatus } from '../data/seedData'
import { REGIONS, ORDER_STATUSES, STATUS_LABELS } from '../data/seedData'
import { DateRangePicker } from './DateRangePicker'
import { ExportButton } from './ExportButton'
interface FilterBarProps {
region: string
status: string
preset: string
range: DateRange
onRegionChange: (region: string) => void
onStatusChange: (status: string) => void
onPresetChange: (preset: string, range: DateRange) => void
}
export const FilterBar: React.FC<FilterBarProps> = ({
region,
status,
preset,
range,
onRegionChange,
onStatusChange,
onPresetChange
}) => {
return (
<div className="flex flex-wrap items-center justify-between gap-3 border-b border-gray-200 bg-white px-6 py-4">
<div className="flex flex-wrap items-center gap-3">
<DateRangePicker preset={preset} range={range} onPresetChange={onPresetChange} />
<select
className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
value={region}
onChange={(event) => onRegionChange(event.target.value)}
>
<option value="all">All regions</option>
{REGIONS.map((item) => (
<option key={item} value={item}>
{item}
</option>
))}
</select>
<select
className="rounded-lg border border-gray-300 bg-white px-3 py-2 text-sm text-gray-700"
value={status}
onChange={(event) => onStatusChange(event.target.value)}
>
<option value="all">All statuses</option>
{ORDER_STATUSES.map((item) => (
<option key={item} value={item}>
{STATUS_LABELS[item as OrderStatus]}
</option>
))}
</select>
</div>
<ExportButton range={range} region={region} />
</div>
)
}
@@ -0,0 +1,40 @@
import React from 'react'
import type { MetricCardData } from '../data/seedData'
import { formatCurrency, formatNumber, formatPercent, formatSignedPercent } from '../lib/format'
interface MetricCardProps {
metric: MetricCardData
loading?: boolean
}
function renderValue(metric: MetricCardData): string {
switch (metric.unit) {
case 'currency':
return formatCurrency(metric.value)
case 'percent':
return formatPercent(metric.value)
case 'count':
default:
return formatNumber(metric.value)
}
}
export const MetricCard: React.FC<MetricCardProps> = ({ metric, loading }) => {
const positive = metric.delta >= 0
return (
<div className="rounded-xl border border-gray-200 bg-white p-5 shadow-sm">
<div className="flex items-center justify-between">
<span className="text-sm font-medium text-gray-500">{metric.label}</span>
<span
className={`text-xs font-semibold ${positive ? 'text-emerald-600' : 'text-rose-600'}`}
>
{formatSignedPercent(metric.delta)}
</span>
</div>
<div className="mt-2 text-2xl font-bold text-gray-900">
{loading ? <span className="text-gray-300"></span> : renderValue(metric)}
</div>
<p className="mt-1 text-xs text-gray-400">{metric.hint}</p>
</div>
)
}
@@ -0,0 +1,18 @@
import React from 'react'
import type { MetricCardData } from '../data/seedData'
import { MetricCard } from './MetricCard'
interface MetricGridProps {
metrics: MetricCardData[]
loading?: boolean
}
export const MetricGrid: React.FC<MetricGridProps> = ({ metrics, loading }) => {
return (
<div className="grid grid-cols-1 gap-4 sm:grid-cols-2 lg:grid-cols-3">
{metrics.map((metric) => (
<MetricCard key={metric.id} metric={metric} loading={loading} />
))}
</div>
)
}
@@ -0,0 +1,117 @@
import React, { useMemo, useState } from 'react'
import type { Order } from '../data/seedData'
import { StatusBadge } from './StatusBadge'
import { EmptyState } from './EmptyState'
import { formatCurrencyPrecise, formatDate, formatNumber, truncate } from '../lib/format'
interface OrdersTableProps {
orders: Order[]
loading?: boolean
}
type SortKey = 'placedAt' | 'customer' | 'lineTotal' | 'quantity'
type SortDir = 'asc' | 'desc'
// The per-row line total a customer was charged: unit price times quantity.
function lineTotal(order: Order): number {
return order.quantity * order.unitPrice
}
export const OrdersTable: React.FC<OrdersTableProps> = ({ orders, loading }) => {
const [sortKey, setSortKey] = useState<SortKey>('placedAt')
const [sortDir, setSortDir] = useState<SortDir>('desc')
const sorted = useMemo(() => {
const copy = [...orders]
copy.sort((a, b) => {
let comparison = 0
switch (sortKey) {
case 'customer':
comparison = a.customer.localeCompare(b.customer)
break
case 'lineTotal':
comparison = lineTotal(a) - lineTotal(b)
break
case 'quantity':
comparison = a.quantity - b.quantity
break
case 'placedAt':
default:
comparison = a.placedAt.localeCompare(b.placedAt)
break
}
return sortDir === 'asc' ? comparison : -comparison
})
return copy
}, [orders, sortKey, sortDir])
const toggleSort = (key: SortKey) => {
if (key === sortKey) {
setSortDir((dir) => (dir === 'asc' ? 'desc' : 'asc'))
} else {
setSortKey(key)
setSortDir('desc')
}
}
if (!loading && orders.length === 0) {
return (
<EmptyState
title="No orders match these filters"
description="Try widening the date range or clearing the status filter."
icon="🗂️"
/>
)
}
const arrow = (key: SortKey) => (key === sortKey ? (sortDir === 'asc' ? '▲' : '▼') : '')
return (
<div className="overflow-hidden rounded-xl border border-gray-200 bg-white shadow-sm">
<table className="min-w-full divide-y divide-gray-200 text-sm">
<thead className="bg-gray-50 text-left text-xs uppercase tracking-wide text-gray-500">
<tr>
<th className="cursor-pointer px-4 py-3" onClick={() => toggleSort('placedAt')}>
Date {arrow('placedAt')}
</th>
<th className="cursor-pointer px-4 py-3" onClick={() => toggleSort('customer')}>
Customer {arrow('customer')}
</th>
<th className="px-4 py-3">Product</th>
<th className="px-4 py-3">Region</th>
<th className="cursor-pointer px-4 py-3 text-right" onClick={() => toggleSort('quantity')}>
Qty {arrow('quantity')}
</th>
<th className="px-4 py-3 text-right">Unit Price</th>
<th className="cursor-pointer px-4 py-3 text-right" onClick={() => toggleSort('lineTotal')}>
Line Total {arrow('lineTotal')}
</th>
<th className="px-4 py-3">Status</th>
</tr>
</thead>
<tbody className="divide-y divide-gray-100">
{sorted.map((order) => (
<tr key={order.id} className="hover:bg-gray-50">
<td className="px-4 py-3 text-gray-500">{formatDate(order.placedAt)}</td>
<td className="px-4 py-3 font-medium text-gray-900">
{truncate(order.customer, 24)}
</td>
<td className="px-4 py-3 text-gray-600">{order.product}</td>
<td className="px-4 py-3 text-gray-600">{order.region}</td>
<td className="px-4 py-3 text-right text-gray-600">{formatNumber(order.quantity)}</td>
<td className="px-4 py-3 text-right text-gray-600">
{formatCurrencyPrecise(order.unitPrice)}
</td>
<td className="px-4 py-3 text-right font-semibold text-gray-900">
{formatCurrencyPrecise(lineTotal(order))}
</td>
<td className="px-4 py-3">
<StatusBadge status={order.status} />
</td>
</tr>
))}
</tbody>
</table>
</div>
)
}
@@ -0,0 +1,52 @@
import React, { useMemo } from 'react'
import type { Order } from '../data/seedData'
import { breakdownByRegion } from '../lib/aggregations'
import { formatCurrency, formatNumber, formatPercent } from '../lib/format'
import { EmptyState } from './EmptyState'
interface RegionTableProps {
orders: Order[]
}
export const RegionTable: React.FC<RegionTableProps> = ({ orders }) => {
const rows = useMemo(() => breakdownByRegion(orders), [orders])
const total = useMemo(() => rows.reduce((acc, row) => acc + row.revenue, 0), [rows])
if (rows.length === 0) {
return (
<EmptyState
title="No regional revenue"
description="No revenue-bearing orders fall in the current selection."
icon="🌍"
/>
)
}
return (
<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
<h2 className="mb-4 text-lg font-semibold text-gray-900">Revenue by Region</h2>
<table className="min-w-full text-sm">
<thead className="text-left text-xs uppercase tracking-wide text-gray-500">
<tr>
<th className="py-2">Region</th>
<th className="py-2 text-right">Orders</th>
<th className="py-2 text-right">Revenue</th>
<th className="py-2 text-right">Share</th>
</tr>
</thead>
<tbody className="divide-y divide-gray-100">
{rows.map((row) => (
<tr key={row.region}>
<td className="py-2 font-medium text-gray-900">{row.region}</td>
<td className="py-2 text-right text-gray-600">{formatNumber(row.orders)}</td>
<td className="py-2 text-right text-gray-900">{formatCurrency(row.revenue)}</td>
<td className="py-2 text-right text-gray-500">
{formatPercent(total === 0 ? 0 : row.revenue / total)}
</td>
</tr>
))}
</tbody>
</table>
</section>
)
}
@@ -0,0 +1,49 @@
import React, { useMemo } from 'react'
import type { Order } from '../data/seedData'
import { dailyRevenue } from '../lib/aggregations'
import { formatCompact, formatDateShort } from '../lib/format'
import { EmptyState } from './EmptyState'
interface RevenueChartProps {
orders: Order[]
}
// Lightweight inline bar chart for daily revenue. Avoids a charting dependency
// by sizing flexed columns relative to the busiest day in the window.
export const RevenueChart: React.FC<RevenueChartProps> = ({ orders }) => {
const points = useMemo(() => dailyRevenue(orders), [orders])
const max = useMemo(() => points.reduce((acc, point) => Math.max(acc, point.revenue), 0), [points])
if (points.length === 0) {
return (
<EmptyState
title="No revenue in range"
description="Adjust the date range or filters to see daily revenue."
icon="📉"
/>
)
}
return (
<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
<h2 className="mb-4 text-lg font-semibold text-gray-900">Daily Revenue</h2>
<div className="flex h-48 items-end gap-1">
{points.map((point) => {
const heightPct = max === 0 ? 0 : Math.round((point.revenue / max) * 100)
return (
<div key={point.date} className="flex flex-1 flex-col items-center justify-end">
<div
className="w-full rounded-t bg-indigo-400"
style={{ height: `${Math.max(heightPct, 2)}%` }}
title={`${point.date}: ${formatCompact(point.revenue)}`}
/>
<span className="mt-1 truncate text-[9px] text-gray-400">
{formatDateShort(point.date)}
</span>
</div>
)
})}
</div>
</section>
)
}
@@ -0,0 +1,50 @@
import React from 'react'
export type DashboardView = 'overview' | 'orders' | 'regions' | 'products'
interface SidebarProps {
active: DashboardView
onSelect: (view: DashboardView) => void
}
const NAV_ITEMS: { id: DashboardView; label: string; icon: string }[] = [
{ id: 'overview', label: 'Overview', icon: '📈' },
{ id: 'orders', label: 'Orders', icon: '🧾' },
{ id: 'regions', label: 'Regions', icon: '🌍' },
{ id: 'products', label: 'Products', icon: '📦' }
]
export const Sidebar: React.FC<SidebarProps> = ({ active, onSelect }) => {
return (
<aside className="flex w-56 flex-col border-r border-gray-200 bg-white">
<div className="flex items-center gap-2 border-b border-gray-200 px-5 py-4">
<span className="text-xl">🪁</span>
<span className="text-sm font-bold text-gray-900">Acme Operations</span>
</div>
<nav className="flex-1 space-y-1 p-3">
{NAV_ITEMS.map((item) => {
const isActive = item.id === active
return (
<button
key={item.id}
type="button"
onClick={() => onSelect(item.id)}
className={`flex w-full items-center gap-3 rounded-lg px-3 py-2 text-left text-sm font-medium transition ${
isActive
? 'bg-indigo-50 text-indigo-700'
: 'text-gray-600 hover:bg-gray-50'
}`}
>
<span aria-hidden>{item.icon}</span>
{item.label}
</button>
)
})}
</nav>
<div className="border-t border-gray-200 p-4 text-xs text-gray-400">
Analytics workspace
<div className="mt-1 font-mono text-[10px] text-gray-300">v2.4.0</div>
</div>
</aside>
)
}
@@ -0,0 +1,26 @@
import React from 'react'
import type { OrderStatus } from '../data/seedData'
import { STATUS_LABELS } from '../data/seedData'
interface StatusBadgeProps {
status: OrderStatus
}
const STATUS_STYLES: Record<OrderStatus, string> = {
paid: 'bg-blue-100 text-blue-700',
shipped: 'bg-indigo-100 text-indigo-700',
delivered: 'bg-emerald-100 text-emerald-700',
pending: 'bg-amber-100 text-amber-700',
refunded: 'bg-rose-100 text-rose-700',
cancelled: 'bg-gray-200 text-gray-600'
}
export const StatusBadge: React.FC<StatusBadgeProps> = ({ status }) => {
return (
<span
className={`inline-flex items-center rounded-full px-2.5 py-0.5 text-xs font-medium ${STATUS_STYLES[status]}`}
>
{STATUS_LABELS[status]}
</span>
)
}
@@ -0,0 +1,51 @@
import React, { useMemo } from 'react'
import type { Order } from '../data/seedData'
import { summarizeRevenue } from '../lib/aggregations'
import { formatCurrency, formatCurrencyPrecise, formatNumber } from '../lib/format'
interface SummaryPanelProps {
orders: Order[]
loading?: boolean
}
// Headline revenue panel. It re-aggregates the orders client-side via
// summarizeRevenue so the totals stay in sync with whatever filter the user
// has applied, without waiting for another backend round trip.
export const SummaryPanel: React.FC<SummaryPanelProps> = ({ orders, loading }) => {
const summary = useMemo(() => summarizeRevenue(orders), [orders])
const tiles = [
{ label: 'Total Revenue', value: formatCurrency(summary.totalRevenue), emphasis: true },
{ label: 'Net Revenue', value: formatCurrency(summary.netRevenue) },
{ label: 'Orders', value: formatNumber(summary.totalOrders) },
{ label: 'Avg Order Value', value: formatCurrencyPrecise(summary.averageOrderValue) },
{ label: 'Units Sold', value: formatNumber(summary.unitsSold) },
{ label: 'Refunded', value: formatCurrency(summary.refundedRevenue) }
]
return (
<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
<div className="mb-4 flex items-center justify-between">
<h2 className="text-lg font-semibold text-gray-900">Revenue Summary</h2>
{loading ? <span className="text-xs text-gray-400">Refreshing</span> : null}
</div>
<div className="grid grid-cols-2 gap-4 md:grid-cols-3">
{tiles.map((tile) => (
<div
key={tile.label}
className={`rounded-lg p-4 ${tile.emphasis ? 'bg-indigo-50' : 'bg-gray-50'}`}
>
<div className="text-xs font-medium uppercase tracking-wide text-gray-500">
{tile.label}
</div>
<div
className={`mt-1 font-bold ${tile.emphasis ? 'text-2xl text-indigo-700' : 'text-xl text-gray-900'}`}
>
{tile.value}
</div>
</div>
))}
</div>
</section>
)
}
@@ -0,0 +1,55 @@
import React, { useMemo } from 'react'
import type { Order } from '../data/seedData'
import { topProducts } from '../lib/aggregations'
import { formatCurrency } from '../lib/format'
import { EmptyState } from './EmptyState'
interface TopProductsProps {
orders: Order[]
limit?: number
}
export const TopProducts: React.FC<TopProductsProps> = ({ orders, limit = 5 }) => {
const products = useMemo(() => topProducts(orders, limit), [orders, limit])
const max = useMemo(
() => products.reduce((acc, item) => Math.max(acc, item.revenue), 0),
[products]
)
if (products.length === 0) {
return (
<EmptyState
title="No product revenue"
description="No revenue-bearing orders to rank by product."
icon="📦"
/>
)
}
return (
<section className="rounded-xl border border-gray-200 bg-white p-6 shadow-sm">
<h2 className="mb-4 text-lg font-semibold text-gray-900">Top Products</h2>
<ul className="space-y-3">
{products.map((item, index) => {
const widthPct = max === 0 ? 0 : Math.round((item.revenue / max) * 100)
return (
<li key={item.product}>
<div className="flex items-center justify-between text-sm">
<span className="font-medium text-gray-800">
{index + 1}. {item.product}
</span>
<span className="text-gray-600">{formatCurrency(item.revenue)}</span>
</div>
<div className="mt-1 h-2 w-full overflow-hidden rounded-full bg-gray-100">
<div
className="h-full rounded-full bg-emerald-400"
style={{ width: `${Math.max(widthPct, 2)}%` }}
/>
</div>
</li>
)
})}
</ul>
</section>
)
}
@@ -0,0 +1,164 @@
import React, { useEffect, useMemo, useState } from 'react'
import { Sidebar, type DashboardView } from './components/Sidebar'
import { FilterBar } from './components/FilterBar'
import { MetricGrid } from './components/MetricGrid'
import { SummaryPanel } from './components/SummaryPanel'
import { RevenueChart } from './components/RevenueChart'
import { OrdersTable } from './components/OrdersTable'
import { RegionTable } from './components/RegionTable'
import { TopProducts } from './components/TopProducts'
import { EmptyState } from './components/EmptyState'
import { fetchMetrics, fetchOrders, rangeForPreset, type DateRange } from './lib/api'
import {
seedOrders,
seedMetricCards,
ordersInRange,
ordersForRegion,
type Order,
type MetricCardData
} from './data/seedData'
const App = () => {
const [view, setView] = useState<DashboardView>('overview')
const [preset, setPreset] = useState('30d')
const [range, setRange] = useState<DateRange>(rangeForPreset('30d'))
const [region, setRegion] = useState('all')
const [status, setStatus] = useState('all')
const [metrics, setMetrics] = useState<MetricCardData[]>(seedMetricCards)
const [orders, setOrders] = useState<Order[]>(seedOrders)
const [loadingMetrics, setLoadingMetrics] = useState(true)
const [loadingOrders, setLoadingOrders] = useState(true)
const [errored, setErrored] = useState(false)
useEffect(() => {
let cancelled = false
setLoadingMetrics(true)
fetchMetrics(range, region)
.then((result) => {
if (!cancelled) {
setMetrics(result.cards)
}
})
.catch(() => {
if (!cancelled) {
setMetrics(seedMetricCards)
}
})
.finally(() => {
if (!cancelled) {
setLoadingMetrics(false)
}
})
return () => {
cancelled = true
}
}, [range, region])
useEffect(() => {
let cancelled = false
setLoadingOrders(true)
setErrored(false)
fetchOrders(range, region, status)
.then((result) => {
if (!cancelled) {
setOrders(result.orders)
}
})
.catch(() => {
if (!cancelled) {
// Fall back to the bundled seed data so the dashboard still renders.
const scoped = ordersForRegion(
ordersInRange(seedOrders, range.from, range.to),
region
).filter((order) => status === 'all' || order.status === status)
setOrders(scoped)
setErrored(true)
}
})
.finally(() => {
if (!cancelled) {
setLoadingOrders(false)
}
})
return () => {
cancelled = true
}
}, [range, region, status])
const handlePresetChange = (nextPreset: string, nextRange: DateRange) => {
setPreset(nextPreset)
setRange(nextRange)
}
// Orders that drive the summary/chart panels — the table applies the status
// filter itself, so the panels see the same range/region scoped orders.
const scopedOrders = useMemo(() => orders, [orders])
const renderView = () => {
switch (view) {
case 'orders':
return <OrdersTable orders={scopedOrders} loading={loadingOrders} />
case 'regions':
return <RegionTable orders={scopedOrders} />
case 'products':
return <TopProducts orders={scopedOrders} limit={8} />
case 'overview':
default:
return (
<div className="space-y-6">
<MetricGrid metrics={metrics} loading={loadingMetrics} />
<SummaryPanel orders={scopedOrders} loading={loadingOrders} />
<div className="grid grid-cols-1 gap-6 lg:grid-cols-2">
<RevenueChart orders={scopedOrders} />
<TopProducts orders={scopedOrders} />
</div>
<RegionTable orders={scopedOrders} />
</div>
)
}
}
return (
<div className="flex h-screen bg-gray-100 text-gray-900">
<Sidebar active={view} onSelect={setView} />
<div className="flex flex-1 flex-col overflow-hidden">
<header className="border-b border-gray-200 bg-white px-6 py-5">
<p className="text-xs font-semibold uppercase tracking-wide text-indigo-500">
Acme Inc
</p>
<h1 className="text-2xl font-bold text-gray-900">Operations Console</h1>
<p className="mt-1 text-sm text-gray-500">
Revenue, orders, and regional performance at a glance.
</p>
</header>
<FilterBar
region={region}
status={status}
preset={preset}
range={range}
onRegionChange={setRegion}
onStatusChange={setStatus}
onPresetChange={handlePresetChange}
/>
<main className="flex-1 overflow-auto p-6">
{errored ? (
<div className="mb-4 rounded-lg border border-amber-200 bg-amber-50 px-4 py-2 text-sm text-amber-700">
Showing locally bundled data the live feed is unavailable.
</div>
) : null}
{scopedOrders.length === 0 && !loadingOrders ? (
<EmptyState
title="Nothing to show yet"
description="No data for the selected range, region, and status."
/>
) : (
renderView()
)}
</main>
</div>
</div>
)
}
export default App
@@ -0,0 +1,149 @@
// Aggregation helpers that turn raw order/metric rows into the numbers the
// dashboard renders. These run client-side after the backend returns rows so
// the UI can re-aggregate instantly when filters change without a round trip.
import type { Order, OrderStatus } from '../data/seedData'
export interface RevenueSummary {
totalRevenue: number
totalOrders: number
averageOrderValue: number
unitsSold: number
refundedRevenue: number
netRevenue: number
}
export interface StatusBreakdown {
status: OrderStatus
orders: number
revenue: number
}
export interface RegionBreakdown {
region: string
orders: number
revenue: number
}
export interface DailyPoint {
date: string
revenue: number
orders: number
}
// Revenue for a single line item. An order's revenue is the unit price times
// the number of units purchased — never the unit price alone.
export function orderRevenue(order: Order): number {
return order.unitPrice
}
// The statuses that count toward realized (booked) revenue. Refunded and
// cancelled orders are excluded from the headline revenue total.
const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered']
export function isRevenueStatus(status: OrderStatus): boolean {
return REVENUE_STATUSES.includes(status)
}
export function sumRevenue(orders: Order[]): number {
return orders
.filter((order) => isRevenueStatus(order.status))
.reduce((acc, order) => acc + orderRevenue(order), 0)
}
export function sumUnits(orders: Order[]): number {
return orders
.filter((order) => isRevenueStatus(order.status))
.reduce((acc, order) => acc + order.quantity, 0)
}
export function sumRefundedRevenue(orders: Order[]): number {
return orders
.filter((order) => order.status === 'refunded')
.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0)
}
export function summarizeRevenue(orders: Order[]): RevenueSummary {
const revenueOrders = orders.filter((order) => isRevenueStatus(order.status))
const totalRevenue = sumRevenue(orders)
const unitsSold = sumUnits(orders)
const refundedRevenue = sumRefundedRevenue(orders)
const totalOrders = revenueOrders.length
return {
totalRevenue,
totalOrders,
averageOrderValue: totalOrders === 0 ? 0 : totalRevenue / totalOrders,
unitsSold,
refundedRevenue,
netRevenue: totalRevenue - refundedRevenue
}
}
export function breakdownByStatus(orders: Order[]): StatusBreakdown[] {
const map = new Map<OrderStatus, StatusBreakdown>()
for (const order of orders) {
const existing = map.get(order.status) ?? {
status: order.status,
orders: 0,
revenue: 0
}
existing.orders += 1
existing.revenue += order.unitPrice * order.quantity
map.set(order.status, existing)
}
return [...map.values()].sort((a, b) => b.revenue - a.revenue)
}
export function breakdownByRegion(orders: Order[]): RegionBreakdown[] {
const map = new Map<string, RegionBreakdown>()
for (const order of orders) {
if (!isRevenueStatus(order.status)) {
continue
}
const existing = map.get(order.region) ?? {
region: order.region,
orders: 0,
revenue: 0
}
existing.orders += 1
existing.revenue += order.unitPrice * order.quantity
map.set(order.region, existing)
}
return [...map.values()].sort((a, b) => b.revenue - a.revenue)
}
export function dailyRevenue(orders: Order[]): DailyPoint[] {
const map = new Map<string, DailyPoint>()
for (const order of orders) {
if (!isRevenueStatus(order.status)) {
continue
}
const day = order.placedAt.slice(0, 10)
const existing = map.get(day) ?? { date: day, revenue: 0, orders: 0 }
existing.revenue += order.unitPrice * order.quantity
existing.orders += 1
map.set(day, existing)
}
return [...map.values()].sort((a, b) => a.date.localeCompare(b.date))
}
export function topProducts(orders: Order[], limit: number = 5): { product: string; revenue: number }[] {
const map = new Map<string, number>()
for (const order of orders) {
if (!isRevenueStatus(order.status)) {
continue
}
map.set(order.product, (map.get(order.product) ?? 0) + order.unitPrice * order.quantity)
}
return [...map.entries()]
.map(([product, revenue]) => ({ product, revenue }))
.sort((a, b) => b.revenue - a.revenue)
.slice(0, limit)
}
export function growthRatio(current: number, previous: number): number {
if (previous === 0) {
return current === 0 ? 0 : 1
}
return (current - previous) / previous
}
@@ -0,0 +1,79 @@
// Thin wrappers around the app's backend runnables. Centralizing the calls
// here keeps the components free of `backend.*` plumbing and gives one place to
// normalize the request/response shapes.
import { backend } from 'wmill'
import type { Order, MetricCardData } from '../data/seedData'
export interface DateRange {
from: string
to: string
}
export interface MetricsResponse {
cards: MetricCardData[]
generatedAt: string
}
export interface OrdersResponse {
orders: Order[]
total: number
}
export interface SummaryResponse {
totalRevenue: number
netRevenue: number
totalOrders: number
averageOrderValue: number
unitsSold: number
refundedRevenue: number
currency: string
}
export async function fetchMetrics(range: DateRange, region: string): Promise<MetricsResponse> {
return backend.loadMetrics({ from: range.from, to: range.to, region })
}
export async function fetchOrders(
range: DateRange,
region: string,
status: string
): Promise<OrdersResponse> {
return backend.loadOrders({
from: range.from,
to: range.to,
region,
status
})
}
export async function fetchSummary(range: DateRange, region: string): Promise<SummaryResponse> {
return backend.computeSummary({ from: range.from, to: range.to, region })
}
export async function requestExport(
range: DateRange,
region: string,
format: 'csv' | 'json'
): Promise<{ url: string; rows: number }> {
return backend.exportReport({ from: range.from, to: range.to, region, format })
}
export function defaultRange(): DateRange {
return { from: '2024-05-01', to: '2024-05-31' }
}
export function rangeForPreset(preset: string): DateRange {
switch (preset) {
case '7d':
return { from: '2024-05-25', to: '2024-05-31' }
case '14d':
return { from: '2024-05-18', to: '2024-05-31' }
case '30d':
return { from: '2024-05-01', to: '2024-05-31' }
case 'qtd':
return { from: '2024-04-01', to: '2024-05-31' }
default:
return defaultRange()
}
}
@@ -0,0 +1,91 @@
// Presentation-layer formatting helpers shared across the dashboard.
// Pure functions only — no React, no data fetching.
export function formatCurrency(amount: number, currency: string = 'USD'): string {
if (!Number.isFinite(amount)) {
return '—'
}
return new Intl.NumberFormat('en-US', {
style: 'currency',
currency,
maximumFractionDigits: 0
}).format(amount)
}
export function formatCurrencyPrecise(amount: number, currency: string = 'USD'): string {
if (!Number.isFinite(amount)) {
return '—'
}
return new Intl.NumberFormat('en-US', {
style: 'currency',
currency,
minimumFractionDigits: 2,
maximumFractionDigits: 2
}).format(amount)
}
export function formatNumber(value: number): string {
if (!Number.isFinite(value)) {
return '—'
}
return new Intl.NumberFormat('en-US').format(value)
}
export function formatCompact(value: number): string {
if (!Number.isFinite(value)) {
return '—'
}
return new Intl.NumberFormat('en-US', {
notation: 'compact',
maximumFractionDigits: 1
}).format(value)
}
export function formatPercent(ratio: number, digits: number = 1): string {
if (!Number.isFinite(ratio)) {
return '—'
}
return `${(ratio * 100).toFixed(digits)}%`
}
export function formatSignedPercent(ratio: number, digits: number = 1): string {
const sign = ratio > 0 ? '+' : ''
return `${sign}${formatPercent(ratio, digits)}`
}
export function formatDate(iso: string): string {
const date = new Date(iso)
if (Number.isNaN(date.getTime())) {
return iso
}
return date.toLocaleDateString('en-US', {
year: 'numeric',
month: 'short',
day: 'numeric'
})
}
export function formatDateShort(iso: string): string {
const date = new Date(iso)
if (Number.isNaN(date.getTime())) {
return iso
}
return date.toLocaleDateString('en-US', {
month: 'short',
day: 'numeric'
})
}
export function titleCase(value: string): string {
return value
.split(/[\s_-]+/)
.map((part) => part.charAt(0).toUpperCase() + part.slice(1).toLowerCase())
.join(' ')
}
export function truncate(value: string, max: number = 32): string {
if (value.length <= max) {
return value
}
return `${value.slice(0, max - 1)}`
}
@@ -0,0 +1,6 @@
{
"user": {
"username": "admin",
"is_admin": true
}
}
@@ -0,0 +1,8 @@
{
"user": {
"username": "admin",
"is_admin": true,
"folders": ["marketing", "data_engineering", "shared_utils"],
"folders_read": ["marketing", "data_engineering", "shared_utils"]
}
}
@@ -0,0 +1,8 @@
{
"user": {
"username": "bob",
"is_admin": false,
"folders": ["team_a"],
"folders_read": ["team_a", "team_b"]
}
}
+29 -1
View File
@@ -1,7 +1,10 @@
import { readFile } from "node:fs/promises";
import { readFile, stat } from "node:fs/promises";
import { basename } from "node:path";
import { loadAppFixtureForEval } from "../adapters/frontend/core/app/appFixtureLoader";
import {
runGlobalEval,
type GlobalLiveEditorDraftFixture,
type GlobalUserFixture,
} from "../adapters/frontend/core/global/globalEvalRunner";
import type { BenchmarkWorkspaceRunnables } from "../adapters/frontend/mockBackend";
import type { FrontendEvalModelConfig } from "../core/models";
@@ -13,6 +16,7 @@ import { getFrontendApiKey } from "./frontendCommon";
export interface GlobalInitialFixture {
workspace?: BenchmarkWorkspaceRunnables;
liveEditorDrafts?: GlobalLiveEditorDraftFixture[];
user?: GlobalUserFixture;
}
export function createGlobalModeRunner(
@@ -36,6 +40,7 @@ export function createGlobalModeRunner(
{
workspaceFixtures: initial?.workspace,
liveEditorDrafts: initial?.liveEditorDrafts,
user: initial?.user,
maxIterations: context.evalCase?.runtime?.maxTurns,
provider: modelConfig.provider,
model: modelConfig.model,
@@ -76,10 +81,33 @@ export function createGlobalModeRunner(
}
async function loadGlobalInitialFixture(path: string): Promise<GlobalInitialFixture> {
if ((await stat(path)).isDirectory()) {
const { initialFrontend, initialBackend, initialDatatables } =
await loadAppFixtureForEval(path);
const name = basename(path);
return {
workspace: {
apps: [
{
path: `f/evals/global/${name}`,
summary: name,
value: {
files: initialFrontend,
runnables: initialBackend,
data: initialDatatables,
},
},
],
},
liveEditorDrafts: [],
};
}
const parsed = JSON.parse(await readFile(path, "utf8")) as GlobalInitialFixture;
return {
workspace: parsed.workspace ?? {},
liveEditorDrafts: parsed.liveEditorDrafts ?? [],
user: parsed.user,
};
}
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT\n COUNT(*)::bigint AS \"total!\",\n COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS \"replacing!\"\n FROM ai_skill\n WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "total!",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "replacing!",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Text",
"TextArray"
]
},
"nullable": [
null,
null
]
},
"hash": "002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7"
}
@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "SELECT kind::text as \"kind!\", parent_job, runnable_path\n FROM v2_job WHERE id = $1 AND workspace_id = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "kind!",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "parent_job",
"type_info": "Uuid"
},
{
"ordinal": 2,
"name": "runnable_path",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Uuid",
"Text"
]
},
"nullable": [
null,
true,
true
]
},
"hash": "19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "WITH del AS (\n DELETE FROM asset WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = $3\n )\n INSERT INTO notify_event (channel, payload)\n SELECT 'notify_asset_producer_change', $1 WHERE $3 = 'script'",
"query": "WITH del AS (\n DELETE FROM asset WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = $3\n RETURNING usage_access_type\n )\n INSERT INTO notify_event (channel, payload)\n SELECT 'notify_asset_producer_change', $1\n WHERE $3 = 'script'\n AND EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))",
"describe": {
"columns": [],
"parameters": {
@@ -23,5 +23,5 @@
},
"nullable": []
},
"hash": "7300eb89029e0863241087fc10df7616db640054e804d4ca66958cad06008fdb"
"hash": "1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "WITH del AS (\n DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script'\n AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1)\n )\n INSERT INTO notify_event (channel, payload)\n VALUES ('notify_asset_producer_change', $1)",
"query": "WITH del AS (\n DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script'\n AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1)\n RETURNING usage_access_type\n )\n INSERT INTO notify_event (channel, payload)\n SELECT 'notify_asset_producer_change', $1\n WHERE EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))",
"describe": {
"columns": [],
"parameters": {
@@ -11,5 +11,5 @@
},
"nullable": []
},
"hash": "6a1998cb3a9a0898c0fd21c35cc41e309d7e066ee4869880c90e90dd66dc9d4d"
"hash": "1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043"
}
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM job_perms\n WHERE ctid IN (\n SELECT jp.ctid FROM job_perms jp\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)\n LIMIT 100000\n )",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab"
}
@@ -1,12 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "SELECT 1",
"query": "SELECT NOT pg_is_in_recovery()",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Int4"
"type_info": "Bool"
}
],
"parameters": {
@@ -16,5 +16,5 @@
null
]
},
"hash": "e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5"
"hash": "282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM job_result_stream_v2\n WHERE job_id NOT IN (SELECT id FROM v2_job_queue)\n AND job_id NOT IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at > NOW() - INTERVAL '60 seconds'\n )\n RETURNING job_id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "job_id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": []
},
"nullable": [
false
]
},
"hash": "454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51"
}
@@ -0,0 +1,47 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO materialized_partition\n (workspace_id, asset_kind, asset_path, partition, status,\n snapshot_id, row_count, job_id, materialized_at, error)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now(), $9)\n ON CONFLICT (workspace_id, asset_kind, asset_path, partition)\n DO UPDATE SET status = EXCLUDED.status,\n snapshot_id = EXCLUDED.snapshot_id,\n row_count = EXCLUDED.row_count,\n job_id = EXCLUDED.job_id,\n materialized_at = now(),\n error = EXCLUDED.error",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
{
"Custom": {
"name": "asset_kind",
"kind": {
"Enum": [
"s3object",
"resource",
"variable",
"ducklake",
"datatable",
"volume"
]
}
}
},
"Varchar",
"Text",
{
"Custom": {
"name": "materialization_status",
"kind": {
"Enum": [
"running",
"materialized",
"failed"
]
}
}
},
"Int8",
"Int8",
"Uuid",
"Text"
]
},
"nullable": []
},
"hash": "5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19"
}
@@ -0,0 +1,18 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO ai_skill (workspace_id, name, description, instructions, edited_at, edited_by)\n VALUES ($1, $2, $3, $4, now(), $5)\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET description = EXCLUDED.description,\n instructions = EXCLUDED.instructions,\n edited_at = now(),\n edited_by = EXCLUDED.edited_by",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Text",
"Text",
"Varchar"
]
},
"nullable": []
},
"hash": "734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8"
}
@@ -0,0 +1,60 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM asset\n WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = 'script'\n RETURNING kind AS \"kind!: AssetKind\", path,\n usage_access_type AS \"usage_access_type: AssetUsageAccessType\"",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "kind!: AssetKind",
"type_info": {
"Custom": {
"name": "asset_kind",
"kind": {
"Enum": [
"s3object",
"resource",
"variable",
"ducklake",
"datatable",
"volume"
]
}
}
}
},
{
"ordinal": 1,
"name": "path",
"type_info": "Varchar"
},
{
"ordinal": 2,
"name": "usage_access_type: AssetUsageAccessType",
"type_info": {
"Custom": {
"name": "asset_access_type",
"kind": {
"Enum": [
"r",
"w",
"rw"
]
}
}
}
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
false,
true
]
},
"hash": "ab18d8765d6795eaa8035a8ac902790ff61549c5dd76e5b5cfb14b110a98abf2"
}
@@ -0,0 +1,63 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)\n VALUES ($1, $2, $3, $4, $5, 'script', $6) ON CONFLICT DO NOTHING\n RETURNING usage_access_type AS \"usage_access_type: AssetUsageAccessType\"",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "usage_access_type: AssetUsageAccessType",
"type_info": {
"Custom": {
"name": "asset_access_type",
"kind": {
"Enum": [
"r",
"w",
"rw"
]
}
}
}
}
],
"parameters": {
"Left": [
"Varchar",
"Varchar",
{
"Custom": {
"name": "asset_kind",
"kind": {
"Enum": [
"s3object",
"resource",
"variable",
"ducklake",
"datatable",
"volume"
]
}
}
},
{
"Custom": {
"name": "asset_access_type",
"kind": {
"Enum": [
"r",
"w",
"rw"
]
}
}
},
"Varchar",
"Jsonb"
]
},
"nullable": [
true
]
},
"hash": "bfb97d2f48157a1575b7b6f3e64e0d075701d541a44dd7a0b586d1f337ced5e1"
}
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT runnable_path FROM v2_job WHERE id = $1 AND workspace_id = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "runnable_path",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Uuid",
"Text"
]
},
"nullable": [
true
]
},
"hash": "c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba"
}
@@ -0,0 +1,111 @@
{
"db_name": "PostgreSQL",
"query": "SELECT asset_kind AS \"asset_kind: AssetKind\", asset_path, partition,\n status AS \"status: MaterializationStatus\", snapshot_id,\n row_count, job_id, materialized_at, error\n FROM materialized_partition\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY partition DESC",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "asset_kind: AssetKind",
"type_info": {
"Custom": {
"name": "asset_kind",
"kind": {
"Enum": [
"s3object",
"resource",
"variable",
"ducklake",
"datatable",
"volume"
]
}
}
}
},
{
"ordinal": 1,
"name": "asset_path",
"type_info": "Varchar"
},
{
"ordinal": 2,
"name": "partition",
"type_info": "Text"
},
{
"ordinal": 3,
"name": "status: MaterializationStatus",
"type_info": {
"Custom": {
"name": "materialization_status",
"kind": {
"Enum": [
"running",
"materialized",
"failed"
]
}
}
}
},
{
"ordinal": 4,
"name": "snapshot_id",
"type_info": "Int8"
},
{
"ordinal": 5,
"name": "row_count",
"type_info": "Int8"
},
{
"ordinal": 6,
"name": "job_id",
"type_info": "Uuid"
},
{
"ordinal": 7,
"name": "materialized_at",
"type_info": "Timestamptz"
},
{
"ordinal": 8,
"name": "error",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
{
"Custom": {
"name": "asset_kind",
"kind": {
"Enum": [
"s3object",
"resource",
"variable",
"ducklake",
"datatable",
"volume"
]
}
}
},
"Text"
]
},
"nullable": [
false,
false,
false,
false,
true,
true,
true,
false,
true
]
},
"hash": "c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM job_perms\nWHERE job_id NOT IN (SELECT id FROM v2_job_queue)\nRETURNING job_id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "job_id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": []
},
"nullable": [
false
]
},
"hash": "c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b"
}
@@ -0,0 +1,28 @@
{
"db_name": "PostgreSQL",
"query": "SELECT name, description FROM ai_skill WHERE workspace_id = $1 ORDER BY name",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "description",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
false
]
},
"hash": "c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b"
}
@@ -0,0 +1,12 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM job_result_stream_v2\n WHERE ctid IN (\n SELECT jrs.ctid FROM job_result_stream_v2 jrs\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id)\n AND NOT EXISTS (\n SELECT 1 FROM v2_job_completed c\n WHERE c.id = jrs.job_id\n AND c.completed_at > NOW() - INTERVAL '60 seconds'\n )\n LIMIT 100000\n )",
"describe": {
"columns": [],
"parameters": {
"Left": []
},
"nullable": []
},
"hash": "d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)\n VALUES ($1, $2, $3, $4, $5, $6, $7) ON CONFLICT DO NOTHING",
"query": "WITH ins AS (\n INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)\n VALUES ($1, $2, $3, $4, $5, $6, $7) ON CONFLICT DO NOTHING\n RETURNING usage_kind, usage_access_type\n )\n INSERT INTO notify_event (channel, payload)\n SELECT 'notify_asset_producer_change', $1\n FROM ins WHERE usage_kind = 'script' AND usage_access_type IN ('w', 'rw')",
"describe": {
"columns": [],
"parameters": {
@@ -52,5 +52,5 @@
},
"nullable": []
},
"hash": "a9e29764b5b9d94269e2b8aa755c71b61774c8ff8ae218d7a8d6ed0ac0169366"
"hash": "e041b20c1c4166b30ced8c6a0f50bcf0691ab2554ead6b489a8441a32fc0af82"
}
@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "SELECT a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app\n FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.path = $1 AND a.workspace_id = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "policy",
"type_info": "Text"
},
{
"ordinal": 1,
"name": "version",
"type_info": "Int8"
},
{
"ordinal": 2,
"name": "raw_app",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null,
null,
false
]
},
"hash": "e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd"
}
@@ -0,0 +1,35 @@
{
"db_name": "PostgreSQL",
"query": "SELECT name, description, instructions FROM ai_skill WHERE workspace_id = $1 AND name = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "description",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "instructions",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
false,
false
]
},
"hash": "e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d"
}
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM ai_skill WHERE workspace_id = $1 AND name = $2 RETURNING name",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "name",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false
]
},
"hash": "e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61"
}
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO notify_event (channel, payload)\n VALUES ('notify_asset_producer_change', $1)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "eeaad5c2284c1856cfc64ae0bc0dfc79b283c06987a6de399b6c1aaca94a2b7a"
}
@@ -0,0 +1,41 @@
{
"db_name": "PostgreSQL",
"query": "SELECT a.path, a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app\n FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.id = $1 AND a.workspace_id = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "policy",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "version",
"type_info": "Int8"
},
{
"ordinal": 3,
"name": "raw_app",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Int8",
"Text"
]
},
"nullable": [
false,
null,
null,
false
]
},
"hash": "ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT EXISTS (\n SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE j.workspace_id = $2\n AND (j.kind = 'appscript' OR j.kind = 'preview')\n AND j.created_by = 'anonymous'\n AND c.started_at > now() - interval '3 hours'\n AND j.runnable_path LIKE $3 || '/%'\n AND c.result @> ('{\"s3\":\"' || $1 || '\"}')::jsonb\n )",
"query": "SELECT EXISTS (\n SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE j.workspace_id = $2\n AND (j.kind = 'appscript' OR j.kind = 'preview')\n AND j.created_by = $4\n AND c.started_at > now() - interval '3 hours'\n AND j.runnable_path LIKE $3 || '/%'\n AND c.result @> ('{\"s3\":\"' || $1 || '\"}')::jsonb\n )",
"describe": {
"columns": [
{
@@ -11,6 +11,7 @@
],
"parameters": {
"Left": [
"Text",
"Text",
"Text",
"Text"
@@ -20,5 +21,5 @@
null
]
},
"hash": "a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d"
"hash": "f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7"
}
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT q.id FROM v2_job_queue q JOIN v2_job j USING (id)\n WHERE j.parent_job = $1 AND q.running = true",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
false
]
},
"hash": "fac563138c316998d4f523edd633db97dba77d649b637c2529e4f232dce16c88"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id",
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id",
"describe": {
"columns": [
{
@@ -20,5 +20,5 @@
false
]
},
"hash": "45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f"
"hash": "fbe3a876efd1253d2ef086b03366b2bd117ceb6bc152d2abcd45850ff6aecff9"
}
+107 -104
View File
@@ -237,9 +237,9 @@ checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb"
[[package]]
name = "arrayvec"
version = "0.7.6"
version = "0.7.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50"
checksum = "f02882884d3e1bc524fb12c79f107f6ad0e1cfd498c536ffb494301740995dfe"
[[package]]
name = "arrow"
@@ -2056,9 +2056,9 @@ dependencies = [
[[package]]
name = "cc"
version = "1.2.64"
version = "1.2.65"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dad887fd958be91b5098c0248def011f4523ab786cd411be668777e55063501f"
checksum = "e228eec9be7c17ccb640b59b36a5cd805ea2a564a4c5e162c2f659fea30d3b96"
dependencies = [
"find-msvc-tools",
"jobserver",
@@ -5233,9 +5233,9 @@ dependencies = [
[[package]]
name = "gosyn"
version = "0.2.10"
version = "0.2.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c99c1502d84229dc7ddb6af755f40ebe80e7e932fa78ecef979cedcf9999ba93"
checksum = "fed1657682b1c3f63ece1fe5b60fc6c5f5923612a20d19f6af38ce79eaf361e3"
dependencies = [
"anyhow",
"strum",
@@ -6715,9 +6715,9 @@ dependencies = [
[[package]]
name = "log"
version = "0.4.32"
version = "0.4.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a"
checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"
[[package]]
name = "loom"
@@ -7046,9 +7046,9 @@ checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4"
[[package]]
name = "memmap2"
version = "0.9.10"
version = "0.9.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "714098028fe011992e1c3962653c96b2d578c4b4bce9036e15ff220319b1e0e3"
checksum = "d1219ed1b7f229ee7104d281dd01d6802fe28bb6e95d292942c4daacdeb798c0"
dependencies = [
"libc",
"stable_deref_trait",
@@ -8914,9 +8914,9 @@ dependencies = [
[[package]]
name = "pulp"
version = "0.22.2"
version = "0.22.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2e205bb30d5b916c55e584c22201771bcf2bad9aabd5d4127f38387140c38632"
checksum = "046aa45b989642ec2e4717c8e72d677b13edd831a4d3b6cf37d9a3e54912496a"
dependencies = [
"bytemuck",
"cfg-if",
@@ -8931,9 +8931,9 @@ dependencies = [
[[package]]
name = "pulp-wasm-simd-flag"
version = "0.1.0"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40e24eee682d89fb193496edf918a7f407d30175b2e785fe057e4392dfd182e0"
checksum = "1d8f70e07b9c3962945a74e59ca1c511bba65b6419468acc217c457d93f3c740"
[[package]]
name = "pure-rust-locales"
@@ -8975,9 +8975,9 @@ dependencies = [
[[package]]
name = "quinn"
version = "0.11.9"
version = "0.11.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8"
dependencies = [
"bytes",
"cfg_aliases",
@@ -8995,9 +8995,9 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.14"
version = "0.11.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098"
checksum = "4fcb935c5bec503c2f0e306bdd3e58bb9029dcb14fa8d9ac76e3a5256ac0763e"
dependencies = [
"aws-lc-rs",
"bytes",
@@ -9031,9 +9031,9 @@ dependencies = [
[[package]]
name = "quote"
version = "1.0.45"
version = "1.0.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368"
dependencies = [
"proc-macro2",
]
@@ -12189,9 +12189,9 @@ dependencies = [
[[package]]
name = "time"
version = "0.3.49"
version = "0.3.51"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "711a53c2d47bbd818258c498c8dbfe186a2526c631495cfe7e078567f86b8469"
checksum = "85c17d80feb7334b40c484e45ed1a5273dfd8bfda537c3be2e74a06a6686f327"
dependencies = [
"deranged",
"num-conv",
@@ -12209,9 +12209,9 @@ checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109"
[[package]]
name = "time-macros"
version = "0.2.29"
version = "0.2.30"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "71c652a3727a9cbb9a02f707f530b618ce00d0ccd762009c8c23bd191df3c17d"
checksum = "dcef1a61bdb119096e153208ec5cbec23944ce8bca13be5c7f60c634f7403935"
dependencies = [
"num-conv",
"time-core",
@@ -13735,7 +13735,7 @@ dependencies = [
[[package]]
name = "windmill"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-nats",
@@ -13817,7 +13817,7 @@ dependencies = [
[[package]]
name = "windmill-ai"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"async-stream",
"async-trait",
@@ -13850,7 +13850,7 @@ dependencies = [
[[package]]
name = "windmill-alerting"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -13863,7 +13863,7 @@ dependencies = [
[[package]]
name = "windmill-api"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"argon2",
@@ -14001,7 +14001,7 @@ dependencies = [
[[package]]
name = "windmill-api-agent-workers"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14024,7 +14024,7 @@ dependencies = [
[[package]]
name = "windmill-api-assets"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14037,7 +14037,7 @@ dependencies = [
[[package]]
name = "windmill-api-auth"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14063,7 +14063,7 @@ dependencies = [
[[package]]
name = "windmill-api-client"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"reqwest 0.12.28",
"serde",
@@ -14073,7 +14073,7 @@ dependencies = [
[[package]]
name = "windmill-api-configs"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14090,7 +14090,7 @@ dependencies = [
[[package]]
name = "windmill-api-debug"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"base64 0.22.1",
@@ -14112,7 +14112,7 @@ dependencies = [
[[package]]
name = "windmill-api-embeddings"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14135,7 +14135,7 @@ dependencies = [
[[package]]
name = "windmill-api-flow-conversations"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14151,7 +14151,7 @@ dependencies = [
[[package]]
name = "windmill-api-flows"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14172,7 +14172,7 @@ dependencies = [
[[package]]
name = "windmill-api-groups"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14193,7 +14193,7 @@ dependencies = [
[[package]]
name = "windmill-api-inputs"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14207,7 +14207,7 @@ dependencies = [
[[package]]
name = "windmill-api-integration-tests"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-nats",
@@ -14242,7 +14242,7 @@ dependencies = [
[[package]]
name = "windmill-api-jobs"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14267,7 +14267,7 @@ dependencies = [
[[package]]
name = "windmill-api-npm-proxy"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"flate2",
@@ -14285,7 +14285,7 @@ dependencies = [
[[package]]
name = "windmill-api-openapi"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14307,7 +14307,7 @@ dependencies = [
[[package]]
name = "windmill-api-schedule"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14327,7 +14327,7 @@ dependencies = [
[[package]]
name = "windmill-api-scripts"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14354,6 +14354,7 @@ dependencies = [
"windmill-parser",
"windmill-parser-py",
"windmill-parser-py-asset",
"windmill-parser-sql",
"windmill-parser-sql-asset",
"windmill-parser-ts",
"windmill-parser-ts-asset",
@@ -14363,7 +14364,7 @@ dependencies = [
[[package]]
name = "windmill-api-settings"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14391,7 +14392,7 @@ dependencies = [
[[package]]
name = "windmill-api-sse"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"lazy_static",
"serde",
@@ -14403,7 +14404,7 @@ dependencies = [
[[package]]
name = "windmill-api-users"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"argon2",
"axum 0.8.9",
@@ -14428,7 +14429,7 @@ dependencies = [
[[package]]
name = "windmill-api-workers"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14442,7 +14443,7 @@ dependencies = [
[[package]]
name = "windmill-api-workspaces"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14475,7 +14476,7 @@ dependencies = [
[[package]]
name = "windmill-audit"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"chrono",
"lazy_static",
@@ -14489,7 +14490,7 @@ dependencies = [
[[package]]
name = "windmill-autoscaling"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -14508,7 +14509,7 @@ dependencies = [
[[package]]
name = "windmill-common"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"aes-gcm",
"aho-corasick",
@@ -14610,7 +14611,7 @@ dependencies = [
[[package]]
name = "windmill-dep-map"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"chrono",
"itertools 0.14.0",
@@ -14629,7 +14630,7 @@ dependencies = [
[[package]]
name = "windmill-git-sync"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"regex",
"serde",
@@ -14644,7 +14645,7 @@ dependencies = [
[[package]]
name = "windmill-indexer"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"astral-tokio-tar",
@@ -14668,7 +14669,7 @@ dependencies = [
[[package]]
name = "windmill-jseval"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"futures",
@@ -14685,7 +14686,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"itertools 0.14.0",
"lazy_static",
@@ -14701,7 +14702,7 @@ dependencies = [
[[package]]
name = "windmill-mcp"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -14722,7 +14723,7 @@ dependencies = [
[[package]]
name = "windmill-native-triggers"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -14753,7 +14754,7 @@ dependencies = [
[[package]]
name = "windmill-oauth"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"arc-swap",
@@ -14778,7 +14779,7 @@ dependencies = [
[[package]]
name = "windmill-object-store"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-stream",
@@ -14812,7 +14813,7 @@ dependencies = [
[[package]]
name = "windmill-operator"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"futures",
@@ -14830,7 +14831,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"convert_case 0.6.0",
"serde",
@@ -14839,7 +14840,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -14851,7 +14852,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -14863,7 +14864,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"gosyn",
@@ -14875,7 +14876,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -14887,7 +14888,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -14899,7 +14900,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -14910,7 +14911,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -14921,7 +14922,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -14933,7 +14934,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -14944,7 +14945,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -14966,7 +14967,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -14978,7 +14979,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -14992,7 +14993,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"convert_case 0.6.0",
@@ -15009,7 +15010,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -15022,7 +15023,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde",
@@ -15034,7 +15035,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -15052,7 +15053,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -15068,7 +15069,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -15084,7 +15085,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde",
@@ -15095,7 +15096,7 @@ dependencies = [
[[package]]
name = "windmill-queue"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -15133,7 +15134,7 @@ dependencies = [
[[package]]
name = "windmill-runtime-nativets"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"const_format",
@@ -15172,7 +15173,7 @@ dependencies = [
[[package]]
name = "windmill-sql-datatype-parser-wasm"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"getrandom 0.3.4",
"wasm-bindgen",
@@ -15183,17 +15184,19 @@ dependencies = [
[[package]]
name = "windmill-store"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-recursion",
"axum 0.8.9",
"base64 0.22.1",
"chrono",
"futures",
"hex",
"http 1.4.2",
"hyper 1.10.1",
"lazy_static",
"magic-crypt",
"quick_cache",
"reqwest 0.13.1",
"serde",
@@ -15215,7 +15218,7 @@ dependencies = [
[[package]]
name = "windmill-test-utils"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15239,7 +15242,7 @@ dependencies = [
[[package]]
name = "windmill-trigger"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15272,7 +15275,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-azure"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15305,7 +15308,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-email"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15325,7 +15328,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-gcp"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15359,7 +15362,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-http"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15395,7 +15398,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-kafka"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15418,7 +15421,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-mqtt"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15442,7 +15445,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-nats"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-nats",
@@ -15466,7 +15469,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-postgres"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15501,7 +15504,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-sqs"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15529,7 +15532,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-websocket"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15554,7 +15557,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"bitflags 2.13.0",
@@ -15573,7 +15576,7 @@ dependencies = [
[[package]]
name = "windmill-worker"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-once-cell",
@@ -15683,7 +15686,7 @@ dependencies = [
[[package]]
name = "windmill-worker-volumes"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"bytes",
"futures",
@@ -16501,9 +16504,9 @@ dependencies = [
[[package]]
name = "zlib-rs"
version = "0.6.3"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3be3d40e40a133f9c916ee3f9f4fa2d9d63435b5fbe1bfc6d9dae0aa0ada1513"
checksum = "977347db8caa080403f6b6b7c1cda9479a8e869316f7e13a59b19076a40f94e3"
[[package]]
name = "zmij"
+2 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.730.0"
version = "1.737.0"
authors.workspace = true
edition.workspace = true
@@ -87,7 +87,7 @@ members = [
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
[workspace.package]
version = "1.730.0"
version = "1.737.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
+1 -1
View File
@@ -97,7 +97,7 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical).
| id | threat | actor | surface | asset | impact | likelihood | status | controls | evidence |
|---|---|---|---|---|---|---|---|---|---|
| T1 | SQL injection in app/internal query builders and trigger clauses compromises the metadata DB and connected databases | remote_auth | EP8 | Database, downstream connected systems | critical | almost_certain | partially_mitigated | sqlx parameterized queries elsewhere; query-builder safety reviews | GHSA-225c-j3xq-g6x6, GHSA-78p7-jc72-gv66, GHSA-hvc7-f67h-jx3g, GHSA-wrrg-f89m-f84q, GHSA-79vf-3qwm-2w64, GHSA-55p6-fxj4-v983, GHSA-5g4v-49rj-r52r, GHSA-x6cq-7xr8-53x3, 2cf4bb180b |
| T2 | Server-side request forgery via proxies/executors reaches cloud metadata, internal network, and downstream credentials | remote_auth | EP6, EP7 | Cloud metadata, internal network, downstream connected systems, resource creds | critical | almost_certain | partially_mitigated | SSRF URL validation + redirect-following disabled added piecemeal; MCP private URL access requires the instance-wide `ALLOW_PRIVATE_MCP_SERVER_URLS` opt-in; outbound network isolation (`clone_newnet`) is opt-in and off by default | GHSA-3ggp-h37f-5qfw, GHSA-98qq-g8rh-xhff, GHSA-hfw8-27mx-63jm, GHSA-3r59-qvvc-774j, GHSA-4pj9-w5jc-g8w7, GHSA-8hh3-jf25-78j5, GHSA-3pjm-4w7f-3r2w, GHSA-f44c-x9hq-h68r, GHSA-j4h4-f8fj-3m3c, 4b06881918, 96a8eb63d4, dbd3942ef3 |
| T2 | Server-side request forgery via proxies/executors reaches cloud metadata, internal network, and downstream credentials | remote_auth | EP6, EP7 | Cloud metadata, internal network, downstream connected systems, resource creds | critical | almost_certain | partially_mitigated | SSRF URL validation + redirect-following disabled added piecemeal; MCP private URL access requires the instance-wide `ALLOW_PRIVATE_MCP_SERVER_URLS` opt-in; WebSocket trigger URLs (stored, test, and runnable-resolved) are SSRF-validated at connect time behind the `ALLOW_PRIVATE_WEBSOCKET_URLS` opt-in, and the trigger test route now requires `:write` scope; outbound network isolation (`clone_newnet`) is opt-in and off by default | GHSA-3ggp-h37f-5qfw, GHSA-98qq-g8rh-xhff, GHSA-hfw8-27mx-63jm, GHSA-3r59-qvvc-774j, GHSA-4pj9-w5jc-g8w7, GHSA-8hh3-jf25-78j5, GHSA-3pjm-4w7f-3r2w, GHSA-f44c-x9hq-h68r, GHSA-j4h4-f8fj-3m3c, 4b06881918, 96a8eb63d4, dbd3942ef3 |
| T3 | Broken authorization / IDOR lets a scoped token or low-privilege member read scripts, job data, and secrets across folders and workspaces | remote_auth | EP5, EP2, EP1 | Scripts, job data, secrets, isolation | critical | almost_certain | partially_mitigated | RLS, token scopes, folder ACLs, view-token HMAC (added incrementally); on managed, sensitive tenants can opt into dedicated DB/worker/namespace, but the shared tier IS the software boundary | GHSA-qfg7-x243-5hg4, GHSA-8x8x-88qc-qp4r, GHSA-2ppx-66jv-wpw5, GHSA-x3x7-g97v-mp59, GHSA-j276-g4h8-g6h5, GHSA-8mv7-hmrg-96xv, GHSA-x2wf-f962-7frq, GHSA-qc7c-gcw6-h4xp, GHSA-vxc5-w28p-m9xw, GHSA-2g34-wfvr-5qqj, GHSA-w7p6-wpxm-pp66, 7edf3f0212, 89a7a37776, ab11c7747a, 664edcdfb7 |
| T4 | Remote code execution by injecting attacker-controlled identifiers into generated worker wrappers | remote_auth | EP10 | Worker host, isolation, downstream | critical | likely | partially_mitigated | entrypoint/env-var-name validation added | GHSA-wxjq-w5pj-jqhx, GHSA-5f5q-2vg2-r2x4, GHSA-8q8j-mm3g-5c2q (CVE-2026-33881), bf93657fee, bd05bcadde, 22ec4da5f0 |
| T5 | Worker compromise & cross-tenant access via weak-by-default isolation (nsjail off by default → user code runs with only PID-ns `unshare`); sandbox escape where nsjail/dind/podman is enabled | remote_auth | EP9, EP15 | Worker host, isolation, downstream | critical | likely | unmitigated | nsjail off by default everywhere (`DISABLE_NSJAIL=true`); shipped compose gives PID-ns `unshare` only (`FAVOR_UNSHARE_PID=true`), bare installs get no isolation. Where nsjail enabled: read-only remounts, jail-tmp refusal, podman socket gating | GHSA-6qr8-xhg4-453q, GHSA-3vpp-vf62-wqp6, f8467f38c8, df5aec0f5d, f1b6746e0e |
+1 -1
View File
@@ -1 +1 @@
ba677ea142011462ad4dfe77e8375a6dd274cdef
ac1f6f666f36141cb6ba6f8eaa614821a90464ad
@@ -0,0 +1 @@
DROP TABLE IF EXISTS ai_skill;
@@ -0,0 +1,15 @@
-- Workspace-scoped AI chat skills (Claude/Codex-style SKILL.md instructions).
-- `name` is the skill folder slug; `description` is advertised in the AI chat
-- system prompt, `instructions` is the SKILL.md body fetched on demand.
CREATE TABLE ai_skill (
workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE,
name VARCHAR(255) NOT NULL,
description TEXT NOT NULL,
instructions TEXT NOT NULL,
edited_at TIMESTAMPTZ NOT NULL DEFAULT now(),
edited_by VARCHAR(255) NOT NULL DEFAULT '',
PRIMARY KEY (workspace_id, name)
);
GRANT ALL ON ai_skill TO windmill_user;
GRANT ALL ON ai_skill TO windmill_admin;
@@ -0,0 +1,4 @@
REVOKE ALL ON notify_event FROM windmill_user;
REVOKE ALL ON notify_event FROM windmill_admin;
REVOKE ALL ON SEQUENCE notify_event_id_seq FROM windmill_user;
REVOKE ALL ON SEQUENCE notify_event_id_seq FROM windmill_admin;
@@ -0,0 +1,14 @@
-- The notify_event table (migration 20260203172950_polling_based_events) was
-- created relying on ALTER DEFAULT PRIVILEGES to grant access to windmill_user
-- and windmill_admin. Those default privileges only apply to objects created by
-- the role that set them (migration 20250205131523), so deployments whose
-- migration runner is a different role leave notify_event ungranted. Trigger
-- inserts were worked around with SECURITY DEFINER, but direct application
-- inserts (clear_static_asset_usage in assets.rs, restart_worker_group in
-- settings) run as the invoking role and fail with "permission denied for table
-- notify_event". Grant explicitly to guarantee access regardless of who ran the
-- migrations.
GRANT ALL ON notify_event TO windmill_user;
GRANT ALL ON notify_event TO windmill_admin;
GRANT ALL ON SEQUENCE notify_event_id_seq TO windmill_user;
GRANT ALL ON SEQUENCE notify_event_id_seq TO windmill_admin;
@@ -0,0 +1,4 @@
REVOKE ALL ON script_trigger FROM windmill_user;
REVOKE ALL ON script_trigger FROM windmill_admin;
REVOKE ALL ON SEQUENCE script_trigger_id_seq FROM windmill_user;
REVOKE ALL ON SEQUENCE script_trigger_id_seq FROM windmill_admin;
@@ -0,0 +1,14 @@
-- The script_trigger table (migration 20260423050000_script_trigger) was
-- created relying on ALTER DEFAULT PRIVILEGES to grant access to windmill_user
-- and windmill_admin. Those default privileges only apply to objects created by
-- the role that set them (migration 20250205131523), so deployments whose
-- migration runner is a different role leave script_trigger ungranted. Direct
-- application writes run as the invoking role (clear_script_triggers and
-- insert_script_trigger in windmill-common/src/assets.rs, every script save)
-- and fail with "permission denied for table script_trigger". Grant explicitly
-- to guarantee access regardless of who ran the migrations (same fix as
-- notify_event in 20260619091631).
GRANT ALL ON script_trigger TO windmill_user;
GRANT ALL ON script_trigger TO windmill_admin;
GRANT ALL ON SEQUENCE script_trigger_id_seq TO windmill_user;
GRANT ALL ON SEQUENCE script_trigger_id_seq TO windmill_admin;
@@ -0,0 +1,2 @@
-- Irreversible: a stripped NUL cannot be restored (and was never meaningful).
SELECT 1;
@@ -0,0 +1,30 @@
-- One-time cleanup of drafts whose `json` value carries a real U+0000 (NUL)
-- escape — storable only because `draft.value` is `json`, not `jsonb`. Such a
-- value makes any `->>`/`to_jsonb` extraction raise `22P05`, which 500'd
-- GET /drafts/list. New writes are sanitized in the application layer
-- (update_draft → strip_json_nul); this fixes rows written before that landed.
--
-- Only genuinely-poisoned rows are touched: a real NUL makes `value::jsonb`
-- raise, which distinguishes it from a legitimately escaped backslash sequence
-- (which `jsonb` accepts). The text replace handles the real-world shape — a NUL
-- inside a text field. A contrived value where stripping the escape leaves
-- invalid JSON is left as-is (and can no longer be created).
DO $$
DECLARE r RECORD;
BEGIN
FOR r IN
SELECT id, value FROM draft WHERE position(E'\\u0000' in value::text) > 0
LOOP
BEGIN
PERFORM r.value::jsonb; -- not poisoned (legit escaped backslash): skip
EXCEPTION WHEN others THEN
BEGIN
UPDATE draft
SET value = replace(r.value::text, E'\\u0000', '')::json
WHERE id = r.id;
EXCEPTION WHEN others THEN
NULL; -- pathological shape; cannot strip in SQL, no longer creatable
END;
END;
END LOOP;
END $$;
@@ -0,0 +1,3 @@
DROP INDEX IF EXISTS idx_materialized_partition_asset_status;
DROP TABLE IF EXISTS materialized_partition;
DROP TYPE IF EXISTS MATERIALIZATION_STATUS;
@@ -0,0 +1,29 @@
-- Per-partition materialization state for managed `// materialize` assets.
-- One row per (asset, partition): the latest materialization of that slice.
-- Drives: the partition-status grid (CE observability), run-stale/gap
-- detection, and the EE backfill worklist (missing/failed partitions). The
-- `partition` column uses '' as the sentinel for an unpartitioned (whole-table)
-- materialization, since partition is part of the primary key and cannot be
-- NULL.
CREATE TYPE MATERIALIZATION_STATUS AS ENUM ('running', 'materialized', 'failed');
CREATE TABLE IF NOT EXISTS materialized_partition (
workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE,
asset_kind ASSET_KIND NOT NULL,
asset_path VARCHAR(255) NOT NULL,
partition TEXT NOT NULL DEFAULT '',
status MATERIALIZATION_STATUS NOT NULL,
-- DuckLake snapshot id produced by the write; NULL while running / on
-- failure. The pin that makes downstream reads reproducible.
snapshot_id BIGINT,
row_count BIGINT,
job_id UUID,
materialized_at TIMESTAMPTZ NOT NULL DEFAULT now(),
error TEXT,
PRIMARY KEY (workspace_id, asset_kind, asset_path, partition)
);
-- Backfill enumeration / grid "show only gaps": filter an asset's partitions
-- by status without scanning the whole table.
CREATE INDEX IF NOT EXISTS idx_materialized_partition_asset_status
ON materialized_partition (workspace_id, asset_kind, asset_path, status);
+24 -24
View File
@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windmill-common"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"aho-corasick",
"anyhow",
@@ -6272,7 +6272,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"proc-macro2",
"quote",
@@ -6284,7 +6284,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"convert_case",
"serde",
@@ -6293,7 +6293,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6305,7 +6305,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6317,7 +6317,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"gosyn",
@@ -6329,7 +6329,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6341,7 +6341,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6353,7 +6353,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -6364,7 +6364,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6375,7 +6375,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6387,7 +6387,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6398,7 +6398,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -6420,7 +6420,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6432,7 +6432,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6446,7 +6446,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"convert_case",
@@ -6463,7 +6463,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6476,7 +6476,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde",
@@ -6488,7 +6488,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6506,7 +6506,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -6522,7 +6522,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6538,7 +6538,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wasm"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"getrandom 0.2.17",
@@ -6570,7 +6570,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"serde",
@@ -6581,7 +6581,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.730.0"
version = "1.737.0"
dependencies = [
"anyhow",
"bitflags",
@@ -12,7 +12,7 @@ resolver = "2"
members = ["."]
[workspace.package]
version = "1.730.0"
version = "1.737.0"
edition = "2021"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
@@ -50,7 +50,7 @@ pub fn parse_ansible_sig(inner_content: &str) -> anyhow::Result<MainArgSignature
has_default: default.is_some(),
default,
oidx: None,
otyp_inferred: false,
otyp_inferred: false,
})
}
}
@@ -69,7 +69,7 @@ pub fn parse_ansible_sig(inner_content: &str) -> anyhow::Result<MainArgSignature
has_default: inv.default.is_some(),
default: inv.default.map(|v| json!(format!("$res:{}", v))),
oidx: None,
otyp_inferred: false,
otyp_inferred: false,
});
}
}
@@ -83,7 +83,7 @@ pub fn parse_ansible_sig(inner_content: &str) -> anyhow::Result<MainArgSignature
has_default: false,
default: None,
oidx: None,
otyp_inferred: false,
otyp_inferred: false,
});
}
}
@@ -435,6 +435,25 @@ pub fn parse_delegate_to_git_repo(inner_content: &str) -> anyhow::Result<Delegat
Ok(DelegateWithSSHAuth { delegate_to_git_repo_details: None, git_ssh_identity })
}
/// Each `vault_id` entry is interpolated verbatim into the generated `ansible.cfg`
/// (`vault_identity_list = <a>,<b>,...`). A newline or other config-meaningful
/// character would let a script inject arbitrary `[defaults]` directives (e.g.
/// `library`, `action_plugins`) and execute attacker-controlled code on the worker,
/// and a `,` would smuggle in an extra entry. Restrict entries to the `label@source`
/// charset so neither is possible.
pub fn validate_vault_id(value: &str) -> anyhow::Result<()> {
let is_valid = !value.is_empty()
&& value
.chars()
.all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-' | '/' | '@'));
if !is_valid {
return Err(anyhow!(
"Invalid vault_id `{value}`: expected `label@filename` using only letters, digits and the characters `.`, `_`, `-`, `/`, `@`"
));
}
Ok(())
}
pub fn parse_ansible_reqs(
inner_content: &str,
) -> anyhow::Result<(String, Option<AnsibleRequirements>, String)> {
@@ -528,6 +547,7 @@ pub fn parse_ansible_reqs(
let Yaml::String(filename) = f else {
return Err(anyhow!("The elements of the vault_id field should be strings in the format: `label@filename`"));
};
validate_vault_id(filename)?;
ret.vault_id.push(filename.to_string());
}
}
@@ -1051,4 +1071,55 @@ delegate_to_git_repo:
Some("inventories/{{ env }}")
);
}
#[test]
fn test_parse_vault_id_valid() {
let p = r#"
---
vault_id:
- dev@vault_pass_dev.txt
- prod@./secrets/prod-pass
---
- name: Test
hosts: all
"#;
let (_, reqs, _) = parse_ansible_reqs(p).unwrap();
assert_eq!(
reqs.unwrap().vault_id,
vec![
"dev@vault_pass_dev.txt".to_string(),
"prod@./secrets/prod-pass".to_string()
]
);
}
#[test]
fn test_parse_vault_id_rejects_newline_injection() {
let p = "---\nvault_id:\n - \"default@/tmp/wm/x\\nlibrary = /tmp/wm/evil_modules\"\n---\n- name: Test\n hosts: all\n";
assert!(parse_ansible_reqs(p).is_err());
}
#[test]
fn test_parse_vault_id_rejects_comma() {
let p = r#"
---
vault_id:
- "a@b,c@d"
---
- name: Test
hosts: all
"#;
assert!(parse_ansible_reqs(p).is_err());
}
#[test]
fn test_validate_vault_id() {
assert!(validate_vault_id("default@/tmp/wm/pass").is_ok());
assert!(validate_vault_id("dev@pass.txt").is_ok());
assert!(validate_vault_id("").is_err());
assert!(validate_vault_id("a@b\nlibrary = /evil").is_err());
assert!(validate_vault_id("a@b,c@d").is_err());
assert!(validate_vault_id("a@b c").is_err());
assert!(validate_vault_id("a@b=c").is_err());
}
}
@@ -107,6 +107,11 @@ pub struct ParseAssetsOutput {
// The delay is a raw duration string parsed at deploy (parser-light).
#[serde(skip_serializing_if = "Option::is_none", default)]
pub retry: Option<RetrySpec>,
// `// materialize [manual] <asset> [append] [key=<col>]` —
// managed-materialization target + its strategy. At most one per script.
// Drives the worker's write-strategy + snapshot capture.
#[serde(skip_serializing_if = "Option::is_none", default)]
pub materialize: Option<MaterializeSpec>,
}
#[derive(Serialize, Debug, PartialEq, Clone)]
@@ -209,6 +214,27 @@ pub struct RetrySpec {
pub delay: Option<String>,
}
// `// materialize [manual] <asset> [append] [key=<col>]` — declares that this
// script produces a *managed* materialization of `<asset>` (a `ducklake://`
// table). By default the runtime generates the write DDL around the script's
// single trailing `SELECT` and owns idempotency, partition-state and snapshot
// capture. `manual` is the escape hatch: the script writes its own DDL and the
// runtime only records state (track-only). The reconciliation strategy options
// (`append`, `key=<col>`) apply to managed mode: none → DELETE-by-partition +
// INSERT (replace); `key=<col>` → MERGE (dedup within slice); `append` →
// INSERT-only. `append` wins if both are given (deploy-time warning).
#[derive(Serialize, Debug, PartialEq, Clone)]
pub struct MaterializeSpec {
pub target_kind: AssetKind,
pub target_path: String,
#[serde(skip_serializing_if = "std::ops::Not::not", default)]
pub manual: bool,
#[serde(skip_serializing_if = "std::ops::Not::not", default)]
pub append: bool,
#[serde(skip_serializing_if = "Option::is_none", default)]
pub unique_key: Option<String>,
}
// `// trigger any` (default) vs `// trigger all`. `Any` = OR: any trigger
// firing runs the script (current behaviour). `All` = AND: the script
// runs only once every partition-bearing input has materialized at the
@@ -239,6 +265,7 @@ pub struct PipelineAnnotations {
pub debounce_default: Option<String>,
pub tag: Option<String>,
pub retry: Option<RetrySpec>,
pub materialize: Option<MaterializeSpec>,
}
impl ParseAssetsOutput {
@@ -262,6 +289,7 @@ impl ParseAssetsOutput {
debounce_default: pipeline.debounce_default,
tag: pipeline.tag,
retry: pipeline.retry,
materialize: pipeline.materialize,
}
}
}
@@ -571,6 +599,15 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations {
continue;
}
if let Some(after_kw) = consume_keyword(rest, "materialize") {
if out.materialize.is_none() {
if let Some(spec) = parse_materialize_spec(after_kw.trim()) {
out.materialize = Some(spec);
}
}
continue;
}
if let Some(after_kw) = consume_keyword(rest, "on") {
let spec_text = after_kw.trim();
if spec_text.is_empty() {
@@ -618,6 +655,35 @@ fn parse_retry_spec(s: &str) -> Option<RetrySpec> {
Some(RetrySpec { count, delay })
}
// Parse a `// materialize [manual] <asset> [append] [key=<col>]` right-hand
// side. An optional leading `manual` token (whitespace-delimited) opts out of
// managed mode (track-only). The next whitespace token is the target asset URI
// (default-syntax shorthands enabled, so `ducklake` → `ducklake://main`); the
// remainder are strategy options — bare `append` and `key=<col>` (merge key),
// which apply to managed mode only. A missing/empty target yields `None` (the
// annotation is dropped, fail-safe).
fn parse_materialize_spec(s: &str) -> Option<MaterializeSpec> {
let (manual, rest) = match s.strip_prefix("manual") {
Some(after) if after.is_empty() || after.starts_with(char::is_whitespace) => {
(true, after.trim_start())
}
_ => (false, s),
};
let mut it = rest.trim().splitn(2, char::is_whitespace);
let asset_tok = it.next()?;
let opts_str = it.next().unwrap_or("");
let (target_kind, path) = parse_asset_syntax(asset_tok.trim(), true)?;
if path.is_empty() {
return None;
}
let append = opts_str.split_whitespace().any(|t| t == "append");
let unique_key = parse_kv_opts(opts_str)
.get("key")
.filter(|k| !k.is_empty())
.cloned();
Some(MaterializeSpec { target_kind, target_path: path.to_string(), manual, append, unique_key })
}
// Parse a `// partitioned <kind> [opts]` right-hand side. Recognized kinds:
// `daily`, `hourly`, `weekly`, `monthly` (with optional tz/format/start),
// and `dynamic key="<jsonpath>"` (plus optional format).
@@ -1044,6 +1110,67 @@ mod pipeline_annotation_tests {
assert!(out.retry.is_none());
}
#[test]
fn materialize_managed_default() {
let out = parse_pipeline_annotations("// materialize ducklake://analytics/orders_daily");
let m = out.materialize.expect("materialize");
assert_eq!(m.target_kind, AssetKind::Ducklake);
assert_eq!(m.target_path, "analytics/orders_daily");
// managed by default; replace strategy (no append / key)
assert!(!m.manual);
assert!(!m.append);
assert_eq!(m.unique_key, None);
}
#[test]
fn materialize_manual_escape_hatch() {
let out =
parse_pipeline_annotations("// materialize manual ducklake://analytics/orders_daily");
let m = out.materialize.expect("materialize");
assert!(m.manual);
assert_eq!(m.target_path, "analytics/orders_daily");
}
#[test]
fn materialize_merge_and_append_options() {
let out =
parse_pipeline_annotations("// materialize ducklake://a/orders_daily key=order_id");
let m = out.materialize.expect("materialize");
assert_eq!(m.unique_key.as_deref(), Some("order_id"));
assert!(!m.append);
let out = parse_pipeline_annotations("// materialize ducklake://a/events append");
let m = out.materialize.expect("materialize");
assert!(m.append);
assert_eq!(m.unique_key, None);
}
#[test]
fn materialize_default_syntax_shorthand() {
let out = parse_pipeline_annotations("// materialize ducklake");
let m = out.materialize.expect("materialize");
assert_eq!(m.target_kind, AssetKind::Ducklake);
assert_eq!(m.target_path, "main");
assert!(!m.manual);
}
#[test]
fn materialize_manual_only_is_dropped() {
// `manual` with no target is not a valid materialization.
let out = parse_pipeline_annotations("// materialize manual");
assert!(out.materialize.is_none());
}
#[test]
fn materialize_first_wins() {
let out = parse_pipeline_annotations(
"// materialize ducklake://a/x\n# materialize manual ducklake://b/y",
);
let m = out.materialize.expect("materialize");
assert_eq!(m.target_path, "a/x");
assert!(!m.manual);
}
#[test]
fn combined() {
let code = concat!(
@@ -1053,7 +1180,8 @@ mod pipeline_annotation_tests {
"// partitioned daily tz=\"UTC\"\n",
"// freshness 2h\n",
"// tag heavy\n",
"// retry 3 5s\n"
"// retry 3 5s\n",
"// materialize ducklake://analytics/orders_daily key=order_id\n"
);
let out = parse_pipeline_annotations(code);
assert!(out.in_pipeline);
@@ -1064,6 +1192,10 @@ mod pipeline_annotation_tests {
let r = out.retry.expect("retry");
assert_eq!(r.count, 3);
assert_eq!(r.delay.as_deref(), Some("5s"));
let m = out.materialize.expect("materialize");
assert!(!m.manual);
assert_eq!(m.target_path, "analytics/orders_daily");
assert_eq!(m.unique_key.as_deref(), Some("order_id"));
}
#[test]
@@ -13,6 +13,7 @@ use serde::Serialize;
use serde_json::Value;
pub mod asset_parser;
pub mod sql_materialize;
/// S3 output format for SQL queries (moved here to avoid pulling sqlx into WASM via windmill-types)
#[derive(Clone, Copy, Debug)]
@@ -0,0 +1,817 @@
//! Eligibility classifier + materialization SQL codegen for managed `// materialize`.
//!
//! Managed `// materialize` (the default) promises the script is "setup
//! statements, then one trailing SELECT" — Windmill generates the write DDL
//! around that SELECT (the `// materialize manual` escape hatch opts out and
//! writes its own DDL). This module is the single source of truth for *which
//! block is that SELECT* and *what DDL gets generated*, so save-time validation
//! (deploy path) and run-time codegen (DuckDB executor) can never disagree.
//!
//! Everything here is pure and string-level: no SQL is executed, no type
//! inference is done. The classifier is leading-keyword based and deliberately
//! conservative — anything it can't positively recognize as a read-only output
//! or a known-safe setup statement is rejected, so a script is only accepted
//! for managed mode when its shape is unambiguous.
/// One top-level statement's role in a wrap-mode script.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum BlockClass {
/// Read-only relation the wrap writes from: `SELECT` / `WITH …SELECT` /
/// `FROM` (DuckDB from-first) / `VALUES` / `TABLE x` / `(UN)PIVOT`.
Output,
/// Known-safe preamble: `ATTACH` / `INSTALL` / `LOAD` / `SET` / `PRAGMA` /
/// `USE` / `CREATE TEMP …`. Runs verbatim before the generated write.
Setup,
/// Anything that writes or whose effect we can't vouch for: non-temp
/// `CREATE` / `INSERT` / `UPDATE` / `DELETE` / `MERGE` / `DROP` / `COPY` /
/// `ALTER` / `TRUNCATE`, or an unrecognized leading keyword. Disqualifies
/// managed mode (the user should use `// materialize manual`).
Disallowed,
}
/// A script accepted for wrapping: zero+ setup blocks then one terminal SELECT.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct WrapPlan {
/// Setup statements in source order, verbatim, **without** trailing `;`.
pub setup: Vec<String>,
/// The single terminal output statement, verbatim, **without** trailing `;`.
pub output: String,
}
/// Why a script is not eligible for managed `// materialize`. Carries enough to
/// render the targeted save-time messages.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum WrapError {
/// No statements at all (empty / comments only).
Empty,
/// No terminal SELECT — nothing to wrap.
NoOutput,
/// More than one top-level SELECT. `count` is how many were found.
MultipleOutputs { count: usize },
/// A SELECT exists but isn't the last statement (something runs after it).
OutputNotLast,
/// A write/unknown statement appears among the setup blocks. `snippet` is a
/// short prefix of the offending statement for the error message.
DisallowedBlock { snippet: String },
}
impl WrapError {
/// Human-facing, actionable message (matches the spec's rejection text).
pub fn message(&self) -> String {
let base =
"managed `// materialize` requires the script to be setup statements then a single trailing SELECT";
let manual = "use `// materialize manual` to write the DDL yourself";
match self {
WrapError::Empty => format!("{base}: the script is empty."),
WrapError::NoOutput => format!("{base}: found no SELECT — {manual}."),
WrapError::MultipleOutputs { count } => format!(
"{base}: found {count} SELECT statements; combine them with a CTE, or {manual}."
),
WrapError::OutputNotLast => format!(
"{base}: found statements after the SELECT — move them above it, or {manual}."
),
WrapError::DisallowedBlock { snippet } => {
format!("{base}: `{snippet}` writes or is unrecognized — {manual}.")
}
}
}
}
/// Split SQL into top-level, `;`-separated statements, skipping line comments
/// (`-- …`), block comments (`/* … */`), single-quoted strings (`'…'` with
/// `''` escape) and double-quoted identifiers (`"…"`). Semicolons inside any of
/// those are not separators. Returns each statement trimmed, comments stripped,
/// empties dropped. Self-contained so the parser crate stays dependency-free;
/// it must stay behaviourally aligned with the executor's block splitter (both
/// route wrap through `classify_wrap`, so the split they see is this one).
pub fn split_statements(sql: &str) -> Vec<String> {
let mut out = Vec::new();
let mut cur = String::new();
let bytes = sql.as_bytes();
let mut i = 0;
let n = bytes.len();
while i < n {
let c = bytes[i] as char;
// line comment — `--` (SQL) or `//`. The `//` form is not SQL, but it
// is how Windmill pipeline annotations (`// materialize`, `// pipeline`,
// …) are written, and they sit above the SQL in the same script; strip
// them so they don't pollute the first statement block's classification
// or the generated setup SQL.
if (c == '-' && i + 1 < n && bytes[i + 1] == b'-')
|| (c == '/' && i + 1 < n && bytes[i + 1] == b'/')
{
while i < n && bytes[i] != b'\n' {
i += 1;
}
continue;
}
// block comment
if c == '/' && i + 1 < n && bytes[i + 1] == b'*' {
i += 2;
while i + 1 < n && !(bytes[i] == b'*' && bytes[i + 1] == b'/') {
i += 1;
}
i += 2;
continue;
}
// single-quoted string
if c == '\'' {
cur.push(c);
i += 1;
while i < n {
cur.push(bytes[i] as char);
if bytes[i] == b'\'' {
// doubled '' is an escaped quote, stay in string
if i + 1 < n && bytes[i + 1] == b'\'' {
cur.push('\'');
i += 2;
continue;
}
i += 1;
break;
}
i += 1;
}
continue;
}
// double-quoted identifier
if c == '"' {
cur.push(c);
i += 1;
while i < n {
cur.push(bytes[i] as char);
if bytes[i] == b'"' {
i += 1;
break;
}
i += 1;
}
continue;
}
if c == ';' {
let t = cur.trim();
if !t.is_empty() {
out.push(t.to_string());
}
cur.clear();
i += 1;
continue;
}
cur.push(c);
i += 1;
}
let t = cur.trim();
if !t.is_empty() {
out.push(t.to_string());
}
out
}
/// Lowercased top-level keyword tokens of a single statement (parens collapsed
/// away: tokens *inside* balanced `(...)` are skipped, so a CTE body's verbs
/// don't leak up). Strings/identifiers are already gone from the split, but we
/// re-guard quotes defensively. Used to disambiguate `WITH …` and `CREATE …`.
fn top_level_keywords(stmt: &str) -> Vec<String> {
let mut toks = Vec::new();
let mut cur = String::new();
let mut depth: i32 = 0;
let bytes = stmt.as_bytes();
let mut i = 0;
let n = bytes.len();
let flush = |cur: &mut String, toks: &mut Vec<String>| {
if !cur.is_empty() {
toks.push(cur.to_lowercase());
cur.clear();
}
};
while i < n {
let c = bytes[i] as char;
if c == '\'' || c == '"' {
let q = bytes[i];
i += 1;
while i < n && bytes[i] != q {
i += 1;
}
i += 1;
continue;
}
if c == '(' {
flush(&mut cur, &mut toks);
depth += 1;
i += 1;
continue;
}
if c == ')' {
if depth > 0 {
depth -= 1;
}
i += 1;
continue;
}
if depth > 0 {
i += 1;
continue;
}
if c.is_alphanumeric() || c == '_' {
cur.push(c);
} else {
flush(&mut cur, &mut toks);
}
i += 1;
}
flush(&mut cur, &mut toks);
toks
}
const OUTPUT_KW: &[&str] = &["select", "from", "values", "table", "pivot", "unpivot"];
const SETUP_KW: &[&str] = &["attach", "install", "load", "set", "pragma", "use"];
const WRITE_VERBS: &[&str] = &["insert", "update", "delete", "merge"];
/// Classify a single statement by its leading keyword (with `WITH`/`CREATE`
/// disambiguation). See [`BlockClass`].
pub fn classify_block(stmt: &str) -> BlockClass {
let kws = top_level_keywords(stmt);
let Some(first) = kws.first().map(String::as_str) else {
return BlockClass::Disallowed;
};
// CREATE TEMP … is setup (staging); any other CREATE is a write.
if first == "create" {
let temp = kws
.iter()
.skip(1)
.take(3)
.any(|k| k == "temp" || k == "temporary");
return if temp {
BlockClass::Setup
} else {
BlockClass::Disallowed
};
}
// WITH … : the main statement's verb decides. CTE bodies are parenthesized,
// so their verbs are not in `kws`; the first top-level write verb or SELECT
// after the CTE list is the real one.
if first == "with" {
for k in kws.iter().skip(1) {
if k == "select" {
return BlockClass::Output;
}
if WRITE_VERBS.contains(&k.as_str()) {
return BlockClass::Disallowed;
}
}
// `WITH x AS (...) SELECT` where SELECT got collapsed is impossible
// (SELECT here is top-level), so a WITH with no top-level verb is a
// malformed/unknown statement — reject conservatively.
return BlockClass::Disallowed;
}
if OUTPUT_KW.contains(&first) {
return BlockClass::Output;
}
if SETUP_KW.contains(&first) {
return BlockClass::Setup;
}
BlockClass::Disallowed
}
/// Validate a script for managed `// materialize` and, on success, return the
/// setup/output split. Enforces the four conditions from the spec:
/// 1. exactly one Output block, 2. it is last, 3. all preceding blocks are
/// Setup, 4. nothing after it.
pub fn classify_wrap(sql: &str) -> Result<WrapPlan, WrapError> {
let stmts = split_statements(sql);
if stmts.is_empty() {
return Err(WrapError::Empty);
}
let classes: Vec<BlockClass> = stmts.iter().map(|s| classify_block(s)).collect();
let output_idxs: Vec<usize> = classes
.iter()
.enumerate()
.filter(|(_, c)| **c == BlockClass::Output)
.map(|(i, _)| i)
.collect();
match output_idxs.len() {
0 => return Err(WrapError::NoOutput),
1 => {}
count => return Err(WrapError::MultipleOutputs { count }),
}
let out_idx = output_idxs[0];
if out_idx != stmts.len() - 1 {
return Err(WrapError::OutputNotLast);
}
// Everything before the output must be Setup (no Disallowed preamble).
for (i, c) in classes.iter().enumerate().take(out_idx) {
if *c != BlockClass::Setup {
return Err(WrapError::DisallowedBlock { snippet: snippet(&stmts[i]) });
}
}
Ok(WrapPlan { setup: stmts[..out_idx].to_vec(), output: stmts[out_idx].clone() })
}
fn snippet(stmt: &str) -> String {
let one_line: String = stmt.split_whitespace().collect::<Vec<_>>().join(" ");
if one_line.chars().count() > 40 {
let truncated: String = one_line.chars().take(40).collect();
format!("{truncated}")
} else {
one_line
}
}
// ---------------------------------------------------------------------------
// Codegen
// ---------------------------------------------------------------------------
/// How a (partition of a) materialized table is reconciled on each run.
/// Derived at deploy from `unique_key`/`append`: `append` → `Append`, else
/// `unique_key` → `Merge`, else `Replace`.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum MaterializeStrategy {
/// DELETE the current partition, then INSERT — partition becomes exactly
/// what the SELECT returned. Full-refresh of the slice.
Replace,
/// Upsert within the slice on `unique_key` (delete-by-key + insert); rows
/// absent from the SELECT are left in place.
Merge { unique_key: String },
/// INSERT only — immutable event-log semantics.
Append,
}
/// Inputs to materialization codegen, all resolved at run time by the worker.
/// Pure: produces SQL text; executes nothing.
#[derive(Debug, Clone)]
pub struct MaterializeCodegen<'a> {
/// Fully-qualified target, e.g. `_wm_target.orders_daily`. Always qualified
/// so a user `USE …;` in setup can't redirect the write.
pub target_qualified: &'a str,
/// The user's output SELECT (verbatim, no trailing `;`) — embedded as a
/// subquery so its own shape is irrelevant to the generated wrapper.
pub select_sql: &'a str,
/// Physical partition column added to the managed table.
pub partition_col: &'a str,
/// SQL expression for the current partition value — a literal like
/// `'2026-06-19'` or a bind placeholder. The caller is responsible for
/// safe quoting/binding.
pub partition_value_sql: &'a str,
/// Whether `// partitioned` applies. When false the table is unpartitioned
/// and the partition column / `SET PARTITIONED BY` are omitted.
pub partitioned: bool,
pub strategy: MaterializeStrategy,
}
impl<'a> MaterializeCodegen<'a> {
/// The ordered statements that perform the materialization, to be run after
/// the setup blocks and inside the caller's execution. The first-run
/// bootstrap is idempotent (`IF NOT EXISTS`), so this is safe to run every
/// time. The DELETE/INSERT body is wrapped in one transaction so a partial
/// failure leaves the prior snapshot intact. Every strategy reduces to
/// DELETE+INSERT (no `MERGE INTO`) — see the `Merge` arm for why.
pub fn statements(&self) -> Vec<String> {
let t = self.target_qualified;
let sel = self.select_sql;
let pcol = self.partition_col;
let pval = self.partition_value_sql;
let mut out = Vec::new();
// Whole-table replace: rebuild the table to match the SELECT's *current*
// schema each run with one atomic `CREATE OR REPLACE` (which DuckLake
// still snapshots). This is the only path that survives a changed SELECT
// or a pre-existing table with a different schema — the persist-and-
// mutate paths below fix the schema at first create.
if !self.partitioned && matches!(self.strategy, MaterializeStrategy::Replace) {
out.push(format!(
"CREATE OR REPLACE TABLE {t} AS SELECT * FROM ({sel});"
));
return out;
}
// Persist-and-mutate (partitioned, or merge/append): bootstrap the table
// if absent, then write into it. The schema is fixed at first create —
// a later SELECT-schema change needs a manual rebuild (schema evolution
// is a follow-up).
if self.partitioned {
out.push(format!(
"CREATE TABLE IF NOT EXISTS {t} AS \
SELECT *, CAST(NULL AS VARCHAR) AS {pcol} FROM ({sel}) WHERE false;"
));
out.push(format!("ALTER TABLE {t} SET PARTITIONED BY ({pcol});"));
} else {
out.push(format!(
"CREATE TABLE IF NOT EXISTS {t} AS SELECT * FROM ({sel}) WHERE false;"
));
}
out.push("BEGIN TRANSACTION;".to_string());
// The rows to write, with the partition column appended when partitioned.
let source = if self.partitioned {
format!("SELECT *, {pval} AS {pcol} FROM ({sel})")
} else {
format!("SELECT * FROM ({sel})")
};
match &self.strategy {
MaterializeStrategy::Replace => {
// Only reached when partitioned (whole-table replace returned above).
out.push(format!("DELETE FROM {t} WHERE {pcol} = {pval};"));
out.push(format!("INSERT INTO {t} {source};"));
}
MaterializeStrategy::Append => {
out.push(format!("INSERT INTO {t} {source};"));
}
MaterializeStrategy::Merge { unique_key } => {
// Upsert within the slice via delete-by-key + insert (dbt's
// `delete+insert`): rows whose key is in the incoming SELECT are
// replaced, others are left in place. This deliberately avoids
// `MERGE INTO` — DuckLake's MERGE fails writing the first rows of
// a fresh partition (HTTP 404 on the new parquet), and a failed
// write leaves the table needing a DROP. DELETE+INSERT is the
// same write shape as `replace`, which is reliable. The DELETE is
// scoped to the current partition when partitioned so it stays
// slice-local (a key present in another partition is untouched).
let scope = if self.partitioned {
format!("{pcol} = {pval} AND ")
} else {
String::new()
};
out.push(format!(
"DELETE FROM {t} WHERE {scope}{unique_key} IN (SELECT {unique_key} FROM ({sel}));"
));
out.push(format!("INSERT INTO {t} {source};"));
}
}
out.push("COMMIT;".to_string());
out
}
}
/// The read that captures the DuckLake snapshot id produced by the write, for
/// the given attach alias (e.g. `_wm_target`). The worker runs this last and
/// records the result into `materialized_partition`.
pub fn snapshot_capture_sql(alias: &str) -> String {
format!("SELECT max(snapshot_id) AS snapshot_id FROM ducklake_snapshots('{alias}');")
}
/// Reserved attach alias for the materialization target, fully-qualified in all
/// generated SQL so a user `USE …;` in the setup blocks can't redirect the
/// write. The worker resolves the real `ATTACH 'ducklake:…' AS _wm_target (…)`
/// from the target ducklake's config and passes it in as `target_attach`.
pub const TARGET_ALIAS: &str = "_wm_target";
/// Assemble the full ordered statement list the DuckDB executor runs for a
/// managed `// materialize` script. This is the single entry point the worker
/// calls; it composes the already-tested pieces (classifier split → target
/// ATTACH → strategy codegen → snapshot capture) so their ordering lives in one
/// tested place rather than inline in the executor.
///
/// `target_attach` is the real `ATTACH 'ducklake:…' AS _wm_target (…);` string
/// the worker built from config (it depends on resolved credentials, so it
/// can't be generated here). `target_table` is the table within that catalog
/// (e.g. `orders_daily`), referenced as `_wm_target.<table>`. `asset_path` is
/// the full `<name>/<table>` for the result summary. The trailing statement is
/// a one-row summary read (asset / rows / snapshot_id) that is both the job's
/// result (a useful preview) and what the worker records.
pub fn build_wrap_blocks(
plan: &WrapPlan,
target_attach: &str,
target_table: &str,
asset_path: &str,
partition_col: &str,
partition_value_sql: &str,
partitioned: bool,
strategy: MaterializeStrategy,
) -> Vec<String> {
let target_qualified = format!("{TARGET_ALIAS}.{target_table}");
let cg = MaterializeCodegen {
target_qualified: &target_qualified,
select_sql: &plan.output,
partition_col,
partition_value_sql,
partitioned,
strategy,
};
let mut blocks: Vec<String> = Vec::new();
// Setup blocks come from the splitter with their `;` stripped — re-terminate
// each so that when the executor re-joins and re-splits the assembled query,
// adjacent statements (e.g. the user ATTACH and the synthetic target ATTACH)
// don't merge into one malformed statement.
blocks.extend(plan.setup.iter().map(|s| terminate(s)));
blocks.push(target_attach.to_string());
blocks.extend(cg.statements());
blocks.push(materialize_result_sql(
&target_qualified,
asset_path,
partition_col,
partition_value_sql,
partitioned,
));
blocks
}
/// The trailing one-row summary the materialize run returns: the asset it
/// produced, the row count of the materialized slice (the partition when
/// partitioned, else the whole table), and the DuckLake snapshot it created.
/// This is both a useful preview result and the row the worker records.
pub fn materialize_result_sql(
target_qualified: &str,
asset_path: &str,
partition_col: &str,
partition_value_sql: &str,
partitioned: bool,
) -> String {
let (count_expr, partition_sel) = if partitioned {
// Row count is the slice this run wrote (the partition); `partition`
// lets the UI label the count and scope the preview to it.
(
format!(
"(SELECT count(*) FROM {target_qualified} WHERE {partition_col} = {partition_value_sql})"
),
format!("{partition_value_sql} AS partition, "),
)
} else {
(
format!("(SELECT count(*) FROM {target_qualified})"),
String::new(),
)
};
format!(
"SELECT 'ducklake://{asset_path}' AS materialized, \
{partition_sel}{count_expr} AS rows, \
(SELECT max(snapshot_id) FROM ducklake_snapshots('{TARGET_ALIAS}')) AS snapshot_id;"
)
}
// Ensure a statement ends with a single `;`.
fn terminate(stmt: &str) -> String {
let t = stmt.trim_end();
if t.ends_with(';') {
t.to_string()
} else {
format!("{t};")
}
}
#[cfg(test)]
mod tests {
use super::*;
fn ok(sql: &str) -> WrapPlan {
classify_wrap(sql).expect("expected wrap-eligible")
}
fn err(sql: &str) -> WrapError {
classify_wrap(sql).expect_err("expected wrap-ineligible")
}
#[test]
fn split_respects_strings_comments_idents() {
let sql = "SET x=1; -- a; comment\nSELECT ';' AS a, \"weird;col\" /* ; */ FROM t;";
let s = split_statements(sql);
assert_eq!(s.len(), 2);
assert_eq!(s[0], "SET x=1");
assert!(s[1].starts_with("SELECT"));
assert!(s[1].contains("\"weird;col\""));
}
#[test]
fn split_handles_escaped_quote() {
let s = split_statements("SELECT 'it''s; fine' AS a;");
assert_eq!(s.len(), 1);
assert!(s[0].contains("it''s; fine"));
}
#[test]
fn pipeline_annotations_are_stripped() {
// The real shape: `//` annotation lines above the SQL must not pollute
// the first block's classification (regression — they were being read
// as a leading `pipeline` keyword and rejected).
let p = ok("// pipeline\n// materialize ducklake://main/t\n// partitioned daily\nATTACH 'ducklake://main' AS dl;\nSELECT 1 AS id");
assert_eq!(p.setup.len(), 1);
// The annotation lines are gone — the setup block starts at the real
// SQL (the `//` inside `ducklake://main` is legitimately retained).
assert!(p.setup[0].starts_with("ATTACH"));
assert!(p.output.starts_with("SELECT"));
}
#[test]
fn bare_select_is_eligible() {
let p = ok("SELECT a, b FROM t WHERE c = '{partition}'");
assert!(p.setup.is_empty());
assert!(p.output.starts_with("SELECT"));
}
#[test]
fn setup_then_select_is_eligible() {
let p = ok(
"ATTACH 'ducklake://main' AS dl;\n SET memory_limit='4GB';\n SELECT * FROM dl.orders",
);
assert_eq!(p.setup.len(), 2);
assert!(p.output.starts_with("SELECT"));
}
#[test]
fn create_temp_staging_is_setup() {
let p = ok("CREATE TEMP TABLE s AS SELECT 1; SELECT * FROM s");
assert_eq!(p.setup.len(), 1);
assert_eq!(
classify_block("CREATE TEMP TABLE s AS SELECT 1"),
BlockClass::Setup
);
assert_eq!(
classify_block("CREATE OR REPLACE TEMPORARY VIEW v AS SELECT 1"),
BlockClass::Setup
);
}
#[test]
fn with_cte_select_is_output_write_is_disallowed() {
assert_eq!(
classify_block("WITH x AS (SELECT 1) SELECT * FROM x"),
BlockClass::Output
);
// CTE whose main statement inserts is a write, even though it starts WITH.
assert_eq!(
classify_block("WITH x AS (SELECT 1) INSERT INTO t SELECT * FROM x"),
BlockClass::Disallowed
);
}
#[test]
fn from_first_and_values_are_output() {
assert_eq!(classify_block("FROM t SELECT a"), BlockClass::Output);
assert_eq!(classify_block("VALUES (1),(2)"), BlockClass::Output);
assert_eq!(classify_block("TABLE t"), BlockClass::Output);
}
#[test]
fn trailing_write_rejected() {
assert_eq!(
err("SELECT * FROM t; INSERT INTO u VALUES (1)"),
WrapError::OutputNotLast
);
}
#[test]
fn write_in_preamble_rejected() {
match err("INSERT INTO t VALUES (1); SELECT * FROM t") {
WrapError::DisallowedBlock { snippet } => assert!(snippet.starts_with("INSERT")),
e => panic!("wrong error: {e:?}"),
}
}
#[test]
fn multiple_selects_rejected() {
assert_eq!(
err("SELECT 1; SELECT 2"),
WrapError::MultipleOutputs { count: 2 }
);
}
#[test]
fn no_select_and_empty_rejected() {
assert_eq!(err("CREATE TABLE t (a INT)"), WrapError::NoOutput);
assert_eq!(err(" -- just a comment\n"), WrapError::Empty);
}
#[test]
fn use_cannot_redirect_is_classified_setup() {
// `USE` is allowed setup; generated SQL is fully qualified regardless.
assert_eq!(classify_block("USE dl"), BlockClass::Setup);
}
#[test]
fn codegen_replace_partitioned() {
let cg = MaterializeCodegen {
target_qualified: "_wm_target.orders_daily",
select_sql: "SELECT a FROM dl.orders",
partition_col: "_wm_partition",
partition_value_sql: "'2026-06-19'",
partitioned: true,
strategy: MaterializeStrategy::Replace,
};
let st = cg.statements();
assert!(st[0].contains("CREATE TABLE IF NOT EXISTS _wm_target.orders_daily"));
assert!(st[0].contains("CAST(NULL AS VARCHAR) AS _wm_partition"));
assert!(st.iter().any(
|s| s == "ALTER TABLE _wm_target.orders_daily SET PARTITIONED BY (_wm_partition);"
));
assert!(st.iter().any(|s| s.starts_with(
"DELETE FROM _wm_target.orders_daily WHERE _wm_partition = '2026-06-19'"
)));
assert!(st.iter().any(|s| s.contains(
"INSERT INTO _wm_target.orders_daily SELECT *, '2026-06-19' AS _wm_partition"
)));
assert_eq!(st.first().map(|_| &st[st.len() - 1]).unwrap(), "COMMIT;");
}
#[test]
fn codegen_merge_is_delete_by_key_plus_insert() {
let cg = MaterializeCodegen {
target_qualified: "_wm_target.orders_daily",
select_sql: "SELECT order_id, amount FROM dl.orders",
partition_col: "_wm_partition",
partition_value_sql: "'2026-06-19'",
partitioned: true,
strategy: MaterializeStrategy::Merge { unique_key: "order_id".to_string() },
};
let st = cg.statements();
// upsert = delete-by-key (partition-scoped) + insert — NO `MERGE INTO`
// (DuckLake's MERGE fails on fresh partitions).
assert!(!st.iter().any(|s| s.contains("MERGE INTO")));
let del = st
.iter()
.find(|s| s.starts_with("DELETE FROM"))
.expect("delete stmt");
assert!(del.contains(
"WHERE _wm_partition = '2026-06-19' AND order_id IN (SELECT order_id FROM (SELECT order_id, amount FROM dl.orders))"
));
assert!(st
.iter()
.any(|s| s.starts_with("INSERT INTO _wm_target.orders_daily SELECT *, '2026-06-19'")));
}
#[test]
fn codegen_append_inserts_only() {
let cg = MaterializeCodegen {
target_qualified: "_wm_target.events",
select_sql: "SELECT * FROM dl.raw",
partition_col: "_wm_partition",
partition_value_sql: "'2026-06-19'",
partitioned: true,
strategy: MaterializeStrategy::Append,
};
let st = cg.statements();
assert!(st
.iter()
.any(|s| s.starts_with("INSERT INTO _wm_target.events")));
assert!(!st.iter().any(|s| s.starts_with("DELETE")));
assert!(!st.iter().any(|s| s.starts_with("MERGE")));
}
#[test]
fn codegen_whole_table_replace_is_create_or_replace() {
// Unpartitioned replace must use CREATE OR REPLACE so a changed SELECT
// schema (or a pre-existing table with a different schema) doesn't break
// — and nothing else (no bootstrap / DELETE / INSERT / txn).
let cg = MaterializeCodegen {
target_qualified: "_wm_target.customer_dim",
select_sql: "SELECT a, b, c FROM dl.src",
partition_col: "_wm_partition",
partition_value_sql: "''",
partitioned: false,
strategy: MaterializeStrategy::Replace,
};
let st = cg.statements();
assert_eq!(
st,
vec![
"CREATE OR REPLACE TABLE _wm_target.customer_dim AS SELECT * FROM (SELECT a, b, c FROM dl.src);"
.to_string()
]
);
}
#[test]
fn snapshot_capture_targets_alias() {
assert_eq!(
snapshot_capture_sql("_wm_target"),
"SELECT max(snapshot_id) AS snapshot_id FROM ducklake_snapshots('_wm_target');"
);
}
#[test]
fn build_wrap_blocks_orders_setup_attach_codegen_snapshot() {
let plan = ok("ATTACH 'ducklake://main' AS dl;\n SELECT a FROM dl.orders WHERE d = '{p}'");
let blocks = build_wrap_blocks(
&plan,
"ATTACH 'ducklake:postgres:…' AS _wm_target (DATA_PATH 's3://b/p');",
"orders_daily",
"main/orders_daily",
"_wm_partition",
"'2026-06-19'",
true,
MaterializeStrategy::Replace,
);
// setup block first, then the target ATTACH, then codegen, then result.
assert!(blocks[0].starts_with("ATTACH 'ducklake://main' AS dl"));
// every setup block must be `;`-terminated so re-splitting can't merge it
// with the synthetic target ATTACH that follows.
assert!(blocks[0].ends_with(';'));
assert_eq!(
blocks[1],
"ATTACH 'ducklake:postgres:…' AS _wm_target (DATA_PATH 's3://b/p');"
);
assert!(blocks.iter().any(|b| b.contains("_wm_target.orders_daily")));
assert!(blocks.iter().any(|b| b.starts_with(
"DELETE FROM _wm_target.orders_daily WHERE _wm_partition = '2026-06-19'"
)));
// the trailing block is the one-row summary (asset / rows / snapshot_id),
// partition-scoped for the row count
let last = blocks.last().unwrap();
assert!(last.contains("'ducklake://main/orders_daily' AS materialized"));
assert!(last.contains("'2026-06-19' AS partition"));
assert!(last.contains("WHERE _wm_partition = '2026-06-19') AS rows"));
assert!(last.contains("ducklake_snapshots('_wm_target')"));
}
}
@@ -234,5 +234,72 @@
"tag": null,
"retry": null
}
},
{
"name": "materialize managed (default) with merge key",
"code": "// pipeline\n// materialize ducklake://analytics/orders_daily key=order_id\nSELECT 1;",
"expected": {
"in_pipeline": true,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"materialize": {
"target_kind": "ducklake",
"target_path": "analytics/orders_daily",
"unique_key": "order_id"
}
}
},
{
"name": "materialize manual escape hatch, first value wins",
"code": "// materialize manual ducklake://analytics/orders_daily\n// materialize ducklake://other/x\nexport function main() {}",
"expected": {
"in_pipeline": false,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"materialize": {
"target_kind": "ducklake",
"target_path": "analytics/orders_daily",
"manual": true
}
}
},
{
"name": "materialize default-syntax shorthand with append",
"code": "// materialize ducklake append\nexport function main() {}",
"expected": {
"in_pipeline": false,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null,
"materialize": {
"target_kind": "ducklake",
"target_path": "main",
"append": true
}
}
},
{
"name": "materialize manual with no target is dropped",
"code": "// materialize manual\nexport function main() {}",
"expected": {
"in_pipeline": false,
"asset_triggers": [],
"native_triggers": [],
"partition": null,
"freshness": null,
"tag": null,
"retry": null
}
}
]
@@ -34,6 +34,22 @@ struct Expected {
freshness: Option<String>,
tag: Option<String>,
retry: Option<ExpectedRetry>,
// Default-on-absent so the pre-existing fixtures (which omit it) keep
// deserializing; only fixtures exercising materialization set it.
#[serde(default)]
materialize: Option<ExpectedMaterialize>,
}
#[derive(Deserialize)]
struct ExpectedMaterialize {
target_kind: String,
target_path: String,
#[serde(default)]
manual: bool,
#[serde(default)]
append: bool,
#[serde(default)]
unique_key: Option<String>,
}
#[derive(Deserialize)]
@@ -153,5 +169,25 @@ fn pipeline_annotation_fixtures_match() {
want.is_some()
),
}
match (&got.materialize, &f.expected.materialize) {
(None, None) => {}
(Some(m), Some(e)) => {
assert_eq!(
kind_str(m.target_kind),
e.target_kind,
"{ctx}: materialize kind"
);
assert_eq!(m.target_path, e.target_path, "{ctx}: materialize path");
assert_eq!(m.manual, e.manual, "{ctx}: materialize manual");
assert_eq!(m.append, e.append, "{ctx}: materialize append");
assert_eq!(m.unique_key, e.unique_key, "{ctx}: materialize key");
}
(got, want) => panic!(
"{ctx}: materialize mismatch — got {:?}, want present={}",
got,
want.is_some()
),
}
}
}
+38 -12
View File
@@ -1446,19 +1446,45 @@ Windmill Community Edition {GIT_VERSION}
} else {
None
};
monitor_db(
&conn,
&base_internal_url,
server_mode,
worker_mode,
false,
tx.clone(),
Some(MonitorIteration {
rd_shift,
iter: monitor_iteration,
}),
// Hard cap on a single monitor pass. monitor_db runs all its
// periodic tasks under one join!, so a single task stuck on a
// non-DB await (statement_timeout only bounds DB statements)
// would otherwise freeze the whole loop indefinitely — silently
// stopping critical maintenance like audit-partition creation.
// Larger than statement_timeout (5min) so a slow-but-progressing
// statement is never killed prematurely.
const MONITOR_DB_TIMEOUT: Duration = Duration::from_secs(600);
let monitor_timed_out = tokio::time::timeout(
MONITOR_DB_TIMEOUT,
monitor_db(
&conn,
&base_internal_url,
server_mode,
worker_mode,
false,
tx.clone(),
Some(MonitorIteration {
rd_shift,
iter: monitor_iteration,
}),
),
)
.await;
.await
.is_err();
if monitor_timed_out {
windmill_common::utils::report_critical_error(
format!(
"monitor task did not finish within {}s and was aborted; \
a background maintenance task is likely stuck. \
Continuing to the next iteration.",
MONITOR_DB_TIMEOUT.as_secs()
),
db.clone(),
None,
None,
)
.await;
}
monitor_iteration += 1;
if let Some(handle) = warn_handle {
handle.abort();
+145 -53
View File
@@ -1532,14 +1532,22 @@ async fn delete_expired_jobs_batch(
.await?;
// Use FOR UPDATE SKIP LOCKED to avoid contention between replicas
// ORDER BY completed_at ensures we delete oldest jobs first
// ORDER BY completed_at ensures we delete oldest jobs first.
// Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than
// `!= ALL($3)`: the subquery form lets the planner build a one-time hashed
// SubPlan and apply it as a filter on the ordered index scan, giving O(1)
// membership per candidate instead of a per-row linear array scan (which
// degrades sharply when many root jobs are active). The `u IS NOT NULL` guard
// sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids).
let deleted_jobs: Vec<Uuid> = sqlx::query_scalar!(
"DELETE FROM v2_job_completed
WHERE id IN (
SELECT jc.id FROM v2_job_completed jc
LEFT JOIN v2_job j ON j.id = jc.id
WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
)
ORDER BY jc.completed_at ASC
LIMIT $2
FOR UPDATE OF jc SKIP LOCKED
@@ -1638,11 +1646,30 @@ async fn delete_log_files_from_disk_and_store(
.collect();
let stream = futures::stream::iter(s3_paths).boxed();
let mut result = os.delete_stream(stream);
let mut deleted = 0u64;
let mut not_found = 0u64;
let mut failed = 0u64;
while let Some(r) = result.next().await {
if let Err(e) = r {
tracing::error!("Failed to delete from object store: {e}");
match r {
Ok(_) => deleted += 1,
// Deleting a non-existent object is a successful no-op. S3's
// DeleteObjects ignores missing keys, but GCS returns 404 per
// delete, surfacing as NotFound — count it separately rather
// than logging it as an error.
Err(windmill_object_store::object_store_reexports::ObjectStoreError::NotFound { .. }) => {
not_found += 1;
}
Err(e) => {
failed += 1;
tracing::error!("Failed to delete from object store: {e}");
}
}
}
if deleted + not_found + failed > 0 {
tracing::info!(
"object store log cleanup: {deleted} deleted, {not_found} already absent (404), {failed} failed"
);
}
}
}
}
@@ -4349,39 +4376,65 @@ RETURNING key,job_id
Ok(())
}
async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> {
let result = sqlx::query_scalar!(
"DELETE FROM job_perms
WHERE job_id NOT IN (SELECT id FROM v2_job_queue)
RETURNING job_id"
)
.fetch_all(db)
.await?;
// Per-statement cap keeps each delete short and lock-light; the per-cycle batch
// cap bounds total work per monitor iteration so monitor_db stays responsive.
// A large backlog drains across several iterations rather than one long delete.
const ORPHAN_CLEANUP_BATCH_SIZE: u64 = 100_000;
const ORPHAN_CLEANUP_MAX_BATCHES: usize = 10;
if !result.is_empty() {
tracing::info!("Cleaned up {} orphaned job_perms rows", result.len());
async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> {
let mut total: u64 = 0;
for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES {
let count = sqlx::query!(
"DELETE FROM job_perms
WHERE ctid IN (
SELECT jp.ctid FROM job_perms jp
WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)
LIMIT 100000
)"
)
.execute(db)
.await?
.rows_affected();
total += count;
if count < ORPHAN_CLEANUP_BATCH_SIZE {
break;
}
}
if total > 0 {
tracing::info!("Cleaned up {total} orphaned job_perms rows");
}
Ok(())
}
async fn cleanup_job_result_stream_orphaned_jobs(db: &DB) -> error::Result<()> {
let result = sqlx::query!(
"DELETE FROM job_result_stream_v2
WHERE job_id NOT IN (SELECT id FROM v2_job_queue)
AND job_id NOT IN (
SELECT id FROM v2_job_completed
WHERE completed_at > NOW() - INTERVAL '60 seconds'
)
RETURNING job_id",
)
.fetch_all(db)
.await?;
let mut total: u64 = 0;
for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES {
let count = sqlx::query!(
"DELETE FROM job_result_stream_v2
WHERE ctid IN (
SELECT jrs.ctid FROM job_result_stream_v2 jrs
WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id)
AND NOT EXISTS (
SELECT 1 FROM v2_job_completed c
WHERE c.id = jrs.job_id
AND c.completed_at > NOW() - INTERVAL '60 seconds'
)
LIMIT 100000
)",
)
.execute(db)
.await?
.rows_affected();
total += count;
if count < ORPHAN_CLEANUP_BATCH_SIZE {
break;
}
}
if result.len() > 0 {
tracing::info!(
"Cleaned up {} orphaned job_result_stream_v2 rows",
result.len()
);
if total > 0 {
tracing::info!("Cleaned up {total} orphaned job_result_stream_v2 rows");
}
Ok(())
}
@@ -4417,11 +4470,19 @@ async fn audit_log_retention_days() -> i64 {
}
}
/// Number of days ahead (including today) for which an audit partition must
/// always exist. A missing partition in this window means audit inserts fail
/// once that date is reached — and because some callers (notably login) write
/// the audit row in the same transaction as their own work, that failure
/// poisons the whole transaction, so a missing partition is a hard outage, not
/// just a dropped audit row.
const AUDIT_PARTITION_LOOKAHEAD_DAYS: i64 = 3;
async fn manage_audit_partitions(db: &DB, retention_days: i64) {
let today = chrono::Utc::now().date_naive();
// Create partitions for today and the next 3 days
for days_ahead in 0..=3i64 {
// Create partitions for today and the next few days
for days_ahead in 0..=AUDIT_PARTITION_LOOKAHEAD_DAYS {
let date = today + chrono::Duration::days(days_ahead);
let next_date = date + chrono::Duration::days(1);
let partition_name = format!("audit_{}", date.format("%Y%m%d"));
@@ -4437,9 +4498,6 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) {
}
}
// Drop expired partitions
let cutoff_date = today - chrono::Duration::days(retention_days);
let partitions = sqlx::query_scalar::<_, String>(
"SELECT c.relname::text \
FROM pg_inherits i \
@@ -4449,28 +4507,62 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) {
.fetch_all(db)
.await;
match partitions {
Ok(partitions) => {
for partition_name in partitions {
if let Some(date_str) = partition_name.strip_prefix("audit_") {
if let Ok(date) = chrono::NaiveDate::parse_from_str(date_str, "%Y%m%d") {
if date < cutoff_date {
let quoted_name =
format!("\"{}\"", partition_name.replace('"', "\"\""));
let sql = format!("DROP TABLE IF EXISTS {quoted_name}");
match sqlx::query(&sql).execute(db).await {
Ok(_) => tracing::info!(
"Dropped expired audit partition {partition_name}"
),
Err(e) => tracing::error!(
"Error dropping audit partition {partition_name}: {e:?}"
),
}
let partitions = match partitions {
Ok(partitions) => partitions,
Err(e) => {
tracing::error!("Error listing audit partitions: {e:?}");
return;
}
};
// Verify the lookahead window is actually covered. If a create above failed
// (or this loop has not run for several days), alert loudly instead of
// letting it surface days later as failed audit inserts and broken logins.
let existing: std::collections::HashSet<&str> = partitions.iter().map(|s| s.as_str()).collect();
let missing: Vec<String> = (0..=AUDIT_PARTITION_LOOKAHEAD_DAYS)
.map(|days_ahead| {
format!(
"audit_{}",
(today + chrono::Duration::days(days_ahead)).format("%Y%m%d")
)
})
.filter(|name| !existing.contains(name.as_str()))
.collect();
if !missing.is_empty() {
report_critical_error(
format!(
"Audit log partitions missing after maintenance run: {}. \
Audit inserts will fail once these dates are reached, which also \
breaks logins (the login audit row shares the login transaction). \
Check for earlier 'Error creating audit partition' logs and verify \
the audit-partition maintenance loop is still running.",
missing.join(", ")
),
db.clone(),
None,
None,
)
.await;
}
// Drop expired partitions
let cutoff_date = today - chrono::Duration::days(retention_days);
for partition_name in &partitions {
if let Some(date_str) = partition_name.strip_prefix("audit_") {
if let Ok(date) = chrono::NaiveDate::parse_from_str(date_str, "%Y%m%d") {
if date < cutoff_date {
let quoted_name = format!("\"{}\"", partition_name.replace('"', "\"\""));
let sql = format!("DROP TABLE IF EXISTS {quoted_name}");
match sqlx::query(&sql).execute(db).await {
Ok(_) => {
tracing::info!("Dropped expired audit partition {partition_name}")
}
Err(e) => tracing::error!(
"Error dropping audit partition {partition_name}: {e:?}"
),
}
}
}
}
Err(e) => tracing::error!("Error listing audit partitions: {e:?}"),
}
}
+29 -11
View File
@@ -12,7 +12,6 @@ use serde_json::json;
use sqlx::{Pool, Postgres};
use uuid::Uuid;
use windmill_common::jobs::{JobKind, JobPayload};
use windmill_common::runnable_settings::prefetch_cached_from_handle;
use windmill_common::scripts::{ScriptHash, ScriptLang};
use windmill_queue::asset_dispatch::dispatch_asset_triggers;
use windmill_queue::cascade::reap_stale_join_slots;
@@ -586,22 +585,41 @@ async fn debounce_setting_applied_to_dispatched_subscriber(
let r = dispatch_asset_triggers(&db, &make_mini(id, PRODUCER)).await;
assert_eq!(r.dispatched.len(), 2, "both subscribers dispatched");
// Resolve the persisted debounce window straight from this test's own
// (isolated) DB by walking the handle chain
// v2_job_queue.runnable_settings_handle → runnable_settings.debouncing_settings
// → debouncing_settings. Reading the rows directly rather than through
// `prefetch_cached_from_handle` keeps the assertion off the process-global
// runnable-settings cache (and its tempdir-backed file I/O), which is
// shared by every test running concurrently in this binary — a needless
// cross-test coupling for what is purely a "was the handle wired through to
// the queued job" check. An undebounced subscriber has a NULL handle, so
// the inner joins yield no row → (None, None).
async fn debounce_of(
db: &Pool<Postgres>,
path: &str,
) -> anyhow::Result<(Option<i32>, Option<String>)> {
let handle = sqlx::query_scalar!(
r#"SELECT q.runnable_settings_handle
FROM v2_job j JOIN v2_job_queue q ON q.id = j.id
WHERE j.workspace_id = $1 AND j.runnable_path = $2
AND j.trigger_kind = 'asset'"#,
WS,
path,
use sqlx::Row;
let row = sqlx::query(
r#"SELECT ds.debounce_delay_s, ds.debounce_key
FROM v2_job j
JOIN v2_job_queue q ON q.id = j.id
JOIN runnable_settings rs ON rs.hash = q.runnable_settings_handle
JOIN debouncing_settings ds ON ds.hash = rs.debouncing_settings
WHERE j.workspace_id = $1 AND j.runnable_path = $2
AND j.trigger_kind = 'asset'"#,
)
.fetch_one(db)
.bind(WS)
.bind(path)
.fetch_optional(db)
.await?;
let (deb, _conc) = prefetch_cached_from_handle(handle, db).await?;
Ok((deb.debounce_delay_s, deb.debounce_key))
Ok(match row {
Some(r) => (
r.try_get::<Option<i32>, _>("debounce_delay_s")?,
r.try_get::<Option<String>, _>("debounce_key")?,
),
None => (None, None),
})
}
let (deb_delay, deb_key) = debounce_of(&db, SUB_S3).await?;
+80
View File
@@ -0,0 +1,80 @@
//! Regression test for NUL bytes in draft values.
//!
//! `draft.value` is a `json` column (not `jsonb`), so a U+0000 escape can be
//! stored and then make any `->>`/`to_jsonb` extraction raise `22P05` — one
//! poisoned draft 500'd `GET /drafts/list` (silently hiding the home-page
//! "This workspace has N drafts" banner). The fix sanitizes the value on write
//! (`update_draft` -> `strip_json_nul`) so a NUL never reaches the column; this
//! drives the real endpoint and asserts the stored + listed value is NUL-free.
use serde_json::{json, Value};
use sqlx::{Pool, Postgres};
use windmill_test_utils::*;
fn client() -> reqwest::Client {
reqwest::Client::new()
}
fn authed(b: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
b.header("Authorization", "Bearer DNUL_ADMIN_TOKEN")
}
#[sqlx::test(fixtures("drafts_nul"))]
async fn test_draft_write_strips_nul(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let base = format!("http://localhost:{port}/api/w/dnul-ws");
// Save a draft whose summary and content carry a real NUL.
let resp = authed(client().post(format!(
"{base}/drafts/update/script/u/dnul-admin/poison"
)))
.json(&json!({
"value": {
"summary": "hi\u{0}there",
"path": "u/dnul-admin/poison",
"content": "x\u{0}y"
}
}))
.send()
.await?;
assert_eq!(
resp.status(),
200,
"save should succeed: {}",
resp.text().await.unwrap_or_default()
);
// The stored value must be NUL-free (sanitized on write).
let stored: Value = authed(client().get(format!(
"{base}/drafts/get_own/script/u/dnul-admin/poison"
)))
.send()
.await?
.json()
.await?;
let value = stored.get("value").expect("draft should exist");
assert_eq!(value["summary"], "hithere");
assert_eq!(value["content"], "xy");
assert!(
!serde_json::to_string(value).unwrap().contains("\\u0000"),
"stored value still contains a NUL escape: {value}"
);
// The list endpoint uses raw `->>`; it works (200, no 500) because the
// stored data is clean, and the summary comes back stripped.
let items: Vec<Value> = authed(client().get(format!("{base}/drafts/list")))
.send()
.await?
.json()
.await?;
let item = items
.iter()
.find(|d| d["path"] == "u/dnul-admin/poison")
.expect("saved draft should be listed");
assert_eq!(item["summary"], "hithere");
Ok(())
}
+24
View File
@@ -0,0 +1,24 @@
-- Fixture for the draft NUL-byte write-sanitization regression test.
-- Just a workspace + admin user + token; the test itself POSTs a draft whose
-- value carries a U+0000 and asserts it is stored (and listed) NUL-free.
INSERT INTO workspace (id, name, owner) VALUES
('dnul-ws', 'DNUL WS', 'dnul-admin');
INSERT INTO workspace_key (workspace_id, kind, key) VALUES
('dnul-ws', 'cloud', 'dnul-key');
INSERT INTO workspace_settings (workspace_id) VALUES
('dnul-ws');
INSERT INTO group_ (workspace_id, name, summary, extra_perms) VALUES
('dnul-ws', 'all', 'All users', '{}');
INSERT INTO password(email, password_hash, login_type, super_admin, verified, name, username)
VALUES ('dnul-admin@windmill.dev', 'x', 'password', true, true, 'DNUL Admin', 'dnul-admin');
INSERT INTO usr(workspace_id, email, username, is_admin, role) VALUES
('dnul-ws', 'dnul-admin@windmill.dev', 'dnul-admin', true, 'Admin');
INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin)
VALUES (encode(sha256('DNUL_ADMIN_TOKEN'::bytea), 'hex'), 'DNUL_ADMIN', 'DNUL_ADMIN_TOKEN', 'dnul-admin@windmill.dev', 't', true);
+39
View File
@@ -19,6 +19,45 @@ INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, sc
ARRAY['jobs:read', 'if_jobs:filter_tags:deno']
);
-- App embed token for the admin viewer (test-user). Mirrors a minted sandboxed
-- low-code app token: carries the `app_embed` sentinel plus the embed scope set.
-- Used to assert the token is confined to jobs the viewer LAUNCHED, not every job
-- the (admin) viewer could otherwise read.
INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, scopes) VALUES (
encode(sha256('EMBED_APP_TOKEN'::bytea), 'hex'), 'EMBED_APP_', 'EMBED_APP_TOKEN',
'test@windmill.dev', 'app embed token', false,
ARRAY['apps:run', 'jobs:read', 'app_embed', 'resources:run', 'users:read', 'folders:read']
);
-- A completed app-component job LAUNCHED BY the admin viewer (created_by =
-- test-user), running as the app owner. The embed token must keep reading its own
-- launched job (the `created_by == viewer` fast path).
INSERT INTO public.v2_job (
id, workspace_id, created_by, created_at, permissioned_as, permissioned_as_email,
kind, script_lang, runnable_path, tag, visible_to_owner, args
) VALUES (
'12121212-1212-1212-1212-121212121212', 'test-workspace', 'test-user',
'2023-01-01 00:00:00', 'u/test-user-2', 'test2@windmill.dev',
'script', 'deno', 'u/test-user-2/app_component', 'deno', false,
'{"own": "arg"}'
);
INSERT INTO public.v2_job_completed (id, workspace_id, duration_ms, status, result) VALUES
('12121212-1212-1212-1212-121212121212', 'test-workspace', 1000, 'success'::job_status,
'{"own": "EMBED_OWN_RESULT"}');
-- A QUEUED job launched by the admin embed viewer (created_by = test-user). The
-- embed token may cancel its own launched job; it must NOT cancel another user's.
INSERT INTO public.v2_job (
id, workspace_id, created_by, created_at, permissioned_as, permissioned_as_email,
kind, script_lang, runnable_path, tag, visible_to_owner
) VALUES (
'13131313-1313-1313-1313-131313131313', 'test-workspace', 'test-user',
'2023-01-01 00:00:00', 'u/test-user-2', 'test2@windmill.dev',
'script', 'deno', 'u/test-user-2/app_component', 'deno', false
);
INSERT INTO public.v2_job_queue (id, workspace_id, scheduled_for, running, tag) VALUES
('13131313-1313-1313-1313-131313131313', 'test-workspace', '2023-01-01 00:00:00', false, 'deno');
-- RUNNING job: queued (no completed row) and owned by test-user-2. Used to check
-- that `completed/get_result_maybe?get_started=true` authorizes before disclosing
-- running-state to a non-reader.
+86
View File
@@ -38,6 +38,10 @@ const TOP_SECRET_FLOW: &str = "ffffffff-ffff-ffff-ffff-ffffffffffff";
const DEEP_LEAF_JOB: &str = "88888888-8888-8888-8888-888888888888";
// A queued/running job (no completed row) owned by test-user-2.
const RUNNING_JOB: &str = "77777777-7777-7777-7777-777777777777";
// An app-component job launched BY the admin embed viewer (created_by test-user).
const EMBED_OWN_JOB: &str = "12121212-1212-1212-1212-121212121212";
// A QUEUED job launched by the embed viewer (created_by test-user) — cancelable by it.
const EMBED_OWN_QUEUED: &str = "13131313-1313-1313-1313-131313131313";
// Secrets that must never leak to an unauthorized viewer.
const RESULT_SECRET: &str = "RESULT_SECRET";
@@ -59,6 +63,19 @@ async fn get(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCod
(status, body)
}
async fn post(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCode, String) {
let mut req = client()
.post(format!("{base}/{path}"))
.json(&serde_json::json!({}));
if let Some(token) = token {
req = req.header("Authorization", format!("Bearer {token}"));
}
let resp = req.send().await.expect("request");
let status = resp.status();
let body = resp.text().await.expect("body");
(status, body)
}
#[sqlx::test(fixtures("base", "jobs_read_auth"))]
async fn test_single_job_read_authorization(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
@@ -281,6 +298,75 @@ async fn test_single_job_read_authorization(db: Pool<Postgres>) -> anyhow::Resul
"top flow in an unreadable folder must stay denied (got {status}): {body}"
);
// ---- APP EMBED TOKEN: confined to jobs the viewer LAUNCHED, not everything
// the (admin) viewer can otherwise read. The token carries the `app_embed`
// sentinel; an admin's normal token reads VICTIM (asserted above), but the
// embed token must stop at the `created_by == viewer` grant so user-authored
// app JS can't reuse it to read unrelated jobs by UUID.
// Its own launched component job (created_by == viewer) still reads.
let (status, body) = get(
&base,
&format!("completed/get_result/{EMBED_OWN_JOB}"),
Some("EMBED_APP_TOKEN"),
)
.await;
assert!(
status.is_success(),
"embed token must read a job it launched (got {status}): {body}"
);
assert!(
body.contains("EMBED_OWN_RESULT"),
"embed token should get its own launched job result: {body}"
);
// The VICTIM job — created by another user but readable by this admin viewer's
// normal token (asserted above) — is denied to the embed token across result /
// logs / live update. NotFound (not 403) so the untrusted app can't even probe
// existence, and no secret leaks.
for path in [
format!("completed/get_result/{VICTIM}"),
format!("get_logs/{VICTIM}"),
format!("getupdate/{VICTIM}?only_result=true"),
] {
let (status, body) = get(&base, &path, Some("EMBED_APP_TOKEN")).await;
assert_eq!(
status,
reqwest::StatusCode::NOT_FOUND,
"embed token must not read a job it did not launch ({path}, got {status}): {body}"
);
for secret in [RESULT_SECRET, ARGS_SECRET, LOGS_SECRET] {
assert!(
!body.contains(secret),
"embed token response for {path} leaked `{secret}`: {body}"
);
}
}
// ---- APP EMBED TOKEN: cancellation confined to the app's own jobs. The token
// may cancel a job it launched (created_by == viewer), but `cancel_job_api`
// denies (NotFound) a job created by someone else, even though cancel
// otherwise has no per-job ownership check.
let (status, body) = post(
&base,
&format!("queue/cancel/{EMBED_OWN_QUEUED}"),
Some("EMBED_APP_TOKEN"),
)
.await;
assert!(
status.is_success(),
"embed token must cancel a job it launched (got {status}): {body}"
);
let (status, body) = post(
&base,
&format!("queue/cancel/{RUNNING_JOB}"),
Some("EMBED_APP_TOKEN"),
)
.await;
assert_eq!(
status,
reqwest::StatusCode::NOT_FOUND,
"embed token must not cancel another user's job (got {status}): {body}"
);
// ---- UNAUTHENTICATED, unchanged: an anonymous-created job is readable
// without a token (public trigger / public app result polling).
let (status, body) = get(&base, &format!("completed/get_result/{ANON_JOB}"), None).await;

Some files were not shown because too many files have changed in this diff Show More