starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-08-19 00:02:03 +00:00
Code Issues Packages Projects Releases Wiki Activity
14,432 Commits 725 Branches 1,509 Tags
fix/hook-tests-worktree-root
Commit Graph

20 Commits

Author SHA1 Message Date
Ruben Fiszel 81b23a2ba0 feat: make the fork lineage the only deploy relationship (#10410) 
* feat: make the fork lineage the only deploy relationship

`workspace_settings.deploy_to` (2023) and `workspace.parent_workspace_id` (2025)
both expressed "which workspace does this one deploy into". Fork creation and
dev-workspace attach seeded both, but nothing kept them in agreement, so every
reader picked one and they disagreed.

Drop `deploy_to`. A migration folds surviving pairs into the lineage: a sole
claimant on a target with no dev workspace becomes that target's dev workspace
and keeps its own job tags, while many-to-one pairs become plain forks. Pairs
that the lineage cannot express -- dangling target, self-reference, chain,
mutual -- are reported and left unlinked.

Job tags were never lineage-aware: `per_workspace_tag` mapped any parented
workspace to its parent while `$workspace` interpolated the raw id, so a fork
running a script tagged `<tag>-$workspace` produced a tag no worker serves and
the job queued forever. Both paths now resolve to the nearest ancestor whose id
an admin would provision workers for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: preserve unconvertible deploy links and sweep tag caches on reparent

Review findings on the deploy_to unification:

- convert chains instead of discarding them, and keep whatever the lineage
  cannot express in workspace_deploy_to_unmigrated so the down migration can
  restore it
- ignore soft-deleted workspaces when choosing between a dev workspace and a
  plain fork; an archived claimant was demoting live pairs
- mirror attach_dev_workspace's git-sync strip, which the migration skipped
- sweep the tag cache over whole subtrees on rename and delete: tag resolution
  now walks ancestors, so a nested fork kept a tag nothing serves
- call a dev workspace a dev workspace in the settings copy
- redirect a root away from ?tab=deploy_to instead of rendering an empty target

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: detect lineage cycles and record archived links in the deploy_to migration

Second review round on the unification:

- detect cycles over the lineage as it would exist after conversion, not over
  the deploy_to graph alone: a root whose target was one of its own forks
  closed a loop that no deploy_to edge revealed
- record an archived source's link instead of filtering it out entirely, which
  dropped it with the column
- treat a fork whose deploy_to merely repeats its parent as redundant rather
  than reporting every pre-existing fork as unmigrated
- read the row count from the lineage update rather than the git-sync one
- sweep the tag cache when archiving a dev workspace, the last site that
  mutates is_dev_workspace without one

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: resolve $workspace on preprocessed flow tags regardless of $args

Third review round on the unification:

- a flow tag containing only `$workspace` skipped interpolation entirely on the
  preprocessed path, because the branch that ran it keys on `$args`. The raw
  tag was written back and named a queue no worker serves. Resolve `$workspace`
  before the branch and leave `$args` to it.
- record the new table's foreign key in the schema summary
- describe what the archive tag sweep actually does: the dev flag is cleared for
  any archived workspace, which is why it is unconditional

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: keep the deploy_to leftovers table only when it holds something

* fix: sweep tag caches on archive only where the dev flag actually changes

* feat: broadcast lineage changes and walk ws_specific ancestors only

- propagate tag-cache invalidation across processes over notify_events: the
  cache is per-process, so replicas kept resolving stale lineage for the TTL.
  The listener clears the whole cache rather than tracking ids, since a single
  mutation invalidates an unbounded set of descendants and lineage changes are
  rare admin actions.
- narrow list_ws_specific_versions to ancestors: walking down as well made a
  root fan out over its entire live fork subtree, and each member costs an
  identity lookup plus an RLS switch and probe. Ancestors are bounded by the
  fork depth limit.
- probe the leftovers table unqualified so rollback restores on a PG_SCHEMA
  install, where search_path is not public
- drop the nativets client method for the removed edit_deploy_to endpoint

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: let a prod see its dev workspace in ws_specific, and stop the walk oscillating

Descending into plain forks made a root fan out over its whole live fork
subtree, but a dev workspace is the paired editable environment rather than a
throwaway copy, so a prod should still see it. There is at most one per parent
and attach rejects nested dev chains, so that edge stays bounded.

The edges run both ways, so the recursion never converged: it bounced
parent<->dev until the depth cap on every call, 33 rows for a two-member set.
A visited-path guard ends the walk when nothing new is reachable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: keep dev pairings unnested, gate the delete broadcast, cover the ws_specific walk

Fifth review round:

- a root that already owns a dev workspace no longer converts: linking it under
  its deploy target would leave that dev nested beneath a fork, the shape
  attach_dev_workspace refuses to create. The link is preserved instead.
- broadcast a lineage change on delete only when descendants are orphaned.
  Deleting a leaf, which ephemeral fork churn does constantly, changes nobody
  else's resolution and was making every replica drop its whole tag cache.
- call list_ws_specific_versions in a test. plpgsql defers everything past a raw
  parse to the first call, so replaying the migration only proved it parses.
- use unwrap_or_default for the descendant sweeps, which run after the
  transaction has committed; a transient failure must not fail the request
- trim the traversal comment to the four-line limit

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: cache the renamed tally query and clear instance alerts on conversion

The integration test's query was never cached: `cargo sqlx prepare` without
--all-targets skips test targets entirely, and renaming its fixture workspace
changed the query text. Regenerated with --all-targets --features
all_sqlx_features,private, which is what lets the EE-gated otel test compile.

Also from review:
- clear error_handler_fallback_to_instance_alerts on converted workspaces.
  Dispatch ignores it once a parent exists, but the settings page keeps
  submitting the stored true, which the API rejects on a fork.
- restore the schema summary row to the file's name: columns format and put it
  back in alphabetical order

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: never cache an unresolvable tag workspace, and unadvertise the removed endpoint

- lookup_tag_workspace cached a "no row" result as self-resolution. A rename
  resolves the new id before its row lands, so a fork could be pinned to its own
  wm-fork-* id -- which nothing serves -- for the whole TTL, and its schedules
  kept re-pushing onto that dead tag. Fall back for the call without caching,
  matching how the error path already behaved.
- change_workspace_id swept its children but never itself. Sweep the new and old
  ids and broadcast unconditionally, since a rename always changes lineage.
- openapi-deref.{json,yaml} are served to clients via include_str!, so they were
  advertising edit_deploy_to after it started 404ing. The audit-action enum
  keeps the entry: historical rows still carry it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: align the served YAML spec with the JSON one and correct two comments

- the YAML deref lost the removed path but kept deploy_to on get_settings,
  so the two served specs disagreed. Both are now identical.
- the rename-sweep comment blamed cached-unresolvable lookups, which the same
  commit stopped caching. The real reason is that workspace ids are
  reclaimable, so a new id can carry a previous occupant's resolution.
- the instance-alert comment claimed the settings page submits the stored true
  and gets a 400. It hides the option on a fork and sends false; the hazard is
  the value outliving the pairing and re-enabling alerts after a detach.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to 82da6cb2bafeda18acd6b70c599013a12117ecb0

This commit updates the EE repository reference after PR #694 was merged in windmill-ee-private.

Previous ee-repo-ref: f9ddf6a75aa13d1c13a3d7216a361a96f75ca435

New ee-repo-ref: 82da6cb2bafeda18acd6b70c599013a12117ecb0

Automated by sync-ee-ref workflow.

* fix: grant the deploy_to preservation table to the windmill roles

* test: drop the one-shot migration tests, keep the ws_specific execution guard

The two conversion tests replayed the migration against the fully-migrated
schema, which is not how it runs -- in production it runs mid-sequence against
the schema as of that point. A later migration touching workspace or
workspace_settings would break them without breaking anything real, and sqlx
checksums already freeze a released migration. They earned their keep finding
the archived-claimant and nested-dev cases during development; there is nothing
left for them to guard.

list_ws_specific_versions is different: it is live, no caller exercises it, and
plpgsql only parses a function body until first call.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: invalidate a reclaimed fork id cluster-wide without flushing every entry

Gating the delete broadcast on orphaned descendants stopped leaf churn flushing
every replica, but fork ids are reclaimable: the deleting process invalidated
locally while every other replica kept the old parent for the TTL, so a job
pushed in a recreated fork routed to the previous parent's tag.

The broadcast payload now carries meaning. A workspace id drops that one entry,
used for leaf deletion where exactly one id changed what it denotes. The `*`
sentinel drops everything, used for attach, detach, archive, rename and
deletions that orphan descendants -- reshaping a subtree no single id names.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: name the right broadcast for each invalidation case

* docs: attach does invalidate the tag cache; the resolver walks the whole chain

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2026-07-30 14:20:27 +00:00
Ruben Fiszel 6d1e12d5e9 feat(nativets): expose the standard web-platform globals deno_web provides (#10112) 
* feat(nativets): expose standard web-platform globals for bun parity

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(nativets): wire bun-present Event subclasses and add construction smoke test

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(nativets): seed performance.timeOrigin per isolate, drop broken reportError

Addresses CI Codex review on #10112:
- performance.timeOrigin was undefined (setTimeOrigin never called); seed it
  per isolate via __wmInitPerIsolate executed from create_nativets_runtime.
- reportError needs a global EventTarget this runtime never installs; drop it.
- reword the namespace-import comment to not describe drafting history.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(nativets): wire DOMException global + broad edge-case smoke sweep

DOMException is present in bun and, more importantly, deno_web references it as
a global: AbortController.abort() with no reason constructs a
DOMException("...", "AbortError"), so the already-wired AbortController/
AbortSignal threw "DOMException is not defined" on abort. Surfaced by a new
functional edge-case sweep (smoke_web_globals_edge_cases) that exercises every
wired global for real (not just presence) — DOMException/abort, AbortSignal.timeout,
EventTarget dispatch, stream tee/reader/writer, all 3 compression formats,
structuredClone Map/Set/Date/circular/reject-function, performance mark/measure,
MessagePort delivery — plus a check that the merged Web Crypto globals still work.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(nativets): restore arg-default smoke tests dropped in merge, drop history comments

Addresses CI Codex/Pi review on the merge commit:
- Merge conflict resolution (checkout --ours) dropped smoke_missing_optional_arg_uses_default
  and smoke_explicit_null_arg_is_preserved (added on main by #10111); restore them.
- Reword edge-case-sweep comments to state the constraint, not how the gaps were found.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(nativets): give reportException a global dispatch target; wire stream reader/controller globals

Addresses CI Codex review on #10112:
- P1: a throwing EventTarget listener (and reportError) is routed through
  deno_web's reportException, which dispatches on a saved global reference.
  With none set, dispatchEvent threw a masking error that hid the original.
  Wire a dedicated EventTarget as that target so the ORIGINAL error is reported
  (async unhandled, matching bun). Does NOT make globalThis an EventTarget (bun's
  isn't either). Re-adds reportError, now functional. Regression test asserts the
  original error is surfaced, not a masking one.
- P2: wire the stream reader/controller globals bun also exposes
  (ReadableStreamDefaultReader/BYOBReader, ReadableStreamDefault/ByteStreamController,
  ReadableStreamBYOBRequest, WritableStreamDefaultWriter/Controller,
  TransformStreamDefaultController) for instanceof parity; sweep verifies via real
  reader/writer/controller instances.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(nativets): make globalThis an EventTarget so globalThis.reportError() works

Addresses follow-up CI Codex review on #10112:
- P1: the prior fix saved a *separate* EventTarget as the global reference, so
  globalThis.reportError() still failed its receiver check (this === globalThis_)
  with 'Illegal invocation'. Make globalThis itself the saved reference by turning
  it into a functional EventTarget (setPrototypeOf to DedicatedWorkerGlobalScope +
  setEventTargetData + webidl brand + saveGlobalThisReference), per isolate in
  __wmInitPerIsolate. Both reportError(e) and globalThis.reportError(e) now surface
  the original error (async, matching bun) instead of throwing. New test
  smoke_report_error_both_call_forms covers both call forms.
- P2: reword the regression-test comment to state the invariant, not the patch history.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(nativets): wire performance constructor globals for bun parity

Addresses the P2 nit in the CI Codex review: bun exposes Performance,
PerformanceEntry, PerformanceMark, and PerformanceMeasure as globals (deno_web
exports all four), so wire them alongside the performance singleton. The
edge-case sweep verifies instanceof against real mark/measure entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(nativets): state global-wiring comment as a constraint, not patch history

Addresses the P2 in the CI Codex review: reword the block comment to describe
the current bun-parity constraint and the deliberate EventSource/ImageData
exclusions, without narrating what was or wasn't wired before (per AGENTS.md).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 09:47:54 +02:00
Ruben Fiszel ba232544e7 feat(nativets): add Web Crypto support via deno_crypto (#10109) 
The nativets in-process runtime (deno_core) exposed no Web Crypto API:
`crypto` was undefined, so scripts could not use `crypto.getRandomValues`,
`crypto.randomUUID`, or `crypto.subtle`, even though the bun runner provides
them. This closes that parity gap by registering the `deno_crypto` extension
and wiring the crypto globals onto `globalThis`.

- Pin `deno_crypto = "0.223.0"`, the sibling release of the already-pinned
  deno_core 0.352 / deno_web 0.240 stack (deps: deno_core ^0.352,
  deno_web ^0.240, deno_error =0.6.1), so the rest of the deno stack is
  untouched.
- Register `deno_crypto::init(None)` after `deno_web` in both the snapshot
  (build.rs) and the runtime (lib.rs) extension lists, keeping the snapshot a
  prefix of the runtime list. deno_crypto declares deps = [deno_webidl,
  deno_web], which the position satisfies.
- Import `ext:deno_crypto/00_crypto.js` in runtime.js and assign
  `crypto` / `Crypto` / `CryptoKey` / `SubtleCrypto` to `globalThis`.
- Add the `smoke_web_crypto` opt-in smoke test asserting the UUIDv4 shape of
  `randomUUID`, a non-zero `getRandomValues` fill, and the known
  SHA-256("abc") vector via `subtle.digest`.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 01:58:56 +02:00
Ruben Fiszel ba7f9c065f fix(nativets): apply parameter defaults for missing args instead of null (#10111) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-15 01:53:43 +02:00
Ruben Fiszel 41562c7d7c fix(nativets): respect custom CA certs in in-process fetch runtime (#9615) 
* fix(nativets): respect custom CA certs in in-process fetch runtime

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(nativets): dedupe CA file paths and clarify DENO_TLS_CA_STORE semantics

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(nativets): resolve CA env vars from worker-group config too

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-16 14:39:36 +02:00
Ruben Fiszel 6637e00375 test(nativets): cover deno_telemetry init contract + http trace export E2E (#9165) 2026-05-14 10:54:19 +00:00
Ruben Fiszel 3cd0eac8c1 deps: bump deno_core / deno_ast / swc to the goldilocks pin set; drop serde ceiling (#9111) 
* deps: bump deno_core / deno_ast / swc to the goldilocks pin set; drop serde ceiling

Bumps every deno_* and swc_* workspace dep to a hand-picked "goldilocks"
combination that drops the serde =1.0.220 ceiling without crashing into
the rustls / aws-sdk resolver wall that the obvious deno v2.6.0 target
hits.

## What's the goldilocks set

| crate            | old      | new       | source                                |
|------------------|----------|-----------|---------------------------------------|
| deno_core        | 0.336.0  | 0.352.0   | deno v2.4.0                           |
| deno_fetch       | 0.214.0  | 0.233.0   | deno v2.4.0                           |
| deno_tls         | 0.177.0  | 0.196.0   | deno v2.4.0 (last permissive-rustls)  |
| deno_console     | 0.190.0  | 0.209.0   | deno v2.4.0                           |
| deno_url         | 0.190.0  | 0.209.0   | deno v2.4.0                           |
| deno_webidl      | 0.190.0  | 0.209.0   | deno v2.4.0                           |
| deno_web         | 0.221.0  | 0.240.0   | deno v2.4.0                           |
| deno_io          | 0.100.0  | 0.119.0   | deno v2.4.0                           |
| deno_net         | 0.182.0  | 0.201.0   | deno v2.4.0                           |
| deno_permissions | 0.49.0   | 0.68.0    | deno v2.4.0                           |
| deno_telemetry   | 0.12.0   | 0.31.0    | deno v2.4.0                           |
| deno_error       | =0.5.5   | =0.6.1    | deno v2.4.0                           |
| deno_ast         | =0.44.0  | =0.51.0   | **override** — see "load-bearing" below |
| deno_fs          | (new)    | 0.119.0   | new workspace dep — FetchPermissions exposes deno_fs::CheckedPath / GetPath as public API |
| v8               | =130.0.7 | =137.1.0  | deno_core 0.352 transitive            |
| swc_common       | =0.37.5  | =14.0.4   | **the load-bearing pin**              |
| swc_ecma_ast     | =0.118.2 | =15.0.0   | matched set with swc_common 14.0.4    |
| swc_ecma_parser  | =0.149.1 | =24.0.3   | matched set                           |
| swc_ecma_visit   | =0.104.8 | =15.0.0   | matched set                           |
| serde            | =1.0.220 | ^1        | **freed** (resolves to 1.0.228+)      |

## Why this combination and not v2.6.0

The obvious target was deno v2.6.0 (with deno_ast 0.52 → swc_common 17,
well past the `__private` ceiling). That hits three resolver collisions:

1. libsqlite3-sys: deno_cache → rusqlite 0.37 → libsqlite3-sys 0.35
   vs sqlx → libsqlite3-sys 0.30. **Already killed by PR #9110** —
   we dropped deno_runtime, which is what pulled in deno_cache.
2. fqdn 0.4.6/0.4.7 yanked, required by deno_permissions 0.81.0. Solvable
   by injecting the yanked entry into Cargo.lock manually but ugly.
3. rustls: deno_tls 0.198+ hard-pins `=0.23.28`, but aws-sdk-bedrockruntime
   1.122.0 → aws-smithy-http-client 1.1.5 wants `^0.23.31`. Within-major
   conflict, no resolver path. The unbeatable wall.

Goldilocks-set choice sidesteps (2) and (3) entirely:

- `deno_tls 0.196.0` was the last version before deno tightened
  `rustls ^0.23.11` (range, accepts 0.23.31) to exact `=0.23.28`. With
  ^0.23.11, the resolver picks rustls 0.23.35 (latest 0.23 patch) which
  satisfies both deno_tls's `>=0.23.11` and aws-sdk's `>=0.23.31`. Verified
  empirically: lockfile has rustls 0.23.35 after this bump.
- `deno_permissions 0.68.0` (v2.4.0's pin) doesn't depend on fqdn at all.
  The fqdn dep was added in a later deno_permissions release.

## Why deno_ast =0.51.0 specifically (not 0.48.0 from v2.4.0)

`swc_common 14.0.4` is the first patch that **drops the
`pub use serde::__private as serde;` line** in `src/private/mod.rs`. Older
14.0.x and all 0.37.5–13.x revisions still have it, and that line is
what was capping `serde = "=1.0.220"` (the workspace pin's "stuck because
of swc" comment). Empirically verified by inspecting the tarballs of
14.0.0 / 14.0.1 / 14.0.2 / 14.0.3 / 14.0.4:

    14.0.0: has hack
    14.0.1: has hack
    14.0.2: has hack
    14.0.3: has hack
    14.0.4: NO HACK    ← inflection point

`deno_ast 0.51.0` pins `swc_common =14.0.4` exactly — older deno_ast
versions pin earlier swc_common patches that still have the hack.
Notably, deno v2.4.0 itself pins `deno_ast =0.48.0` (swc_common 9.2.0,
still has hack) — we deliberately deviate from v2.4.0's deno_ast pin
to escape the swc serde wall, while keeping the rest of v2.4.0's pin
set for resolver compatibility with aws-sdk. deno_ast 0.51 was never
shipped in any deno release (v2.4.5 used 0.49, v2.5.0 jumped to 0.50,
v2.6.0 to 0.52), but it's published on crates.io and compatible with
v2.4.0's deno_core 0.352.

## What this unblocks

- PR #9106's `serde = "=1.0.224"` bump variant can rebase onto this
  and resolve cleanly (MaterializeInc/rust-postgres' `postgres-types`
  needs `serde_core ^1.0.221`, which is satisfied now that we're on
  serde 1.0.228).
- Future deno_* / swc_* bumps no longer need to argue about the serde
  ceiling — it's gone.

## What changes in source code

This commit is Cargo.toml + Cargo.lock only. Source changes that the
new deno_core / deno_fetch API requires live in the follow-up commits:

- `parsers/windmill-parser-{ts,ts-asset,wac}`: swc 0.37 → 14
  (`code.into()` ambiguity fix at 5 sites)
- `windmill-runtime-nativets/build.rs` + `src/lib.rs`: deno_core 0.336
  → 0.352 API moves (`init_ops_and_esm()` → `init()`,
  `FetchPermissions` / `NetPermissions` trait signature updates,
  `deno_tls::Proxy` enum shape change)

A companion change in windmill-ee-private adjusts
`otel_tracing_proxy_ee.rs:521` for `deno_telemetry::init`'s second arg
becoming by-value (was `&OtelConfig`).

* fix(parsers): adapt to swc_common 14 BytesStr ambiguity

swc 0.37.5 → 14.0.4 changed `SourceMap::new_source_file`'s `src` argument
from `String` to `impl Into<BytesStr>`. With `BytesStr` available, the
existing call sites' `code.into()` on a `&str` becomes ambiguous between
`Into<Bytes>` (from the bytes crate) and `Into<BytesStr>` (from
bytes_str). Switch to `code.to_string()` to produce an owned `String`
that satisfies `From<String> for BytesStr` unambiguously.

Five call sites across three crates:
- windmill-parser-ts/src/lib.rs (3 sites)
- windmill-parser-ts-asset/src/lib.rs (1 site)
- windmill-parser-wac/src/typescript.rs (1 site)

* fix(nativets): adapt to deno_core 0.352 / deno_fetch 0.233 API changes

The goldilocks deno bump (deno_core 0.336 → 0.352, deno_fetch 0.214 →
0.233, etc.) ripples through nativets' build.rs and src/lib.rs.
Source-level changes required:

## 1. `extension!` macro: `init_ops_and_esm()` and `init_ops()` removed

deno_core 0.352's `extension!` macro now generates a single `init()`
function on the extension struct (full: ops + esm), plus `lazy_init()`
(ops only, with `needs_lazy_init = true` and a contract that the
caller invokes `JsRuntime::lazy_init_extensions` after construction).

- `build.rs` (snapshot creation, wants both ops and esm baked in):
  `X::init_ops_and_esm(...)` → `X::init(...)`.
- `src/lib.rs:create_nativets_runtime` (runtime, was using `init_ops()`
  because the snapshot already provides esm): also → `X::init(...)`.
  deno_core's snapshot path skips esm re-execution when the snapshot
  provides them, so the esm registration is a no-op at runtime. This
  is how deno's own v2.4.0 runtime works.

Avoided `lazy_init` because it requires plumbing
`JsRuntime::lazy_init_extensions(ext_args_vec)` correctly across the
codebase, which is invasive for no behavioural benefit.

## 2. Local `fetch` extension now declared in both build.rs and lib.rs

deno_core 0.352 validates extension order between snapshot and runtime.
Our snapshot's last extension is the local `fetch` ext (which provides
ext:fetch/src/runtime.js). To avoid a runtime panic:

    "Extensions from snapshot loaded in wrong order: expected fetch but got windmill"

…the runtime extension list now ends with `fetch::init()` matching the
snapshot order. The macro requires the same `esm` argument to type-check,
even though the ESM is not re-executed at runtime (it's in the snapshot).

## 3. `FetchPermissions` and `NetPermissions` trait shape

`deno_fetch::FetchPermissions` (deno_fetch 0.233.0) added new methods
and changed signatures:

- `check_read` / `check_write`: now take `path: Cow<'a, Path>` plus
  a new `get_path: &'a dyn deno_fs::GetPath` parameter, and return
  `Result<deno_fs::CheckedPath<'a>, FsError>` instead of `Result<Cow<Path>, FsError>`.
- New `check_write` (didn't exist) and `check_net_vsock` methods.

`deno_net::NetPermissions` (deno_net 0.201.0) gained `check_vsock`
and `check_write_path` now takes `Cow<'_, Path>`.

For `build.rs`'s `PermissionsContainer` (used only during snapshot
creation, where permissions are never actually checked): all methods
`unreachable!("snapshotting")`.

For `src/lib.rs`'s `PermissionsContainer` (used at runtime — the
nativets policy is "allow everything"): `check_read` / `check_write`
return `Ok(CheckedPath::Unresolved(path))`, `check_*_vsock` return
`Ok(())`. Smoke tests confirm fetch/net/url/web/blob/timers/structuredClone
behaviour is intact end-to-end.

## 4. `deno_tls::Proxy` is now an enum

`deno_tls::Proxy` was a struct, is now an enum with `Http`, `Https`,
`Socks5` variants. Our call site uses HTTP proxies — switched the
struct literal `deno_tls::Proxy { url, basic_auth }` to
`deno_tls::Proxy::Http { url, basic_auth }`.

## 5. New `deno_fs` direct workspace dep

`FetchPermissions` exposes `deno_fs::CheckedPath` and `deno_fs::GetPath`
as part of its public API. We can't avoid naming `deno_fs` directly any
more. Pinned to 0.119.0 (v2.4.0's matched version, transitively present
already through deno_fetch). Added to workspace `[dependencies]` plus
nativets's `[dependencies]` and `[build-dependencies]`.

## Validation

`cargo check --features enterprise,deno_core,duckdb,license,python,rust,scoped_cache,parquet,private,private_registry_test,csharp,php,ruby,mysql,quickjs,mcp,run_inline`
→ clean.

`cargo test -p windmill-runtime-nativets smoke -- --ignored --skip smoke_net_`
→ 8 passed; 0 failed (the full local smoke suite covering fetch,
setTimeout/Promise.all, URL/SearchParams, Blob/btoa/atob, large payload
roundtrip, error propagation, concurrent isolates, TS enum/union
transpile).

Network smoke tests (`smoke_net_fetch_example_com`,
`smoke_net_fetch_json_and_headers`) not run as part of the validation
gate but expected to pass — the change preserves deno_fetch behaviour
through the trait surface.

* chore: update ee-repo-ref to pick up deno_telemetry::init by-value fix

Points at windmill-ee-private branch deps/bump-deno-and-swc-goldilocks
which contains the companion otel_tracing_proxy_ee.rs adjustment for
deno_telemetry 0.12 → 0.31 (second arg of `init` is now by-value).
EE-only file, doesn't affect OSS build.

* chore(nix): bump rusty_v8 in flake.nix to 137.1.0 to match Cargo.toml

Cargo.toml's v8 pin moved from =130.0.7 to =137.1.0 as part of the
deno_core 0.336 → 0.352 bump, but I missed the comment directly above
the version pin:

    # Exact version NOTE: Do not forget to update version and hash in flake.nix

flake.nix provides the prebuilt librusty_v8 binary that the v8 crate
links against. A version mismatch would either fail to fetch (if the
137.1.0 release didn't exist) or cause link-time symbol mismatches.
Nix is used by rust-client-check.yml and rust_on_release.yml in CI,
plus the dev shell — stale flake pin breaks all of those.

Updates x86_64-linux's sha256 to match the actual hash of
librusty_v8_release_x86_64-unknown-linux-gnu.a.gz at the 137.1.0 tag.
Other targets (aarch64-linux, x86_64-darwin, aarch64-darwin) remain
as lib.fakeHash — they were already placeholders in the previous
pin, so we don't regress on them.

Caught by both cubic and Pi reviewers on PR #9111.

* docs(nativets): clarify snapshot-prefix rule in extension-order comment

Claude reviewer caught that the doc comment claimed the runtime
extension list matches the snapshot's order — implying an exact match.
The truth is more permissive: deno_core 0.352 requires the snapshot's
extension list to be a *prefix* of the runtime's, not an exact match.
Runtime is allowed to append extra extensions (which we do — the
windmill `ext` carrying our ops is the last entry at runtime but absent
from the snapshot).

The code is correct as-is; only the comment wording was misleading.

Also fixes the same wording in PR description.
 2026-05-11 21:13:24 +00:00
Ruben Fiszel 36b316d9e8 deps(nativets): inline maybe_transpile_source, drop deno_runtime (#9110) 
windmill-runtime-nativets was the workspace's only consumer of the
deno_runtime crate, and its only use of it was one call site in
build.rs:

    deno_runtime::transpile::maybe_transpile_source(specifier, source)

That function (`deno_runtime-0.198.0/transpile.rs`, ~80 lines) is a pure
deno_ast + deno_core + deno_error wrapper — it doesn't touch any
deno_runtime state. Inline it verbatim into our build.rs and drop the
entire deno_runtime dep.

Why this matters now: deno_runtime transitively pulls in deno_cache →
rusqlite → libsqlite3-sys. From deno_cache 0.128.0 (Feb-Mar 2025)
onwards, rusqlite was bumped to ^0.34, which means libsqlite3-sys ^0.35.
sqlx 0.8 transitively requires libsqlite3-sys ^0.30. Cargo's `links =
"sqlite3"` rule allows only one libsqlite3-sys in a build graph, so the
two crates collide on any deno release ≥ v2.5. Inlining the transpile
helper sidesteps the collision entirely — sqlx-sqlite stays the sole
libsqlite3-sys consumer at 0.30.1.

All other appearances of "deno_runtime" in the source tree are for a
Windmill-internal function named `setup_deno_runtime`, not the crate.

Build artifacts validated:
- `cargo check --features quickjs` → green.
- `cargo test -p windmill-runtime-nativets smoke -- --ignored --skip smoke_net_`
  → 8 passed (the in-process V8 runtime + deno_fetch + deno_web + swc
  transpilation surface still works end-to-end through the inlined
  function).
- `cargo tree --invert deno_cache` → "did not match any packages"
  (gone from the graph).
- Single `libsqlite3-sys` entry in Cargo.lock at 0.30.1 (sqlx's).
 2026-05-11 20:21:46 +00:00
Ruben Fiszel 1e89aff2d6 test(nativets): add #[ignore]'d smoke suite for deno_core / deno_ast / swc bumps (#9108) 
Adds ten `#[ignore]`'d integration tests in
backend/windmill-runtime-nativets/src/smoke_tests.rs exercising the JS
API surface that the existing nativets tests in tests/worker.rs don't
reach. Run them when bumping the deno_core / deno_ast / deno_runtime /
swc_* pins in backend/Cargo.toml, not on every CI:

    cargo test -p windmill-runtime-nativets smoke -- --ignored
    # skip network-dependent ones with `--skip smoke_net_`

Why here and not in tests/worker.rs:
windmill-runtime-nativets is the only consumer of the deno_core crate
in the workspace — ScriptLang::Deno spawns the external `deno` binary
via deno_executor.rs, while ScriptLang::Nativets is the only path that
loads the in-process V8 runtime. So a deno_core / deno_ast bump can
only break things downstream of this crate. Co-locating the smoke
tests with the runtime they exercise means they hit the right surface
directly, skip the entire job-queue / worker / API-server stack, and
run in <1s end-to-end (vs. ~30-60s per test for the worker-level
nativets tests).

The tests use the existing `PrewarmedIsolate::spawn` API (already
public for the dedicated-worker path), which gives a clean
"compile-TS → load module → execute main(args) → return JSON" entry
point with no DB or queue plumbing required.

Coverage:
- smoke_basic_value_passing — args binding + return marshaling
- smoke_transpile_enum_and_union — TS-specific syntax (enums,
  discriminated unions, casts) through swc_ecma_parser / swc_ecma_ast
- smoke_set_timeout_and_promise_all — deno_web timer ops + V8
  microtask drain order
- smoke_url_and_searchparams — deno_url surface
- smoke_web_blob_btoa_atob — deno_web Blob + base64 ops
- smoke_large_payload_roundtrip — 512 KB string in/out through the
  op-table boundary
- smoke_error_propagation_with_message — thrown Error must surface
  in PrewarmedResult::Err with original message
- smoke_concurrent_isolates — 8 isolates spawned in parallel from
  the same tokio runtime; catches V8 isolate-setup races
- smoke_net_fetch_example_com — deno_fetch end-to-end against
  example.com
- smoke_net_fetch_json_and_headers — deno_fetch with custom request
  headers + Response.json() against httpbin.org/anything

`structuredClone` is not currently wired into the nativets global —
documented in the smoke_web_blob_btoa_atob test in case that ever
changes.

All ten tests pass locally against the current pinned versions
(deno_core 0.336.0 / deno_ast =0.44.0 / swc_common =0.37.5).
 2026-05-11 18:43:54 +00:00
Ruben Fiszel e732004180 fix(nativets): forward OTEL-prefixed console logs to tracing events (#8937) 
* fix(nativets): forward OTEL-prefixed console logs to tracing events

Nativets jobs run in-process and bypass the handle_child.rs stdout loop
where `OTEL: ` lines are turned into `tracing::event!` calls when
`OTEL_JOB_LOGS=true`. Apply the same prefix handling in the nativets
log receiver so `console.log("OTEL: ...")` reaches the OTEL exporter
like it does for other runtimes.

Moves `OTEL_JOB_LOGS` and `OTEL_PREFIX` into windmill-common so both
crates share the same definition.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(nativets): emit job_log tracing target so logs reach OTEL bridge

For non-native runtimes, `lines_to_stream` → `process_streaming_log_lines`
(EE) emits every stdout line as `tracing::info!(target: "windmill:job_log", ...)`,
which is picked up by the EE `LogContextBridge` and exported to OTEL
(the bridge's filter is `EnvFilter` only, not the targets filter that
drops `windmill:job_log` from stdout/file sinks).

Nativets delivers logs in-process via a channel, so it never goes
through that path and console.log output only reached the Windmill UI.
Emit the same `windmill:job_log` event per line from the nativets log
receiver.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-24 19:42:05 +00:00
Ruben Fiszel 2d18a68099 feat: add scheduled job deletion with configurable retention period (#8753) 
* feat: add scheduled job deletion with configurable retention period

Extends delete_after_use with delete_after_secs to enable configurable
retention periods for job args/result/logs. At completion, jobs can be
scheduled for future deletion via a new job_delete_schedule table,
processed by a monitor task. Supports per-script, per-flow, and
per-flow-step configuration. Backward compatible.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: add integration tests, revert query! macros, fix review issues

- Add integration tests for resolve_delete_after_secs, schedule_job_deletion,
  flow-level and module-level delete_after_secs, backward compat
- Revert sqlx::query() back to sqlx::query!() macros for compile-time safety
- Regenerate sqlx offline cache
- Fix FlowModule/NewScript/FlowValue constructions in all test files
- Fix autoscaling_ee.rs for updated script_path_to_payload return type

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref.txt for autoscaling_ee fix

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: gate cleanup_scheduled_job_deletions behind enterprise feature

Prevents dead_code warning (which CI treats as error via -D warnings)
when compiling without enterprise feature.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: regenerate sqlx cache after merge with main

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review feedback on scheduled deletion

- Monitor: roll back transaction on any cleanup error so schedule rows
  survive for retry on next cycle (instead of best-effort then discard)
- Migration: add FK with ON DELETE CASCADE to job_delete_schedule.job_id
  to prevent orphan rows when jobs are deleted through other means
- Simplify bool-to-Option conversion with .then_some(true)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: stop setting delete_after_use alongside delete_after_secs

No mixed-version deployment scenario exists, so delete_after_secs alone
is sufficient. The backend's resolve_delete_after_secs handles
(None, Some(secs)) correctly without needing delete_after_use set.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: remove delete_after_use from public API surface

Remove delete_after_use from OpenAPI spec, API client, runtime client,
and workspace export. Only delete_after_secs is exposed going forward.

The field remains in Rust backend types with #[serde(skip_serializing)]
for backward-compatible deserialization of existing scripts/flows that
were saved with delete_after_use: true.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to 1d4b7a31fc115d6aba8640f7cd3fd5a01abe6806

This commit updates the EE repository reference after PR #519 was merged in windmill-ee-private.

Previous ee-repo-ref: 9eba09a13b778caafc6ae65098b90e53c91984d3

New ee-repo-ref: 1d4b7a31fc115d6aba8640f7cd3fd5a01abe6806

Automated by sync-ee-ref workflow.

* fix: regenerate system prompts, remove unused import

- Regenerate auto-generated system prompts after openflow schema change
- Remove unused serde_json::json import in test file (CI -D warnings)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: insert dummy v2_job row in schedule tests for FK constraint

The job_delete_schedule table has a FK to v2_job, so tests need a
real v2_job row before inserting into the schedule table.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: trigger CI re-run

* fix: remove heavy flow integration tests to avoid CI worker contention

The flow integration tests spawn workers that compete for CPU with
the existing relock_skip tests under --test-threads=10, causing
consistent 60s timeouts in CI. Keep only the lightweight unit tests
and DB integration tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: restore correct ee-repo-ref for our branch

The ref was overwritten to main's EE ref during a rebase. Restore to
our branch's EE commit that includes the autoscaling tuple fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: retrigger CI on fresh runner

* fix: remove FK constraint from job_delete_schedule to unblock CI

The FK with ON DELETE CASCADE to v2_job may have caused performance
overhead during test DB setup (each sqlx::test creates a fresh DB
with all migrations). Remove the FK — orphan schedule rows are
harmlessly cleaned by the monitor.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* ee-ref

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2026-04-08 04:15:28 +00:00
Ruben Fiszel a46aa641f9 feat: add R language support (#8263) 
* feat: add R language support

Add R as a new supported scripting language in Windmill, following the
same pattern used for Ruby. Includes:

- Backend: ScriptLang::Rlang enum variant, DB migration, tree-sitter-r
  parser crate with tests, WASM parser binding, R executor with NSJail
  sandboxing, job dispatch and signature parsing
- Frontend: language picker, R icon, syntax highlighting, editor bar
  insertions (Sys.getenv, get_variable, get_resource), schema inference,
  init code template, BETA badge
- CLI: .r extension mapping, sync support, bootstrap template

R scripts use `main <- function(...)` syntax, jsonlite for JSON
serialization, and system curl for the Windmill client helper.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add R package resolution and installation

Parse library()/require() calls from R scripts to extract dependencies.
Resolve versions from CRAN, cache lockfiles in pip_resolution_cache,
and install packages to a shared R library cache. The run step sets
R_LIBS_USER so installed packages are available to the script.

- Parser: parse_r_requirements() extracts package names from AST
- Executor: resolve() generates lockfile, install() installs from CRAN
- Worker lockfiles: wire up R resolve for dependency jobs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add nsjail sandboxing for R resolve and install phases

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: fix R get_variable/get_resource and add sandbox annotation + e2e tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: fix R arg inference with JS fallback parser and get_variable/get_resource

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix flake

* nsjail

* nits

* fix: R install improvements - suppress verbose output, flat lockfile logging, Dockerfile R support, rlimits

- Suppress renv verbose output during resolve and install (controlled by #verbose annotation)
- Filter renv from install list (already loaded, causes noisy restart message)
- Log compact "resolved N packages" instead of full renv.lock JSON
- Add R (r-base, r-cran-renv) to DockerfileFull and DockerfileFullEe
- Use disable_rl for nsjail install config (R compiles from source)
- Reduce default concurrency from 20 to 5
- Add rlang to openflow.openapi.yaml
- Fix MainArgSignature (no_main_func -> auto_kind) after main merge

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* final

* fix: remove accidental R install from multiplayer Dockerfile

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: remove R from Windows build and DockerfileExtra

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: rename R migration to avoid timestamp collision with trigger_filter_logic

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* all

* fix: R install improvements - suppress verbose output, flat lockfile logging, Dockerfile R support, rlimits

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: add clear error when Rscript binary is missing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: fix type errors in R fallback parser, use format! in wrap(), add R system prompts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: pyranota <pyra@duck.com>
 2026-04-01 06:11:37 +00:00
Ruben Fiszel 31d6660d56 feat: script module mode with CLI sync, preview, and WAC UI improvements (#8380) 
* feat: add script module mode with folder model for Bun and Python

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: add missing modules field to RawCode in bun_executor

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* sqlx

* feat: enrich WAC templates with checkpoint and replay semantics

Add prominent comments explaining that all computation must happen
inside task/step/taskScript or it will be replayed on resume/retry.
Clarify that waitForApproval does not hold a worker and that
approve/reject URLs are available in the timeline step details.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(cli): script module sync idempotency, per-module hash tracking, and preview support

- Fix pull→push idempotency: use `??` instead of `||` for module lock
  field so empty strings are preserved (matches API's `lock: ""`)
- Add per-module hash tracking in wmill-lock.yaml following the flow
  inline script pattern (SCRIPT_TOP_HASH + per-module subpath hashes)
- Selective module lock regeneration: only regenerate locks for modules
  whose content actually changed, not all modules
- Use unfiltered rawWorkspaceDependencies for module hashes to match
  what updateModuleLocks passes to fetchScriptLock
- Show changed module names in stale script output for clarity
- Add module support to `script preview` command: read modules from
  __mod/ folder and pass them in the preview API request
- Add preview tests for taskScript pattern (flat and folder layout)
- Update test assertion for module stale detection output

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(frontend): WAC UI improvements — reorder templates, module tab rename, import consolidation

- Reorder WAC template buttons: TypeScript before Python in
  ScriptBuilder, CreateActionsScript, and CreateActionsFlow
- Remove dropdown items from +Script button (simplify to direct link)
- Move "Import Workflow-as-Code" to +Flow dropdown with dedicated drawer
- Add module tab rename: pencil icon on hover opens popover with
  validation, fixed-width icon container prevents layout shift

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: remaining module-mode changes from working branch

- Backend parser updates for WAC detection
- CLI sync/types updates for raw app path and module support
- Frontend UI polish (Dev.svelte, ScriptRow, script hash page)
- Test fixture updates

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test(cli): add test for module modification detection in generate-metadata

Verifies that modifying a single module file re-triggers stale
detection and only the changed module is listed, not all modules.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): critical fixes from PR review

- Fix hardcoded dev path in bun_executor.rs WAC v2 wrapper — use
  "windmill-client" import instead of absolute filesystem path
- Fix missed no_main_func → auto_kind rename in parser TS test
- Add modules column to clone_script SQL (windmill-common and
  windmill-api-workspaces) so cloned scripts retain their modules
- Add modules: None to RawCode structs in worker tests
- Restore complete sqlx cache (merge main's cache + our new queries)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): fix clone warning treated as error in CI

Change `.clone()` on double reference to `*k` dereference in
scripts.rs hash implementation. Update sqlx cache with new query
hashes from modified clone_script SQL.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): use published parser wasm versions for CI build

The local file:// paths for windmill-parser-wasm-py and
windmill-parser-wasm-ts don't exist in the Cloudflare Pages build
environment. Revert to published npm versions (1.655.0).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): update parser wasm packages to 1.657.2

Use newly published windmill-parser-wasm-ts and windmill-parser-wasm-py
v1.657.2 which include auto_kind/WAC detection changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): regenerate package-lock.json for npm ci compatibility

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(frontend): use main's lockfile as base, update only parser wasm packages

Regenerating package-lock.json from scratch pulled different dependency
versions causing svelte-check type errors. Instead, start from main's
lockfile and only update the two changed packages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): add modules column to fetch_script_for_update query

The Script<SR> struct has a modules field (FromRow), but
fetch_script_for_update didn't SELECT modules, causing a runtime
error "no column found for name: modules" when the worker processed
dependency jobs. This was the root cause of the relock_skip test
timeout.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): fix script module execution for Python and Bun

- Fix modules not passed through job queue: inject _MODULES into
  PushArgs.extra when pushing Code jobs so worker can extract them
- Fix Python module imports: use relative imports (from .helper)
  and add sys.path.insert for module directory in wrapper
- Fix Python tests: use relative imports and empty lock to prevent
  pip from resolving module names as packages
- Add local file check in Bun loader for module resolution
- Ignore Bun module test (bundle mode loader integration tracked
  separately)
- Add missing modules column to fetch_script_for_update query

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): remove unnecessary empty lock in Python module tests

Relative imports (from .helper) are not parsed as pip packages,
so the empty lock workaround is not needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(backend): fix module execution for Python and Bun — all tests pass

Python modules:
- Use relative imports (from .helper import greet) since scripts run
  as packages
- Add sys.path.insert for module directory in wrapper to ensure local
  modules take precedence over pip packages with same name

Bun modules:
- Use bundled output (./out/main.js) as wrapper import when modules
  are present — the bundled output has module content inlined by
  Bun.build, avoiding runtime loader resolution issues
- Add local file check in loader.bun.js onResolve to short-circuit
  API URL resolution for module files on disk

Job queue:
- Inject _MODULES into PushArgs.extra when pushing Code jobs so
  the worker can extract them at execution time

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: address PR review — simplify, fix correctness, remove dead code

Critical fixes:
- Replace all CLI `no_main_func` references with `auto_kind` (string)
  to match the backend migration and API changes
- Remove duplicated `compute_python_module_dir` in worker.rs, use
  the canonical version from python_executor.rs

High priority:
- Auto-create `__init__.py` in intermediate directories for nested
  Python modules so imports like `from .utils.math import add` work
  without users manually creating __init__.py files
- Remove redundant `sys_path_insert` — relative imports use Python's
  package system, not sys.path

Medium:
- Fix lock file base name extraction: use regex to strip only the
  final extension (`.replace(/\.[^.]+$/, '')`) instead of `indexOf(".")`
  which breaks for files like `helper.test.ts`

Simplification:
- Remove dead `{#if false}` Popover block in ScriptEditor.svelte
- Guard loader.bun.js local file check to only run for relative paths
  (matching the Windows loader pattern)
- Add clarifying comment on Bun dual mechanism (build + run phases)
- Add maintenance comment on manual Hash impl for NewScript

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: final review fixes — stale cleanup, baseName, auto_kind export

- Fix sync.ts baseName extraction using indexOf(".") → regex
  (same fix as script.ts/metadata.ts, missed this instance)
- Add stale module file cleanup in writeModulesToDisk: removes files
  from __mod/ that are no longer in the modules map before writing,
  fixing the pull→push cycle that couldn't delete modules
- Log warning when _MODULES serialization fails in job push instead
  of silently dropping modules
- Use strict equality (===) for auto_kind comparison
- Exclude auto_kind from workspace export — it is auto-detected by
  the parser at deploy time from script content

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(cli): remove auto_kind from push, comparison, and metadata

auto_kind is auto-detected by the parser at deploy time, so the CLI
should not send it, compare it, or write it to script.yaml.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: remove erroneously added backend/backend/.sqlx directory

Duplicate .sqlx cache was committed at the wrong nested path.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review feedback + fix CI dead_code warning

Frontend (ScriptEditor.svelte):
- Fix switchToMain() missing lastSyncedCode update — prevents stale
  code sync on external changes while editing a module tab
- Fix formatAction saving module code to main script's localStorage
  draft — now saves main code when on a module tab
- Fix non-null assertion on inferModuleLang in renameModule — fall
  back to original language instead of force unwrap
- Remove redundant activeModuleTab truthy check in runTest

CLI (script.ts):
- Clean up empty directories after removing stale module files in
  writeModulesToDisk

Backend:
- Add path traversal guard in write_module_files — reject module
  paths containing ".."
- Fix dead_code warning on auto_kind field in workspace export struct

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(frontend): improve auto_kind UX + address review findings

- Rename "Include without main function" toggle to "Include library
  scripts" in script list (ItemsList.svelte)
- Update NoMainFuncBadge: "No main" → "Library" with clearer tooltip
- Filter module file extensions by main script language — Python
  scripts only allow .py modules, TypeScript only .ts, etc.
- Split flushModuleState into flushModuleContent (no UI side-effect)
  and flushModuleState (flush + reset tab), reducing duplication
- Dynamic placeholder and hint text in add module popover based on
  main script language

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-17 01:20:09 +00:00
hugocasa dc0e59f432 feat: add preprocessor support for dedicated workers and bunnative scripts (#8284) 
* feat: add preprocessor support for dedicated workers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee-repo-ref.txt

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: extract transform_and_run helper in python dedicated wrapper

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add preprocessor support for bunnative scripts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: resolve unused postprocessor variable in python wrapper

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* chore: add workflow_dispatch trigger to backend integration tests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: bunnative fixture lock format and PrewarmedIsolate::spawn callers

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: update generate_dedicated_worker_wrapper callers in bun_jobs test

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use non-dedicated workers in preprocessor integration tests

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: prewarm preprocessor isolate for bunnative dedicated workers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: flatten bunnative dedicated worker preprocessing into single result path

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: use labeled block instead of async block for EE compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update commit reference in ee-repo-ref.txt

* chore: update ee-repo-ref to e36945b987f7904fa984181baf3124e7b2722bd1

This commit updates the EE repository reference after PR #445 was merged in windmill-ee-private.

Previous ee-repo-ref: 8a2625833452aadb8907242bf502b24ca2dffd73

New ee-repo-ref: e36945b987f7904fa984181baf3124e7b2722bd1

Automated by sync-ee-ref workflow.

* Fix merge conflict in ee-repo-ref.txt

Resolve merge conflict in ee-repo-ref.txt

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2026-03-10 18:26:02 +00:00
Ruben Fiszel 424ca59dfe feat: make WINDMILL_DIR configurable via environment variable (#8215) 
* fix: auto-heal corrupted python runtime cache on remote workers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Revert "fix: auto-heal corrupted python runtime cache on remote workers"

This reverts commit 0ea013a554.

* feat: make WINDMILL_DIR configurable via environment variable

Allow users to configure the base directory for Windmill's tmp/cache files
via the WINDMILL_DIR env var (default: /tmp/windmill). This fixes Python
runtime cache corruption on RHEL systems where systemd-tmpfiles-clean
removes files from /tmp.

Converts TMP_DIR (renamed to WINDMILL_DIR) and all derived cache directory
constants from compile-time const &str (concatcp!) to runtime lazy_static
String values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: deref ERROR_DIR lazy_static for AsRef<Path> and Display traits

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref to branch name for CI compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: deref lazy_static constants in all executor files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: panic if WINDMILL_DIR has trailing slash

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: also reject trailing backslash in WINDMILL_DIR for Windows

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: deref GO_BIN_CACHE_DIR in test utils

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: replace remaining hardcoded /tmp/windmill paths and validate empty WINDMILL_DIR

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: nsjail powershell mount dst, Windows path assumptions, pwsh deref consistency

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: restore Windows /tmp path translation in go and bun executors

The Windows path translation replaces /tmp with the Windows temp dir
(e.g. C:\tmp) before normalizing slashes. Without this, the default
WINDMILL_DIR=/tmp/windmill produces paths without a drive letter on
Windows.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee-repo-ref to 6fd5a2ce908235a17975ad4dbdf0051cd89334f3

This commit updates the EE repository reference after PR #436 was merged in windmill-ee-private.

Previous ee-repo-ref: e8c03e16720833230ebd1878b4c63642ecc6c80f

New ee-repo-ref: 6fd5a2ce908235a17975ad4dbdf0051cd89334f3

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2026-03-04 08:53:25 +00:00
hugocasa 37c9acb232 feat: dedicated nativets (#8021) 
* feat: dedicated nativets

* review nits

* prewarm isolates

* ref

* chore: update ee-repo-ref to 5f8105b808f3f0186fdf5132d2ee602d8a14aa17

This commit updates the EE repository reference after PR #424 was merged in windmill-ee-private.

Previous ee-repo-ref: b7906acabb8ce359230bbd3e30dbb3bba4c42adb

New ee-repo-ref: 5f8105b808f3f0186fdf5132d2ee602d8a14aa17

Automated by sync-ee-ref workflow.

---------

Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
 2026-02-20 16:28:40 +00:00
Ruben Fiszel aa9f3da429 fix: make V8 runtime init idempotent and auto-initialize before isolate creation 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-08 14:31:12 +00:00
Ruben Fiszel 90d010347c fix: use unprotected V8 platform to prevent SIGSEGV on x86_64 Linux 
The default V8 platform uses Memory Protection Keys (pkeys) which
require all V8-using threads to be descendants of the thread that
called v8::Initialize. Tokio's spawn_blocking pool threads don't
satisfy this, causing SIGSEGV in WasmCodePointerTable during isolate
creation on x86_64 Linux.

Switch to new_unprotected_default_platform which relaxes the pkey
requirement. Also remove --single-threaded V8 flag (was degrading
performance without fixing the issue) and scope the creation mutex
to just JsRuntime::new() instead of the entire lifecycle.

See: https://github.com/denoland/deno_core/issues/952

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-08 13:35:59 +00:00
Ruben Fiszel 05106d7dee fix: prevent V8 SIGSEGV by serializing isolate creation and fixing use-after-free 
Work around a V8 bug in WasmCodePointerTable::AllocateUninitializedEntry()
that causes SIGSEGV when multiple isolates are created concurrently on
x86_64 Linux (https://github.com/denoland/deno_core/issues/952).

- Add V8_ISOLATE_CREATE_LOCK mutex to serialize JsRuntime::new() calls
- Replace oneshot channel with Arc<Mutex<Option<IsolateHandle>>> shared
  between spawn_blocking and an IsolateDropGuard for proper cancellation
- Remove terminate_execution() call on dead isolate handle in error path
  (was use-after-free: handle dereferenced after JsRuntime already dropped)
- Clear handle before drop(js_runtime) to prevent guard from terminating
  a destroyed isolate

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-08 12:54:47 +00:00
Ruben Fiszel 3f68e3a58e refactor: isolate deno_core into windmill-runtime-nativets subcrate (#7848) 
* refactor: isolate deno_core into windmill-runtime-nativets subcrate

Remove deno_core from flow eval and isolate nativets V8 runtime into a
dedicated subcrate so deno_core compilation no longer blocks
windmill-worker or windmill-api.

- Create windmill-jseval crate: QuickJS-based JS eval for flow
  expressions and batch rerun, extracted from windmill-worker
- Create windmill-runtime-nativets crate: all deno_core/V8 deps and
  nativets script execution, with build.rs snapshot generation
- Simplify windmill-worker: remove all deno_* direct deps, empty
  build.rs, gate nativets behind optional dep
- Update windmill-api: use windmill-jseval for batch rerun instead of
  deno_core, remove deno_core feature entirely
- Add nativets integration tests (nativets_jobs.rs) and parallel
  stress test (nativets_stress.rs, 8 workers x 200 jobs)
- Remove dead code: deno flow eval path, USE_QUICKJS env var,
  parity tests (replaced with 63 standalone expected-value tests)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address PR review feedback for deno_core isolation

- Deduplicate unsafe_raw() into windmill-common/src/utils.rs (single source)
- Delete orphaned runtime.js and windmill-client.js from windmill-worker/src/
- Fix operator precedence in windmill-jseval with explicit parentheses
- Remove unnecessary return keyword in heap limit callback
- Remove redundant as usize casts
- Remove ~150 lines of commented-out code from runtime.js
- Remove commented-out #[cfg] in build.rs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* otel ee

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-08 09:16:37 +00:00