mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-21 08:03:33 +00:00
* feat: add script module mode with folder model for Bun and Python Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add missing modules field to RawCode in bun_executor Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * sqlx * feat: enrich WAC templates with checkpoint and replay semantics Add prominent comments explaining that all computation must happen inside task/step/taskScript or it will be replayed on resume/retry. Clarify that waitForApproval does not hold a worker and that approve/reject URLs are available in the timeline step details. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): script module sync idempotency, per-module hash tracking, and preview support - Fix pull→push idempotency: use `??` instead of `||` for module lock field so empty strings are preserved (matches API's `lock: ""`) - Add per-module hash tracking in wmill-lock.yaml following the flow inline script pattern (SCRIPT_TOP_HASH + per-module subpath hashes) - Selective module lock regeneration: only regenerate locks for modules whose content actually changed, not all modules - Use unfiltered rawWorkspaceDependencies for module hashes to match what updateModuleLocks passes to fetchScriptLock - Show changed module names in stale script output for clarity - Add module support to `script preview` command: read modules from __mod/ folder and pass them in the preview API request - Add preview tests for taskScript pattern (flat and folder layout) - Update test assertion for module stale detection output Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(frontend): WAC UI improvements — reorder templates, module tab rename, import consolidation - Reorder WAC template buttons: TypeScript before Python in ScriptBuilder, CreateActionsScript, and CreateActionsFlow - Remove dropdown items from +Script button (simplify to direct link) - Move "Import Workflow-as-Code" to +Flow dropdown with dedicated drawer - Add module tab rename: pencil icon on hover opens popover with validation, fixed-width icon container prevents layout shift Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: remaining module-mode changes from working branch - Backend parser updates for WAC detection - CLI sync/types updates for raw app path and module support - Frontend UI polish (Dev.svelte, ScriptRow, script hash page) - Test fixture updates Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(cli): add test for module modification detection in generate-metadata Verifies that modifying a single module file re-triggers stale detection and only the changed module is listed, not all modules. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): critical fixes from PR review - Fix hardcoded dev path in bun_executor.rs WAC v2 wrapper — use "windmill-client" import instead of absolute filesystem path - Fix missed no_main_func → auto_kind rename in parser TS test - Add modules column to clone_script SQL (windmill-common and windmill-api-workspaces) so cloned scripts retain their modules - Add modules: None to RawCode structs in worker tests - Restore complete sqlx cache (merge main's cache + our new queries) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix clone warning treated as error in CI Change `.clone()` on double reference to `*k` dereference in scripts.rs hash implementation. Update sqlx cache with new query hashes from modified clone_script SQL. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): use published parser wasm versions for CI build The local file:// paths for windmill-parser-wasm-py and windmill-parser-wasm-ts don't exist in the Cloudflare Pages build environment. Revert to published npm versions (1.655.0). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): update parser wasm packages to 1.657.2 Use newly published windmill-parser-wasm-ts and windmill-parser-wasm-py v1.657.2 which include auto_kind/WAC detection changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): regenerate package-lock.json for npm ci compatibility Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): use main's lockfile as base, update only parser wasm packages Regenerating package-lock.json from scratch pulled different dependency versions causing svelte-check type errors. Instead, start from main's lockfile and only update the two changed packages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): add modules column to fetch_script_for_update query The Script<SR> struct has a modules field (FromRow), but fetch_script_for_update didn't SELECT modules, causing a runtime error "no column found for name: modules" when the worker processed dependency jobs. This was the root cause of the relock_skip test timeout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix script module execution for Python and Bun - Fix modules not passed through job queue: inject _MODULES into PushArgs.extra when pushing Code jobs so worker can extract them - Fix Python module imports: use relative imports (from .helper) and add sys.path.insert for module directory in wrapper - Fix Python tests: use relative imports and empty lock to prevent pip from resolving module names as packages - Add local file check in Bun loader for module resolution - Ignore Bun module test (bundle mode loader integration tracked separately) - Add missing modules column to fetch_script_for_update query Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): remove unnecessary empty lock in Python module tests Relative imports (from .helper) are not parsed as pip packages, so the empty lock workaround is not needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix module execution for Python and Bun — all tests pass Python modules: - Use relative imports (from .helper import greet) since scripts run as packages - Add sys.path.insert for module directory in wrapper to ensure local modules take precedence over pip packages with same name Bun modules: - Use bundled output (./out/main.js) as wrapper import when modules are present — the bundled output has module content inlined by Bun.build, avoiding runtime loader resolution issues - Add local file check in loader.bun.js onResolve to short-circuit API URL resolution for module files on disk Job queue: - Inject _MODULES into PushArgs.extra when pushing Code jobs so the worker can extract them at execution time Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: address PR review — simplify, fix correctness, remove dead code Critical fixes: - Replace all CLI `no_main_func` references with `auto_kind` (string) to match the backend migration and API changes - Remove duplicated `compute_python_module_dir` in worker.rs, use the canonical version from python_executor.rs High priority: - Auto-create `__init__.py` in intermediate directories for nested Python modules so imports like `from .utils.math import add` work without users manually creating __init__.py files - Remove redundant `sys_path_insert` — relative imports use Python's package system, not sys.path Medium: - Fix lock file base name extraction: use regex to strip only the final extension (`.replace(/\.[^.]+$/, '')`) instead of `indexOf(".")` which breaks for files like `helper.test.ts` Simplification: - Remove dead `{#if false}` Popover block in ScriptEditor.svelte - Guard loader.bun.js local file check to only run for relative paths (matching the Windows loader pattern) - Add clarifying comment on Bun dual mechanism (build + run phases) - Add maintenance comment on manual Hash impl for NewScript Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: final review fixes — stale cleanup, baseName, auto_kind export - Fix sync.ts baseName extraction using indexOf(".") → regex (same fix as script.ts/metadata.ts, missed this instance) - Add stale module file cleanup in writeModulesToDisk: removes files from __mod/ that are no longer in the modules map before writing, fixing the pull→push cycle that couldn't delete modules - Log warning when _MODULES serialization fails in job push instead of silently dropping modules - Use strict equality (===) for auto_kind comparison - Exclude auto_kind from workspace export — it is auto-detected by the parser at deploy time from script content Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): remove auto_kind from push, comparison, and metadata auto_kind is auto-detected by the parser at deploy time, so the CLI should not send it, compare it, or write it to script.yaml. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove erroneously added backend/backend/.sqlx directory Duplicate .sqlx cache was committed at the wrong nested path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review feedback + fix CI dead_code warning Frontend (ScriptEditor.svelte): - Fix switchToMain() missing lastSyncedCode update — prevents stale code sync on external changes while editing a module tab - Fix formatAction saving module code to main script's localStorage draft — now saves main code when on a module tab - Fix non-null assertion on inferModuleLang in renameModule — fall back to original language instead of force unwrap - Remove redundant activeModuleTab truthy check in runTest CLI (script.ts): - Clean up empty directories after removing stale module files in writeModulesToDisk Backend: - Add path traversal guard in write_module_files — reject module paths containing ".." - Fix dead_code warning on auto_kind field in workspace export struct Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(frontend): improve auto_kind UX + address review findings - Rename "Include without main function" toggle to "Include library scripts" in script list (ItemsList.svelte) - Update NoMainFuncBadge: "No main" → "Library" with clearer tooltip - Filter module file extensions by main script language — Python scripts only allow .py modules, TypeScript only .ts, etc. - Split flushModuleState into flushModuleContent (no UI side-effect) and flushModuleState (flush + reset tab), reducing duplication - Dynamic placeholder and hint text in add module popover based on main script language Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
262 lines
8.7 KiB
Rust
262 lines
8.7 KiB
Rust
#![cfg_attr(target_arch = "wasm32", feature(c_variadic))]
|
|
|
|
use anyhow::{anyhow, bail};
|
|
use nu_parser::lex;
|
|
|
|
use serde_json::{json, Value};
|
|
use windmill_parser::{Arg, MainArgSignature, ObjectType, Typ};
|
|
|
|
pub fn parse_nu_signature(code: &str) -> anyhow::Result<MainArgSignature> {
|
|
let (tokens, ..) = lex(code.as_bytes(), 0, &[], &[], true);
|
|
let src = code.to_owned();
|
|
#[derive(Debug)]
|
|
enum LastToken {
|
|
None,
|
|
Def,
|
|
Main,
|
|
Args(String),
|
|
}
|
|
let mut last_token = LastToken::None;
|
|
for token in tokens {
|
|
let s = token.span;
|
|
let cont = src.get(s.start..s.end).ok_or(anyhow!("Parsing error"))?;
|
|
last_token = match last_token {
|
|
LastToken::None if cont == "def" => LastToken::Def,
|
|
LastToken::Def if cont == "main" => LastToken::Main,
|
|
LastToken::Main => {
|
|
LastToken::Args(cont.get(1..(cont.len() - 1)).unwrap_or("Error").to_owned())
|
|
}
|
|
LastToken::Args(_) => break,
|
|
_ => LastToken::None,
|
|
};
|
|
}
|
|
|
|
let LastToken::Args(args) = last_token else {
|
|
bail!("Cannot find main function.");
|
|
};
|
|
|
|
let mut sig = MainArgSignature::default();
|
|
sig.auto_kind = None;
|
|
|
|
let batches = args
|
|
.lines()
|
|
.filter_map(|el| {
|
|
if el.trim_start().starts_with('#') {
|
|
None
|
|
} else {
|
|
Some(
|
|
el.split(',')
|
|
.map(|el| el.trim())
|
|
.filter(|el| el != &"")
|
|
.collect::<Vec<&str>>(),
|
|
)
|
|
}
|
|
})
|
|
.flatten()
|
|
.collect::<Vec<&str>>();
|
|
|
|
let mut compensate_lookahead = 0;
|
|
for (i, batch) in batches.iter().enumerate() {
|
|
// parse_default can lookahead and if it does we need to compensate
|
|
// otherwise we would try to parse data already parsed but not yielded by parse_default
|
|
if compensate_lookahead > 0 {
|
|
compensate_lookahead -= 1;
|
|
continue;
|
|
}
|
|
|
|
let type_start = batch.find(":");
|
|
let default_start = batch.find("=");
|
|
|
|
let (name, typ, default) = match (type_start, default_start) {
|
|
(None, None) => (batch.trim(), None, None),
|
|
(None, Some(d)) => (
|
|
batch
|
|
.get(0..d)
|
|
.ok_or(anyhow!("Cannot parse argument ident"))?
|
|
.trim(),
|
|
None,
|
|
Some(parse_default(
|
|
&batch
|
|
.get(d..)
|
|
.ok_or(anyhow!("Cannot parse default value for argument"))?,
|
|
&batches,
|
|
i,
|
|
&mut compensate_lookahead,
|
|
)?),
|
|
),
|
|
(Some(t), None) => (
|
|
batch
|
|
.get(0..t)
|
|
.ok_or(anyhow!("Cannot parse argument ident"))?
|
|
.trim(),
|
|
Some(parse_type(
|
|
&batch
|
|
.get(t..)
|
|
.ok_or(anyhow!("Cannot parse type of argument"))?,
|
|
)?),
|
|
None,
|
|
),
|
|
(Some(t), Some(d)) => {
|
|
if t < d {
|
|
(
|
|
batch
|
|
.get(0..t)
|
|
.ok_or(anyhow!("Cannot parse argument ident"))?
|
|
.trim(),
|
|
Some(parse_type(
|
|
&batch
|
|
.get(t..d)
|
|
.ok_or(anyhow!("Cannot parse type of argument"))?,
|
|
)?),
|
|
Some(parse_default(
|
|
&batch
|
|
.get(d..)
|
|
.ok_or(anyhow!("Cannot parse default value of argument"))?,
|
|
&batches,
|
|
i,
|
|
&mut compensate_lookahead,
|
|
)?),
|
|
)
|
|
} else {
|
|
bail!("Parsing error `:` should be before `=`\nit likely means you are trying to set default value to record or table which is not supported at the moment.")
|
|
}
|
|
}
|
|
};
|
|
|
|
// Check if it is optional
|
|
let optional = {
|
|
let Some(element) = name.chars().last() else {
|
|
bail!("Internal error, cannot check if argument is optional")
|
|
};
|
|
element == '?'
|
|
};
|
|
|
|
// Rest parameters are not supported
|
|
if matches!(name.get(0..3), Some("...")) {
|
|
bail!("Rest (...) parameters are not supported")
|
|
}
|
|
|
|
// Flags are not supported
|
|
if matches!(name.get(0..2), Some("--")) {
|
|
bail!("Flags are not supported")
|
|
}
|
|
|
|
sig.args.push(Arg {
|
|
name: if optional {
|
|
name.get(..name.len() - 1).unwrap_or("Error").to_owned()
|
|
} else {
|
|
name.to_owned()
|
|
},
|
|
typ: typ.unwrap_or(Typ::Unknown),
|
|
otyp: None,
|
|
has_default: default.is_some() || optional,
|
|
default: default.or_else(|| if optional { Some(json!(null)) } else { None }),
|
|
oidx: None,
|
|
});
|
|
}
|
|
|
|
fn parse_type(content: &str) -> anyhow::Result<Typ> {
|
|
let c = content.replace(":", "").trim().to_owned();
|
|
let typ = match c.as_str() {
|
|
"string" => Typ::Str(None),
|
|
"int" => Typ::Int,
|
|
"float" => Typ::Float,
|
|
"number" => Typ::Float,
|
|
"record" => Typ::Object(ObjectType::new(None, Some(vec![]))),
|
|
"table" => Typ::List(Box::new(Typ::Object(ObjectType::new(None, Some(vec![]))))),
|
|
"nothing" => Typ::Unknown,
|
|
// TODO: needs additional work on literal parsing
|
|
// "binary" => Typ::Bytes,
|
|
"datetime" => Typ::Datetime,
|
|
"any" => Typ::Unknown,
|
|
"bool" => Typ::Bool,
|
|
// Lists
|
|
"list" | "list<any>" | "list<nothing>" => Typ::List(Box::new(Typ::Unknown)),
|
|
"list<number>" => Typ::List(Box::new(Typ::Float)),
|
|
"list<bool>" => Typ::List(Box::new(Typ::Bool)),
|
|
"list<string>" => Typ::List(Box::new(Typ::Str(None))),
|
|
// list<float/int> is not supported
|
|
// Records and Tables
|
|
// TODO: Support in V1?
|
|
s if s.contains("record<") => {
|
|
bail!("typed records are not supported, use `ident: record`")
|
|
}
|
|
s if s.contains("table<") => {
|
|
bail!("typed tables are not supported, use `ident: table`")
|
|
}
|
|
s => bail!("{s} is not supported"),
|
|
};
|
|
Ok(typ)
|
|
}
|
|
fn parse_default(
|
|
content: &str,
|
|
ctx: &[&str],
|
|
i: usize,
|
|
skip: &mut usize,
|
|
) -> anyhow::Result<Value> {
|
|
let mut c = content.replace("=", "").trim().to_owned();
|
|
|
|
fn parse_object_literal(
|
|
(open, close): (char, char),
|
|
mut c_2: String,
|
|
ctx: &[&str],
|
|
i: usize,
|
|
skip: &mut usize,
|
|
) -> anyhow::Result<String> {
|
|
// It is list
|
|
// if c.contains("[") {
|
|
let mut closed = false;
|
|
if c_2 != open.to_string() {
|
|
// [a ~ , ~ ...
|
|
// Add ^
|
|
c_2 += ",";
|
|
}
|
|
// else {
|
|
// [
|
|
// a < Do not add ","
|
|
// ...
|
|
// }
|
|
let remainder = &ctx
|
|
.iter()
|
|
.skip(i + 1)
|
|
.map_while(|el| {
|
|
let el = el.trim();
|
|
|
|
if closed {
|
|
None
|
|
} else {
|
|
if el.chars().last() == Some(close) {
|
|
closed = true;
|
|
}
|
|
*skip += 1;
|
|
Some(el)
|
|
}
|
|
})
|
|
.collect::<Vec<&str>>()
|
|
.join(",");
|
|
|
|
if remainder.contains(&['{', '[']) {
|
|
bail!("Nesting is not supported")
|
|
}
|
|
|
|
Ok((c_2 + remainder)
|
|
// Remove trailing comma if there is any
|
|
.replace(&format!(",{close}"), &close.to_string()))
|
|
}
|
|
// It is list
|
|
if c.contains("[") {
|
|
if c.chars().last() != Some(']') {
|
|
c = parse_object_literal(('[', ']'), c.clone(), ctx, i, skip)?;
|
|
}
|
|
}
|
|
// It is record
|
|
if c.contains("{") {
|
|
if c.chars().last() != Some('}') {
|
|
c = parse_object_literal(('{', '}'), c.clone(), ctx, i, skip)?;
|
|
}
|
|
}
|
|
Ok(serde_json::from_str(&c)?)
|
|
}
|
|
Ok(sig)
|
|
}
|