Files
windmill/cli/test/duckdb_macros_unit.test.ts
Ruben Fiszel e3f43033ca fix(cli): macro-library parity in --local pipeline graph + read-only run --dry-run (#9942)
* fix(cli): surface macro libraries in --local pipeline graph + make run --dry-run read-only

* fix(cli): resolve workspace-wide macro libraries in --local graph (shared libs outside the pipeline folder)

* fix(cli): macro-lib consumers + //-prefix parity in --local pipeline graph

Address Codex review P1s: (1) macro libraries that consume another library's
macros now produce lib->lib edges (any folder DuckDB script is a consumer, not
just // pipeline members) so an upstream provider node no longer disappears;
(2) parseMacroAnnotations accepts //, --, and # prefixes like the backend, so a
.duckdb.sql library headed with // macros is detected locally. Both edge
endpoints are forced into the node set. Verified byte-for-byte against deployed.

* fix(cli): exclude non-pipeline macro-consumer nodes from --local run selection

Address Codex P1: buildMacroEdges surfaces macro-consumer nodes (a DuckDB script
calling a macro but not marked // pipeline) for lineage display. Those have no
local file, so pipeline run --local must not treat them as manual roots — a
dry-run listed them and a real run failed resolving local content. Exclude any
--local graph node absent from localScripts (the previewable set) from starts and
selection, alongside the existing macro-library exclusion.

* fix(cli): reject display-only macro consumers in explicit --from (post-merge with #9945)

The mid-DAG --from feature (#9945, now on main) admits any autorun-able script
via validFromStarts/fromEligible, which was filtered only by macroLibPaths. A
non-// pipeline macro-consumer helper (a --local display node) therefore passed
--from eligibility and produced an empty plan. Filter fromEligible by the broader
notRunnablePaths too, and reject such a --from with a clear message instead of a
silent empty plan.

* chore(cli): remove NUL edge-key separator + refresh stale macro comments

Address Codex P2 nits: (1) the macro edge map packed (lib, consumer) into a
string with a literal NUL separator, which made localGraph.ts read as a binary
file to grep/rg — replace with a nested lib->consumer Map (no separator); (2)
comments claiming macro nodes/edges are 'deployed graph only' contradicted this
PR's local derivation — describe the code as it is.

* fix(cli): tag unused // pipeline + // macros libraries so --local run excludes them

Address Codex P1: the deployed builder sets 'macros' on any node whose path
provides macros (edge or not), so a // pipeline + // macros script with no
consumers is still recognized as definition-only. Local enrichment only tagged
edge providers, leaving an unused pipeline macro library as a bare runnable that
pipeline run --local would schedule as a manual root. Also tag any library whose
path is already a runnable; unused non-pipeline libraries stay suppressed.

* fix(pipelines): `// macros` takes precedence over `// pipeline` (a library is never a member)

A macro library is definition-only — its macros are injected into consumers and
running it is a no-op — so marking it `// pipeline` is meaningless and only
produced a confusing state (an unused pipeline macro library appearing as a
manual root). Make `// macros` win: parse_pipeline_annotations forces in_pipeline
false when macros is set. Mirrored in all three parsers that must agree — the Rust
canonical parser (drives deploy membership), the frontend TS parser (live graph),
and the CLI local graph (pinned wasm still reports in_pipeline, so precedence is
applied when skipping members). Shared parity fixture + unit tests on each side.

* docs(cli): trim narrative comment blocks to non-obvious constraints

Address Codex P2: duckdbMacros.ts opened with a ~19-line narrative block whose
parity rationale belongs in the PR description; reduce to the two real constraints
(keep in lockstep with duckdb_macros.rs; dynamic-SQL calls need // use). Per the
AGENTS.md comment policy.

* fix(cli): model macro libraries as pipeline members, matching the deployed graph

Reverts the parser-precedence approach (b398b69): the backend deliberately marks
EVERY macro library auto_kind='pipeline' (scripts.rs:1474, macro_lib_defs), so a
macro library IS a graph member — the // pipeline marker is redundant, not
authoritative. Precedence was a no-op on deploy while diverging the CLI/frontend.

Instead mirror reality in the CLI local graph: an in-folder // macros library is a
member node (in_pipeline=true, with signatures) whether used or not; its // use is
processed (it's a member) so a library that reaches another only via dynamic SQL
still gets the via_use lib->lib edge (fixes the missing-edge case); an out-of-folder
library referenced by an in-folder consumer is a non-member provider node. Macro
libraries stay excluded from runs (via macros) and from the previewable scripts set.

Verified byte-for-byte (incl. in_pipeline) against the deployed graph: unused
in-folder lib, lexical lib->lib chain, // use dynamic-SQL lib->lib, out-of-folder
shared lib.
2026-07-06 01:36:42 +02:00

88 lines
4.3 KiB
TypeScript

import { expect, test } from "bun:test";
import {
detectMacroCalls,
parseCreateMacro,
parseMacroAnnotations,
parseMacroLibrary,
splitStatements,
} from "../src/commands/pipeline/duckdbMacros.ts";
// These mirror the accepted-shape cases of the backend parser
// (backend/parsers/windmill-parser/src/duckdb_macros.rs) so the local graph
// derives the same macro registry the server records at deploy.
test("parseCreateMacro: scalar, OR REPLACE / TEMP / TABLE, FUNCTION alias", () => {
expect(parseCreateMacro("CREATE MACRO surrogate_key(a, b) AS md5(concat_ws('||', a, b))")).toEqual(
{ name: "surrogate_key", params: "a, b", isTable: false },
);
expect(parseCreateMacro("CREATE OR REPLACE TEMP MACRO top_n(t_max) AS TABLE SELECT * FROM t")).toEqual(
{ name: "top_n", params: "t_max", isTable: true },
);
// FUNCTION is DuckDB's alias for MACRO; name lowercased
expect(parseCreateMacro("create function Dbl(a) as a * 2")).toEqual(
{ name: "dbl", params: "a", isTable: false },
);
});
test("parseCreateMacro: default params + nested/string parens in the param list", () => {
expect(
parseCreateMacro("CREATE MACRO safe_div(a, b, fallback := (0)) AS CASE WHEN b = 0 THEN fallback ELSE a / b END")?.params,
).toBe("a, b, fallback := (0)");
expect(parseCreateMacro("CREATE MACRO f(sep := '(') AS concat(sep, 'x')")?.params).toBe("sep := '('");
});
test("parseCreateMacro: rejects non-macro, qualified/quoted names, missing params/body", () => {
expect(parseCreateMacro("SELECT 1")).toBeNull();
expect(parseCreateMacro("CREATE TABLE t(x int)")).toBeNull();
expect(parseCreateMacro("CREATE MACRO lake.m(a) AS a")).toBeNull();
expect(parseCreateMacro('CREATE MACRO "weird name"(a) AS a')).toBeNull();
expect(parseCreateMacro("CREATE MACRO m AS 1")).toBeNull(); // no params
expect(parseCreateMacro("CREATE MACRO m(a) AS")).toBeNull(); // empty body
expect(parseCreateMacro("CREATE OR MACRO m(a) AS a")).toBeNull(); // OR without REPLACE
});
test("splitStatements strips comments/strings and parseMacroLibrary skips setup", () => {
const lib = `-- macros
ATTACH 'x.duckdb' AS ext;
CREATE MACRO m1(a) AS a; -- inline comment ; not a split
CREATE MACRO m2(b) AS b + 1;`;
expect(splitStatements("SELECT 1; -- c ; still\nSELECT ';' AS x;").length).toBe(2);
expect(parseMacroLibrary(lib).map((m) => m.name)).toEqual(["m1", "m2"]);
});
test("detectMacroCalls: word-boundary, case-insensitive, skips qualified/strings/comments", () => {
const ns = new Set(["dbl", "avg_x"]);
const found = detectMacroCalls("SELECT DBL(1), my_dbl(2), avg_x (3) FROM t", ns);
expect([...found].sort()).toEqual(["avg_x", "dbl"]);
expect(detectMacroCalls("SELECT lake.dbl(1)", new Set(["dbl"])).size).toBe(0);
expect(detectMacroCalls("SELECT 'dbl(1)'", new Set(["dbl"])).size).toBe(0);
expect(detectMacroCalls("-- dbl(1)\nSELECT 1", new Set(["dbl"])).size).toBe(0);
expect(detectMacroCalls("SELECT dbl FROM t", new Set(["dbl"])).size).toBe(0); // no call parens
});
test("parseMacroAnnotations: leading-header only, marker stands alone, `// use` needs a path token", () => {
expect(parseMacroAnnotations("-- macros\nCREATE MACRO m(a) AS a;")).toEqual({
macros: true,
useLibs: [],
});
// marker with trailing prose is not the macros marker
expect(parseMacroAnnotations("-- macros are below\nSELECT 1;").macros).toBe(false);
// `// use` accumulates path-shaped tokens, dedups, and stops at the body
expect(
parseMacroAnnotations("-- pipeline\n-- use f/lib/a\n-- use f/lib/a\n-- use notapath\nSELECT 1;").useLibs,
).toEqual(["f/lib/a"]);
// an annotation after code is ignored (leading header only)
expect(parseMacroAnnotations("SELECT 1;\n-- use f/lib/a").useLibs).toEqual([]);
});
test("parseMacroAnnotations accepts `//`, `--`, and `#` prefixes (backend parity, any language)", () => {
// The canonical backend parser strips `//`/`--`/`#` regardless of language, so
// a DuckDB library headed with `// macros` (not `-- macros`) must be detected.
expect(parseMacroAnnotations("// macros\nCREATE MACRO m(a) AS a;").macros).toBe(true);
expect(parseMacroAnnotations("# macros\nSELECT 1;").macros).toBe(true);
expect(parseMacroAnnotations("// use f/lib/a\nSELECT 1;").useLibs).toEqual(["f/lib/a"]);
// mixed prefixes in one header both register
expect(parseMacroAnnotations("// pipeline\n-- use f/lib/a\nSELECT 1;").useLibs).toEqual(["f/lib/a"]);
});