fix(cli): macro-library parity in --local pipeline graph + read-only run --dry-run (#9942)

* fix(cli): surface macro libraries in --local pipeline graph + make run --dry-run read-only

* fix(cli): resolve workspace-wide macro libraries in --local graph (shared libs outside the pipeline folder)

* fix(cli): macro-lib consumers + //-prefix parity in --local pipeline graph

Address Codex review P1s: (1) macro libraries that consume another library's
macros now produce lib->lib edges (any folder DuckDB script is a consumer, not
just // pipeline members) so an upstream provider node no longer disappears;
(2) parseMacroAnnotations accepts //, --, and # prefixes like the backend, so a
.duckdb.sql library headed with // macros is detected locally. Both edge
endpoints are forced into the node set. Verified byte-for-byte against deployed.

* fix(cli): exclude non-pipeline macro-consumer nodes from --local run selection

Address Codex P1: buildMacroEdges surfaces macro-consumer nodes (a DuckDB script
calling a macro but not marked // pipeline) for lineage display. Those have no
local file, so pipeline run --local must not treat them as manual roots — a
dry-run listed them and a real run failed resolving local content. Exclude any
--local graph node absent from localScripts (the previewable set) from starts and
selection, alongside the existing macro-library exclusion.

* fix(cli): reject display-only macro consumers in explicit --from (post-merge with #9945)

The mid-DAG --from feature (#9945, now on main) admits any autorun-able script
via validFromStarts/fromEligible, which was filtered only by macroLibPaths. A
non-// pipeline macro-consumer helper (a --local display node) therefore passed
--from eligibility and produced an empty plan. Filter fromEligible by the broader
notRunnablePaths too, and reject such a --from with a clear message instead of a
silent empty plan.

* chore(cli): remove NUL edge-key separator + refresh stale macro comments

Address Codex P2 nits: (1) the macro edge map packed (lib, consumer) into a
string with a literal NUL separator, which made localGraph.ts read as a binary
file to grep/rg — replace with a nested lib->consumer Map (no separator); (2)
comments claiming macro nodes/edges are 'deployed graph only' contradicted this
PR's local derivation — describe the code as it is.

* fix(cli): tag unused // pipeline + // macros libraries so --local run excludes them

Address Codex P1: the deployed builder sets 'macros' on any node whose path
provides macros (edge or not), so a // pipeline + // macros script with no
consumers is still recognized as definition-only. Local enrichment only tagged
edge providers, leaving an unused pipeline macro library as a bare runnable that
pipeline run --local would schedule as a manual root. Also tag any library whose
path is already a runnable; unused non-pipeline libraries stay suppressed.

* fix(pipelines): `// macros` takes precedence over `// pipeline` (a library is never a member)

A macro library is definition-only — its macros are injected into consumers and
running it is a no-op — so marking it `// pipeline` is meaningless and only
produced a confusing state (an unused pipeline macro library appearing as a
manual root). Make `// macros` win: parse_pipeline_annotations forces in_pipeline
false when macros is set. Mirrored in all three parsers that must agree — the Rust
canonical parser (drives deploy membership), the frontend TS parser (live graph),
and the CLI local graph (pinned wasm still reports in_pipeline, so precedence is
applied when skipping members). Shared parity fixture + unit tests on each side.

* docs(cli): trim narrative comment blocks to non-obvious constraints

Address Codex P2: duckdbMacros.ts opened with a ~19-line narrative block whose
parity rationale belongs in the PR description; reduce to the two real constraints
(keep in lockstep with duckdb_macros.rs; dynamic-SQL calls need // use). Per the
AGENTS.md comment policy.

* fix(cli): model macro libraries as pipeline members, matching the deployed graph

Reverts the parser-precedence approach (b398b69): the backend deliberately marks
EVERY macro library auto_kind='pipeline' (scripts.rs:1474, macro_lib_defs), so a
macro library IS a graph member — the // pipeline marker is redundant, not
authoritative. Precedence was a no-op on deploy while diverging the CLI/frontend.

Instead mirror reality in the CLI local graph: an in-folder // macros library is a
member node (in_pipeline=true, with signatures) whether used or not; its // use is
processed (it's a member) so a library that reaches another only via dynamic SQL
still gets the via_use lib->lib edge (fixes the missing-edge case); an out-of-folder
library referenced by an in-folder consumer is a non-member provider node. Macro
libraries stay excluded from runs (via macros) and from the previewable scripts set.

Verified byte-for-byte (incl. in_pipeline) against the deployed graph: unused
in-folder lib, lexical lib->lib chain, // use dynamic-SQL lib->lib, out-of-folder
shared lib.
This commit is contained in:
Ruben Fiszel
2026-07-06 01:36:42 +02:00
committed by GitHub
parent b13113964a
commit e3f43033ca
6 changed files with 961 additions and 27 deletions
+46 -3
View File
@@ -43,7 +43,7 @@ async function fetchDeployedGraph(
}
// Render the pipeline graph as a markdown document.
function generatePipelineMarkdown(
export function generatePipelineMarkdown(
folder: string,
graph: AssetGraph,
datatableSchemas: any[],
@@ -73,7 +73,26 @@ function generatePipelineMarkdown(
}
}
const scripts = graph.runnables.filter((r) => r.usage_kind === "script").map((r) => r.path).sort();
// `// macros` libraries are definition-only nodes (not runnable pipeline
// steps), so they get their own section below and are excluded from the
// per-script listing + the script count.
const macroLibs = graph.runnables
.filter((r) => r.usage_kind === "script" && (r.macros?.length ?? 0) > 0)
.sort((a, b) => a.path.localeCompare(b.path));
const macroLibPaths = new Set(macroLibs.map((r) => r.path));
const macroConsumersByLib = new Map<string, { consumer: string; names: string[]; viaUse?: boolean }[]>();
for (const me of graph.macro_edges ?? []) {
(macroConsumersByLib.get(me.lib_path) ?? macroConsumersByLib.set(me.lib_path, []).get(me.lib_path)!).push({
consumer: me.consumer_path,
names: me.macro_names,
viaUse: me.via_use,
});
}
const scripts = graph.runnables
.filter((r) => r.usage_kind === "script" && !macroLibPaths.has(r.path))
.map((r) => r.path)
.sort();
let md = `# Pipeline \`f/${folder}\`
@@ -85,7 +104,7 @@ produces data by reading/writing assets in its body (\`datatable://\`,
\`ducklake://\`, \`s3://\`, \`volume://\`). The cascade runs a producer, then every
downstream subscriber, in topological order.
- **${scripts.length}** script${scripts.length === 1 ? "" : "s"} · **${graph.assets.length}** asset${graph.assets.length === 1 ? "" : "s"}
- **${scripts.length}** script${scripts.length === 1 ? "" : "s"} · **${graph.assets.length}** asset${graph.assets.length === 1 ? "" : "s"}${macroLibs.length > 0 ? ` · **${macroLibs.length}** macro librar${macroLibs.length === 1 ? "y" : "ies"}` : ""}
## Scripts
@@ -107,6 +126,30 @@ downstream subscriber, in topological order.
md += `\n`;
}
// Macro libraries: `// macros` scripts whose macros are injected into consuming
// DuckDB scripts at run time. List each library's signatures and its callers so
// an agent discovers the reuse layer instead of re-inlining the logic.
if (macroLibs.length > 0) {
md += `## Macro libraries\n\n`;
md += `\`// macros\` DuckDB libraries. Their \`CREATE MACRO\` definitions are injected as\nTEMP macros into consuming scripts at run time — call a macro by name, or force the\nwhole library in with \`// use <lib-path>\` (needed for macros only reached via dynamic SQL).\n\n`;
for (const lib of macroLibs) {
md += `### \`${lib.path}\`\n\n`;
for (const m of lib.macros ?? []) {
md += `- \`${m.name}(${m.params ?? ""})\`${m.is_table ? " → TABLE" : ""}\n`;
}
const consumers = [...(macroConsumersByLib.get(lib.path) ?? [])].sort((a, b) =>
a.consumer.localeCompare(b.consumer),
);
if (consumers.length > 0) {
md += `- **Used by:**\n`;
for (const c of consumers) {
md += ` - \`${c.consumer}\` (${c.viaUse ? "via \`// use\`" : `calls ${c.names.map((n) => `\`${n}\``).join(", ")}`})\n`;
}
}
md += `\n`;
}
}
// Datatable schemas, restricted to datatables this pipeline references.
const referencedDatatables = new Set(
graph.assets.filter((a) => a.kind === "datatable").map((a) => a.path.split("/")[0]),
+261
View File
@@ -0,0 +1,261 @@
// Faithful TS port of the lexical rules in
// `backend/parsers/windmill-parser/src/duckdb_macros.rs` — keep the two in
// lockstep. Deliberately lexical, not an AST parse: over-matching a call only
// draws a spurious edge, never a wrong run. A macro reached only through dynamic
// SQL (`query('…')`) is invisible to call detection here as on the server —
// `// use <lib>` forces the whole-library edge instead.
export type ParsedMacro = {
// Lowercased bare identifier (DuckDB identifiers are case-insensitive unquoted;
// qualified / quoted names are rejected, matching the backend).
name: string;
// Verbatim text inside the parameter parens (may be empty).
params: string;
isTable: boolean;
};
// Case-insensitive whole-word prefix strip (whitespace-bounded). Returns the
// remainder with leading whitespace trimmed, or null when `s` does not start with
// `kw` as a whole word. `kw` must be lowercase. Mirrors `strip_kw`.
function stripKw(s: string, kw: string): string | null {
if (s.length < kw.length) return null;
if (s.slice(0, kw.length).toLowerCase() !== kw) return null;
const after = s.slice(kw.length);
if (after.length === 0 || /^\s/.test(after)) return after.replace(/^\s+/, "");
return null;
}
function isIdent(s: string): boolean {
return /^[A-Za-z_][A-Za-z0-9_]*$/.test(s);
}
// Split a script body into `;`-terminated statements with line comments
// (`--`, `//`), block comments (`/* */`), and quoted spans stripped/preserved as
// in the SQL executor. Char-wise (not byte-wise) so multi-byte text survives.
// Mirrors `split_statements`.
export function splitStatements(sql: string): string[] {
const out: string[] = [];
let cur = "";
const chars = [...sql];
let i = 0;
const n = chars.length;
while (i < n) {
const c = chars[i];
// line comment: `--` (SQL) or `//` (Windmill annotation prefix)
if (
(c === "-" && i + 1 < n && chars[i + 1] === "-") ||
(c === "/" && i + 1 < n && chars[i + 1] === "/")
) {
while (i < n && chars[i] !== "\n") i += 1;
continue;
}
// block comment
if (c === "/" && i + 1 < n && chars[i + 1] === "*") {
i += 2;
while (i + 1 < n && !(chars[i] === "*" && chars[i + 1] === "/")) i += 1;
i += 2;
continue;
}
// single-quoted string ('' escapes an embedded quote)
if (c === "'") {
cur += c;
i += 1;
while (i < n) {
cur += chars[i];
if (chars[i] === "'") {
if (i + 1 < n && chars[i + 1] === "'") {
cur += "'";
i += 2;
continue;
}
i += 1;
break;
}
i += 1;
}
continue;
}
// double-quoted identifier
if (c === '"') {
cur += c;
i += 1;
while (i < n) {
cur += chars[i];
if (chars[i] === '"') {
i += 1;
break;
}
i += 1;
}
continue;
}
if (c === ";") {
const t = cur.trim();
if (t !== "") out.push(t);
cur = "";
i += 1;
continue;
}
cur += c;
i += 1;
}
const t = cur.trim();
if (t !== "") out.push(t);
return out;
}
// Parse one comment-free, `;`-less statement as a CREATE MACRO. Returns null when
// the statement is not a well-formed macro definition (unlike the backend, which
// distinguishes "not macro-shaped" from "malformed" to surface deploy errors —
// the local graph is lenient and simply skips anything it can't read). Mirrors
// `parse_create_macro` for the shape it accepts.
export function parseCreateMacro(stmt: string): ParsedMacro | null {
let rest = stripKw(stmt.trim(), "create");
if (rest === null) return null;
const afterOr = stripKw(rest, "or");
if (afterOr !== null) {
const afterReplace = stripKw(afterOr, "replace");
if (afterReplace === null) return null; // CREATE OR <not replace>
rest = afterReplace;
}
const afterTemp = stripKw(rest, "temp") ?? stripKw(rest, "temporary");
if (afterTemp !== null) rest = afterTemp;
// `FUNCTION` is DuckDB's alias for `MACRO`.
const afterMacro = stripKw(rest, "macro") ?? stripKw(rest, "function");
if (afterMacro === null) return null;
rest = afterMacro;
const nameEndMatch = rest.match(/[\s(]/);
const nameEnd = nameEndMatch ? nameEndMatch.index! : rest.length;
const rawName = rest.slice(0, nameEnd);
if (rawName === "" || rawName.includes(".") || rawName.includes('"') || !isIdent(rawName)) {
return null;
}
const name = rawName.toLowerCase();
rest = rest.slice(nameEnd).replace(/^\s+/, "");
if (!rest.startsWith("(")) return null;
// Balanced-paren scan for the verbatim param list; skip quoted spans (a default
// value may contain a paren).
const pchars = [...rest];
let depth = 0;
let j = 0;
let close = -1;
while (j < pchars.length) {
const ch = pchars[j];
if (ch === "(") depth += 1;
else if (ch === ")") {
depth -= 1;
if (depth === 0) {
close = j;
break;
}
} else if (ch === "'" || ch === '"') {
const q = ch;
j += 1;
while (j < pchars.length && pchars[j] !== q) j += 1;
}
j += 1;
}
if (close === -1) return null; // unbalanced
const params = pchars.slice(1, close).join("").trim();
const afterParams = pchars.slice(close + 1).join("").replace(/^\s+/, "");
let body = stripKw(afterParams, "as");
if (body === null) return null;
let isTable = false;
const afterTable = stripKw(body, "table");
if (afterTable !== null) {
body = afterTable;
isTable = true;
}
const trimmedBody = body.replace(/;+\s*$/, "").trim();
if (trimmedBody === "") return null; // empty body
return { name, params, isTable };
}
// All well-formed macro definitions in a `// macros` library body, in source
// order. Non-macro statements (setup: ATTACH/INSTALL/… ) are skipped.
export function parseMacroLibrary(sql: string): ParsedMacro[] {
const out: ParsedMacro[] = [];
for (const stmt of splitStatements(sql)) {
const m = parseCreateMacro(stmt);
if (m) out.push(m);
}
return out;
}
// Names from `names` that `sql` calls: an identifier token immediately followed
// by `(` (after optional whitespace), not `.`-qualified, outside strings and
// comments. Lexical (over-matching only adds an unused edge). Mirrors
// `detect_macro_calls`.
export function detectMacroCalls(sql: string, names: Set<string>): Set<string> {
const found = new Set<string>();
if (names.size === 0) return found;
for (const stmt of splitStatements(sql)) {
const chars = [...stmt];
const n = chars.length;
let i = 0;
let prev: string | undefined = undefined;
while (i < n) {
const c = chars[i];
if (c === "'" || c === '"') {
const q = c;
i += 1;
while (i < n && chars[i] !== q) i += 1;
i += 1;
prev = q;
continue;
}
const isIdentStart = /[A-Za-z_]/.test(c);
const prevBlocks = prev !== undefined && /[.A-Za-z0-9_]/.test(prev);
if (isIdentStart && !prevBlocks) {
const start = i;
while (i < n && /[A-Za-z0-9_]/.test(chars[i])) i += 1;
const word = chars.slice(start, i).join("").toLowerCase();
let k = i;
while (k < n && /\s/.test(chars[k])) k += 1;
if (k < n && chars[k] === "(" && names.has(word)) found.add(word);
prev = chars[i - 1];
continue;
}
prev = c;
i += 1;
}
}
return found;
}
// Scan the LEADING comment header for the `// macros` marker and `// use <lib>`
// annotations, independent of the wasm parser (which drops both). Faithful mirror
// of the canonical `parse_pipeline_annotations` (asset_parser.rs): each line's
// prefix is stripped as `//`, `--`, or `#` REGARDLESS of language (a DuckDB
// library may legitimately head its annotations with `// macros`), scanning stops
// at the first non-comment line, the marker must stand alone on its line, and a
// `// use` target is a single whitespace-free token containing `/`.
export function parseMacroAnnotations(
content: string,
): { macros: boolean; useLibs: string[] } {
let macros = false;
const useLibs: string[] = [];
for (const raw of content.split("\n")) {
const line = raw.replace(/^\s+/, "");
if (line === "") continue;
let rest: string;
if (line.startsWith("//") || line.startsWith("--")) rest = line.slice(2);
else if (line.startsWith("#")) rest = line.slice(1);
else break; // first line of actual code ends the annotation header
rest = rest.replace(/^\s+/, "");
// marker: `macros` as the only content on the line (rejects `macros_v2`,
// `macros are below`)
if (/^macros\s*$/.test(rest)) {
macros = true;
continue;
}
// `use <lib>`: a single whitespace-free path token (must contain `/`, so
// prose like `use this to …` is dropped)
const u = rest.match(/^use\s+(\S+)\s*$/);
if (u && u[1].includes("/") && !useLibs.includes(u[1])) useLibs.push(u[1]);
}
return { macros, useLibs };
}
+191 -3
View File
@@ -19,6 +19,12 @@ import * as log from "../../core/log.ts";
import { exts, removeExtensionToPath } from "../script/script.ts";
import { inferContentTypeFromFilePath } from "../../utils/script_common.ts";
import { getWmillYamlPath } from "../../core/conf.ts";
import {
detectMacroCalls,
parseMacroAnnotations,
parseMacroLibrary,
type ParsedMacro,
} from "./duckdbMacros.ts";
// Resolve the workspace root (the directory containing wmill.yaml) for reading
// local files, falling back to the current directory.
@@ -53,8 +59,9 @@ export type GraphRunnable = {
// schema-contract warnings; only present when set (default `warn` is absent),
// mirroring the deployed graph node.
materialize_on_schema_change?: string;
// `// macros` library: the macros it defines (deployed graph only — the wasm
// asset parser does not emit them). Non-empty ⇒ definition-only node.
// `// macros` library: the macros it defines. Derived locally by
// `buildMacroEdges` (the wasm asset parser emits neither the marker nor the
// registry). Non-empty ⇒ definition-only node.
macros?: { name: string; params?: string; is_table?: boolean }[];
};
export type GraphEdge = {
@@ -89,7 +96,7 @@ export type AssetGraph = {
edges: GraphEdge[];
triggers: GraphTrigger[];
// ƒ edges from a `// macros` library to the scripts calling its macros
// (deployed graph only).
// (present on both the deployed graph and the locally-derived one).
macro_edges?: {
lib_path: string;
consumer_path: string;
@@ -456,6 +463,10 @@ export async function buildLocalPipelineGraph(args: {
const folderDir = path.join(args.root, "f", folderClean);
const all = await collectScripts(folderDir, args.root, args.defaultTs);
// `// macros` DuckDB libraries across the whole workspace (a pipeline may use a
// shared library outside its folder).
const libMacros = collectMacroLibraries(args.root);
const runnables: GraphRunnable[] = [];
const edges: GraphEdge[] = [];
const triggers: GraphTrigger[] = [];
@@ -471,6 +482,26 @@ export async function buildLocalPipelineGraph(args: {
for (const s of all) {
const out = await inferScriptAssets(s.content, s.language);
// A `// macros` library is a pipeline-member node carrying its macro
// signatures — whether or not any consumer uses it — matching the deployed
// graph, which marks every macro library `auto_kind='pipeline'`. It is
// definition-only (no assets/triggers, never run), so we add just the node;
// it is NOT a previewable script (excluded from `pipelineScripts`).
const macroLibDefs = libMacros.get(s.path);
if (macroLibDefs) {
runnables.push({
path: s.path,
usage_kind: "script",
in_pipeline: true,
...(out.tag ? { tag: out.tag } : {}),
macros: macroLibDefs.map((m) => ({
name: m.name,
params: m.params,
is_table: m.isTable,
})),
});
continue;
}
if (!out.in_pipeline) continue; // not a pipeline member
const retry = normalizeRetry(out.retry);
const nativeTriggers = recoverHeaderNativeTriggers(s.content, s.language);
@@ -598,6 +629,8 @@ export async function buildLocalPipelineGraph(args: {
}
}
const macroEdges = buildMacroEdges(all, libMacros, runnables);
const assets = [...assetSet.entries()].map(([key, a]) => {
const derived_from = a.derived_from ?? derivedFromByKey.get(key);
return derived_from ? { ...a, derived_from } : a;
@@ -609,7 +642,162 @@ export async function buildLocalPipelineGraph(args: {
assets,
edges,
triggers,
...(macroEdges.length > 0 ? { macro_edges: macroEdges } : {}),
},
scripts: pipelineScripts,
};
}
// Every `// macros` DuckDB library in the workspace, keyed by its windmill path,
// mapped to its parsed macro definitions. Walks the whole `f/` tree (not just the
// pipeline folder) because the deployed graph resolves consumers against the
// workspace-wide macro registry — a pipeline may `// use` / call a shared library
// living outside its own folder. Only `*.duckdb.sql` files are read (macros are
// DuckDB-only), so the extra walk stays cheap.
function collectMacroLibraries(root: string): Map<string, ParsedMacro[]> {
const out = new Map<string, ParsedMacro[]>();
const walk = (dir: string) => {
let entries: fs.Dirent[];
try {
entries = fs.readdirSync(dir, { withFileTypes: true });
} catch {
return;
}
for (const e of entries) {
if (e.name.startsWith(".") || e.name === "node_modules") continue;
const abs = path.join(dir, e.name);
if (e.isDirectory()) {
walk(abs);
continue;
}
if (!e.isFile() || !e.name.endsWith(".duckdb.sql")) continue;
let content: string;
try {
content = fs.readFileSync(abs, "utf-8");
} catch {
continue;
}
if (!parseMacroAnnotations(content).macros) continue;
const relFromRoot = path.relative(root, abs).replaceAll("\\", "/");
out.set(removeExtensionToPath(relFromRoot), parseMacroLibrary(content));
}
};
walk(path.join(root, "f"));
return out;
}
// Derive lib→consumer edges (lexical calls + `// use`). In-folder libraries are
// already member nodes; this adds any out-of-folder provider referenced by an
// in-folder consumer. Mirrors the deployed `asset_graph`: libraries resolve
// workspace-wide, consumers are folder-scoped.
function buildMacroEdges(
all: LocalScript[],
libMacros: Map<string, ParsedMacro[]>,
runnables: GraphRunnable[],
): NonNullable<AssetGraph["macro_edges"]> {
if (libMacros.size === 0) return [];
const useLibsByScript = new Map<string, string[]>();
for (const s of all) {
if (s.language !== "duckdb") continue;
const { useLibs } = parseMacroAnnotations(s.content);
if (useLibs.length > 0) useLibsByScript.set(s.path, useLibs);
}
// Macro name → providing library. Names are workspace-unique (the deploy path
// enforces this); on a local collision, last-writer-wins, harmlessly.
const providerByName = new Map<string, string>();
for (const [lib, macros] of libMacros) {
for (const m of macros) providerByName.set(m.name, lib);
}
const allMacroNames = new Set(providerByName.keys());
// Aggregate per (lib, consumer): the set of called macro names and whether the
// edge came (also) from a whole-library `// use`. Consumers are folder-scoped
// (`all` is this folder's scripts).
const pipelinePaths = new Set(runnables.map((r) => r.path));
type EdgeAgg = { names: Set<string>; viaUse: boolean };
// lib_path → consumer_path → aggregate. Nested (not a packed single-string key)
// so no separator can ever collide with a path.
const edgeMap = new Map<string, Map<string, EdgeAgg>>();
const aggFor = (lib: string, consumer: string): EdgeAgg => {
let byConsumer = edgeMap.get(lib);
if (!byConsumer) {
byConsumer = new Map();
edgeMap.set(lib, byConsumer);
}
let agg = byConsumer.get(consumer);
if (!agg) {
agg = { names: new Set(), viaUse: false };
byConsumer.set(consumer, agg);
}
return agg;
};
for (const s of all) {
if (s.language !== "duckdb") continue;
// Lexical call edges: the deploy path records `macro_usage` for EVERY DuckDB
// script in the folder (not only pipeline members) — including a macro
// library that calls another library's macros — so a lib→lib edge and its
// upstream provider node survive. Mirror that: any folder DuckDB script is a
// candidate consumer here.
for (const name of detectMacroCalls(s.content, allMacroNames)) {
const lib = providerByName.get(name)!;
if (lib === s.path) continue; // a library calling its own macro is not an edge
aggFor(lib, s.path).names.add(name);
}
// `// use` whole-library edges: the deployed graph re-parses these only from
// pipeline members (`annotations_by_path`), so scope them the same way.
if (!pipelinePaths.has(s.path)) continue;
for (const lib of useLibsByScript.get(s.path) ?? []) {
const macros = libMacros.get(lib);
// An out-of-tree / unknown `// use` target can't be resolved locally.
if (!macros || lib === s.path) continue;
const agg = aggFor(lib, s.path);
agg.viaUse = true;
for (const m of macros) agg.names.add(m.name);
}
}
const edges = [...edgeMap.entries()]
.flatMap(([lib_path, byConsumer]) =>
[...byConsumer.entries()].map(([consumer_path, agg]) => ({
lib_path,
consumer_path,
macro_names: [...agg.names].sort(),
// `via_use` is always present (the deployed `MacroEdge` serializes it
// unconditionally) so `--json` matches byte-for-byte.
via_use: agg.viaUse,
})),
)
.sort((a, b) =>
a.lib_path.localeCompare(b.lib_path) ||
a.consumer_path.localeCompare(b.consumer_path),
);
// In-folder libraries are already member nodes (added above with `in_pipeline`
// + macros). An OUT-OF-folder library referenced by an in-folder consumer is
// added here as a non-member provider node (no `in_pipeline`), matching the
// deployed graph, which folder-scopes membership but pulls the out-of-folder
// provider in as the edge's endpoint.
const libPaths = new Set<string>(edges.map((e) => e.lib_path));
for (const lib of libPaths) {
if (runnables.some((r) => r.path === lib)) continue;
const macros = (libMacros.get(lib) ?? []).map((m) => ({
name: m.name,
params: m.params,
is_table: m.isTable,
}));
runnables.push({ path: lib, usage_kind: "script", macros });
}
// Force every edge's CONSUMER endpoint into the node set too, like the deployed
// builder, so no edge dangles at a missing runnable. Members and libraries are
// already nodes; only a non-member DuckDB helper (calls a macro but isn't
// `// pipeline` and isn't itself a library) needs a bare node here.
for (const consumer of new Set(edges.map((e) => e.consumer_path))) {
if (!runnables.some((r) => r.path === consumer)) {
runnables.push({ path: consumer, usage_kind: "script" });
}
}
runnables.sort((a, b) => a.path.localeCompare(b.path));
return edges;
}
+50 -21
View File
@@ -304,8 +304,9 @@ async function renderGraph(
}
// `// macros` libraries: badge the node with the macros it defines and list
// its consumers as ƒ edges. Deployed graphs only — the local wasm parse does
// not emit `macros`/`macro_edges`, so local mode renders them as plain nodes.
// its consumers as ƒ edges. Populated on both the deployed graph and the local
// graph (localGraph.ts derives `macros`/`macro_edges` from the working tree,
// since the wasm asset parser emits neither).
const macrosByLib = new Map(
graph.runnables
.filter((r) => (r.macros?.length ?? 0) > 0)
@@ -590,19 +591,25 @@ async function run(
// Resolve relative imports from local (not-yet-deployed) content so a script
// that imports a sibling workspace lib previews against local edits. Same
// trick as `wmill dev` / `wmill app dev`; degrades to undefined gracefully.
try {
const codebases = await listSyncCodebases(merged);
const { buildPreviewTempScriptRefs } = await import(
"../generate-metadata/generate-metadata.ts"
);
tempScriptRefs = await buildPreviewTempScriptRefs(
workspace,
merged,
codebases,
{ kind: "all" },
);
} catch {
// best-effort: relative imports fall back to deployed versions
// Skipped for `--dry-run`: it locks/uploads scripts as a side effect (it
// regenerates `*.script.yaml` / `*.script.lock` / `wmill-lock.yaml`), and a
// plan preview must stay READ-ONLY — the refs are only consumed when a
// preview job actually runs below, which dry-run never reaches.
if (!opts.dryRun) {
try {
const codebases = await listSyncCodebases(merged);
const { buildPreviewTempScriptRefs } = await import(
"../generate-metadata/generate-metadata.ts"
);
tempScriptRefs = await buildPreviewTempScriptRefs(
workspace,
merged,
codebases,
{ kind: "all" },
);
} catch {
// best-effort: relative imports fall back to deployed versions
}
}
} else {
graph = await apiGet<BCGraph>(
@@ -671,6 +678,20 @@ async function run(
.filter((r) => r.usage_kind === "script" && (r.macros?.length ?? 0) > 0)
.map((r) => r.path),
);
// Non-runnable graph nodes: the macro libraries above, plus (under `--local`)
// any script node with no local file. The local graph surfaces macro-consumer
// nodes for lineage display — a DuckDB script that calls a macro but isn't a
// `// pipeline` member (so has no previewable content). They must never enter
// the run: as a manual root they'd be scheduled, and a preview would fail with
// no local content to send.
const notRunnablePaths = new Set<string>(macroLibPaths);
if (opts.local && localScripts) {
for (const r of graph.runnables) {
if (r.usage_kind === "script" && !localScripts.has(r.path)) {
notRunnablePaths.add(r.path);
}
}
}
// Resolve the start. Two eligibility sets:
// `starts` — schedule/manual roots (+ `--upload`-bound handlers). The
@@ -684,12 +705,12 @@ async function run(
// unless `--upload`-bound.
const starts = new Set(
[...validStarts(graph), ...boundNodeIds].filter(
(id) => !macroLibPaths.has(scriptPathOf(id)),
(id) => !notRunnablePaths.has(scriptPathOf(id)),
),
);
const fromEligible = new Set(
[...validFromStarts(graph), ...boundNodeIds].filter(
(id) => !macroLibPaths.has(scriptPathOf(id)),
(id) => !notRunnablePaths.has(scriptPathOf(id)),
),
);
// `runAll` = no `--from` on a multi-root pipeline (fan-in): run the whole
@@ -718,6 +739,14 @@ async function run(
`--from '${opts.from}' is a \`// macros\` library — definition-only, injected into consuming scripts at run time, never a runnable step.`,
);
}
// A `--local` display-only node (a non-`// pipeline` DuckDB script surfaced
// only because it calls a macro) has no previewable content — reject it
// clearly rather than admitting it and silently producing an empty plan.
if (notRunnablePaths.has(scriptPathOf(resolved))) {
throw new Error(
`--from '${opts.from}' isn't a \`// pipeline\` script — it appears in the local graph only as a macro consumer (lineage). Mark it \`// pipeline\` to run it.`,
);
}
if (!resolved.startsWith("script:")) {
throw new Error(
`--from '${opts.from}' is an asset, not a runnable — start a run from a script (assets are produced, not run).`,
@@ -829,10 +858,10 @@ async function run(
}
}
// Selections can still pull a macro library in via graph reachability (e.g.
// whole-pipeline mode collects every root's closure) — drop them before
// ordering so they never run.
for (const p of macroLibPaths) selectedScripts.delete(p);
// Selections can still pull a non-runnable node in via graph reachability (e.g.
// whole-pipeline mode collects every root's closure) — drop macro libraries and
// local-only display nodes before ordering so they never run.
for (const p of notRunnablePaths) selectedScripts.delete(p);
const { order, cyclic } = topoOrder(graph, selectedScripts);
if (cyclic.length > 0) {
+87
View File
@@ -0,0 +1,87 @@
import { expect, test } from "bun:test";
import {
detectMacroCalls,
parseCreateMacro,
parseMacroAnnotations,
parseMacroLibrary,
splitStatements,
} from "../src/commands/pipeline/duckdbMacros.ts";
// These mirror the accepted-shape cases of the backend parser
// (backend/parsers/windmill-parser/src/duckdb_macros.rs) so the local graph
// derives the same macro registry the server records at deploy.
test("parseCreateMacro: scalar, OR REPLACE / TEMP / TABLE, FUNCTION alias", () => {
expect(parseCreateMacro("CREATE MACRO surrogate_key(a, b) AS md5(concat_ws('||', a, b))")).toEqual(
{ name: "surrogate_key", params: "a, b", isTable: false },
);
expect(parseCreateMacro("CREATE OR REPLACE TEMP MACRO top_n(t_max) AS TABLE SELECT * FROM t")).toEqual(
{ name: "top_n", params: "t_max", isTable: true },
);
// FUNCTION is DuckDB's alias for MACRO; name lowercased
expect(parseCreateMacro("create function Dbl(a) as a * 2")).toEqual(
{ name: "dbl", params: "a", isTable: false },
);
});
test("parseCreateMacro: default params + nested/string parens in the param list", () => {
expect(
parseCreateMacro("CREATE MACRO safe_div(a, b, fallback := (0)) AS CASE WHEN b = 0 THEN fallback ELSE a / b END")?.params,
).toBe("a, b, fallback := (0)");
expect(parseCreateMacro("CREATE MACRO f(sep := '(') AS concat(sep, 'x')")?.params).toBe("sep := '('");
});
test("parseCreateMacro: rejects non-macro, qualified/quoted names, missing params/body", () => {
expect(parseCreateMacro("SELECT 1")).toBeNull();
expect(parseCreateMacro("CREATE TABLE t(x int)")).toBeNull();
expect(parseCreateMacro("CREATE MACRO lake.m(a) AS a")).toBeNull();
expect(parseCreateMacro('CREATE MACRO "weird name"(a) AS a')).toBeNull();
expect(parseCreateMacro("CREATE MACRO m AS 1")).toBeNull(); // no params
expect(parseCreateMacro("CREATE MACRO m(a) AS")).toBeNull(); // empty body
expect(parseCreateMacro("CREATE OR MACRO m(a) AS a")).toBeNull(); // OR without REPLACE
});
test("splitStatements strips comments/strings and parseMacroLibrary skips setup", () => {
const lib = `-- macros
ATTACH 'x.duckdb' AS ext;
CREATE MACRO m1(a) AS a; -- inline comment ; not a split
CREATE MACRO m2(b) AS b + 1;`;
expect(splitStatements("SELECT 1; -- c ; still\nSELECT ';' AS x;").length).toBe(2);
expect(parseMacroLibrary(lib).map((m) => m.name)).toEqual(["m1", "m2"]);
});
test("detectMacroCalls: word-boundary, case-insensitive, skips qualified/strings/comments", () => {
const ns = new Set(["dbl", "avg_x"]);
const found = detectMacroCalls("SELECT DBL(1), my_dbl(2), avg_x (3) FROM t", ns);
expect([...found].sort()).toEqual(["avg_x", "dbl"]);
expect(detectMacroCalls("SELECT lake.dbl(1)", new Set(["dbl"])).size).toBe(0);
expect(detectMacroCalls("SELECT 'dbl(1)'", new Set(["dbl"])).size).toBe(0);
expect(detectMacroCalls("-- dbl(1)\nSELECT 1", new Set(["dbl"])).size).toBe(0);
expect(detectMacroCalls("SELECT dbl FROM t", new Set(["dbl"])).size).toBe(0); // no call parens
});
test("parseMacroAnnotations: leading-header only, marker stands alone, `// use` needs a path token", () => {
expect(parseMacroAnnotations("-- macros\nCREATE MACRO m(a) AS a;")).toEqual({
macros: true,
useLibs: [],
});
// marker with trailing prose is not the macros marker
expect(parseMacroAnnotations("-- macros are below\nSELECT 1;").macros).toBe(false);
// `// use` accumulates path-shaped tokens, dedups, and stops at the body
expect(
parseMacroAnnotations("-- pipeline\n-- use f/lib/a\n-- use f/lib/a\n-- use notapath\nSELECT 1;").useLibs,
).toEqual(["f/lib/a"]);
// an annotation after code is ignored (leading header only)
expect(parseMacroAnnotations("SELECT 1;\n-- use f/lib/a").useLibs).toEqual([]);
});
test("parseMacroAnnotations accepts `//`, `--`, and `#` prefixes (backend parity, any language)", () => {
// The canonical backend parser strips `//`/`--`/`#` regardless of language, so
// a DuckDB library headed with `// macros` (not `-- macros`) must be detected.
expect(parseMacroAnnotations("// macros\nCREATE MACRO m(a) AS a;").macros).toBe(true);
expect(parseMacroAnnotations("# macros\nSELECT 1;").macros).toBe(true);
expect(parseMacroAnnotations("// use f/lib/a\nSELECT 1;").useLibs).toEqual(["f/lib/a"]);
// mixed prefixes in one header both register
expect(parseMacroAnnotations("// pipeline\n-- use f/lib/a\nSELECT 1;").useLibs).toEqual(["f/lib/a"]);
});
+326
View File
@@ -321,6 +321,300 @@ test("`# volume:` producer connects to its `# on volume://` consumer", async ()
);
});
test("`// macros` library: node with signatures + call-detected consumer edge, counted", async () => {
// The wasm asset parser drops `// macros`/`// use`; the CLI-side lexical scan
// must surface the library node, its macro signatures, and the caller edge so
// `--local` reaches parity with the deployed graph.
await withFolder(
{
"macros_finance.duckdb.sql":
`-- macros\nCREATE MACRO net_revenue(gross, refunds) AS gross - refunds;\nCREATE OR REPLACE MACRO safe_div(a, b) AS TABLE SELECT a / b;\n`,
"fct.duckdb.sql":
`-- pipeline\n-- on datatable://main/orders\nSELECT net_revenue(gross, refunds) AS rev FROM main.orders;\n`,
},
async (root, folder) => {
const { graph } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
// the library is a node carrying its macro signatures
const lib = graph.runnables.find((r) => r.path === "f/mypipe/macros_finance");
expect(lib?.macros).toEqual([
{ name: "net_revenue", params: "gross, refunds", is_table: false },
{ name: "safe_div", params: "a, b", is_table: true },
]);
// library counts toward the script total (2 scripts, not just the 1 pipeline member)
expect(graph.runnables.map((r) => r.path).sort()).toEqual([
"f/mypipe/fct",
"f/mypipe/macros_finance",
]);
// the caller edge names only the macro actually called (safe_div is not)
expect(graph.macro_edges).toEqual([
{
lib_path: "f/mypipe/macros_finance",
consumer_path: "f/mypipe/fct",
macro_names: ["net_revenue"],
via_use: false,
},
]);
},
);
});
test("`// use <lib>` forces a whole-library edge even with no lexical call", async () => {
// Dynamic-SQL callers annotate `// use`; the edge lists every macro and is
// flagged via_use (mirrors the deployed graph).
await withFolder(
{
"stats.duckdb.sql":
`-- macros\nCREATE MACRO zscore(x, m, s) AS (x - m) / s;\n`,
"report.duckdb.sql":
`-- pipeline\n-- on datatable://main/metrics\n-- use f/mypipe/stats\nSELECT query('SELECT zscore(v, 0, 1) FROM main.metrics');\n`,
},
async (root, folder) => {
const { graph } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
expect(graph.macro_edges).toEqual([
{
lib_path: "f/mypipe/stats",
consumer_path: "f/mypipe/report",
macro_names: ["zscore"],
via_use: true,
},
]);
// the library still becomes a node
expect(graph.runnables.some((r) => r.path === "f/mypipe/stats")).toBe(true);
},
);
});
test("an UNUSED in-folder `// macros` library is still a member node (deployed parity)", async () => {
// The deployed graph marks every macro library `auto_kind='pipeline'`, so an
// in-folder library is a member node (in_pipeline, with signatures) whether or
// not a consumer uses it. It is still excluded from runs (via `macros`).
await withFolder(
{
"unused.duckdb.sql": `-- macros\nCREATE MACRO helper(a) AS a + 1;\n`,
"solo.duckdb.sql": `-- pipeline\nSELECT 1;\n`,
},
async (root, folder) => {
const { graph, scripts } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
const lib = graph.runnables.find((r) => r.path === "f/mypipe/unused");
expect(lib?.in_pipeline).toBe(true);
expect(lib?.macros).toEqual([{ name: "helper", params: "a", is_table: false }]);
expect(graph.macro_edges).toBeUndefined();
// but it is not a previewable/runnable script — only `solo` is
expect(scripts.map((s) => s.path)).toEqual(["f/mypipe/solo"]);
},
);
});
test("a shared macro library OUTSIDE the pipeline folder is resolved (workspace-wide)", async () => {
// The deployed graph loads the macro registry workspace-wide and only
// folder-scopes consumers, so a pipeline in `f/mypipe` can use a shared library
// in `f/shared`. Local discovery must walk the whole `f/` tree, not just the
// folder, or the edge/node/docs are missing (parity gap).
const root = mkdtempSync(join(tmpdir(), "wm-pl-"));
writeFileSync(join(root, "wmill.yaml"), "defaultTs: bun\n");
mkdirSync(join(root, "f", "shared"), { recursive: true });
mkdirSync(join(root, "f", "mypipe"), { recursive: true });
writeFileSync(
join(root, "f", "shared", "stats.duckdb.sql"),
`-- macros\nCREATE MACRO zscore(x, m, s) AS (x - m) / s;\n`,
);
// one consumer calls the shared macro lexically, another pulls it via `// use`
writeFileSync(
join(root, "f", "mypipe", "fct.duckdb.sql"),
`-- pipeline\n-- on datatable://main/metrics\nSELECT zscore(v, 0, 1) FROM main.metrics;\n`,
);
writeFileSync(
join(root, "f", "mypipe", "dyn.duckdb.sql"),
`-- pipeline\n-- on datatable://main/metrics\n-- use f/shared/stats\nSELECT query('SELECT zscore(v, 0, 1)');\n`,
);
try {
const { graph } = await buildLocalPipelineGraph({ root, folder: "mypipe", defaultTs: "bun" });
expect(graph.macro_edges).toEqual([
{
lib_path: "f/shared/stats",
consumer_path: "f/mypipe/dyn",
macro_names: ["zscore"],
via_use: true,
},
{
lib_path: "f/shared/stats",
consumer_path: "f/mypipe/fct",
macro_names: ["zscore"],
via_use: false,
},
]);
// the out-of-folder library is surfaced as a node with its signatures
expect(graph.runnables.find((r) => r.path === "f/shared/stats")?.macros).toEqual([
{ name: "zscore", params: "x, m, s", is_table: false },
]);
} finally {
rmSync(root, { recursive: true, force: true });
}
});
test("macro library that CONSUMES another library gets a lib→lib edge (both nodes surface)", async () => {
// The deploy path records macro_usage for any DuckDB script, including a macro
// library calling another library's macros. A `base → derived` edge must exist
// (and `base`, an in-folder library, is a member node not just an edge stub).
await withFolder(
{
"base.duckdb.sql": `-- macros\nCREATE MACRO base_add(a, b) AS a + b;\n`,
// derived is a `// macros` library (NOT `// pipeline`) whose body calls base_add
"derived.duckdb.sql":
`-- macros\nCREATE MACRO derived_sum(a, b) AS base_add(a, b) * 2;\n`,
// the pipeline consumer calls derived_sum
"report.duckdb.sql":
`-- pipeline\n-- on datatable://main/t\nSELECT derived_sum(x, y) FROM main.t;\n`,
},
async (root, folder) => {
const { graph } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
expect(graph.macro_edges).toEqual([
// base → derived: the intermediate library is itself a consumer
{
lib_path: "f/mypipe/base",
consumer_path: "f/mypipe/derived",
macro_names: ["base_add"],
via_use: false,
},
// derived → report: the pipeline consumer
{
lib_path: "f/mypipe/derived",
consumer_path: "f/mypipe/report",
macro_names: ["derived_sum"],
via_use: false,
},
]);
// both libraries surface as nodes with their signatures (base does NOT
// disappear just because it is only reached transitively)
expect(graph.runnables.find((r) => r.path === "f/mypipe/base")?.macros).toEqual([
{ name: "base_add", params: "a, b", is_table: false },
]);
expect(graph.runnables.find((r) => r.path === "f/mypipe/derived")?.macros).toEqual([
{ name: "derived_sum", params: "a, b", is_table: false },
]);
// both intermediate libraries are members (in_pipeline), like the deployed graph
expect(graph.runnables.find((r) => r.path === "f/mypipe/base")?.in_pipeline).toBe(true);
expect(graph.runnables.find((r) => r.path === "f/mypipe/derived")?.in_pipeline).toBe(true);
},
);
});
test("macro library reaching another only via dynamic SQL gets a `// use` lib→lib edge", async () => {
// A library that calls another's macro only inside a `query('…')` string is
// invisible to lexical detection; `// use` forces the lib→lib edge. Exercises
// the case where the CONSUMER is itself a `// macros` library.
await withFolder(
{
"base.duckdb.sql": `-- macros\nCREATE MACRO base_add(a, b) AS a + b;\n`,
"derived.duckdb.sql":
`-- macros\n-- use f/mypipe/base\nCREATE MACRO wrap(a, b) AS query('SELECT base_add(1, 2)');\n`,
"report.duckdb.sql":
`-- pipeline\n-- on datatable://main/t\nSELECT wrap(x, y) FROM main.t;\n`,
},
async (root, folder) => {
const { graph } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
expect(graph.macro_edges).toEqual([
// base → derived comes from `// use` (via_use), NOT a lexical call
{
lib_path: "f/mypipe/base",
consumer_path: "f/mypipe/derived",
macro_names: ["base_add"],
via_use: true,
},
{
lib_path: "f/mypipe/derived",
consumer_path: "f/mypipe/report",
macro_names: ["wrap"],
via_use: false,
},
]);
},
);
});
test("a `// macros` (slash-prefix) DuckDB library is detected (backend prefix parity)", async () => {
// A `.duckdb.sql` library may head its annotation with `// macros` (the backend
// accepts `//`/`--`/`#` for any language); the local scan must too.
await withFolder(
{
"lib.duckdb.sql": `// macros\nCREATE MACRO dbl(a) AS a * 2;\n`,
"use_it.duckdb.sql": `-- pipeline\n-- on datatable://main/t\nSELECT dbl(x) FROM main.t;\n`,
},
async (root, folder) => {
const { graph } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
expect(graph.macro_edges).toEqual([
{
lib_path: "f/mypipe/lib",
consumer_path: "f/mypipe/use_it",
macro_names: ["dbl"],
via_use: false,
},
]);
expect(graph.runnables.find((r) => r.path === "f/mypipe/lib")?.macros).toEqual([
{ name: "dbl", params: "a", is_table: false },
]);
},
);
});
test("a non-pipeline DuckDB macro consumer is a display-only node, never a run step", async () => {
// A `.duckdb.sql` helper that calls a macro but isn't `// pipeline` surfaces as
// a graph node (lineage parity) but is NOT a runnable pipeline script: it must
// be absent from `scripts` (the previewable set `run --local` selects from), so
// it can never be scheduled or fail a preview for lack of local content.
await withFolder(
{
"lib.duckdb.sql": `-- macros\nCREATE MACRO dbl(a) AS a * 2;\n`,
"helper.duckdb.sql": `-- on ducklake://main/src\nSELECT dbl(x) FROM main.src;\n`,
"root.duckdb.sql": `-- pipeline\n-- materialize ducklake://main/out\nSELECT dbl(1) AS v;\n`,
},
async (root, folder) => {
const { graph, scripts } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
// the helper IS a node (macro consumer, for lineage display) …
const helper = graph.runnables.find((r) => r.path === "f/mypipe/helper");
expect(helper).toBeDefined();
// … but not a pipeline member (no local file to preview) and carries no macros
expect(helper?.in_pipeline).toBeFalsy();
expect(helper?.macros).toBeUndefined();
// the previewable set (what `run --local` can schedule) is ONLY the member
expect(scripts.map((s) => s.path)).toEqual(["f/mypipe/root"]);
// and the macro edge still connects lib → helper for the lineage view
expect(graph.macro_edges).toContainEqual({
lib_path: "f/mypipe/lib",
consumer_path: "f/mypipe/helper",
macro_names: ["dbl"],
via_use: false,
});
},
);
});
test("`// pipeline` on a macros library is redundant — it's a library node either way", async () => {
// The backend marks a macro library `auto_kind='pipeline'` regardless of the
// `// pipeline` marker, so a `-- pipeline -- macros` script behaves exactly like
// a plain `-- macros` one: a member library node (in_pipeline + signatures),
// excluded from runs (via `macros`) and from the previewable `scripts` set.
await withFolder(
{
"lib.duckdb.sql": `-- pipeline\n-- macros\nCREATE MACRO dbl(a) AS a * 2;\n`,
"root.duckdb.sql": `-- pipeline\n-- materialize ducklake://main/out\nSELECT 1 AS v;\n`,
},
async (root, folder) => {
const { graph, scripts } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
const lib = graph.runnables.find((r) => r.path === "f/mypipe/lib");
expect(lib?.in_pipeline).toBe(true);
expect(lib?.macros).toEqual([{ name: "dbl", params: "a", is_table: false }]);
// unused here → no edges, but still a node; not previewable/runnable
expect(graph.macro_edges).toBeUndefined();
expect(scripts.map((s) => s.path)).toEqual(["f/mypipe/root"]);
},
);
});
test("`#`-comment languages (ruby) use the `#` annotation fallback (no wasm parser)", async () => {
await withFolder(
{ "ingest.rb": `# pipeline\n# on s3://demo/raw.csv\nputs "hi"\n` },
@@ -335,3 +629,35 @@ test("`#`-comment languages (ruby) use the `#` annotation fallback (no wasm pars
},
);
});
test("pipeline docs renders a `Macro libraries` section (call + `// use`)", async () => {
const { generatePipelineMarkdown } = await import(
"../src/commands/pipeline/docs.ts"
);
await withFolder(
{
"macros_finance.duckdb.sql":
`-- macros\nCREATE MACRO net_revenue(gross, refunds) AS gross - refunds;\nCREATE MACRO safe_div(a, b) AS TABLE SELECT a / b;\n`,
"fct.duckdb.sql":
`-- pipeline\n-- on datatable://main/orders\nSELECT net_revenue(gross, refunds) FROM main.orders;\n`,
"report.duckdb.sql":
`-- pipeline\n-- on datatable://main/orders\n-- use f/mypipe/macros_finance\nSELECT query('SELECT safe_div(1, 2)');\n`,
},
async (root, folder) => {
const { graph } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
const md = generatePipelineMarkdown(folder, graph, [], true);
// count line mentions the library; two pipeline scripts, one library
expect(md).toContain("**2** scripts · **1** asset · **1** macro library");
// dedicated section with signatures (TABLE marker) and callers
expect(md).toContain("## Macro libraries");
expect(md).toContain("### `f/mypipe/macros_finance`");
expect(md).toContain("- `net_revenue(gross, refunds)`");
expect(md).toContain("- `safe_div(a, b)` → TABLE");
expect(md).toContain("`f/mypipe/fct` (calls `net_revenue`)");
expect(md).toContain("`f/mypipe/report` (via `// use`)");
// the library is NOT double-listed under `## Scripts`
expect(md).not.toContain("### `f/mypipe/macros_finance`\n\n- **");
},
);
});