diff --git a/.agents/skills/UPSTREAM.md b/.agents/skills/UPSTREAM.md new file mode 100644 index 0000000000..83ba47964e --- /dev/null +++ b/.agents/skills/UPSTREAM.md @@ -0,0 +1,63 @@ +# Vendored skills + +These five skills are copied from an external repository, not written here: + +- `grill-me`, `grilling` +- `improve-codebase-architecture`, `codebase-design`, `domain-modeling` + +Source: https://github.com/mattpocock/skills +Pinned at commit `84fdeffd12f2ee307994d1eb6feb48173b6e0502`. + +They form one dependency closure — `grill-me` is a stub that runs `grilling`, and +`improve-codebase-architecture` draws its vocabulary from `codebase-design` and its +CONTEXT.md upkeep from `domain-modeling`. Removing any one breaks the others. + +Local changes on top of upstream, kept to the minimum so a refresh stays a diff: + +- Flattened the upstream `skills/engineering/` and `skills/productivity/` split, since this + repo's skills are flat. +- Replaced each SKILL.md's markdown links to its own bundled files with plain repo-root paths + in prose (`.agents/skills//FILE.md`). Upstream's sibling-relative links break when the + file is read through the `.claude/skills//SKILL.md` symlink, which mirrors only + SKILL.md — and a repo-root *link* is equally wrong, since a markdown target resolves relative + to the file containing it. Companion files keep their sibling-relative links; they are only + ever read at their real path, never through the symlink. +- Dropped the upstream `agents/openai.yaml` files — Codex packaging metadata for that repo's + own plugin distribution, unused here. +- **Removed every ADR path.** Upstream, `domain-modeling` offers to write Architecture Decision + Records into `docs/adr/` and `improve-codebase-architecture` reads and cites them. This repo has + not adopted ADRs, and a skill that offers to create them is how the practice arrives by side + effect rather than by decision. Deleted `domain-modeling/ADR-FORMAT.md`, its "Offer ADRs + sparingly" section, and the `docs/adr/` entries in its file-structure diagrams; dropped the ADR + clauses from `improve-codebase-architecture` (intro, explore step, "ADR conflicts", the + offer-an-ADR bullet in the grilling loop) and the ADR callout row in `HTML-REPORT.md`. Also cut + "record an architectural decision" from `domain-modeling`'s description, since that phrase is an + invocation trigger. What remains is CONTEXT.md and ubiquitous-language work only. + +To refresh, diff against the same paths at a newer commit and re-apply these four changes. The +ADR removal is the one that needs judgement: if the team later adopts ADRs, take upstream's +version of those sections back rather than rewriting them here. + +## License + +MIT License + +Copyright (c) 2026 Matt Pocock + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/.agents/skills/codebase-design/DEEPENING.md b/.agents/skills/codebase-design/DEEPENING.md new file mode 100644 index 0000000000..3938457b88 --- /dev/null +++ b/.agents/skills/codebase-design/DEEPENING.md @@ -0,0 +1,37 @@ +# Deepening + +How to deepen a cluster of shallow modules safely, given its dependencies. Assumes the vocabulary in [SKILL.md](SKILL.md) — **module**, **interface**, **seam**, **adapter**. + +## Dependency categories + +When assessing a candidate for deepening, classify its dependencies. The category determines how the deepened module is tested across its seam. + +### 1. In-process + +Pure computation, in-memory state, no I/O. Always deepenable — merge the modules and test through the new interface directly. No adapter needed. + +### 2. Local-substitutable + +Dependencies that have local test stand-ins (PGLite for Postgres, in-memory filesystem). Deepenable if the stand-in exists. The deepened module is tested with the stand-in running in the test suite. The seam is internal; no port at the module's external interface. + +### 3. Remote but owned (Ports & Adapters) + +Your own services across a network boundary (microservices, internal APIs). Define a **port** (interface) at the seam. The deep module owns the logic; the transport is injected as an **adapter**. Tests use an in-memory adapter. Production uses an HTTP/gRPC/queue adapter. + +Recommendation shape: *"Define a port at the seam, implement an HTTP adapter for production and an in-memory adapter for testing, so the logic sits in one deep module even though it's deployed across a network."* + +### 4. True external (Mock) + +Third-party services (Stripe, Twilio, etc.) you don't control. The deepened module takes the external dependency as an injected port; tests provide a mock adapter. + +## Seam discipline + +- **One adapter means a hypothetical seam. Two adapters means a real one.** Don't introduce a port unless at least two adapters are justified (typically production + test). A single-adapter seam is just indirection. +- **Internal seams vs external seams.** A deep module can have internal seams (private to its implementation, used by its own tests) as well as the external seam at its interface. Don't expose internal seams through the interface just because tests use them. + +## Testing strategy: replace, don't layer + +- Old unit tests on shallow modules become waste once tests at the deepened module's interface exist — delete them. +- Write new tests at the deepened module's interface. The **interface is the test surface**. +- Tests assert on observable outcomes through the interface, not internal state. +- Tests should survive internal refactors — they describe behaviour, not implementation. If a test has to change when the implementation changes, it's testing past the interface. diff --git a/.agents/skills/codebase-design/DESIGN-IT-TWICE.md b/.agents/skills/codebase-design/DESIGN-IT-TWICE.md new file mode 100644 index 0000000000..8419ad6fa9 --- /dev/null +++ b/.agents/skills/codebase-design/DESIGN-IT-TWICE.md @@ -0,0 +1,44 @@ +# Design It Twice + +When the user wants to explore alternative interfaces for a chosen deepening candidate, use this parallel sub-agent pattern. Based on "Design It Twice" (Ousterhout) — your first idea is unlikely to be the best. + +Uses the vocabulary in [SKILL.md](SKILL.md) — **module**, **interface**, **seam**, **adapter**, **leverage**. + +## Process + +### 1. Frame the problem space + +Before spawning sub-agents, write a user-facing explanation of the problem space for the chosen candidate: + +- The constraints any new interface would need to satisfy +- The dependencies it would rely on, and which category they fall into (see [DEEPENING.md](DEEPENING.md)) +- A rough illustrative code sketch to ground the constraints — not a proposal, just a way to make the constraints concrete + +Show this to the user, then immediately proceed to Step 2. The user reads and thinks while the sub-agents work in parallel. + +### 2. Spawn sub-agents + +Spawn 3+ sub-agents in parallel. Each must produce a **radically different** interface for the deepened module. + +Prompt each sub-agent with a separate technical brief (file paths, coupling details, dependency category from [DEEPENING.md](DEEPENING.md), what sits behind the seam). The brief is independent of the user-facing problem-space explanation in Step 1. Give each agent a different design constraint: + +- Agent 1: "Minimize the interface — aim for 1–3 entry points max. Maximise leverage per entry point." +- Agent 2: "Maximise flexibility — support many use cases and extension." +- Agent 3: "Optimise for the most common caller — make the default case trivial." +- Agent 4 (if applicable): "Design around ports & adapters for cross-seam dependencies." + +Include both [SKILL.md](SKILL.md) vocabulary and CONTEXT.md vocabulary in the brief so each sub-agent names things consistently with the architecture language and the project's domain language. + +Each sub-agent outputs: + +1. Interface (types, methods, params — plus invariants, ordering, error modes) +2. Usage example showing how callers use it +3. What the implementation hides behind the seam +4. Dependency strategy and adapters (see [DEEPENING.md](DEEPENING.md)) +5. Trade-offs — where leverage is high, where it's thin + +### 3. Present and compare + +Present designs sequentially so the user can absorb each one, then compare them in prose. Contrast by **depth** (leverage at the interface), **locality** (where change concentrates), and **seam placement**. + +After comparing, give your own recommendation: which design you think is strongest and why. If elements from different designs would combine well, propose a hybrid. Be opinionated — the user wants a strong read, not a menu. diff --git a/.agents/skills/codebase-design/SKILL.md b/.agents/skills/codebase-design/SKILL.md new file mode 100644 index 0000000000..b7cedf4732 --- /dev/null +++ b/.agents/skills/codebase-design/SKILL.md @@ -0,0 +1,114 @@ +--- +name: codebase-design +description: Shared vocabulary for designing deep modules. Use when the user wants to design or improve a module's interface, find deepening opportunities, decide where a seam goes, make code more testable or AI-navigable, or when another skill needs the deep-module vocabulary. +--- + +# Codebase Design + +Design **deep modules**: a lot of behaviour behind a small interface, placed at a clean seam, testable through that interface. Use this language and these principles wherever code is being designed or restructured. The aim is leverage for callers, locality for maintainers, and testability for everyone. + +## Glossary + +Use these terms exactly — don't substitute "component," "service," "API," or "boundary." Consistent language is the whole point. + +**Module** — anything with an interface and an implementation. Deliberately scale-agnostic: a function, class, package, or tier-spanning slice. _Avoid_: unit, component, service. + +**Interface** — everything a caller must know to use the module correctly: the type signature, but also invariants, ordering constraints, error modes, required configuration, and performance characteristics. _Avoid_: API, signature (too narrow — they refer only to the type-level surface). + +**Implementation** — what's inside a module, its body of code. Distinct from **Adapter**: a thing can be a small adapter with a large implementation (a Postgres repo) or a large adapter with a small implementation (an in-memory fake). Reach for "adapter" when the seam is the topic; "implementation" otherwise. + +**Depth** — leverage at the interface: the amount of behaviour a caller (or test) can exercise per unit of interface they have to learn. A module is **deep** when a large amount of behaviour sits behind a small interface, **shallow** when the interface is nearly as complex as the implementation. + +**Seam** _(Michael Feathers)_ — a place where you can alter behaviour without editing in that place; the *location* at which a module's interface lives. Where to put the seam is its own design decision, distinct from what goes behind it. _Avoid_: boundary (overloaded with DDD's bounded context). + +**Adapter** — a concrete thing that satisfies an interface at a seam. Describes *role* (what slot it fills), not substance (what's inside). + +**Leverage** — what callers get from depth: more capability per unit of interface they learn. One implementation pays back across N call sites and M tests. + +**Locality** — what maintainers get from depth: change, bugs, knowledge, and verification concentrate in one place rather than spreading across callers. Fix once, fixed everywhere. + +## Deep vs shallow + +**Deep module** = small interface + lots of implementation: + +``` +┌─────────────────────┐ +│ Small Interface │ ← Few methods, simple params +├─────────────────────┤ +│ │ +│ Deep Implementation│ ← Complex logic hidden +│ │ +└─────────────────────┘ +``` + +**Shallow module** = large interface + little implementation (avoid): + +``` +┌─────────────────────────────────┐ +│ Large Interface │ ← Many methods, complex params +├─────────────────────────────────┤ +│ Thin Implementation │ ← Just passes through +└─────────────────────────────────┘ +``` + +When designing an interface, ask: + +- Can I reduce the number of methods? +- Can I simplify the parameters? +- Can I hide more complexity inside? + +## Principles + +- **Depth is a property of the interface, not the implementation.** A deep module can be internally composed of small, mockable, swappable parts — they just aren't part of the interface. A module can have **internal seams** (private to its implementation, used by its own tests) as well as the **external seam** at its interface. +- **The deletion test.** Imagine deleting the module. If complexity vanishes, it was a pass-through. If complexity reappears across N callers, it was earning its keep. +- **The interface is the test surface.** Callers and tests cross the same seam. If you want to test *past* the interface, the module is probably the wrong shape. +- **One adapter means a hypothetical seam. Two adapters means a real one.** Don't introduce a seam unless something actually varies across it. + +## Designing for testability + +Good interfaces make testing natural: + +1. **Accept dependencies, don't create them.** + + ```typescript + // Testable + function processOrder(order, paymentGateway) {} + + // Hard to test + function processOrder(order) { + const gateway = new StripeGateway(); + } + ``` + +2. **Return results, don't produce side effects.** + + ```typescript + // Testable + function calculateDiscount(cart): Discount {} + + // Hard to test + function applyDiscount(cart): void { + cart.total -= discount; + } + ``` + +3. **Small surface area.** Fewer methods = fewer tests needed. Fewer params = simpler test setup. + +## Relationships + +- A **Module** has exactly one **Interface** (the surface it presents to callers and tests). +- **Depth** is a property of a **Module**, measured against its **Interface**. +- A **Seam** is where a **Module**'s **Interface** lives. +- An **Adapter** sits at a **Seam** and satisfies the **Interface**. +- **Depth** produces **Leverage** for callers and **Locality** for maintainers. + +## Rejected framings + +- **Depth as ratio of implementation-lines to interface-lines** (Ousterhout): rewards padding the implementation. We use depth-as-leverage instead. +- **"Interface" as the TypeScript `interface` keyword or a class's public methods**: too narrow — interface here includes every fact a caller must know. +- **"Boundary"**: overloaded with DDD's bounded context. Say **seam** or **interface**. + +## Going deeper + +- **Deepening a cluster given its dependencies** — see `.agents/skills/codebase-design/DEEPENING.md` (path from the repo root): dependency categories, seam discipline, and replace-don't-layer testing. +- **Exploring alternative interfaces** — see `.agents/skills/codebase-design/DESIGN-IT-TWICE.md` (path from the repo root): spin up parallel sub-agents to design the interface several radically different ways, then compare on depth, locality, and seam placement. diff --git a/.agents/skills/domain-modeling/CONTEXT-FORMAT.md b/.agents/skills/domain-modeling/CONTEXT-FORMAT.md new file mode 100644 index 0000000000..eaf2a18573 --- /dev/null +++ b/.agents/skills/domain-modeling/CONTEXT-FORMAT.md @@ -0,0 +1,60 @@ +# CONTEXT.md Format + +## Structure + +```md +# {Context Name} + +{One or two sentence description of what this context is and why it exists.} + +## Language + +**Order**: +{A one or two sentence description of the term} +_Avoid_: Purchase, transaction + +**Invoice**: +A request for payment sent to a customer after delivery. +_Avoid_: Bill, payment request + +**Customer**: +A person or organization that places orders. +_Avoid_: Client, buyer, account +``` + +## Rules + +- **Be opinionated.** When multiple words exist for the same concept, pick the best one and list the others under `_Avoid_`. +- **Keep definitions tight.** One or two sentences max. Define what it IS, not what it does. +- **Only include terms specific to this project's context.** General programming concepts (timeouts, error types, utility patterns) don't belong even if the project uses them extensively. Before adding a term, ask: is this a concept unique to this context, or a general programming concept? Only the former belongs. +- **Group terms under subheadings** when natural clusters emerge. If all terms belong to a single cohesive area, a flat list is fine. + +## Single vs multi-context repos + +**Single context (most repos):** One `CONTEXT.md` at the repo root. + +**Multiple contexts:** A `CONTEXT-MAP.md` at the repo root lists the contexts, where they live, and how they relate to each other: + +```md +# Context Map + +## Contexts + +- [Ordering](./src/ordering/CONTEXT.md) — receives and tracks customer orders +- [Billing](./src/billing/CONTEXT.md) — generates invoices and processes payments +- [Fulfillment](./src/fulfillment/CONTEXT.md) — manages warehouse picking and shipping + +## Relationships + +- **Ordering → Fulfillment**: Ordering emits `OrderPlaced` events; Fulfillment consumes them to start picking +- **Fulfillment → Billing**: Fulfillment emits `ShipmentDispatched` events; Billing consumes them to generate invoices +- **Ordering ↔ Billing**: Shared types for `CustomerId` and `Money` +``` + +The skill infers which structure applies: + +- If `CONTEXT-MAP.md` exists, read it to find contexts +- If only a root `CONTEXT.md` exists, single context +- If neither exists, create a root `CONTEXT.md` lazily when the first term is resolved + +When multiple contexts exist, infer which one the current topic relates to. If unclear, ask. diff --git a/.agents/skills/domain-modeling/SKILL.md b/.agents/skills/domain-modeling/SKILL.md new file mode 100644 index 0000000000..b0372a62b8 --- /dev/null +++ b/.agents/skills/domain-modeling/SKILL.md @@ -0,0 +1,57 @@ +--- +name: domain-modeling +description: Build and sharpen a project's domain model. Use when the user wants to pin down domain terminology or a ubiquitous language, or when another skill needs to maintain the domain model. +--- + +# Domain Modeling + +Actively build and sharpen the project's domain model as you design. This is the *active* discipline — challenging terms, inventing edge-case scenarios, and writing the glossary and decisions down the moment they crystallise. (Merely *reading* `CONTEXT.md` for vocabulary is not this skill — that's a one-line habit any skill can do. This skill is for when you're changing the model, not just consuming it.) + +## File structure + +Most repos have a single context: + +``` +/ +├── CONTEXT.md +└── src/ +``` + +If a `CONTEXT-MAP.md` exists at the root, the repo has multiple contexts. The map points to where each one lives: + +``` +/ +├── CONTEXT-MAP.md +└── src/ + ├── ordering/ + │ └── CONTEXT.md + └── billing/ + └── CONTEXT.md +``` + +Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. + +## During the session + +### Challenge against the glossary + +When the user uses a term that conflicts with the existing language in `CONTEXT.md`, call it out immediately. "Your glossary defines 'cancellation' as X, but you seem to mean Y — which is it?" + +### Sharpen fuzzy language + +When the user uses vague or overloaded terms, propose a precise canonical term. "You're saying 'account' — do you mean the Customer or the User? Those are different things." + +### Discuss concrete scenarios + +When domain relationships are being discussed, stress-test them with specific scenarios. Invent scenarios that probe edge cases and force the user to be precise about the boundaries between concepts. + +### Cross-reference with code + +When the user states how something works, check whether the code agrees. If you find a contradiction, surface it: "Your code cancels entire Orders, but you just said partial cancellation is possible — which is right?" + +### Update CONTEXT.md inline + +When a term is resolved, update `CONTEXT.md` right there. Don't batch these up — capture them as they happen. Use the format in `.agents/skills/domain-modeling/CONTEXT-FORMAT.md` (path from the repo root). + +`CONTEXT.md` should be totally devoid of implementation details. Do not treat `CONTEXT.md` as a spec, a scratch pad, or a repository for implementation decisions. It is a glossary and nothing else. + diff --git a/.agents/skills/grill-me/SKILL.md b/.agents/skills/grill-me/SKILL.md new file mode 100644 index 0000000000..9470cfcfe2 --- /dev/null +++ b/.agents/skills/grill-me/SKILL.md @@ -0,0 +1,7 @@ +--- +name: grill-me +description: A relentless interview to sharpen a plan or design. +disable-model-invocation: true +--- + +Run a `/grilling` session. diff --git a/.agents/skills/grilling/SKILL.md b/.agents/skills/grilling/SKILL.md new file mode 100644 index 0000000000..95bd01ee90 --- /dev/null +++ b/.agents/skills/grilling/SKILL.md @@ -0,0 +1,22 @@ +--- +name: grilling +description: Grill the user relentlessly about a plan, decision, or idea. Use when the user wants to stress-test their thinking, or uses any 'grill' trigger phrases. +--- + +Interview the user relentlessly until you reach a shared understanding. Map this as a **design tree**: every decision branches into the decisions that hang off it. + +Work the tree in **rounds**. The **frontier** is every decision whose prerequisites are already settled — the questions you can ask _now_ without guessing at answers you haven't heard yet. Ask the whole frontier in one round: number each question and give your recommended answer. Then wait for the user's answers before the next round. + +Each question should be formatted like so: + +``` +❓ **Q1** - ****: + +➡️ +``` + +Each round the user answers reshapes the tree — settled decisions push the frontier outward and unblock questions that depended on them. Recompute the frontier and ask the next round. A question whose answer depends on another question still open in this round belongs to a _later_ round, not this one. + +Finding _facts_ is your job, never the user's. When a frontier question needs a fact from the environment (filesystem, tools, etc.), dispatch a sub-agent to find it — don't ask the user for anything you could look up yourself. Don't block on it: a running exploration is an unsettled prerequisite, so only the questions downstream of it wait for the sub-agent to report — ask the rest of the frontier now. The _decisions_ are the user's — put each to them and wait. + +The session is done when the frontier is empty: every branch of the design tree visited, nothing left silently assumed. Do not act on it until the user confirms you have reached a shared understanding. diff --git a/.agents/skills/improve-codebase-architecture/HTML-REPORT.md b/.agents/skills/improve-codebase-architecture/HTML-REPORT.md new file mode 100644 index 0000000000..ecec59a00f --- /dev/null +++ b/.agents/skills/improve-codebase-architecture/HTML-REPORT.md @@ -0,0 +1,122 @@ +# HTML Report Format + +The architectural review is rendered as a single self-contained HTML file in the OS temp directory. Tailwind and Mermaid both come from CDNs. Mermaid handles graph-shaped diagrams reliably; hand-built divs and inline SVG handle the more editorial visuals (mass diagrams, cross-sections). Mix the two — don't lean on Mermaid for everything, it'll start to look generic. + +## Scaffold + +```html + + + + + Architecture review — {{repo name}} + + + + + +
+
...
+
...
+
...
+
+ + +``` + +## Header + +Repo name, date, and a compact legend: solid box = module, dashed line = seam, red arrow = leakage, thick dark box = deep module. No introduction paragraph — straight into the candidates. + +## Candidate card + +The diagrams carry the weight. Prose is sparse, plain, and uses the glossary terms (from the `/codebase-design` skill) without ceremony. + +Each candidate is one `
`: + +- **Title** — short, names the deepening (e.g. "Collapse the Order intake pipeline"). +- **Badge row** — recommendation strength (`Strong` = emerald, `Worth exploring` = amber, `Speculative` = slate), plus a tag for the dependency category (`in-process`, `local-substitutable`, `ports & adapters`, `mock`). +- **Files** — monospaced list, `font-mono text-sm`. +- **Before / After diagram** — the centrepiece. Two columns, side by side. See patterns below. +- **Problem** — one sentence. What hurts. +- **Solution** — one sentence. What changes. +- **Wins** — bullets, ≤6 words each. e.g. "Tests hit one interface", "Pricing logic stops leaking", "Delete 4 shallow wrappers". + +No paragraphs of explanation. If the diagram needs a paragraph to be understood, redraw the diagram. + +## Diagram patterns + +Pick the pattern that fits the candidate. Mix them. Don't make every diagram look the same — variety is part of the point. + +### Mermaid graph (the workhorse for dependencies / call flow) + +Use a Mermaid `flowchart` or `graph` when the point is "X calls Y calls Z, and look at the mess." Wrap it in a Tailwind-styled card so it doesn't feel parachuted in. Style with classDef to colour leakage edges red and the deep module dark. Sequence diagrams work well for "before: 6 round-trips; after: 1." + +```html +
+ 
+    flowchart LR
+      A[OrderHandler] --> B[OrderValidator]
+      B --> C[OrderRepo]
+      C -.leak.-> D[PricingClient]
+      classDef leak stroke:#dc2626,stroke-width:2px;
+      class C,D leak
+
+
+``` + +### Hand-built boxes-and-arrows (when Mermaid's layout fights you) + +Modules as `
`s with borders and labels. Arrows as inline SVG `` or `` elements positioned absolutely over a relative container. Reach for this when you want the "after" diagram to feel like one thick-bordered deep module with greyed-out internals — Mermaid won't render that with the right weight. + +### Cross-section (good for layered shallowness) + +Stack horizontal bands (`h-12 border-l-4`) to show layers a call passes through. Before: 6 thin layers each doing nothing. After: 1 thick band labelled with the consolidated responsibility. + +### Mass diagram (good for "interface as wide as implementation") + +Two rectangles per module — one for interface surface area, one for implementation. Before: interface rectangle is nearly as tall as the implementation rectangle (shallow). After: interface rectangle is short, implementation rectangle is tall (deep). + +### Call-graph collapse + +Before: a tree of function calls rendered as nested boxes. After: the same tree collapsed into one box, with the now-internal calls shown faded inside it. + +## Style guidance + +- Lean editorial, not corporate-dashboard. Generous whitespace. Serif optional for headings (`font-serif` works well with stone/slate). +- Colour sparingly: one accent (emerald or indigo) plus red for leakage and amber for warnings. +- Keep diagrams ~320px tall so before/after sits comfortably side by side without scrolling. +- Use `text-xs uppercase tracking-wider` for module labels inside diagrams — they should read as schematic, not as UI. +- The only scripts are the Tailwind CDN and the Mermaid ESM import. The report is otherwise static — no app code, no interactivity beyond Mermaid's own rendering. + +## Top recommendation section + +One larger card. Candidate name, one sentence on why, anchor link to its card. That's it. + +## Tone + +Plain English, concise — but the architectural nouns and verbs come straight from the `/codebase-design` skill. Concision is not an excuse to drift. + +**Use exactly:** module, interface, implementation, depth, deep, shallow, seam, adapter, leverage, locality. + +**Never substitute:** component, service, unit (for module) · API, signature (for interface) · boundary (for seam) · layer, wrapper (for module, when you mean module). + +**Phrasings that fit the style:** + +- "Order intake module is shallow — interface nearly matches the implementation." +- "Pricing leaks across the seam." +- "Deepen: one interface, one place to test." +- "Two adapters justify the seam: HTTP in prod, in-memory in tests." + +**Wins bullets** name the gain in glossary terms: *"locality: bugs concentrate in one module"*, *"leverage: one interface, N call sites"*, *"interface shrinks; implementation absorbs the wrappers"*. Don't write *"easier to maintain"* or *"cleaner code"* — those terms aren't in the glossary and don't earn their place. + +No hedging, no throat-clearing, no "it's worth noting that…". If a sentence could be a bullet, make it a bullet. If a bullet could be cut, cut it. If a term isn't in the `/codebase-design` glossary, reach for one that is before inventing a new one. diff --git a/.agents/skills/improve-codebase-architecture/SKILL.md b/.agents/skills/improve-codebase-architecture/SKILL.md new file mode 100644 index 0000000000..488c850990 --- /dev/null +++ b/.agents/skills/improve-codebase-architecture/SKILL.md @@ -0,0 +1,68 @@ +--- +name: improve-codebase-architecture +description: Scan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick. +disable-model-invocation: true +--- + +# Improve Codebase Architecture + +Surface architectural friction and propose **deepening opportunities** — refactors that turn shallow modules into deep ones. The aim is testability and AI-navigability. + +This command is _informed_ by the project's domain model and built on a shared design vocabulary: + +- Run the `/codebase-design` skill for the architecture vocabulary (**module**, **interface**, **depth**, **seam**, **adapter**, **leverage**, **locality**) and its principles (the deletion test, "the interface is the test surface", "one adapter = hypothetical seam, two = real"). Use these terms exactly in every suggestion — don't drift into "component," "service," "API," or "boundary." +- The domain language in `CONTEXT.md` gives names to good seams. + +## Process + +### 1. Explore + +**Scope before you scan — YAGNI.** Deepening a module pays off by making future changes to it easier, so put extra weight on the parts of the codebase that have recently changed. Decide *where* to look before you look: + +- If the user named a direction — a module, a subsystem, a pain point — take it, and skip the inference below. +- Otherwise, walk back a good stretch of the commit history (`git log --oneline`) to find the codebase's hot spots — the files and areas that keep coming up — and let those paths pull your attention first. If the changes are scattered with no clear hot spot, widen the net. + +Read the project's domain glossary (`CONTEXT.md`) first. + +Then spawn a sub-agent to walk the codebase. Don't follow rigid heuristics — explore organically and note where you experience friction: + +- Where does understanding one concept require bouncing between many small modules? +- Where are modules **shallow** — interface nearly as complex as the implementation? +- Where have pure functions been extracted just for testability, but the real bugs hide in how they're called (no **locality**)? +- Where do tightly-coupled modules leak across their seams? +- Which parts of the codebase are untested, or hard to test through their current interface? + +Apply the **deletion test** to anything you suspect is shallow: would deleting it concentrate complexity, or just move it? A "yes, concentrates" is the signal you want. + +### 2. Present candidates as an HTML report + +Write a self-contained HTML file to the OS temp directory so nothing lands in the repo. Resolve the temp dir from `$TMPDIR`, falling back to `/tmp` (or `%TEMP%` on Windows), and write to `/architecture-review-.html` so each run gets a fresh file. Open it for the user — `xdg-open ` on Linux, `open ` on macOS, `start ` on Windows — and tell them the absolute path. + +The report uses **Tailwind via CDN** for layout and styling, and **Mermaid via CDN** for diagrams where a graph/flow/sequence reliably communicates the structure. Mix Mermaid with hand-crafted CSS/SVG visuals — use Mermaid when relationships are graph-shaped (call graphs, dependencies, sequences), and hand-built divs/SVG when you want something more editorial (mass diagrams, cross-sections, collapse animations). Each candidate gets a **before/after visualisation**. Be visual. + +For each candidate, render a card with: + +- **Files** — which files/modules are involved +- **Problem** — why the current architecture is causing friction +- **Solution** — plain English description of what would change +- **Benefits** — explained in terms of locality and leverage, and how tests would improve +- **Before / After diagram** — side-by-side, custom-drawn, illustrating the shallowness and the deepening +- **Recommendation strength** — one of `Strong`, `Worth exploring`, `Speculative`, rendered as a badge + +End the report with a **Top recommendation** section: which candidate you'd tackle first and why. + +**Use CONTEXT.md vocabulary for the domain, and the `/codebase-design` vocabulary for the architecture.** If `CONTEXT.md` defines "Order," talk about "the Order intake module" — not "the FooBarHandler," and not "the Order service." + +See `.agents/skills/improve-codebase-architecture/HTML-REPORT.md` (path from the repo root) for the full HTML scaffold, diagram patterns, and styling guidance. + +Do NOT propose interfaces yet. After the file is written, ask the user: "Which of these would you like to explore?" + +### 3. Grilling loop + +Once the user picks a candidate, run the `/grilling` skill to walk the decision tree with them — constraints, dependencies, the shape of the deepened module, what sits behind the seam, what tests survive. + +Side effects happen inline as decisions crystallize — run the `/domain-modeling` skill to keep the domain model current as you go: + +- **Naming a deepened module after a concept not in `CONTEXT.md`?** Add the term to `CONTEXT.md`. Create the file lazily if it doesn't exist. +- **Sharpening a fuzzy term during the conversation?** Update `CONTEXT.md` right there. +- **Want to explore alternative interfaces for the deepened module?** Run the `/codebase-design` skill and use its design-it-twice parallel sub-agent pattern. diff --git a/.agents/skills/pr/SKILL.md b/.agents/skills/pr/SKILL.md index bd40b6c472..6a72949c96 100644 --- a/.agents/skills/pr/SKILL.md +++ b/.agents/skills/pr/SKILL.md @@ -62,7 +62,7 @@ If `git diff main...HEAD --name-only` matches `^frontend/`, the PR body **must** screenshots of the affected UI. Skip only when there is no visible UI effect (types, tests, build config) — and say so in the body. -1. Verify the change in the browser (AGENTS.md → "Verifying Frontend Changes"). +1. Verify the change in the browser (frontend/AGENTS.md → "Verifying Frontend Changes"). 2. Screenshot each affected page with `mcp__playwright__browser_take_screenshot` (save to a file). 3. Host each image and get its Markdown embed by pushing to the public `windmill-labs/agent-screenshots-internal` repo. **Pipe base64 through stdin** — @@ -130,6 +130,10 @@ and continue once they confirm it's done. ## Review rounds (draft → ready) A PR leaves draft **only after a clean CI review round**. Never run `gh pr ready` before that. +This is the rule in every mode, autonomous included. A clean round is necessary but not always +sufficient — see "Flip, or ask first" below. The one standing exception is an explicit request to +leave that PR in draft (usually so it can be tested first) — honour it for that PR, and don't +carry it over to the next one. 1. **Trigger a round and wait for it**: launch the waiter as a background Bash task (a round takes 10–30 min; you are woken when it exits — do not stop the session or poll in the foreground while it runs): @@ -162,6 +166,77 @@ A PR leaves draft **only after a clean CI review round**. Never run `gh pr ready If any P0/P1 finding is unaddressed or the head moved for reasons other than nit fixes, do **not** post the marker or flip — run another round instead. +### A round that never starts is usually a conflict + +The review workflows don't run on a PR that cannot merge, so a round that produces no verdict is +more often a conflict with `main` than a CI outage. Check before assuming anything is broken: + +```bash +gh pr view --json mergeable,mergeStateStatus +``` + +Resolve by **merging, not rebasing** — a rebase rewrites the head SHA that round verdicts and the +clean-round marker are keyed to, invalidating work you have already paid for: + +```bash +git fetch origin main +git merge origin/main +``` + +**If that merge changed `backend/ee-repo-ref.txt`, move the EE worktree to match.** The file pins +the EE commit CE builds against, so a merge that advances it leaves the EE checkout behind what CE +now expects, and `cargo check --features private` compiles a tree neither you nor CI intends: + +```bash +git -C merge "$(tr -d '[:space:]' < backend/ee-repo-ref.txt)" +``` + +Push both, then start a fresh round — the head moved, so the earlier verdicts no longer apply. + +### Flip, or ask first + +A clean round earns the flip; it does not always earn it *unattended*. Judge the blast radius from +the diff first — `git diff --name-only main...HEAD` answers most of these. + +**Ask before flipping** when the change: + +- touches `*_ee.rs` (it spans the EE repo through symlinks and has a companion PR) +- adds a migration under `backend/migrations/` +- changes `openapi.yaml`, `openflow.openapi.yaml`, or the generated client +- touches auth, permission, or token paths +- changes shared worker infrastructure — the job poller, `handle_child`, an executor +- trips `REVIEW.md`'s "Checklist for new public surfaces" + +**Flip without asking** when it is self-contained: a single-file fix, test-only, docs-only, one +call site, no new public surface. + +Unattended (webmux oneshot) there is nobody to ask, so the judgement holds and the action +degrades: flip the self-contained ones, and leave the rest at a clean draft with a line in the PR +description saying why — `left in draft: adds a migration, wants a human look before ready`. +Don't flip a wide-blast-radius change just because the round came back clean, and don't ask a +question nobody will read. + +`AGENTS.local.md` (gitignored, so it may not exist) carries a "PR ready calibration" section +recording how past ambiguous calls went. Read it before deciding; when a call is still genuinely +ambiguous, ask, then append the answer there so the next one is less ambiguous. + +### When rounds stop converging + +Three or more rounds without a clean verdict usually means the change's shape is wrong, not that +there is an endless supply of independent bugs. The tells: + +- findings keep landing in the same files round after round +- fixing one finding creates the next +- the findings are about coupling, duplication, or state threaded through many places, rather + than logic errors + +When that pattern holds, stop running rounds — each one costs a CI cycle and is not going to +converge. Say plainly that the remaining findings look structural rather than incidental, and +name the module or seam they cluster around. With a user present, suggest they run +`/improve-codebase-architecture` over that area: it is slash-only so you cannot invoke it +yourself, and reshaping the code is a scope change they should choose. Unattended, put the +diagnosis in the PR description and stop there rather than grinding out more rounds. + ## EE Companion PR (when `*_ee.rs` files were modified) The `*_ee.rs` files in the windmill repo are **symlinks** to `windmill-ee-private` — changes won't appear in `git diff` of the windmill repo. Instead, check the EE repo for uncommitted or unpushed changes. diff --git a/.agents/skills/refine/SKILL.md b/.agents/skills/refine/SKILL.md index aaf747cd29..51b29564c6 100644 --- a/.agents/skills/refine/SKILL.md +++ b/.agents/skills/refine/SKILL.md @@ -17,7 +17,6 @@ Reflect on the current session and update documentation with lessons learned. 2. **Read current docs**: Read the docs that were relevant to this session: - `docs/validation.md` - `docs/enterprise.md` - - `docs/autonomous-mode.md` - Any skills that were invoked 3. **Propose updates**: For each piece of friction, decide if it warrants a doc update: diff --git a/.agents/skills/rust-backend/SKILL.md b/.agents/skills/rust-backend/SKILL.md index f0c52002bc..2c6f077f0f 100644 --- a/.agents/skills/rust-backend/SKILL.md +++ b/.agents/skills/rust-backend/SKILL.md @@ -94,6 +94,13 @@ Use `tokio::sync::mpsc` (bounded) for channels. Avoid `std::thread::sleep` in as Always use rust-analyzer LSP for go-to-definition, find-references, and type info. Do not guess at module paths. +## Feature Telemetry + +`FEATURE_USAGE_KINDS` in `windmill-api-workspaces/src/workspaces.rs` is an allowlist: a +`(feature, kind)` pair missing from it is dropped by `valid_feature_usage_event` with a bare +`continue` — no error, and the route still returns 204. Adding a counter on the frontend without +registering it here records nothing. See `docs/feature-telemetry.md`. + ## Axum Handlers Destructure extractors directly in function signatures: diff --git a/.agents/skills/svelte-frontend/SKILL.md b/.agents/skills/svelte-frontend/SKILL.md index b0c4b39939..6aceedc25c 100644 --- a/.agents/skills/svelte-frontend/SKILL.md +++ b/.agents/skills/svelte-frontend/SKILL.md @@ -7,9 +7,47 @@ description: Svelte coding guidelines for the Windmill frontend. MUST use when w Apply these Windmill-specific patterns when writing Svelte code in `frontend/`. For general Svelte 5 syntax (runes, snippets, event handling), use the Svelte MCP server. +## Before writing any UI (MUST) + +Do both of these before the first line of markup — not after, and not only when something +looks unfamiliar. + +**1. Find the component that already exists.** `frontend/src/lib/components/common/index.ts` +is the design-system barrel — 28 lines, read it in full. It exports far more than the three +documented below: `Alert`, `Badge`, `Breadcrumb`, `Drawer`/`DrawerContent`, `Menu`/`MenuItem`, +`Tabs`/`Tab`/`TabContent`, `Skeleton`, `FileInput`, `RadioCard`, `Section`, `Kbd`, `ActionRow`, +`ClearableInput`, `CopyButton`, `SecondsInput`, `UndoRedo`, `Url`. + +The barrel is not the full picture either: `common/` has 34 subdirectories and only 23 exports, +so `modal/`, `popup/`, `stepper/`, `tooltip/`, `checkbox/`, `table/`, `contextmenu/`, +`confirmationModal/`, `calendarPicker/`, `fileUpload/`, `toggleButton-v2/` and more exist but +must be imported by path. Selects, text inputs and melt-based primitives sit next to `common/` +in `components/select/`, `components/text_input/`, `components/meltComponents/`. + +The tree holds 1,600+ components — grep `frontend/src/lib/components` for the thing you're about +to build; it almost certainly exists. Building a new one is the last resort, not the first move. + +**2. Read the guideline for what you're building.** `frontend/brand-guidelines.md` is the +authority on how it should look and read. Don't load all 34k chars — jump to the section: + +| Building | Section to read | +|---|---| +| Any new screen or component | `# Components` (Core Rules, Quick Reference) | +| Buttons, CTAs | `## Buttons` — hierarchy matters, only one Accent per view | +| Colors, surfaces, borders | `# Color system` (Quick Reference, Do's and Don'ts) | +| Text, labels, headings | `# Typography` — note `## Text Casing`, sentence case throughout | +| Spacing, grids, page structure | `# Spacing & Layout`; `# Layout` → `## Form` for forms | +| Shadows, overlays, depth | `# Elevation` | +| Icons | `# Iconography` | +| Wording of any UI copy | `# Voice & Communication`, `# Tone of Voice` | + +Get the line range with `grep -n '^#' frontend/brand-guidelines.md`, then read just that span. + ## Windmill UI Components (MUST use) -Always use Windmill's design-system components. Never use raw HTML elements. +Always use Windmill's design-system components. Never use raw HTML elements. The three below +are the ones you'll reach for most often — they are examples, not the catalog. For anything +else, go back to the barrel and grep. ### Buttons — ` + Not recording + + + Passwords are masked. Mark sensitive elements with data-wm-no-record +
+ + + + + +`; +} diff --git a/cli/src/commands/app/devRecorderBundle.gen.ts b/cli/src/commands/app/devRecorderBundle.gen.ts new file mode 100644 index 0000000000..0e7f004b4b --- /dev/null +++ b/cli/src/commands/app/devRecorderBundle.gen.ts @@ -0,0 +1,15 @@ +// Generated by cli/generate-dev-recorder.ts. Do not edit. +// Run `bun run gen:dev-recorder` from cli/ to rebuild it from +// frontend/src/lib/components/recording/. + +/** Repo-relative sources bundled below. */ +export const DEV_RECORDER_SOURCES = [ + "frontend/src/lib/components/recording/rawAppRecording.svelte.ts", + "frontend/src/lib/components/recording/rawAppSnapshot.ts" +]; + +/** SHA-256 of those sources, as of this build. */ +export const DEV_RECORDER_SOURCE_HASH = "c93b8b23455528be0a03da303fb573536bd3f0b8386e6f5f1078a6b988f59082"; + +/** IIFE exposing `createRawAppRecording` on `window.__wmillRecorder`. */ +export const DEV_RECORDER_BUNDLE = "var __wmillRecorder=(()=>{var ne=Object.defineProperty;var Fe=Object.getOwnPropertyDescriptor;var Xe=Object.getOwnPropertyNames;var Be=Object.prototype.hasOwnProperty;var je=(t,o)=>{for(var n in o)ne(t,n,{get:o[n],enumerable:!0})},Ke=(t,o,n,s)=>{if(o&&typeof o==\"object\"||typeof o==\"function\")for(let l of Xe(o))!Be.call(t,l)&&l!==n&&ne(t,l,{get:()=>o[l],enumerable:!(s=Fe(o,l))||s.enumerable});return t};var We=t=>Ke(ne({},\"__esModule\",{value:!0}),t);var mt={};je(mt,{createRawAppRecording:()=>Ie});var G=\"data-wm-rec-target\",D=\"data-wm-no-record\";function Ye(t,o){let n=(u,c,f)=>{let d=f.trim();if(!d||/^(data:|blob:|about:|https?:|\\/\\/|#)/i.test(d))return u;try{return`url(${c}${new URL(d,o).href}${c})`}catch{return u}},s=\"\",l=0;for(;ln?String.fromCodePoint(parseInt(n,16)):s)}function Qe(t){let o=new Set,n=new Set;for(let s of Array.from(t.querySelectorAll(\"style\"))){if(_(s))continue;let l=s.textContent??\"\";for(let u of l.matchAll(Ge))o.add(Ee(u[1]));for(let u of l.matchAll(Je))n.add(Ee(u[1]))}return{classes:o,ids:n}}function Ze(t,o){let n=Qe(o),s=[...o.hasAttribute(D)?[o]:[],...Array.from(o.querySelectorAll(`[${D}]`))];for(let l of s){l.replaceChildren(t.createTextNode(\"\\u2022\\u2022\\u2022\")),l.setAttribute(D,\"\");for(let u of Array.from(l.attributes)){if(u.name===D)continue;let c=u.localName.toLowerCase();if(!ze.has(c))l.removeAttributeNode(u);else if(c===\"class\"){let f=u.value.split(/\\s+/).filter(d=>d&&n.classes.has(d));f.length?l.setAttribute(\"class\",f.join(\" \")):l.removeAttributeNode(u)}else c===\"id\"&&!n.ids.has(u.value)&&l.removeAttributeNode(u)}}}var et=4e6,tt=8e6;function nt(t,o){let n=t.querySelectorAll(\"canvas\"),s=o.querySelectorAll(\"canvas\");if(n.length!==s.length)return;let l=tt;for(let u=0;uet||f>l)continue;l-=f;let d;try{d=c.toDataURL(\"image/webp\",.85)}catch{continue}if(!d.startsWith(\"data:image/\"))continue;let m=c.getBoundingClientRect();if(!m.width||!m.height)continue;let y=t.defaultView?.getComputedStyle(c).display,v=!y||y===\"inline\"?\"inline-block\":y,F=s[u],X=F.getAttribute(\"style\");F.setAttribute(\"style\",`${X?X+\";\":\"\"}display:${v};box-sizing:border-box;width:${m.width}px;height:${m.height}px;background-image:url(\"${d}\");background-size:100% 100%;background-repeat:no-repeat`)}}function rt(t,o){let n=t.querySelectorAll(\"select\"),s=o.querySelectorAll(\"select\");if(n.length===s.length)for(let l=0;l_(f)))continue;let c=t.createElement(\"option\");c.setAttribute(\"selected\",\"\"),c.textContent=\"\\u2022\\u2022\\u2022\",s[l].replaceChildren(c)}}function it(t,o){let n=\"input, textarea, select\",s=t.querySelectorAll(n),l=o.querySelectorAll(n);if(s.length===l.length)for(let u=0;u{let n=o.styleSheet;if(!n)return o.cssText;try{let s=be(n.cssRules),l=n.media?.mediaText;return l?`@media ${l} {\n${s}\n}`:s}catch{return o.cssText}}).join(`\n`)}function ot(t,o){let n=be(o);t.href&&(n=Ye(n,t.href));let s=t.media?.mediaText;return s&&(n=`@media ${s} {\n${n}\n}`),n}function st(t,o,n){for(let s of Array.from(t.styleSheets)){let l=s.ownerNode;if(!C(l))continue;if(s.disabled){let m=re(o,l),y=m?ie(n,m):void 0;y&&(y.setAttribute(\"media\",\"not all\"),y.tagName===\"STYLE\"&&(y.textContent=\"\"));continue}if(_(l))continue;let u;try{let m=s.cssRules;if(!m)continue;u=m}catch{continue}let c=re(o,l);if(!c)continue;let f=ie(n,c);if(!f)continue;let d=ot(s,u);if(l.tagName===\"LINK\"){let m=t.createElement(\"style\");m.textContent=d,f.replaceWith(m)}else l.tagName===\"STYLE\"&&(f.textContent=d)}}function Se(t,o={}){let n=t.documentElement,s=n.cloneNode(!0);if(it(t,s),st(t,n,s),nt(t,s),rt(t,s),Ze(t,s),o.target){let d=re(n,o.target);(d?ie(s,d):void 0)?.setAttribute(G,\"\")}s.querySelectorAll(\"template, noscript\").forEach(d=>d.remove()),s.querySelectorAll(\"script\").forEach(d=>d.remove()),s.querySelectorAll('meta[http-equiv=\"refresh\" i]').forEach(d=>d.remove()),s.querySelectorAll(\"*\").forEach(d=>{for(let m of Array.from(d.attributes))m.name.toLowerCase().startsWith(\"on\")&&d.removeAttribute(m.name)});let l=t.defaultView,u=Math.round(l?.scrollY??t.documentElement.scrollTop??0),c=Math.round(l?.scrollX??t.documentElement.scrollLeft??0);if(u>0||c>0){let d=t.createElement(\"style\");d.textContent=`html { margin-top: -${u}px !important; margin-left: -${c}px !important; }`,s.querySelector(\"head\")?.appendChild(d)}let f=s.querySelector(\"head\");if(o.baseHref&&f&&!f.querySelector(\"base\")){let d=t.createElement(\"base\");d.setAttribute(\"href\",o.baseHref),f.prepend(d)}return`${s.outerHTML}`}var gt=`[${G}] {\n\toutline: 3px solid #ef4444 !important;\n\toutline-offset: 2px !important;\n\tbox-shadow: 0 0 0 6px rgba(239, 68, 68, 0.25) !important;\n}`;function oe(t){if(!t||_(t))return\"\";let o=t;return t.querySelector(`[${D}]`)&&(o=t.cloneNode(!0),o.querySelectorAll(`[${D}]`).forEach(n=>n.remove())),(o.textContent??\"\").replace(/\\s+/g,\" \").trim()}function ye(t,o=40){let n=oe(t);return n.length>o?`${n.slice(0,o)}\\u2026`:n}function we(t){let o=t.tagName.toLowerCase(),n=(t.getAttribute(\"type\")??\"text\").toLowerCase(),s=o===\"input\"?`input[${n}]`:o,l=t.labels?.[0],u=t.getAttribute(\"aria-label\")||(l&&!_(l)?ye(l):\"\")||(o===\"input\"&&[\"button\",\"submit\",\"reset\"].includes(n)?t.getAttribute(\"value\"):\"\")||t.getAttribute(\"placeholder\")||t.getAttribute(\"title\")||ye(t)||t.getAttribute(\"name\")||t.getAttribute(\"id\")||\"\";return u?`${s} \"${u}\"`:s}function Re(t){let o=[],n=t,s=0;for(;n&&s<5;){let l=n.tagName.toLowerCase();if(n.id){o.unshift(`#${n.id}`);break}let u=typeof n.className==\"string\"?n.className.trim().split(/\\s+/).filter(Boolean)[0]:void 0,c=n.parentElement,f=u?`${l}.${u}`:l;if(c){let d=Array.from(c.children).filter(m=>m.tagName===n.tagName);d.length>1&&(f+=`:nth-of-type(${d.indexOf(n)+1})`)}o.unshift(f),n=c,s++}return o.join(\" > \")}function Le(t,o,n){switch(t){case\"click\":return`Clicked ${o}`;case\"fill\":return`Filled ${o} with \"${n??\"\"}\"`;case\"select\":return`Selected \"${n??\"\"}\" in ${o}`;case\"toggle\":return n?`${n===\"checked\"?\"Checked\":\"Unchecked\"} ${o}`:`Toggled ${o}`;case\"submit\":return`Submitted ${o}`;case\"key\":return`Pressed ${n??\"key\"} in ${o}`;case\"navigate\":return n?`Navigated to ${n}`:\"Reloaded the app\"}}var V=400,ct=3e3,ve=6e4,Ce=800,_e=new Set([\"button\",\"submit\",\"reset\",\"image\"]),ut=new Set([\"range\",\"color\",\"date\",\"time\",\"datetime-local\",\"month\",\"week\"]),dt=new Set([\"\",\"text\",\"search\",\"url\",\"tel\",\"email\",\"password\",\"number\"]),J=200,ft=250,xe=500;function Ie(){let t=!1,o=0,n=0,s=\"\",l,u,c=[],f=[],d=new Map,m=0,y=!1,v=!1,F={width:0,height:0},X=\"\",z=[],A,M,O=0,L,b,h,q=new Set,Q,N=0,B,Z=!1;function ke(e){return new Promise(a=>{let i=()=>{if(N===0||Date.now()-e>=ve||!P()){a();return}setTimeout(i,V)};i()})}let Me=e=>new Promise(a=>setTimeout(a,e));function P(){try{return u?.contentDocument??void 0}catch{return}}function $(e){if(e===void 0)return;let a=d.get(e);if(a!==void 0)return a;if(m+e.length>41943040){y=!0,v=!0;return}let i=f.length;return f.push(e),d.set(e,i),m+=e.length,i}function S(e){if(v)return;let a=P();if(a)try{return Se(a,{target:e,baseHref:X})}catch(i){console.warn(\"raw app recorder: snapshot failed\",i);return}}function Ne(e){return e.replace(` ${G}=\"\"`,\"\")}function j(){h&&(h.observer.disconnect(),clearTimeout(h.timer),clearTimeout(h.cap),h=void 0)}function se(e){j();let a=P();if(!a)return;let i=()=>{if(N>0&&h&&Date.now()-h.startedAt{h&&(clearTimeout(h.timer),h.timer=setTimeout(i,V))});r.observe(a,{subtree:!0,childList:!0,attributes:!0,characterData:!0}),h={step:e,observer:r,startedAt:Date.now(),timer:setTimeout(i,V),cap:setTimeout(i,ct)}}function ae(e){if(!h)return;let a=h.step;j();let i=e!==void 0&&(L?.html===e||b?.html===e);a.after=$(i?Ne(e):S())}function x(e,a,i,r,w=!1){if(!t)return;let E=Date.now()-n,p=c[c.length-1],R=!!p&&!!a&&ue(M,a)&&p.kind===e&&(w||Ue(a)&&E-O=500&&!R){y=!0,v=!0;return}let T=!!a&&_(a),H=le(a?T?Ae(a):we(a):\"the app\")??\"the app\",W=r&&r.length>J?`${r.slice(0,J)}\\u2026`:r,k=!T||!W?W:e===\"toggle\"?void 0:Y(W),ge=Le(e,H,k);if(R&&p){p.value=k,p.label=ge,O=E,se(p);return}let he={t:E,kind:e,label:ge,target:H,selector:a&&!T?le(Re(a)):void 0,value:k,before:$(i??(e===\"key\"?S(a):void 0))};c.push(he),i!==void 0&&L?.html===i&&(L=void 0),i!==void 0&&b?.html===i&&(b=void 0),M=a,O=E,o=c.length,se(he)}function Pe(e){let i=e.closest(\"label\")?.control;return!i||e===i||i.contains(e)?!1:!e.closest(\"a, button, input, select, textarea\")}function $e(e){return e.ctrlKey||e.metaKey||e.altKey?!1:e.key.length===1||[\" \",\"Enter\",\"ArrowUp\",\"ArrowDown\",\"ArrowLeft\",\"ArrowRight\",\"Home\",\"End\"].includes(e.key)}function ee(e){if(g(e,\"SELECT\"))return!0;if(!g(e,\"INPUT\"))return!1;let a=e.type;return!K(e)&&!_e.has(a)}function He(e){return g(e,\"BUTTON\")?(e.type||\"submit\")===\"submit\":g(e,\"INPUT\")&&[\"submit\",\"image\"].includes(e.type)}function De(e){let a=c[c.length-1];return a?.kind===\"key\"&&a.value===\"Enter\"&&!!e&&!!M&&e.contains(M)&&Date.now()-n-OJ?`${e.slice(0,J)}\\u2026`:e}function K(e){return g(e,\"TEXTAREA\")||e.isContentEditable?!0:g(e,\"INPUT\")&&dt.has(e.type)}function ce(e){let a=g(e,\"INPUT\")||g(e,\"TEXTAREA\")?e.value:oe(e);return g(e,\"INPUT\")&&e.type===\"password\"||_(e)?Y(a):a}function U(e){let a=L?.el;if(!a)return;if(a===e||a.contains(e)||e.contains(a))return L?.html;let i=e.labels;if(i&&Array.from(i).some(r=>r===a||r.contains(a)))return L?.html}function ue(e,a){if(!e||!a)return!1;if(e===a)return!0;let i=e,r=a;return i.type===\"radio\"&&r.type===\"radio\"&&!!i.name&&i.name===r.name&&i.form===r.form}function te(e){if(b)return ue(b.el,e)?b.html:void 0}function I(){if(!A)return;let{el:e,before:a}=A;clearTimeout(A.timer),A=void 0,U(e)!==void 0&&(L=void 0),b?.el===e&&(b=void 0),x(\"fill\",e,a,ce(e))}function de(e){let a=(i,r)=>{e.addEventListener(i,r,!0),z.push(()=>e.removeEventListener(i,r,!0))};a(\"pointerdown\",i=>{let r=C(i.target)?i.target:void 0;r&&(L={el:r,html:S(r)})}),a(\"click\",i=>{let r=C(i.target)?i.target:void 0;r&&(A&&A.el!==r&&I(),!(K(r)||ee(r)||g(r,\"OPTION\"))&&(i.detail===0&&He(r)&&De(r.closest(\"form\"))||Pe(r)||x(\"click\",r,U(r)??S(r))))}),a(\"focusin\",i=>{let r=C(i.target)?i.target:void 0;L&&(!r||U(r)===void 0)&&(L=void 0)}),a(\"beforeinput\",i=>{let r=C(i.target)?i.target:void 0;!r||!K(r)||A?.el===r||U(r)===void 0&&(b={el:r,html:S(r),repeat:!1})}),a(\"input\",i=>{let r=C(i.target)?i.target:void 0;if(!(!r||!K(r)))if(A&&A.el!==r&&I(),A)clearTimeout(A.timer),A.timer=setTimeout(I,Ce);else{let w=U(r),E=te(r),p=w??E??S(r);ae(p),A={el:r,before:p,timer:setTimeout(I,Ce)}}}),a(\"change\",i=>{let r=C(i.target)?i.target:void 0;if(!r)return;if(K(r)){I();return}A&&I();let w=U(r),E=te(r),p=w??E,R=w===void 0&&E!==void 0&&!!b?.repeat;if(g(r,\"SELECT\")){let T=Array.from(r.selectedOptions),H=T.map(k=>k.label||k.value).join(\", \"),W=T.some(k=>_(k));x(\"select\",r,p,W?Y(H):H,R)}else if(g(r,\"INPUT\")){let T=r;[\"checkbox\",\"radio\"].includes(T.type)?x(\"toggle\",r,p,T.checked?\"checked\":\"unchecked\",R):T.type===\"file\"?x(\"fill\",r,p,Array.from(T.files??[]).map(H=>H.name).join(\", \")):x(\"fill\",r,p,ce(r),R)}}),a(\"submit\",i=>{let r=C(i.target)?i.target:void 0;I();let w=c[c.length-1];(w?.kind===\"click\"||w?.kind===\"key\"&&w.value===\"Enter\")&&M&&r&&r.contains(M)&&Date.now()-n-O{let r=C(i.target)?i.target:void 0;r&&ee(r)&&$e(i)&&(i.repeat&&b&&te(r)!==void 0?b.repeat=!0:b={el:r,html:S(r),repeat:i.repeat}),!(i.key!==\"Enter\"&&i.key!==\"Escape\")&&(i.key===\"Enter\"&&r&&(ee(r)||qe(r)||Oe(r))||(I(),x(\"key\",r,i.repeat?void 0:S(r),i.key,i.repeat)))})}function fe(){z.forEach(e=>e()),z=[]}function me(e){let a=e.contentWindow,i=E=>{let p=E.data;if(!p||typeof p!=\"object\"||E.source!==window)return;let{type:R,reqId:T}=p;typeof R!=\"string\"||!R.endsWith(\"Res\")||(q.delete(T),N=q.size)},r=()=>{let E=P();!E||E===Q||(a?.addEventListener(\"message\",i),Q=E)},w=E=>{let p=E.data;if(!p||typeof p!=\"object\"||E.source!==a)return;let{type:R,reqId:T}=p;typeof R!=\"string\"||R.endsWith(\"Res\")||T===void 0||(r(),q.add(T),N=q.size)};return window.addEventListener(\"message\",w),r(),()=>{window.removeEventListener(\"message\",w),a?.removeEventListener(\"message\",i),Q=void 0}}function pe(){fe();let e=h?.step;j(),A&&clearTimeout(A.timer),A=void 0,L=void 0,b=void 0;let a=P();if(!a)return;de(a),u&&(B?.(),B=me(u));let i=S();if(e&&(e.after=$(i)),f.length===0){$(i);return}x(\"navigate\",void 0,i,a.location?.hash||void 0)}return{get active(){return t},get stepCount(){return o},get stopping(){return Z},start(e,a){u=e;let i=P();return i?.documentElement?(t=!0,n=Date.now(),s=a.appPath,l=a.workspace,c=[],M=void 0,O=0,o=0,f=[],d=new Map,m=0,y=!1,v=!1,X=typeof window<\"u\"?window.location.origin:\"\",F={width:e.clientWidth||i.documentElement.clientWidth,height:e.clientHeight||i.documentElement.clientHeight},i.readyState===\"complete\"&&i.location?.href!==\"about:blank\"&&$(S()),de(i),e.addEventListener(\"load\",pe),q.clear(),N=0,B=me(e),!0):(u=void 0,!1)},async stop(){if(I(),fe(),t=!1,h&&N>0){Z=!0;let a=h.startedAt;await ke(a),P()&&await Me(V),Z=!1}if(h){let a=h.step;j(),a.after=$(S())}B?.(),B=void 0,q.clear(),N=0,u?.removeEventListener(\"load\",pe),L=void 0,b=void 0,u=void 0;let e={version:1,type:\"app\",recorded_at:new Date().toISOString(),app_path:s,workspace:l,total_duration_ms:Date.now()-n,viewport:F,frames:f,steps:c,truncated:y||void 0};return c=[],f=[],d=new Map,m=0,e},download(e){let a=new Blob([JSON.stringify(e)],{type:\"application/json\"}),i=URL.createObjectURL(a),r=document.createElement(\"a\");r.href=i,r.download=`app-recording-${(e.app_path||\"untitled\").replace(/\\//g,\"-\")}-${Date.now()}.json`,r.click(),URL.revokeObjectURL(i)}}}return We(mt);})();\n"; diff --git a/cli/src/commands/app/raw_apps.ts b/cli/src/commands/app/raw_apps.ts index fcfc68915e..9f83d0322a 100644 --- a/cli/src/commands/app/raw_apps.ts +++ b/cli/src/commands/app/raw_apps.ts @@ -1,5 +1,6 @@ import { requireLogin } from "../../core/auth.ts"; import { resolveWorkspace, validatePath } from "../../core/context.ts"; +import { mergeConfigWithConfigFile } from "../../core/conf.ts"; import { colors } from "@cliffy/ansi/colors"; import * as log from "../../core/log.ts"; import { sep as SEP } from "node:path"; @@ -16,7 +17,7 @@ import { deepEqual, readTextFile } from "../../utils/utils.ts"; import { replaceInlineScripts, repopulateFields } from "./app.ts"; import { createBundle, detectFrameworks } from "./bundle.ts"; -import { APP_BACKEND_FOLDER } from "./app_metadata.ts"; +import { APP_BACKEND_FOLDER, RECORDINGS_FOLDER } from "./app_metadata.ts"; import { writeIfChanged } from "../../utils/utils.ts"; import { yamlOptions } from "../sync/sync.ts"; import { applyExtraPermsDiff } from "../../core/extra_perms.ts"; @@ -143,7 +144,8 @@ function getRunnableIdFromCodeFile(fileName: string): string | undefined { * Returns an empty object if the backend folder doesn't exist. * * @param backendPath - Path to the backend folder - * @param defaultTs - Default TypeScript runtime ("bun" or "deno") + * @param defaultTs - TypeScript runtime a bare `.ts` denotes. Must match what + * newRawAppPathAssigner used to write the file, or the round-trip relabels it. */ export async function loadRunnablesFromBackend( backendPath: string, @@ -317,6 +319,12 @@ async function collectAppFiles( ) { continue; } + // Session recordings, which the dev server only ever writes at the app + // root. Matched there alone, so an app of its own with a `recordings/` + // component folder still ships it. + if (basePath === "/" && entry.name === RECORDINGS_FOLDER) { + continue; + } await readDirRecursive(fullPath + SEP, relativePath + "/"); } else if (entry.isFile()) { // Skip generated/metadata files that shouldn't be part of the app @@ -344,6 +352,7 @@ export async function pushRawApp( remotePath: string, localPath: string, message?: string, + defaultTs: "bun" | "deno" = "bun", ): Promise { if (alreadySynced.includes(localPath)) { return; @@ -377,7 +386,10 @@ export async function pushRawApp( // Load runnables from separate YAML files in the backend folder // Falls back to reading from raw_app.yaml if no separate files exist (backward compat) const backendPath = path.join(localPath, APP_BACKEND_FOLDER); - const runnablesFromBackend = await loadRunnablesFromBackend(backendPath); + const runnablesFromBackend = await loadRunnablesFromBackend( + backendPath, + defaultTs, + ); let runnables: Record; if (Object.keys(runnablesFromBackend).length > 0) { @@ -539,7 +551,14 @@ async function pushRawAppCommand( } const workspace = await resolveWorkspace(opts); await requireLogin(opts); + const merged = await mergeConfigWithConfigFile(opts); - await pushRawApp(workspace.workspaceId, remotePath, filePath); + await pushRawApp( + workspace.workspaceId, + remotePath, + filePath, + undefined, + merged.defaultTs, + ); log.info(colors.bold.underline.green("Raw app pushed")); } diff --git a/cli/src/commands/app/wmillTsDev.ts b/cli/src/commands/app/wmillTsDev.ts index cd77652f53..7cb2ea328b 100644 --- a/cli/src/commands/app/wmillTsDev.ts +++ b/cli/src/commands/app/wmillTsDev.ts @@ -68,11 +68,45 @@ function initWebSocket() { initWebSocket() +/** A runnable call leaves this page over the WebSocket without touching the DOM, + * so the session recorder of \`wmill app dev --recording\` (which frames the app) + * has nothing else to tell it a step is still waiting on the backend. Announcing + * the request and its answer to the shell mirrors what the deployed runner posts + * across the same boundary. */ +const framed = typeof window !== 'undefined' && window.parent !== window + +function notifyRecorder(type: string, reqId: string) { + if (framed) window.parent.postMessage({ type, reqId }, window.location.origin) +} + +// A reload takes the previous context and its WebSocket with it, so whatever it +// had in flight can never answer. Announcing a fresh module is how the shell +// learns those calls are dead: a message posted from the unloading document +// would be dropped with the realm that sent it, and this runs before any app +// code can issue a call of its own. +if (framed) { + window.parent.postMessage({ type: 'wmillDevReady' }, window.location.origin) +} + +function tracked(type: string, reqId: string, resolve: (v: any) => void, reject: (e: any) => void) { + notifyRecorder(type, reqId) + let settled = false + const done = () => { + if (settled) return + settled = true + notifyRecorder(type + 'Res', reqId) + } + return { + resolve: (v: any) => { done(); resolve(v) }, + reject: (e: any) => { done(); reject(e) } + } +} + async function doRequest(type: string, o: object) { await wsReady return new Promise((resolve, reject) => { const reqId = Math.random().toString(36) - reqs[reqId] = { resolve, reject } + reqs[reqId] = tracked(type, reqId, resolve, reject) ws?.send(JSON.stringify({ ...o, type, reqId })) }) } @@ -119,7 +153,7 @@ export function streamJob( return new Promise(async (resolve, reject) => { await wsReady const reqId = Math.random().toString(36) - reqs[reqId] = { resolve, reject, onUpdate } + reqs[reqId] = { ...tracked('streamJob', reqId, resolve, reject), onUpdate } ws?.send(JSON.stringify({ jobId, type: 'streamJob', reqId })) }) } diff --git a/cli/src/commands/dev/dev.ts b/cli/src/commands/dev/dev.ts index f588f5841d..00a6f72f0e 100644 --- a/cli/src/commands/dev/dev.ts +++ b/cli/src/commands/dev/dev.ts @@ -20,7 +20,19 @@ import { SyncOptions, mergeConfigWithConfigFile, } from "../../core/conf.ts"; -import { exts, removeExtensionToPath } from "../script/script.ts"; +import { + exts, + hasScriptExt, + readModulesFromDisk, + removeExtensionToPath, +} from "../script/script.ts"; +import type { ScriptModule } from "../../../gen/types.gen.ts"; +import { + DBT_DESCRIPTOR_NAME, + DBT_MODULE_SUFFIX, + getScriptBasePathFromModulePath, + isDbtModulePath, +} from "../../utils/resource_folders.ts"; import { inferContentTypeFromFilePath } from "../../utils/script_common.ts"; import { OpenFlow } from "../../../gen/types.gen.ts"; import { FlowFile } from "../flow/flow.ts"; @@ -108,7 +120,7 @@ function findFlowFolderPrefix(cpath: string): string | undefined { return undefined; } -async function listWorkspacePaths(): Promise { +export async function listWorkspacePaths(): Promise { // Walk first, capturing each item's metadata file path. Then read summaries in // parallel — one tree pass plus N file reads is faster than a serialized walk. const items: (WmPathItem & { _metaPath?: string })[] = []; @@ -136,6 +148,19 @@ async function listWorkspacePaths(): Promise { items.push({ path: stripFolderSuffix(childRel, APP_SUFFIXES), kind: "raw_app" }); continue; } + // A dbt script IS the project directory: its descriptor is optional, so + // there may be no file here to recognize it by. Not descended into + // either — the project's own `.sql` models would otherwise each be + // listed as a script of their own. + if (entry.name.endsWith(DBT_MODULE_SUFFIX)) { + const base = childRel.slice(0, -DBT_MODULE_SUFFIX.length); + items.push({ + path: base, + kind: "script", + _metaPath: childAbs.slice(0, -DBT_MODULE_SUFFIX.length) + ".script.yaml", + }); + continue; + } await walk(childAbs, childRel); } else if (entry.isFile()) { const matchedExt = exts.find((ext) => entry.name.endsWith(ext)); @@ -282,12 +307,40 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { const flowMetadataFile = getMetadataFileName("flow", "yaml"); async function loadPaths(pathsToLoad: string[]) { - const paths = pathsToLoad.filter((p) => - exts.some( - (ext) => p.endsWith(ext) - || p.endsWith(".flow/" + flowMetadataFile) - || p.endsWith("__flow/" + flowMetadataFile) - ) + // A change ANYWHERE inside a dbt project is a change to that script: the + // bundle is the project, so the whole thing is re-read and rebroadcast. + // Treating the file as a script of its own would drop the edit — a bare + // `.sql` has no language to infer, and a `.yml` or `.csv` is filtered out + // below — leaving the browser previewing the snapshot taken at startup. + const rest: string[] = []; + const dbtProjects = new Set(); + for (const raw of pathsToLoad) { + const rel = (await realpath(raw).catch(() => raw)) + .replace(base + SEP, "") + .replaceAll("\\", "/"); + const wmPath = isDbtModulePath(rel) + ? getScriptBasePathFromModulePath(rel) + : undefined; + // Every project in the batch, and each only once: a save-all or a + // `git checkout` touches many files at once, and returning on the first + // would drop both the other projects and whatever else changed with them. + if (wmPath) dbtProjects.add(wmPath); + else rest.push(raw); + } + for (const wmPath of dbtProjects) { + const edit = await loadWmPath(wmPath); + if (edit) { + log.info("Updated " + wmPath + " (dbt project)"); + broadcastChanges(edit); + } + } + if (rest.length === 0) return; + pathsToLoad = rest; + const paths = pathsToLoad.filter( + (p) => + hasScriptExt(p) || + p.endsWith(".flow/" + flowMetadataFile) || + p.endsWith("__flow/" + flowMetadataFile) ); if (paths.length == 0) { return; @@ -380,6 +433,10 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { tag?: string; lock?: string; temp_script_refs?: Record; + /** The bundle the dev page forwards to a preview run. A dbt project cannot + * run without it: the worker looks for `dbt_project.yml` in the bundle and + * refuses the job when it is not there. */ + modules?: Record; }; type LastEditFlow = { @@ -441,8 +498,23 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { for (const ext of exts) { const filePath = wmPath + ext; try { - await access(filePath); - const content = await readTextFile(filePath); + // A dbt project's descriptor is optional, so what says "this is a dbt + // script" is the project beside it. Requiring the descriptor to exist + // would list such a project in the picker and then refuse to load it. + const isAbsentDbtDescriptor = + ext === "__dbt/" + DBT_DESCRIPTOR_NAME && + !(await access(filePath).then( + () => true, + () => false + )) && + (await access(wmPath + DBT_MODULE_SUFFIX + "/dbt_project.yml").then( + () => true, + () => false + )); + if (!isAbsentDbtDescriptor) { + await access(filePath); + } + const content = isAbsentDbtDescriptor ? "" : await readTextFile(filePath); const lang = inferContentTypeFromFilePath(filePath, opts.defaultTs); const typed = (await parseMetadataFile(removeExtensionToPath(filePath), undefined))?.payload; const edit: LastEditScript = { @@ -453,6 +525,18 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { tag: typed?.tag, lock: typed?.lock, temp_script_refs: tempScriptRefs, + // Read VERBATIM for dbt, the way push does: the project's `.sql` and + // `.yml` files are dbt's, and inferring a language for each would + // drop the ones that are not Windmill scripts. + modules: + lang === "dbt" + ? await readModulesFromDisk( + wmPath + DBT_MODULE_SUFFIX, + opts.defaultTs, + true, + true + ) + : undefined, }; currentLastEdit = edit; return edit; diff --git a/cli/src/commands/flow/flow.ts b/cli/src/commands/flow/flow.ts index 17a5e1febb..6ace1eaf48 100644 --- a/cli/src/commands/flow/flow.ts +++ b/cli/src/commands/flow/flow.ts @@ -13,7 +13,12 @@ import { mkdirSync, writeFileSync } from "node:fs"; import { buildFolderPath, getMetadataFileName, loadNonDottedPathsSetting } from "../../utils/resource_folders.ts"; import { requireLogin } from "../../core/auth.ts"; -import { resolveWorkspace, validatePath } from "../../core/context.ts"; +import { + assertRemotePath, + resolveWorkspace, + toSyncRootRelativePath, + validatePath, +} from "../../core/context.ts"; import { resolve, track_job, pollForJobResult } from "../script/script.ts"; import { defaultFlowDefinition } from "../../../bootstrap/flow_bootstrap.ts"; import { SyncOptions, mergeConfigWithConfigFile } from "../../core/conf.ts"; @@ -232,11 +237,19 @@ export async function pushFlow( const hasOnBehalfOf = (localFlow as any).has_on_behalf_of ?? !!localFlow.on_behalf_of_email; delete (localFlow as any).has_on_behalf_of; + // The authorization half of the identity is never exported to the repo (the + // workspace tarball strips it); it only ever travels back from the remote row. + delete (localFlow as any).on_behalf_of; - const preserveFields: { on_behalf_of_email?: string; preserve_on_behalf_of?: boolean } = {}; + const preserveFields: { + on_behalf_of_email?: string; + on_behalf_of?: string; + preserve_on_behalf_of?: boolean; + } = {}; if (permissionedAsContext?.userIsAdminOrDeployer && hasOnBehalfOf) { if (flow && flow.on_behalf_of_email) { preserveFields.on_behalf_of_email = flow.on_behalf_of_email; + preserveFields.on_behalf_of = (flow as any).on_behalf_of; preserveFields.preserve_on_behalf_of = true; log.info(`Preserving ${flow.on_behalf_of_email} as on_behalf_of for flow ${remotePath}`); } @@ -597,6 +610,8 @@ async function preview( log.setSilent(true); } const useLocalPathScripts = !opts.remote; + // Captured before the config read, which chdirs to the wmill.yaml root. + const cwdBeforeConfig = process.cwd(); if (useLocalPathScripts) { opts = await mergeConfigWithConfigFile(opts); } @@ -604,6 +619,9 @@ async function preview( await requireLogin(opts); const codebases = useLocalPathScripts ? listSyncCodebases(opts) : []; + const argPath = flowPath; + flowPath = toSyncRootRelativePath(flowPath, cwdBeforeConfig); + // Normalize path - ensure it's a directory path to a .flow or __flow folder const isFlowDir = flowPath.endsWith(".flow") || flowPath.endsWith(".flow" + SEP) || flowPath.endsWith("__flow") || flowPath.endsWith("__flow" + SEP); @@ -625,6 +643,14 @@ async function preview( flowPath += SEP; } + // The flow's windmill path (e.g. "f/cli_smoke/myrelflow"). It is what the + // preview job runs under, and the anchor for relative-import resolution: + // inline scripts in this flow are treated as living at + // "/", so "./util" resolves to + // "/util" — matching the keys in temp_script_refs. + const flowWmPath = stripFlowSuffix(flowPath).replaceAll(SEP, "/"); + assertRemotePath(flowWmPath, argPath); + // Read and parse the flow definition const localFlow = (await yamlParseFile(flowPath + "flow.yaml")) as FlowFile; @@ -695,12 +721,6 @@ async function preview( // too — PathScript modules have already been rewritten to inline rawscript // when `useLocalPathScripts` is set, and tempScriptRefs covers relative // imports in inline scripts. - // Compute the flow's windmill path (e.g. "f/cli_smoke/myrelflow"). Used as - // the anchor for relative-import resolution: inline scripts in this flow are - // treated as living at "/", so "./util" resolves to - // "/util" — matching the keys in temp_script_refs. - const flowWmPath = stripFlowSuffix(flowPath).replaceAll(SEP, "/"); - if (opts.step) { await previewStep(opts.step, localFlow, flowWmPath, workspace, input, tempScriptRefs, opts.silent, opts.tag); return; @@ -1218,6 +1238,8 @@ const command = new Command() ...remote, path: flowPath, on_behalf_of_email: email, + // Derived server-side; see the script command for why. + on_behalf_of: undefined, preserve_on_behalf_of: true, // Preserve any user draft at this path (see backend skip_draft_deletion). skip_draft_deletion: true, diff --git a/cli/src/commands/generate-metadata/generate-metadata.ts b/cli/src/commands/generate-metadata/generate-metadata.ts index 7eab79b52d..7310efc483 100644 --- a/cli/src/commands/generate-metadata/generate-metadata.ts +++ b/cli/src/commands/generate-metadata/generate-metadata.ts @@ -3,6 +3,7 @@ import { Confirm } from "@cliffy/prompt/confirm"; import { colors } from "@cliffy/ansi/colors"; import { sep as SEP } from "node:path"; import { GlobalOptions, isDatatableMigrationPath } from "../../types.ts"; +import { isFileResource, isFilesetResource } from "../../utils/utils.ts"; import { SyncOptions, mergeConfigWithConfigFile } from "../../core/conf.ts"; import { resolveWorkspace } from "../../core/context.ts"; import { requireLogin } from "../../core/auth.ts"; @@ -18,11 +19,12 @@ import { import { generateFlowLockInternal, FlowLocksResult } from "../flow/flow_metadata.ts"; import { generateAppLocksInternal, AppLocksResult } from "../app/app_metadata.ts"; import { + dedupeLockfilesOnDisk, elementsToMap, FSFSElement, ignoreF, } from "../sync/sync.ts"; -import { exts } from "../script/script.ts"; +import { hasScriptExt } from "../script/script.ts"; import { isFolderResourcePathAnyFormat, isScriptModulePath, isModuleEntryPoint, scriptPathToRemotePath } from "../../utils/resource_folders.ts"; import { listSyncCodebases, SyncCodebase } from "../../utils/codebase.ts"; import { @@ -51,11 +53,14 @@ async function walkLocalScripts( const elems = await elementsToMap( await FSFSElement(process.cwd(), codebases, false), (p, isD) => - (!isD && !exts.some((ext) => p.endsWith(ext))) || + (!isD && !hasScriptExt(p)) || ignore(p, isD) || isFolderResourcePathAnyFormat(p) || // Datatable migration `.sql` files aren't Windmill scripts. isDatatableMigrationPath(p) || + // Neither are file/fileset resource content files (.sql, .ts, …). + isFileResource(p) || + isFilesetResource(p) || (isScriptModulePath(p) && !isModuleEntryPoint(p)), false, {}, @@ -221,10 +226,13 @@ function categorizeLocalFiles( ) { appPaths.push(p); } else if ( - exts.some((ext) => p.endsWith(ext)) && + hasScriptExt(p) && !isFolderResourcePathAnyFormat(p) && // Datatable migration `.sql` files aren't Windmill scripts. !isDatatableMigrationPath(p) && + // Neither are file/fileset resource content files (.sql, .ts, …). + !isFileResource(p) && + !isFilesetResource(p) && !(isScriptModulePath(p) && !isModuleEntryPoint(p)) ) { scripts.push(p); @@ -404,6 +412,51 @@ export async function rehashOnly( return counts; } +/** + * Normalize the tree's shared lockfiles, unless the run asked to stay inside one + * folder: shared lockfiles are workspace-wide, so the pass reads and rewrites + * metadata outside it, which `--strict-folder-boundaries` promises not to do. + */ +async function maybeDedupeLockfiles( + opts: GlobalOptions & SyncOptions & { strictFolderBoundaries?: boolean }, + workspace: Workspace, + codebases: SyncCodebase[], + ignore: (p: string, isD: boolean) => boolean, + rawWorkspaceDependencies: Record, + tree: DoubleLinkedDependencyTree, + folder: string | undefined, + failed: string[] = [], +): Promise { + if (!opts.dedupeLockfiles) return; + if (folder && opts.strictFolderBoundaries) { + log.info( + colors.yellow( + `Skipping lockfile deduplication: it spans the whole workspace, and --strict-folder-boundaries keeps this run inside "${folder}".`, + ), + ); + return; + } + const args = { + opts, + workspace, + codebases, + ignore, + rawWorkspaceDependencies, + tree, + failed, + }; + if (opts.dryRun) { + await dedupeLockfilesOnDisk({ ...args, dryRun: true }); + return; + } + await beginLockfileBatch(); + try { + await dedupeLockfilesOnDisk(args); + } finally { + await flushLockfileBatch(); + } +} + export async function generateMetadata( opts: GlobalOptions & { yes?: boolean; @@ -603,6 +656,10 @@ export async function generateMetadata( // === Show stale items and confirm === if (filteredItems.length === 0) { log.info(colors.green("All metadata up-to-date")); + // Turning `dedupeLockfiles` on in a repo whose metadata is already current + // is exactly the case where nothing is stale, and the conversion still has + // to happen — the sync compares against a deduplicated remote either way. + await maybeDedupeLockfiles(opts, workspace, codebases, ignore, rawWorkspaceDependencies, tree, folder); return; } @@ -630,6 +687,9 @@ export async function generateMetadata( printItems("Apps", apps); if (opts.dryRun) { + // The preview belongs on this path too: the conversion is what a stale tree + // is about to get, and only the up-to-date path reported it. + await maybeDedupeLockfiles(opts, workspace, codebases, ignore, rawWorkspaceDependencies, tree, folder); return; } @@ -768,10 +828,18 @@ export async function generateMetadata( // Persist all stale workspace dep hashes (not just filtered — deps are global, not folder-scoped) const allStaleDeps = staleItems.filter((i) => i.type === "dependencies"); await tree.persistDepsHashes(allStaleDeps.map((d) => d.path)); + } finally { await flushLockfileBatch(); } + // The scripts whose generation failed keep whatever lock they had, and must + // not be re-hashed as though this run had refreshed them. + await maybeDedupeLockfiles( + opts, workspace, codebases, ignore, rawWorkspaceDependencies, tree, folder, + errors.map((e) => e.path), + ); + const succeeded = total - errors.length; log.info(""); if (errors.length > 0) { diff --git a/cli/src/commands/gitsync-settings/converter.ts b/cli/src/commands/gitsync-settings/converter.ts index 6861603bb4..868fa1b330 100644 --- a/cli/src/commands/gitsync-settings/converter.ts +++ b/cli/src/commands/gitsync-settings/converter.ts @@ -34,6 +34,7 @@ export class GitSyncSettingsConverter { includeSettings: includeTypes.includes("settings"), includeKey: includeTypes.includes("key"), skipWorkspaceDependencies: !includeTypes.includes("workspacedependencies"), + skipDatatableMigrations: !includeTypes.includes("datatablemigration"), }; // Only include extraIncludes if it has content @@ -63,6 +64,7 @@ export class GitSyncSettingsConverter { if (opts.includeSettings) includeTypes.push("settings"); if (opts.includeKey) includeTypes.push("key"); if (!opts.skipWorkspaceDependencies) includeTypes.push("workspacedependencies"); + if (!opts.skipDatatableMigrations) includeTypes.push("datatablemigration"); const result: BackendGitSyncSettings = { include_path: opts.includes || [], @@ -102,6 +104,7 @@ export class GitSyncSettingsConverter { includeSettings: opts.includeSettings ?? false, includeKey: opts.includeKey ?? false, skipWorkspaceDependencies: opts.skipWorkspaceDependencies ?? false, + skipDatatableMigrations: opts.skipDatatableMigrations ?? false, }; } @@ -126,6 +129,7 @@ export class GitSyncSettingsConverter { includeSettings: opts.includeSettings, includeKey: opts.includeKey, skipWorkspaceDependencies: opts.skipWorkspaceDependencies, + skipDatatableMigrations: opts.skipDatatableMigrations, }; } diff --git a/cli/src/commands/gitsync-settings/types.ts b/cli/src/commands/gitsync-settings/types.ts index 7bebb1f1a1..2a74966ccf 100644 --- a/cli/src/commands/gitsync-settings/types.ts +++ b/cli/src/commands/gitsync-settings/types.ts @@ -38,6 +38,7 @@ export const GIT_SYNC_FIELDS = [ "includeSettings", "includeKey", "skipWorkspaceDependencies", + "skipDatatableMigrations", ] as const; export type GitSyncField = typeof GIT_SYNC_FIELDS[number]; @@ -59,6 +60,7 @@ export const INCLUDE_TYPE_MAPPINGS = { settings: "includeSettings", key: "includeKey", workspacedependencies: "skipWorkspaceDependencies", + datatablemigration: "skipDatatableMigrations", } as const; // Write mode for branch-based configuration diff --git a/cli/src/commands/init/template.ts b/cli/src/commands/init/template.ts index b0c83d7039..9032cee397 100644 --- a/cli/src/commands/init/template.ts +++ b/cli/src/commands/init/template.ts @@ -97,6 +97,7 @@ export const CONFIG_REFERENCE: ConfigOption[] = [ { name: "skipApps", type: "boolean", default: "false", description: "Skip syncing apps" }, { name: "skipFolders", type: "boolean", default: "false", description: "Skip syncing folders" }, { name: "skipWorkspaceDependencies", type: "boolean", default: "false", description: "Skip syncing workspace dependencies" }, + { name: "skipDatatableMigrations", type: "boolean", default: "false", description: "Skip syncing data table SQL migrations" }, { name: "includeSchedules", type: "boolean", default: "false", description: "Include schedules in sync", commented: true, templateValue: "true", groupNote: "Uncomment to include these (excluded by default):" }, @@ -116,6 +117,8 @@ export const CONFIG_REFERENCE: ConfigOption[] = [ section: "Sync behavior", commented: true, templateValue: "4" }, { name: "locksRequired", type: "boolean", default: "false", description: "Require lock files for all scripts", commented: true, templateValue: "true" }, + { name: "dedupeLockfiles", type: "boolean", default: "false", description: "Share one lockfile per workspace dependency file (locks/.lock), instead of an identical .script.lock per script", + commented: true, templateValue: "true" }, { name: "lint", type: "boolean", default: "false", description: "Run linting before push", commented: true, templateValue: "true" }, { name: "plainSecrets", type: "boolean", default: "false", description: "Handle secrets as plain text (not recommended)", diff --git a/cli/src/commands/lint/lint.ts b/cli/src/commands/lint/lint.ts index d3db402833..f6a35429ef 100644 --- a/cli/src/commands/lint/lint.ts +++ b/cli/src/commands/lint/lint.ts @@ -26,6 +26,7 @@ import { inferContentTypeFromFilePath, languageNeedsLock, ScriptLanguage, + isSharedLockPath, } from "../../utils/script_common.ts"; import { isFlowInlineScriptPath, @@ -133,6 +134,7 @@ function formatYamlDiagnostics(parsed: { diagnostics?: Array<{ message?: string async function isLockResolved( lockValue: string | string[] | undefined, baseDir: string, + sharedLockBase?: string, ): Promise { if (lockValue === undefined) return false; @@ -141,7 +143,11 @@ async function isLockResolved( const joined = lockValue.join("\n"); if (joined === "") return false; if (joined.startsWith("!inline ")) { - return await checkInlineFile(joined.substring("!inline ".length), baseDir); + return await checkInlineFile( + joined.substring("!inline ".length), + baseDir, + sharedLockBase, + ); } return true; } @@ -150,7 +156,11 @@ async function isLockResolved( // Inline file reference if (lockValue.startsWith("!inline ")) { - return await checkInlineFile(lockValue.substring("!inline ".length), baseDir); + return await checkInlineFile( + lockValue.substring("!inline ".length), + baseDir, + sharedLockBase, + ); } // Embedded lock content @@ -160,8 +170,17 @@ async function isLockResolved( async function checkInlineFile( relativePath: string, baseDir: string, + sharedLockBase?: string, ): Promise { - const fullPath = path.join(baseDir, relativePath.trim()); + const trimmed = relativePath.trim(); + // A shared lockfile (`dedupeLockfiles`) is referenced from the sync root, not + // from the directory being linted. Only a standalone script's metadata passes + // a base for it: a flow or app inline lock is folder-relative even when its + // name happens to look like one. + const fullPath = + sharedLockBase !== undefined && isSharedLockPath(trimmed) + ? path.resolve(sharedLockBase, trimmed) + : path.join(baseDir, trimmed); try { const s = await stat(fullPath); return s.size > 0; @@ -170,6 +189,21 @@ async function checkInlineFile( } } +/** Where a repo-root-relative reference resolves from: the directory holding + * wmill.yaml at or above the linted one, and that directory itself when there + * is none. */ +async function findSyncRoot(dir: string): Promise { + let current = path.resolve(dir); + while (true) { + if (await stat(path.join(current, "wmill.yaml")).then(() => true).catch(() => false)) { + return current; + } + const parent = path.dirname(current); + if (parent === current) return path.resolve(dir); + current = parent; + } +} + /** * Recursively find rawscript modules in a flow's module tree. */ @@ -481,6 +515,7 @@ export async function checkMissingLocks( } // Check standalone scripts + const syncRoot = await findSyncRoot(targetDirectory); for (const yamlPath of scriptYamls) { const basePath = yamlPath.replace(/\.script\.yaml$/, ""); @@ -506,6 +541,7 @@ export async function checkMissingLocks( const lockResolved = await isLockResolved( metadata?.lock, targetDirectory, + syncRoot, ); if (!lockResolved) { issues.push({ diff --git a/cli/src/commands/pipeline/docs.ts b/cli/src/commands/pipeline/docs.ts index abee64cbf3..1ae38d5e71 100644 --- a/cli/src/commands/pipeline/docs.ts +++ b/cli/src/commands/pipeline/docs.ts @@ -17,11 +17,12 @@ import { colors } from "@cliffy/ansi/colors"; import { GlobalOptions } from "../../types.ts"; import { type AssetGraph, + hideDbtRunnables, buildLocalPipelineGraph, workspaceRoot, } from "./localGraph.ts"; -const ASSET_KINDS = "s3object,ducklake,datatable,volume"; +const ASSET_KINDS = "s3object,ducklake,datatable,volume,dbt"; function assetUri(kind: string, p: string): string { const prefix = kind === "s3object" ? "s3" : kind; @@ -39,7 +40,7 @@ async function fetchDeployedGraph( if (!res.ok) { throw new Error(`GET assets/graph -> ${res.status}: ${await res.text()}`); } - return (await res.json()) as AssetGraph; + return hideDbtRunnables((await res.json()) as AssetGraph); } // Render the pipeline graph as a markdown document. diff --git a/cli/src/commands/pipeline/localGraph.ts b/cli/src/commands/pipeline/localGraph.ts index bb51cd98eb..5f9e713f42 100644 --- a/cli/src/commands/pipeline/localGraph.ts +++ b/cli/src/commands/pipeline/localGraph.ts @@ -63,6 +63,10 @@ export type GraphRunnable = { // `buildMacroEdges` (the wasm asset parser emits neither the marker nor the // registry). Non-empty ⇒ definition-only node. macros?: { name: string; params?: string; is_table?: boolean }[]; + // Set by the deployed graph on a dbt script: it owns a whole project, so the + // node counts models rather than reading as a single-output script. Never set + // locally — a dbt descriptor has no asset parser here. + dbt?: { model_count: number }; }; export type GraphEdge = { runnable_kind: string; @@ -229,6 +233,7 @@ function commentPrefix(language: string): string { language === "ansible" || language === "ruby" || language === "rlang" || + language === "dbt" || language === "nu" || language === "powershell" ) @@ -342,7 +347,11 @@ function normalizeRetry(retry: ParseAssetsRaw["retry"]): ParseAssetsRaw["retry"] // Read-asset kinds whose read auto-derives a cascade trigger edge inside a // `// pipeline`. Mirror of backend `is_auto_trigger_kind` (windmill-common // assets.rs) / frontend `AUTO_TRIGGER_KINDS` (resolveGraph.ts) — ducklake -// tables and s3 objects only; resource/datatable/volume stay explicit-`// on`. +// tables and s3 objects; resource/datatable/volume/table stay explicit-`// on`. +// A local graph out of step with that set shows an edge the deploy will not +// cascade along, and the generated pipeline docs then describe the wrong DAG. +// `table` is excluded everywhere: a dbt run does not dispatch, and nothing else +// writes a warehouse relation. const AUTO_TRIGGER_KINDS = new Set(["ducklake", "s3object"]); // Asset-URI prefixes accepted by `// mute `, in lockstep with the @@ -550,6 +559,75 @@ export async function collectScripts( return out; } +// The structural minimum the dbt filter needs. Declared instead of taking +// `AssetGraph` so the bounded-cascade view of the same payload (`BCGraph`, a +// narrower shape over identical JSON) passes through without a cast. +type DbtFilterableGraph = { + runnables: { path: string; usage_kind: string; dbt?: unknown }[]; + edges: { runnable_kind: string; runnable_path: string }[]; + triggers: { runnable_kind: string; runnable_path: string }[]; + macro_edges?: { lib_path: string; consumer_path: string }[]; + test_edges?: { + producer_kind: string; + producer_path: string; + runnable_kind: string; + runnable_path: string; + }[]; +}; + +/** + * The pipeline's view of a deployed graph whose folder also holds a dbt project. + * + * `/assets/graph` is asset-usage driven, not membership driven: it lists every + * script that reads or writes a relation, so a dbt script appears there like any + * producer. That is right for the endpoint — the node is what attributes a + * relation to what builds it — but a dbt project is not a pipeline, so it must + * not be rendered as one of its scripts. Mirrors the frontend's + * `hideDbtRunnables`; the local builder drops these nodes at the source. + * + * Its relations stay: they are what a downstream pipeline script reads. + */ +export function hideDbtRunnables(graph: G): G { + // Keyed by `(usage_kind, path)`, the graph's identity for a runnable — a + // script and a flow may share a path, and keying on path alone would take the + // flow's node, edges and triggers down with the dbt script's. + const key = (usage_kind: string, path: string) => `${usage_kind}:${path}`; + const dbtKeys = new Set( + (graph.runnables ?? []) + .filter((r) => r.dbt) + .map((r) => key(r.usage_kind, r.path)), + ); + if (dbtKeys.size === 0) return graph; + const kept = (usage_kind: string, path: string) => + !dbtKeys.has(key(usage_kind, path)); + return { + ...graph, + runnables: graph.runnables.filter((r) => kept(r.usage_kind, r.path)), + edges: graph.edges.filter((e) => kept(e.runnable_kind, e.runnable_path)), + triggers: graph.triggers.filter((t) => + kept(t.runnable_kind, t.runnable_path), + ), + // A macro library is a script; a dbt script defines no macros, so neither + // end can be a flow. + ...(graph.macro_edges + ? { + macro_edges: graph.macro_edges.filter( + (m) => kept("script", m.lib_path) && kept("script", m.consumer_path), + ), + } + : {}), + ...(graph.test_edges + ? { + test_edges: graph.test_edges.filter( + (t) => + kept(t.producer_kind, t.producer_path) && + kept(t.runnable_kind, t.runnable_path), + ), + } + : {}), + }; +} + // Build the full pipeline asset-graph from local files in `f//`. // Only `// pipeline` scripts become graph nodes (pipeline membership), mirroring // the deployed graph endpoint. Returns the graph plus the in-pipeline scripts' @@ -617,6 +695,11 @@ export async function buildLocalPipelineGraph(args: { }); continue; } + // A dbt project is not a data pipeline: the deploy never marks a dbt script + // `auto_kind='pipeline'` and the pipeline canvas drops its node, so the + // local graph must not invent one. Its models don't belong here either — + // they come from the manifest the deploy derives, not from the descriptor. + if (s.language === "dbt") continue; if (!out.in_pipeline) continue; // not a pipeline member if (out.data_tests && out.data_tests.length > 0) { dataTestsByPath.set(s.path, out.data_tests); diff --git a/cli/src/commands/pipeline/pipeline.ts b/cli/src/commands/pipeline/pipeline.ts index 0738283fba..8d46119e14 100644 --- a/cli/src/commands/pipeline/pipeline.ts +++ b/cli/src/commands/pipeline/pipeline.ts @@ -24,6 +24,7 @@ import { } from "./boundedCascade.ts"; import { type AssetGraph, + hideDbtRunnables, type GraphTrigger, type LocalScript, buildLocalPipelineGraph, @@ -96,7 +97,7 @@ async function list(opts: GlobalOptions & { json?: boolean }) { } } -const ASSET_KINDS = "s3object,ducklake,datatable,volume"; +const ASSET_KINDS = "s3object,ducklake,datatable,volume,dbt"; function assetUri(kind: string, path: string): string { const prefix = kind === "s3object" ? "s3" : kind; @@ -132,8 +133,10 @@ async function show( } else { const workspace = await resolveWorkspace(opts); await requireLogin(opts); - graph = await apiGet( - `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + graph = hideDbtRunnables( + await apiGet( + `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + ), ); enrich = (nativeByScript, roots) => enrichRootMarkers(workspace.workspaceId, graph, nativeByScript, roots); } @@ -612,8 +615,10 @@ async function run( } } } else { - graph = await apiGet( - `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + graph = hideDbtRunnables( + await apiGet( + `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + ), ); // Recover marker-only `data_upload`/`webhook`/`email` triggers the graph // endpoint can't emit, so input-only entrypoints are cut here as they are diff --git a/cli/src/commands/resource/resource.ts b/cli/src/commands/resource/resource.ts index f954b35adb..7cef4caf1f 100644 --- a/cli/src/commands/resource/resource.ts +++ b/cli/src/commands/resource/resource.ts @@ -46,6 +46,32 @@ async function readFilesetDirectory(dirPath: string): Promise.fileset` — that is the only layout the sync diff engine + * can round-trip (remote state is always rendered there, including for + * workspace-specific resources). Any other pointer breaks change detection: + * children are planned as full delete/re-add churn and adds under the custom + * directory are dropped, which manifests as erased or stale fileset content. + */ +export function validateFilesetPointer( + dirPath: string, + remotePath: string, +): void { + const normalize = (p: string) => + p.replaceAll("\\", "/").replace(/\/+$/, ""); + const pointer = normalize(dirPath); + const expected = normalize(remotePath.replaceAll(SEP, "/")) + ".fileset"; + if (pointer !== expected) { + throw new Error( + `Resource ${remotePath.replaceAll(SEP, "/")} uses '!inline_fileset ${dirPath}', ` + + `but a fileset directory must live next to its resource file, at '${expected}'. ` + + `Move the directory there (e.g. 'git mv ${pointer} ${expected}') and update the ` + + `'!inline_fileset' value to match.`, + ); + } +} + export async function pushResource( workspace: string, remotePath: string, @@ -53,6 +79,10 @@ export async function pushResource( localResource: ResourceFile, originalLocalPath?: string, wsSpecific?: boolean, + // Sync pushes reject non-canonical fileset pointers (the diff engine can + // only round-trip the canonical layout); the standalone `resource push` + // command pushes a single explicit file, where any pointer is fine. + enforceCanonicalFileset?: boolean, ): Promise { remotePath = removeType(remotePath, "resource"); try { @@ -68,6 +98,9 @@ export async function pushResource( const resolveInlineContent = async () => { if (typeof localResource.value === "string" && localResource.value.startsWith("!inline_fileset ")) { const dirPath = localResource.value.split(" ")[1]; + if (enforceCanonicalFileset) { + validateFilesetPointer(dirPath, remotePath); + } localResource.value = await readFilesetDirectory(dirPath.replaceAll("/", SEP)); } else if (localResource.value["content"]?.startsWith("!inline ")) { const basePath = localResource.value["content"].split(" ")[1]; diff --git a/cli/src/commands/script/script.ts b/cli/src/commands/script/script.ts index c699e3d58f..8c93bdb963 100644 --- a/cli/src/commands/script/script.ts +++ b/cli/src/commands/script/script.ts @@ -1,6 +1,11 @@ import { GlobalOptions } from "../../types.ts"; import { requireLogin } from "../../core/auth.ts"; -import { resolveWorkspace, validatePath } from "../../core/context.ts"; +import { + assertRemotePath, + resolveWorkspace, + toSyncRootRelativePath, + validatePath, +} from "../../core/context.ts"; import type { PermissionedAsContext } from "../../core/permissioned_as.ts"; import { applyExtraPermsDiff } from "../../core/extra_perms.ts"; import { writeFile, stat, mkdir } from "node:fs/promises"; @@ -13,7 +18,7 @@ import * as log from "../../core/log.ts"; import { sep as SEP } from "node:path"; import * as path from "node:path"; import { stringify as yamlStringify } from "yaml"; -import { deepEqual, getHeaders, readTextFile, readTextFileSync } from "../../utils/utils.ts"; +import { deepEqual, getHeaders, isFileResource, isFilesetResource, readTextFile, readTextFileSync } from "../../utils/utils.ts"; import { detectAuthGatewayChallenge } from "../../utils/http_guards.ts"; import * as wmill from "../../../gen/services.gen.ts"; import * as specificItems from "../../core/specific_items.ts"; @@ -71,10 +76,19 @@ import { isScriptModulePath, buildModuleFolderPath, getModuleFolderSuffix, + dbtGeneratedDirs, + isUnderGeneratedDir, + isLocalSecretFile, + moduleFileExclusion, + oversizedModuleFileError, + MAX_MODULE_BYTES, isModuleEntryPoint, getScriptBasePathFromModulePath, scriptPathToRemotePath, isRawAppPath, + DBT_DESCRIPTOR_NAME, + isDbtDescriptorPath, + isMissingDbtDescriptor, } from "../../utils/resource_folders.ts"; export interface ScriptFile { @@ -159,9 +173,18 @@ async function push(opts: PushOptions, filePath: string) { return; } - const fstat = await stat(filePath); - if (!fstat.isFile()) { - throw new Error("file path must refer to a file."); + // A dbt project's descriptor is optional, so the one content path a + // descriptor-less project has is deliberately not on disk. The project beside + // it is what says the script is real. + const absentDescriptor = await stat(filePath).then( + () => false, + (e) => isMissingDbtDescriptor(filePath, e) + ); + if (!absentDescriptor) { + const fstat = await stat(filePath); + if (!fstat.isFile()) { + throw new Error("file path must refer to a file."); + } } if (filePath.endsWith(".script.json") || filePath.endsWith(".script.yaml")) { @@ -170,11 +193,17 @@ async function push(opts: PushOptions, filePath: string) { ); } + if (isFileResource(filePath) || isFilesetResource(filePath)) { + throw Error( + "Cannot push a file/fileset resource content file as a script, push its .resource.yaml with 'wmill resource push' instead" + ); + } + await requireLogin(opts); // Warn about metadata state before pushing try { - const content = await readTextFile(filePath); + const content = await readScriptContent(filePath); const remotePath = removeExtensionToPath(filePath).replaceAll(SEP, "/"); const contentHash = await computePushMetadataHash(filePath, content); const conf = await readLockfile(); @@ -272,7 +301,7 @@ const MODULE_ENTRY_META_RE = /([\\/])script\.(yaml|json|lock)$/; * `__mod/`, so this only narrows it to the metadata extensions: a `script.yaml` * nested deeper in the module tree is a module file, not the script's metadata. */ -function isModuleEntryMetadata(p: string): boolean { +export function isModuleEntryMetadata(p: string): boolean { return isModuleEntryPoint(p) && MODULE_ENTRY_META_RE.test(p); } @@ -326,6 +355,12 @@ export async function handleFile( codebases: SyncCodebase[], permissionedAsContext?: PermissionedAsContext ): Promise { + // A file/fileset resource's content file can carry a script extension + // (.sql, .ts, …) but belongs to its parent resource, never to a + // standalone script. + if (isFileResource(path) || isFilesetResource(path)) { + return false; + } // Detect module entry point: e.g., my_script__mod/script.ts const moduleEntryPoint = isModuleEntryPoint(path); if ( @@ -335,7 +370,7 @@ export async function handleFile( // standalone scripts — pushed via pushRawApp, not here. !isRawAppPath(path) && (!isScriptModulePath(path) || moduleEntryPoint) && - exts.some((exts) => path.endsWith(exts)) + hasScriptExt(path) ) { if (alreadySynced.includes(path)) { return true; @@ -345,6 +380,29 @@ export async function handleFile( alreadySynced.push(path); const remotePath = scriptPathToRemotePath(path); + // Before anything is written: `.py` and `__dbt/` deploy to ONE + // remote path, so whichever is pushed last replaces the other's script. + // Refused from either side — the descriptor is exempt only from finding its + // OWN project (it is that project's content file, so its base resolves to + // the same `dbt_project.yml`), never from an ordinary sibling. + // A folder-layout script is `__mod/script.ts`, so stripping its + // extension yields `__mod/script`, not the base both layouts deploy + // to. Wrong base, and the probe below looks in a directory that cannot + // exist — which is how a `__mod` script and a dbt project at one path were + // both pushed, each replacing the other. + const base = isScriptModulePath(path) + ? getScriptBasePathFromModulePath(path) ?? removeExtensionToPath(path) + : removeExtensionToPath(path); + const isDescriptor = isDbtDescriptorPath(path); + const other = isDescriptor + ? await collidingOrdinaryScript(base) + : await collidingDbtProject(base); + if (other) { + throw isDescriptor + ? dbtPathCollisionError(path, other) + : dbtPathCollisionError(other, path); + } + const language = inferContentTypeFromFilePath(path, opts?.defaultTs); const codebase = @@ -479,7 +537,7 @@ export async function handleFile( } catch { log.debug(`Script ${remotePath} does not exist on remote`); } - const content = await readTextFile(path); + const content = await readScriptContent(path); if (opts?.skipScriptsMetadata) { // if (codebase) { @@ -499,8 +557,14 @@ export async function handleFile( const scriptBasePath = moduleEntryPoint ? getScriptBasePathFromModulePath(path)! : path.substring(0, path.indexOf(".")); - const moduleFolderPath = scriptBasePath + getModuleFolderSuffix(); - const modules = await readModulesFromDisk(moduleFolderPath, opts?.defaultTs, moduleEntryPoint); + const isDbt = language === "dbt"; + const moduleFolderPath = scriptBasePath + getModuleFolderSuffix(language); + const modules = await readModulesFromDisk( + moduleFolderPath, + opts?.defaultTs, + moduleEntryPoint, + isDbt, + ); // A concurrent_limit of <= 0 means "concurrency disabled", not "zero slots" (which // would brick the runnable at the queue's concurrency gate). Emit it as omitted rather @@ -518,12 +582,17 @@ export async function handleFile( path: remotePath.replaceAll(SEP, "/"), summary: typed?.summary ?? "", kind: typed?.kind, - lock: typed?.lock, + // A dbt lock pins a resolved commit and engine versions that only a + // dependency job can determine, and that job is also what publishes the + // script's manifest graph. Sending one suppresses that job, so the push + // would deploy a stale lock AND leave the graph unpublished. + lock: language === "dbt" ? undefined : typed?.lock, schema: typed?.schema, tag: typed?.tag, ws_error_handler_muted: typed?.ws_error_handler_muted, dedicated_worker: typed?.dedicated_worker, cache_ttl: typed?.cache_ttl, + cache_ignore_s3_path: typed?.cache_ignore_s3_path, concurrency_time_window_s: normConcurrencyTimeWindowS, concurrent_limit: normConcurrentLimit, deployment_message: message, @@ -534,8 +603,14 @@ export async function handleFile( concurrency_key: typed?.concurrency_key, debounce_key: typed?.debounce_key, debounce_delay_s: typed?.debounce_delay_s, + debounce_args_to_accumulate: typed?.debounce_args_to_accumulate, + max_total_debouncing_time: typed?.max_total_debouncing_time, + max_total_debounces_amount: typed?.max_total_debounces_amount, codebase: await codebase?.getDigest(forceTar), timeout: nonePositiveInt(typed?.timeout), + // 0 means "delete immediately after completion", so it must survive as 0 + // rather than being folded into "unset" the way the positive-only settings are. + delete_after_secs: typed?.delete_after_secs, on_behalf_of_email: typed?.on_behalf_of_email, envs: typed?.envs, modules: modules, @@ -544,10 +619,16 @@ export async function handleFile( const hasOnBehalfOf = (typed as any)?.has_on_behalf_of ?? !!typed?.on_behalf_of_email; delete (typed as any)?.has_on_behalf_of; + // The authorization half of the identity is never exported to the repo (the + // workspace tarball strips it); it only ever travels back from the remote row. + delete (typed as any)?.on_behalf_of; if (permissionedAsContext?.userIsAdminOrDeployer && hasOnBehalfOf) { if (remote && remote.on_behalf_of_email) { requestBodyCommon.on_behalf_of_email = remote.on_behalf_of_email; + (requestBodyCommon as any).on_behalf_of = ( + remote as any + ).on_behalf_of; (requestBodyCommon as any).preserve_on_behalf_of = true; log.info(`Preserving ${remote.on_behalf_of_email} as on_behalf_of for script ${remotePath}`); } @@ -561,6 +642,12 @@ export async function handleFile( (typed.description === remote.description && typed.summary === remote.summary && typed.kind == remote.kind && + // A `.ts` file changes language when defaultTs flips, content untouched. + // bun and bunnative share that extension, so the inferred language is always + // bun; the server derives bunnative back from the `//native` annotation in + // the content, which is compared above. + language == + (remote.language === "bunnative" ? "bun" : remote.language) && !remote.archived && (Array.isArray(remote?.lock) ? remote?.lock?.join("\n") @@ -572,6 +659,8 @@ export async function handleFile( remote.ws_error_handler_muted && typed.dedicated_worker == remote.dedicated_worker && typed.cache_ttl == remote.cache_ttl && + Boolean(typed.cache_ignore_s3_path) == + Boolean(remote.cache_ignore_s3_path) && normConcurrencyTimeWindowS == normalizeConcurrency( remote.concurrent_limit, @@ -585,15 +674,23 @@ export async function handleFile( Boolean(remote.visible_to_runner_only) && Boolean(typed.has_preprocessor) == Boolean(remote.has_preprocessor) && - typed.priority == Boolean(remote.priority) && + typed.priority == remote.priority && nonePositiveInt(typed.timeout) == nonePositiveInt(remote.timeout) && + typed.delete_after_secs == remote.delete_after_secs && //@ts-ignore typed.concurrency_key == remote["concurrency_key"] && typed.debounce_key == remote["debounce_key"] && typed.debounce_delay_s == remote["debounce_delay_s"] && + deepEqual( + typed.debounce_args_to_accumulate ?? null, + remote.debounce_args_to_accumulate ?? null + ) && + typed.max_total_debouncing_time == remote.max_total_debouncing_time && + typed.max_total_debounces_amount == remote.max_total_debounces_amount && typed.codebase == remote.codebase && (hasOnBehalfOf ? true : typed.on_behalf_of_email == remote.on_behalf_of_email) && deepEqual(typed.envs, remote.envs) && + deepEqual(typed.labels ?? null, remote.labels ?? null) && deepEqual(modules ?? null, remote.modules ?? null)) ) { log.info(colors.green(`Script ${remotePath} is up to date`)); @@ -679,6 +776,11 @@ export async function readModulesFromDisk( moduleFolderPath: string, defaultTs: "bun" | "deno" | undefined, folderLayout: boolean = false, + // A dbt project rides in its module folder as-is: `.sql` models (which the + // language inference below rejects as an ambiguous dialect), `.yml` schemas + // and `.csv` seeds are all part of the project and none is a Windmill script. + // Verbatim, or dbt receives a project missing exactly the files it needs. + verbatim: boolean = false, ): Promise | undefined> { if (!fs.existsSync(moduleFolderPath) || !fs.statSync(moduleFolderPath).isDirectory()) { return undefined; @@ -686,9 +788,18 @@ export async function readModulesFromDisk( const modules: Record = {}; + const skipDirs = verbatim + ? dbtGeneratedDirs(moduleFolderPath) + : new Set(); + // In folder layout mode, skip the entry point files (script.*, script.yaml, etc.) const isEntryPointFile = (name: string, isTopLevel: boolean) => { - if (!folderLayout || !isTopLevel) return false; + if (!isTopLevel) return false; + // A dbt project's descriptor is the script's CONTENT, so it must not also + // ride along as a module: the push would send the same text twice and dbt + // would find a stray file at its project root. + if (verbatim) return name === DBT_DESCRIPTOR_NAME; + if (!folderLayout) return false; return ( name.startsWith("script.") || name === "script.lock" || @@ -705,10 +816,63 @@ export async function readModulesFromDisk( const isTopLevel = relPrefix === ""; if (entry.isDirectory()) { + // A configured `target-path` may be nested (`build/target`), so the + // comparison is on the project-relative path, not the entry name. + if (skipDirs.size > 0 && isUnderGeneratedDir(relPath, skipDirs)) continue; readDir(fullPath, relPath); - } else if (entry.isFile() && !entry.name.endsWith(".lock") && !isEntryPointFile(entry.name, isTopLevel)) { - // Skip lock files — they're handled as the `lock` field on ScriptModule - if (exts.some((ext) => entry.name.endsWith(ext))) { + // `.lock` is the script's own lockfile in a `__mod` bundle (the `lock` + // field on ScriptModule) — but a dbt project's files are its author's, + // and one may legitimately be named `uv.lock`. Dropping it would break + // the unmodified-project round trip this bundle exists to keep. + } else if ( + entry.isFile() && + (verbatim || !entry.name.endsWith(".lock")) && + !isEntryPointFile(entry.name, isTopLevel) + ) { + if (verbatim) { + // Secrets stay on the machine that holds them. Skipped before the + // read, and loudly: a `.env` swept into the bundle is a credential + // stored in every version of the script and handed back on pull. + if (isLocalSecretFile(entry.name)) { + log.warn( + `Skipping ${relPath}: a local secrets file is not part of the dbt project — ` + + `dbt reads its values from the environment, so set them in the script's ` + + `environment variables or the descriptor's \`env\``, + ); + continue; + } + // A dbt project's authored files are text. A binary one -- an image + // under `docs/`, a `.DS_Store`, a parquet seed -- would be read as + // mojibake and, if it carries a NUL, rejected by Postgres with an + // opaque `unsupported Unicode escape sequence`, which the push then + // reports as success. Skip it, loudly: dbt does not read it either. + // + // Asked BEFORE reading: the predicate only stats the file and reads + // its first 8 KB, so a multi-gigabyte seed next to the project costs + // that rather than being loaded whole just to be rejected. + const exclusion = moduleFileExclusion(fullPath); + if (exclusion !== undefined) { + // Over the limit but readable as text — a large seed CSV is the + // realistic case — is refused rather than skipped: dbt WOULD have + // read it, so shipping the project without it deploys something that + // compiles here and fails at run time with a missing relation. + if (exclusion === "oversized") { + throw oversizedModuleFileError(relPath, fs.statSync(fullPath).size); + } + log.warn( + `Skipping ${relPath}: not a text file, so it is not part of the dbt project the ` + + `bundle carries — dbt does not read it either`, + ); + continue; + } + // `language` is a required field of the API type and is not used for + // these: the worker writes them to their relative path and dbt reads + // the tree. + modules[relPath] = { + content: fs.readFileSync(fullPath).toString("utf-8"), + language: "dbt" as ScriptModule["language"], + }; + } else if (exts.some((ext) => entry.name.endsWith(ext))) { const content = readTextFileSync(fullPath); const language = inferContentTypeFromFilePath(entry.name, defaultTs); @@ -889,6 +1053,90 @@ async function createScript( */ export class UnresolvableScriptContentFileError extends Error {} +/** + * A path claimed by both a dbt project and an ordinary script. + * + * Its own class because the module push tolerates "no parent found" and must + * NOT tolerate this: swallowed, the command reports success while deploying + * nothing. + */ +export class DbtPathCollisionError extends UnresolvableScriptContentFileError {} + +/** + * The dbt project a path would collide with, if there is one. + * + * `.py` and `__dbt/` deploy to the SAME remote path, so whichever + * is pushed last wins and replaces the other's script. The descriptor is + * optional, so `dbt_project.yml` — not the descriptor — is what says a project + * is there. Asked on BOTH push paths: an ordinary file goes straight to + * `handleFile`, a model reaches its parent through `findContentFile`, and a + * guard on one of them leaves the other silently overwriting. + */ +export async function collidingDbtProject( + basePath: string +): Promise { + const project = basePath + "__dbt/dbt_project.yml"; + return (await stat(project).then(() => true).catch(() => false)) + ? project + : undefined; +} + +/** + * The ordinary script file sharing a base with a dbt project, if there is one — + * the same collision as [`collidingDbtProject`], seen from the dbt side. + * + * Needed because a descriptor may be pushed DIRECTLY (`wmill script push + * __dbt/wm_dbt.yaml`), which never passes through the metadata resolution + * that would otherwise catch it. + */ +export async function collidingOrdinaryScript( + basePath: string +): Promise { + for (const ext of exts) { + if (ext === "__dbt/" + DBT_DESCRIPTOR_NAME) continue; + // Both layouts, because both deploy to `basePath`: the flat file, and the + // folder layout's entry point. + for (const candidate of [ + basePath + ext, + `${basePath}${getModuleFolderSuffix()}/script${ext}`, + ]) { + const isFile = await stat(candidate) + .then((s) => s.isFile()) + .catch(() => false); + if (isFile) return candidate; + } + } + return undefined; +} + +export function dbtPathCollisionError( + project: string, + other: string +): DbtPathCollisionError { + return new DbtPathCollisionError( + `${project} and ${other} deploy to the same path, so pushing either one ` + + `replaces the other's script. Keep one: move the dbt project to a path ` + + `of its own, or remove ${other}.` + ); +} + + +/** + * A script's content, tolerating the one content file that may not exist: a dbt + * project's descriptor is optional, and absent means an empty descriptor. + */ +async function readScriptContent(filePath: string): Promise { + try { + return await readTextFile(filePath); + } catch (e) { + // ONLY a missing file is an empty descriptor. A permission or I/O error on a + // descriptor that does exist would otherwise deploy the defaults — the + // `main` warehouse and the whole project — in place of what the file says. + if (isMissingDbtDescriptor(filePath, e)) return ""; + throw e; + } +} + export async function findContentFile(filePath: string) { // Folder layout: __mod/script.yaml -> __mod/script.ts const isModuleFolderMeta = isModuleEntryMetadata(filePath); @@ -924,6 +1172,20 @@ export async function findContentFile(filePath: string) { ) .filter((x) => x.file) .map((x) => x.path); + // A dbt project's descriptor is OPTIONAL, so `dbt_project.yml` is what says a + // dbt script lives at this path — the descriptor is often absent from the + // candidates above while the project is perfectly real. Asked BEFORE the + // counts below: a project beside an ordinary script is not "one candidate", + // it is two scripts claiming one remote path, and returning the ordinary one + // deploys it OVER the dbt script on the next push of any model. + const dbtCandidate = toCandidate("__dbt/" + DBT_DESCRIPTOR_NAME); + const dbtProject = await collidingDbtProject( + dbtCandidate.slice(0, -("__dbt/" + DBT_DESCRIPTOR_NAME).length), + ); + const nonDbtCandidates = validCandidates.filter((c) => c !== dbtCandidate); + if (dbtProject && nonDbtCandidates.length > 0) { + throw dbtPathCollisionError(dbtProject, nonDbtCandidates.join(", ")); + } if (validCandidates.length > 1) { throw new UnresolvableScriptContentFileError( `Multiple script files found next to ${filePath}: ${validCandidates.join(", ")} — ` + @@ -931,6 +1193,11 @@ export async function findContentFile(filePath: string) { ); } if (validCandidates.length < 1) { + // Resolving to the absent descriptor keeps one content path for every + // caller; reading it yields an empty descriptor. + if (dbtProject) { + return dbtCandidate; + } throw new UnresolvableScriptContentFileError( `No script file found next to ${filePath} — a script cannot be deployed from its metadata alone. ` + `Add the matching script file (e.g. ${toCandidate(".ts")} or ${toCandidate( @@ -999,6 +1266,10 @@ export function filePathExtensionFromContentType( return ".rb"; } else if (language === "rlang") { return ".r"; + } else if (language === "dbt") { + // Not an extension but a path suffix: a dbt script's content file lives + // inside the project folder, so ` + this` is where it belongs. + return "__dbt/" + DBT_DESCRIPTOR_NAME; // for related places search: ADD_NEW_LANG } else { throw new Error("Invalid language: " + language); @@ -1031,12 +1302,28 @@ export const exts = [ ".java", ".rb", ".r", + // Not an extension: a dbt script's content file is its descriptor, inside + // the project folder. `.script.yaml` -> `__dbt/wm_dbt.yaml`. + "__dbt/" + DBT_DESCRIPTOR_NAME, // for related places search: ADD_NEW_LANG ]; +/** + * Whether a path is a script's content file. + * + * Separators are normalized first: one "extension" is the path suffix + * `__dbt/wm_dbt.yaml`, which on Windows is spelled `__dbt\wm_dbt.yaml` and + * would match nothing — silently skipping every dbt project on that platform. + */ +export function hasScriptExt(p: string): boolean { + const norm = p.replaceAll("\\", "/"); + return exts.some((ext) => norm.endsWith(ext)); +} + export function removeExtensionToPath(path: string): string { + const norm = path.replaceAll("\\", "/"); for (const ext of exts) { - if (path.endsWith(ext)) { + if (norm.endsWith(ext)) { return path.substring(0, path.length - ext.length); } } @@ -1450,7 +1737,7 @@ export async function generateMetadata( await FSFSElement(process.cwd(), codebases, false), (p, isD) => { return ( - (!isD && !exts.some((ext) => p.endsWith(ext))) || + (!isD && !hasScriptExt(p)) || ignore(p, isD) || isFlowPath(p) || isAppPath(p) || @@ -1561,17 +1848,28 @@ async function preview( if (opts.silent) { log.setSilent(true); } + // Captured before the config read, which chdirs to the wmill.yaml root. + const cwdBeforeConfig = process.cwd(); opts = await mergeConfigWithConfigFile(opts); const workspace = await resolveWorkspace(opts); await requireLogin(opts); - if (!validatePath(filePath)) { - return; - } + const argPath = filePath; + filePath = toSyncRootRelativePath(filePath, cwdBeforeConfig); + const remotePath = scriptPathToRemotePath(filePath); + assertRemotePath(remotePath, argPath); - const fstat = await stat(filePath); - if (!fstat.isFile()) { - throw new Error("file path must refer to a file."); + // Same as push: a descriptor-less dbt project's content path is deliberately + // absent, and the project beside it is what says the script is real. + const absentDescriptor = await stat(filePath).then( + () => false, + (e) => isMissingDbtDescriptor(filePath, e) + ); + if (!absentDescriptor) { + const fstat = await stat(filePath); + if (!fstat.isFile()) { + throw new Error("file path must refer to a file."); + } } if (filePath.endsWith(".script.json") || filePath.endsWith(".script.yaml")) { @@ -1582,15 +1880,23 @@ async function preview( const codebases = await listSyncCodebases(opts); const language = inferContentTypeFromFilePath(filePath, opts?.defaultTs); - const content = await readTextFile(filePath); + const content = await readScriptContent(filePath); const input = opts.data ? await resolve(opts.data) : {}; - // Read modules from __mod/ folder if present + // Read modules from the bundle folder if present. Same suffix and same + // verbatim read as deploy: a dbt project lives in `__dbt/`, and parsing its + // files as scripts would drop the `dbt_project.yml` the executor looks for. const isFolderLayout = isModuleEntryPoint(filePath); + const isDbt = language === "dbt"; const moduleFolderPath = isFolderLayout ? path.dirname(filePath) - : filePath.substring(0, filePath.indexOf(".")) + getModuleFolderSuffix(); - const modules = await readModulesFromDisk(moduleFolderPath, opts?.defaultTs, isFolderLayout); + : filePath.substring(0, filePath.indexOf(".")) + getModuleFolderSuffix(language); + const modules = await readModulesFromDisk( + moduleFolderPath, + opts?.defaultTs, + isFolderLayout, + isDbt + ); // Check if this is a codebase script const codebase = @@ -1606,11 +1912,7 @@ async function preview( const { extractRelativeImports } = await import( "../../utils/relative_imports.ts" ); - const relImports = await extractRelativeImports( - content, - scriptPathToRemotePath(filePath), - language - ); + const relImports = await extractRelativeImports(content, remotePath, language); if (relImports.length > 0) { const { buildPreviewTempScriptRefs } = await import( "../generate-metadata/generate-metadata.ts" @@ -1713,7 +2015,7 @@ async function preview( const form = new FormData(); const previewPayload = { content: content, // Pass the original content (frontend does this too) - path: filePath.substring(0, filePath.indexOf(".")).replaceAll(SEP, "/"), + path: remotePath, args: input, language: language, tag: opts.tag, @@ -1783,7 +2085,7 @@ async function preview( workspace: workspace.workspaceId, requestBody: { content, - path: filePath.substring(0, filePath.indexOf(".")).replaceAll(SEP, "/"), + path: remotePath, args: input, language: language as any, tag: opts.tag, @@ -1872,6 +2174,10 @@ async function setPermissionedAs( lock: Array.isArray(remote.lock) ? remote.lock.join("\n") : remote.lock ?? undefined, parent_hash: remote.hash, on_behalf_of_email: email, + // The principal is derived server-side from the email, which resolves workspace + // members, groups and superadmins acting outside their workspaces alike — a + // client-side `usr` lookup would see only the first of those. + on_behalf_of: undefined, preserve_on_behalf_of: true, // Preserve any user draft at this path (see backend skip_draft_deletion). skip_draft_deletion: true, diff --git a/cli/src/commands/sync/pull.ts b/cli/src/commands/sync/pull.ts index 90a31f4919..2702e13714 100644 --- a/cli/src/commands/sync/pull.ts +++ b/cli/src/commands/sync/pull.ts @@ -84,7 +84,9 @@ export async function downloadZip( includeSettings?: boolean, includeKey?: boolean, skipWorkspaceDependencies?: boolean, - defaultTs?: "bun" | "deno" + skipDatatableMigrations?: boolean, + defaultTs?: "bun" | "deno", + syncBehavior?: string ): Promise { const requestHeaders = new Headers(); requestHeaders.set("Authorization", "Bearer " + workspace.token); @@ -98,6 +100,9 @@ export async function downloadZip( } const includeWorkspaceDependenciesValue = !(skipWorkspaceDependencies ?? false); + // `sync_behavior_version` lets the server skip work this client would only throw away: + // from v1 the on-behalf-of address is stripped below, so the tarball sends the + // `has_on_behalf_of` marker instead and never resolves an address. // `preserve_extra_perms=true` opts the tarball into surfacing granular ACLs // on flow / script / app rows. Default-off on the server protects cross- // workspace tarball imports from carrying ACLs that reference identities @@ -107,7 +112,7 @@ export async function downloadZip( }&skip_secrets=${skipSecrets ?? false}&include_schedules=${includeSchedules ?? false }&include_triggers=${includeTriggers ?? false}&include_users=${includeUsers ?? false }&include_groups=${includeGroups ?? false}&include_settings=${includeSettings ?? false - }&include_key=${includeKey ?? false}&include_workspace_dependencies=${includeWorkspaceDependenciesValue}&default_ts=${defaultTs ?? "bun"}&skip_resource_types=${skipResourceTypes ?? false}&settings_version=v2&preserve_extra_perms=true`; + }&include_key=${includeKey ?? false}&include_workspace_dependencies=${includeWorkspaceDependenciesValue}&skip_datatable_migrations=${skipDatatableMigrations ?? false}&default_ts=${defaultTs ?? "bun"}&skip_resource_types=${skipResourceTypes ?? false}&settings_version=v2&preserve_extra_perms=true&sync_behavior_version=${syncBehavior ?? "v0"}`; const baseUrl = workspace.remote + "api/w/" + workspace.workspaceId + "/workspaces/tarball?"; diff --git a/cli/src/commands/sync/sync.ts b/cli/src/commands/sync/sync.ts index 3e6dd464fc..b1fca78ed2 100644 --- a/cli/src/commands/sync/sync.ts +++ b/cli/src/commands/sync/sync.ts @@ -1,13 +1,28 @@ import { requireLogin } from "../../core/auth.ts"; +import { markRequestsAsSyncOrigin } from "../../core/client.ts"; import { fetchVersion, resolveWorkspace } from "../../core/context.ts"; -import { writeFile, readdir, stat, rm, copyFile, mkdir } from "node:fs/promises"; +import { + writeFile, + readdir, + stat, + rm, + copyFile, + mkdir, +} from "node:fs/promises"; +import { existsSync, type Dirent } from "node:fs"; import { colors } from "@cliffy/ansi/colors"; import { Command } from "@cliffy/command"; import { Confirm } from "@cliffy/prompt/confirm"; import * as log from "../../core/log.ts"; import * as path from "node:path"; import { sep as SEP } from "node:path"; -import { stringify as yamlStringify, type DocumentOptions, type SchemaOptions, type CreateNodeOptions, type ToStringOptions } from "yaml"; +import { + stringify as yamlStringify, + type DocumentOptions, + type SchemaOptions, + type CreateNodeOptions, + type ToStringOptions, +} from "yaml"; import JSZip from "jszip"; import { minimatch } from "minimatch"; import { yamlParseContent } from "../../utils/yaml.ts"; @@ -39,13 +54,15 @@ import { exts, findContentFile, findResourceFile, + isModuleEntryMetadata, handleScriptMetadata, UnresolvableScriptContentFileError, removeExtensionToPath, filePathExtensionFromContentType, + hasScriptExt, } from "../script/script.ts"; -import { handleFile } from "../script/script.ts"; +import { DbtPathCollisionError, handleFile } from "../script/script.ts"; import { deepEqual, fetchRemoteVersion, @@ -55,6 +72,7 @@ import { isRawAppFile, isWorkspaceDependencies, readTextFile, + removeResourceSuffix, } from "../../utils/utils.ts"; import { getEffectiveSettings, @@ -70,6 +88,7 @@ import type { PermissionedAsContext } from "../../core/permissioned_as.ts"; import { preCheckPermissionedAs } from "../../core/permissioned_as.ts"; import { fromWorkspaceSpecificPath, + toWorkspaceSpecificPath, getWorkspaceSpecificPath, getSpecificItemsForCurrentBranch, isWorkspaceSpecificFile, @@ -86,12 +105,16 @@ import { gitSyncDeployPush, deriveGitSyncDeployIncludes, isForkWorkspace, + gitRecordedDatatableMigrationPaths, type GitSyncDeployItem, + type RecordedMigrationPaths, } from "../../utils/git.ts"; import { Workspace } from "../workspace/workspace.ts"; import { removePathPrefix } from "../../types.ts"; import { listSyncCodebases, SyncCodebase } from "../../utils/codebase.ts"; import { + beginLockfileBatch, + flushLockfileBatch, generateScriptMetadataInternal, getRawWorkspaceDependencies, readLockfile, @@ -99,20 +122,31 @@ import { MalformedLockfileError, workspaceDependenciesPathToLanguageAndFilename, } from "../../utils/metadata.ts"; -import { DoubleLinkedDependencyTree, uploadScripts } from "../../utils/dependency_tree.ts"; -import { OpenFlow, NativeServiceName, ScriptModule } from "../../../gen/types.gen.ts"; -import { pushResource } from "../resource/resource.ts"; +import { + DoubleLinkedDependencyTree, + uploadScripts, +} from "../../utils/dependency_tree.ts"; +import { + OpenFlow, + NativeServiceName, + ScriptModule, +} from "../../../gen/types.gen.ts"; +import { pushResource, validateFilesetPointer } from "../resource/resource.ts"; import { newPathAssigner, newRawAppPathAssigner, PathAssigner, } from "../../../windmill-utils-internal/src/path-utils/path-assigner.ts"; -import { extractInlineScripts as extractInlineScriptsForFlows, extractCurrentMapping } from "../../../windmill-utils-internal/src/inline-scripts/extractor.ts"; +import { + extractInlineScripts as extractInlineScriptsForFlows, + extractCurrentMapping, +} from "../../../windmill-utils-internal/src/inline-scripts/extractor.ts"; import { generateFlowLockInternal } from "../flow/flow_metadata.ts"; import { isExecutionModeAnonymous } from "../app/app.ts"; import { APP_BACKEND_FOLDER, generateAppLocksInternal, + RECORDINGS_FOLDER, } from "../app/app_metadata.ts"; import { isFlowPath, @@ -132,11 +166,146 @@ import { getFolderSuffixWithSep, getNonDottedPaths, isScriptModulePath, + oversizedDbtFileError, getModuleFolderSuffix, + isDbtModulePath, + isDbtGeneratedPath, isModuleEntryPoint, getScriptBasePathFromModulePath, hasWrongFormatSuffix, + DBT_DESCRIPTOR_NAME, + isDbtDescriptorPath, } from "../../utils/resource_folders.ts"; +import { isSharedLockPath, SHARED_LOCK_DIR } from "../../utils/script_common.ts"; +import { + applySharedLockPlanToDisk, + applySharedLockPlanToMap, + metadataLockUnreadable, + sharedLockRefOf, + computeSharedLockPlan, + isEmptySharedLockPlan, + scriptsReferencingSharedLock, + type LockDedupOptions, +} from "../../utils/lock_dedup.ts"; + +/** A lockfile belonging to one script, as opposed to a shared one. */ +function isScriptLockPath(p: string): boolean { + const n = p.replaceAll(SEP, "/"); + return n.endsWith(".script.lock") || n.endsWith("__mod/script.lock"); +} + +/** + * Every shared lockfile the tree still reads, from a single walk. + * + * One pull can retire several at once — `--skip-workspace-dependencies` retires + * all of them, and consolidating k dependency files retires k-1 — so a walk per + * deletion would re-read and re-parse the same metadata each time. Read after + * the pull has applied (or refused) every metadata change, because that is the + * only moment the answer is settled. + */ +export type SharedLockReaders = { + /** Reference (`locks/.lock`) to the metadata files reading it. */ + byRef: Map; + /** Metadata whose `lock` cannot be read, which pins every shared lockfile. */ + unreadable: string[]; +}; + +export async function collectSharedLockReaders( + json: boolean, +): Promise { + const metaExt = json ? ".script.json" : ".script.yaml"; + const modMeta = json ? "__mod/script.json" : "__mod/script.yaml"; + const readers: SharedLockReaders = { byRef: new Map(), unreadable: [] }; + const walk = async (dir: string): Promise => { + let entries: Dirent[]; + try { + entries = await readdir(dir, { withFileTypes: true }); + } catch (e) { + // A directory that is not there holds no reader. Anything else hides + // scripts, and a lockfile deleted out from under one resolves to nothing. + if ((e as { code?: string })?.code === "ENOENT") return; + throw e; + } + for (const entry of entries) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) { + if (isNeverWalkedDir(entry.name)) continue; + await walk(full); + continue; + } + const rel = full.replaceAll(SEP, "/"); + if (!rel.endsWith(metaExt) && !rel.endsWith(modMeta)) continue; + const content = await readTextFile(full); + // The `lock` field, not the raw text: a folded line, a summary quoting + // the path, or a stale twin of the other format would each answer wrongly. + const ref = sharedLockRefOf(rel, content, json); + if (ref === undefined) { + if (metadataLockUnreadable(rel, content, json)) readers.unreadable.push(rel); + continue; + } + const existing = readers.byRef.get(ref); + if (existing) existing.push(rel); + else readers.byRef.set(ref, [rel]); + } + }; + for (const root of ["f", "u", "g"]) { + await walk(root); + } + return readers; +} + +/** + * Whether the metadata beside a script lockfile still points at it. Read from + * disk, after the pull has applied (or refused) every metadata change, because + * that is the only moment the answer is settled - and from the `lock` field of + * the twin this sync reads, not the raw text: a folded line, a summary quoting + * the path, or a stale twin of the other format would each answer wrongly. + * + * Returns the reason it is kept, or undefined when nothing reads it. + */ +async function lockStillReadBecause( + lockPath: string, + json: boolean, + sharedReaders: SharedLockReaders, +): Promise { + const n = lockPath.replaceAll(SEP, "/"); + // A shared lockfile is read by any number of scripts, so the whole tree + // answers rather than one sibling. + if (isSharedLockPath(n)) { + const readers = sharedReaders.byRef.get(n)?.length ?? 0; + if (readers > 0) return `${readers} script(s) on disk still reference it`; + if (sharedReaders.unreadable.length > 0) { + // Naming the file matters: on a dependency-file deletion this line is the + // only signal, and "still references it" would point away from the fix. + return `${sharedReaders.unreadable[0]} cannot be parsed, so what it reads is unknown`; + } + return undefined; + } + const metaPath = n.endsWith("__mod/script.lock") + ? n.slice(0, -".lock".length) + (json ? ".json" : ".yaml") + : n.slice(0, -".script.lock".length) + + (json ? ".script.json" : ".script.yaml"); + let content: string; + try { + content = await readTextFile(metaPath.replaceAll("/", SEP)); + } catch { + return undefined; // no metadata: nothing reads it + } + try { + const parsed = json + ? JSON.parse(content) + : yamlParseContent(metaPath, content); + return parsed?.["lock"] === "!inline " + n + ? `${metaPath} still references it` + : undefined; + } catch { + // Unparseable metadata is not proof that nothing reads the lock. + return `${metaPath} cannot be parsed, so what it reads is unknown`; + } +} + +/** Sync maps are keyed with the platform separator; `!inline` refs are not. */ +const toMapKeySep = (refPath: string) => refPath.replaceAll("/", SEP); let branchDeprecationWarned = false; @@ -154,12 +323,12 @@ function configKeyForItemKind( return "resources"; case "variable": return "variables"; - // case "schedule": - // return "schedules"; - // default: - // return kind.endsWith("_trigger") ? "triggers" : null; - } - return null + // case "schedule": + // return "schedules"; + // default: + // return kind.endsWith("_trigger") ? "triggers" : null; + } + return null; } // Fetch ws_specific items from the server and merge their paths into specificItems. @@ -182,8 +351,11 @@ async function mergeWsSpecificFromServer( // 404 = endpoint not present on an older server: expected, log at debug. // Anything else (401/403/network) is a real failure that produces an // incomplete sync — surface it so the user notices. - const isApiError = err && typeof err === "object" && - "name" in err && (err as { name: unknown }).name === "ApiError"; + const isApiError = + err && + typeof err === "object" && + "name" in err && + (err as { name: unknown }).name === "ApiError"; const status = isApiError ? (err as { status?: number }).status : undefined; if (status === 404) { log.debug("listWsSpecific endpoint not available on server, skipping"); @@ -235,9 +407,7 @@ export function computeWsSpecificFlagOnlyPushes( ): Array<{ kind: string; serverPath: string; filePath: string }> { if (!localSpecificItems || serverItems === null) return []; - const serverSet = new Set( - serverItems.map((i) => `${i.item_kind}:${i.path}`), - ); + const serverSet = new Set(serverItems.map((i) => `${i.item_kind}:${i.path}`)); const out: Array<{ kind: string; serverPath: string; filePath: string }> = []; for (const filePath of Object.keys(localMap)) { @@ -261,7 +431,10 @@ export function computeWsSpecificFlagOnlyPushes( // Resolve workspace name from a --branch override (git branch → workspace name). // Falls back to using the branch value as-is (backward compat: old key = branch name). -function resolveWsNameFromBranch(opts: SyncOptions, branchName: string): string { +function resolveWsNameFromBranch( + opts: SyncOptions, + branchName: string, +): string { const match = findWorkspaceByGitBranch(opts.workspaces, branchName); return match ? match[0] : branchName; } @@ -288,17 +461,23 @@ export function resolveWsNameForConfigFromFlags( } // Warn if --workspace overrides auto-detected branch or if workspace not in config. -function warnWorkspaceOverride(opts: SyncOptions, wsNameForConfig: string | undefined): void { +function warnWorkspaceOverride( + opts: SyncOptions, + wsNameForConfig: string | undefined, +): void { if (!wsNameForConfig || !opts.workspaces) return; // Check if workspace exists in config - const wsEntry = (opts.workspaces as any)?.[wsNameForConfig] as WorkspaceEntryConfig | undefined; + const wsEntry = (opts.workspaces as any)?.[wsNameForConfig] as + WorkspaceEntryConfig | undefined; if (!wsEntry) { - const wsNames = Object.keys(opts.workspaces).filter((k) => k !== "commonSpecificItems"); + const wsNames = Object.keys(opts.workspaces).filter( + (k) => k !== "commonSpecificItems", + ); if (wsNames.length > 0) { log.warn( `⚠️ Workspace '${wsNameForConfig}' is not defined in the 'workspaces' section of wmill.yaml.\n` + - ` No workspace-specific overrides will be applied. Available workspaces: ${wsNames.join(", ")}` + ` No workspace-specific overrides will be applied. Available workspaces: ${wsNames.join(", ")}`, ); } return; @@ -308,11 +487,14 @@ function warnWorkspaceOverride(opts: SyncOptions, wsNameForConfig: string | unde if (isGitRepository()) { const currentBranch = getCurrentGitBranch(); if (currentBranch) { - const autoMatch = findWorkspaceByGitBranch(opts.workspaces, currentBranch); + const autoMatch = findWorkspaceByGitBranch( + opts.workspaces, + currentBranch, + ); if (autoMatch && autoMatch[0] !== wsNameForConfig) { log.info( `Current git branch '${currentBranch}' maps to workspace '${autoMatch[0]}', ` + - `but --workspace overrides to '${wsNameForConfig}'.` + `but --workspace overrides to '${wsNameForConfig}'.`, ); } } @@ -327,9 +509,14 @@ function resolveWsNameForFiles(_opts: SyncOptions, wsName: string): string { // After resolveWorkspace, infer the workspace config name from the resolved profile // by matching baseUrl + workspaceId against the workspaces config entries. -function inferWsNameFromProfile(opts: SyncOptions, profile: { remote: string; workspaceId: string }): string | undefined { +function inferWsNameFromProfile( + opts: SyncOptions, + profile: { remote: string; workspaceId: string }, +): string | undefined { if (!opts.workspaces) return undefined; - const wsNames = Object.keys(opts.workspaces).filter((k) => k !== "commonSpecificItems"); + const wsNames = Object.keys(opts.workspaces).filter( + (k) => k !== "commonSpecificItems", + ); for (const name of wsNames) { const entry = (opts.workspaces as any)[name] as WorkspaceEntryConfig; if (!entry?.baseUrl) continue; @@ -362,7 +549,13 @@ async function resolveEffectiveSyncOptions( promotion?: string, workspaceNameOverride?: string, ): Promise { - return await getEffectiveSettings(localConfig, promotion, false, false, workspaceNameOverride); + return await getEffectiveSettings( + localConfig, + promotion, + false, + false, + workspaceNameOverride, + ); } type DynFSElement = { @@ -462,6 +655,27 @@ async function addCodebaseDigestIfRelevant( return content; } +/** + * Whether a script's modules ARE a dbt project. + * + * Keyed on `dbt_project.yml` rather than on the descriptor: the descriptor is + * optional, so its absence says nothing, while a dbt project without + * `dbt_project.yml` is one dbt itself refuses to run. + * + * Its LANGUAGE decides, not its name. A dbt bundle is read verbatim and every + * file in it is stored as `dbt`; an ordinary modular script that happens to + * vendor a dbt project stores that same file as whatever its extension infers, + * and calling it dbt would lay the bundle out as `__dbt` and drop it on the + * next push. + */ +function isDbtModules(modules: unknown): boolean { + if (typeof modules !== "object" || modules === null) return false; + const marker = (modules as Record)[ + "dbt_project.yml" + ]; + return marker?.language === "dbt"; +} + export async function FSFSElement( p: string, codebases: SyncCodebase[], @@ -491,8 +705,14 @@ export async function FSFSElement( } }, async getContentText(): Promise { - const content = await readTextFile(localP); const itemPath = localP.substring(p.length + 1); + // BEFORE the read: an oversized dbt project file stays visible to the + // diff on purpose (so the push reports it rather than silently shipping + // an incomplete project), and buffering a multi-gigabyte seed to reach + // that error is what this refusal exists to avoid. + const oversized = oversizedDbtFileError(localP, itemPath); + if (oversized) throw oversized; + const content = await readTextFile(localP); const r = await addCodebaseDigestIfRelevant( itemPath, content, @@ -525,9 +745,14 @@ function prioritizeName(name: string): string { return name; } -export const yamlOptions: DocumentOptions & SchemaOptions & CreateNodeOptions & ToStringOptions = { +export const yamlOptions: DocumentOptions & + SchemaOptions & + CreateNodeOptions & + ToStringOptions = { sortMapEntries: (a, b) => { - return prioritizeName(String(a.key)).localeCompare(prioritizeName(String(b.key))); + return prioritizeName(String(a.key)).localeCompare( + prioritizeName(String(b.key)), + ); }, aliasDuplicateObjects: false, singleQuote: true, @@ -591,11 +816,15 @@ export function extractFieldsForRawApps(runnables: Record) { * References the raw-app skill for complete documentation and includes instance-specific * data configuration (datatable, schema, whitelisted tables). */ -export function generateAgentsDocumentation(data: { - tables?: string[]; - datatable?: string; - schema?: string; -} | undefined): string { +export function generateAgentsDocumentation( + data: + | { + tables?: string[]; + datatable?: string; + schema?: string; + } + | undefined, +): string { const tables = data?.tables ?? []; const defaultDatatable = data?.datatable; const defaultSchema = data?.schema; @@ -610,15 +839,19 @@ This file contains **app-specific configuration** for this raw app instance. ## Data Configuration -${defaultDatatable - ? `**Default Datatable:** \`${defaultDatatable}\`${defaultSchema ? ` | **Default Schema:** \`${defaultSchema}\`` : ''}` - : '**No default datatable configured.** Set \`data.datatable\` in \`raw_app.yaml\` to enable database access.'} +${ + defaultDatatable + ? `**Default Datatable:** \`${defaultDatatable}\`${defaultSchema ? ` | **Default Schema:** \`${defaultSchema}\`` : ""}` + : "**No default datatable configured.** Set \`data.datatable\` in \`raw_app.yaml\` to enable database access." +} ### Whitelisted Tables -${tables.length > 0 - ? `These tables are accessible to this app:\n\n${tables.map(t => `- \`${t}\``).join('\n')}` - : `**No tables whitelisted.** Add tables to \`data.tables\` in \`raw_app.yaml\`.`} +${ + tables.length > 0 + ? `These tables are accessible to this app:\n\n${tables.map((t) => `- \`${t}\``).join("\n")}` + : `**No tables whitelisted.** Add tables to \`data.tables\` in \`raw_app.yaml\`.` +} ### Adding a Table @@ -626,10 +859,10 @@ Edit \`raw_app.yaml\`: \`\`\`yaml data: - datatable: ${defaultDatatable || 'main'} - ${defaultSchema ? `schema: ${defaultSchema}\n ` : ''}tables: -${tables.length > 0 ? tables.map(t => ` - ${t}`).join('\n') : ' # Add tables here'} - - ${defaultDatatable || 'main'}/${defaultSchema ? defaultSchema + ':' : ''}new_table # ← Add like this + datatable: ${defaultDatatable || "main"} + ${defaultSchema ? `schema: ${defaultSchema}\n ` : ""}tables: +${tables.length > 0 ? tables.map((t) => ` - ${t}`).join("\n") : " # Add tables here"} + - ${defaultDatatable || "main"}/${defaultSchema ? defaultSchema + ":" : ""}new_table # ← Add like this \`\`\` **Table reference formats:** @@ -665,11 +898,15 @@ const rows = await sql\`SELECT * FROM table WHERE id = \${id}\`.fetch(); * Generates a simple DATATABLES.md with just the current configuration summary. * The detailed schema information is generated by generate_datatables.ts command. */ -export function generateDatatablesDocumentation(data: { - tables?: string[]; - datatable?: string; - schema?: string; -} | undefined): string { +export function generateDatatablesDocumentation( + data: + | { + tables?: string[]; + datatable?: string; + schema?: string; + } + | undefined, +): string { const tables = data?.tables ?? []; const defaultDatatable = data?.datatable; const defaultSchema = data?.schema; @@ -683,15 +920,19 @@ Run \`wmill app generate-agents\` to refresh with current workspace schemas. ## Current Configuration -${defaultDatatable - ? `**Default Datatable:** \`${defaultDatatable}\`${defaultSchema ? ` | **Default Schema:** \`${defaultSchema}\`` : ''}` - : '**No default datatable configured.**'} +${ + defaultDatatable + ? `**Default Datatable:** \`${defaultDatatable}\`${defaultSchema ? ` | **Default Schema:** \`${defaultSchema}\`` : ""}` + : "**No default datatable configured.**" +} ## Whitelisted Tables -${tables.length > 0 - ? `${tables.map(t => `- \`${t}\``).join('\n')}` - : `*No tables whitelisted. Add tables to \`data.tables\` in \`raw_app.yaml\`.*`} +${ + tables.length > 0 + ? `${tables.map((t) => `- \`${t}\``).join("\n")}` + : `*No tables whitelisted. Add tables to \`data.tables\` in \`raw_app.yaml\`.*` +} --- @@ -761,11 +1002,17 @@ export function extractInlineScriptsForApps( return []; } -type FileResourceTypeInfo = { format_extension: string | null; is_fileset: boolean }; +type FileResourceTypeInfo = { + format_extension: string | null; + is_fileset: boolean; +}; function parseFileResourceTypeMap( raw: Record, -): { formatExtMap: Record; filesetMap: Record } { +): { + formatExtMap: Record; + filesetMap: Record; +} { const formatExtMap: Record = {}; const filesetMap: Record = {}; for (const [k, v] of Object.entries(raw)) { @@ -782,7 +1029,10 @@ function parseFileResourceTypeMap( return { formatExtMap, filesetMap }; } -async function findFilesetResourceFile(changePath: string): Promise { +export async function findFilesetResourceFile( + changePath: string, + wsName?: string | null, +): Promise { // Extract the base path before .fileset/ const filesetIdx = changePath.indexOf(".fileset" + SEP); if (filesetIdx === -1) { @@ -790,6 +1040,16 @@ async function findFilesetResourceFile(changePath: string): Promise { } const basePath = changePath.substring(0, filesetIdx); const candidates = [basePath + ".resource.json", basePath + ".resource.yaml"]; + // A workspace-specific resource keeps its children at the server-canonical + // `.fileset/` while its metadata file carries the workspace suffix. + // The suffixed file is this workspace's authoritative metadata, so it must + // win over a base file that coexists with it. + if (wsName) { + candidates.unshift( + toWorkspaceSpecificPath(basePath + ".resource.json", wsName), + toWorkspaceSpecificPath(basePath + ".resource.yaml", wsName), + ); + } for (const candidate of candidates) { try { @@ -799,7 +1059,9 @@ async function findFilesetResourceFile(changePath: string): Promise { // not found, try next } } - throw new Error(`No resource metadata file found for fileset resource: ${changePath}`); + throw new Error( + `No resource metadata file found for fileset resource: ${changePath}`, + ); } type FilesetPushResult = @@ -816,7 +1078,7 @@ async function pushFilesetParentResource( ): Promise { let resourceFilePath: string; try { - resourceFilePath = await findFilesetResourceFile(childPath); + resourceFilePath = await findFilesetResourceFile(childPath, cachedWsName); } catch { return { status: "parent-missing" }; } @@ -846,10 +1108,36 @@ async function pushFilesetParentResource( newObj, resourceFilePath, wsSpecific ? true : undefined, + true, ); return { status: "pushed", resourceFilePath }; } +/** + * Join a raw app's author-controlled key (`value.files` path, `value.runnables` + * id) under `baseFolder` and refuse anything that resolves outside it. Keys are + * remote data written to disk on pull, so a `..` segment must not walk a written + * file out of the app's own folder. + */ +export function rawAppPathWithinFolder( + baseFolder: string, + relPath: string, +): string { + const resolved = path.join(baseFolder, relPath); + const rel = path.relative(baseFolder, resolved); + if ( + rel === "" || + rel === ".." || + rel.startsWith(".." + path.sep) || + path.isAbsolute(rel) + ) { + throw new Error( + `raw app path ${JSON.stringify(relPath)} escapes the app folder ${baseFolder}`, + ); + } + return resolved; +} + function ZipFSElement( zip: JSZip, useYaml: boolean, @@ -871,9 +1159,13 @@ function ZipFSElement( const content = await zip.files[filename].async("text"); const parsed = JSON.parse(content); if (parsed.modules && Object.keys(parsed.modules).length > 0) { - _moduleScriptPaths.add( - filename.slice(0, -".script.json".length) - ); + const base = filename.slice(0, -".script.json".length); + // A dbt script's modules ARE its dbt project, so it keeps the flat + // layout: only the project goes in the folder, which is what + // `--project-dir` expects and what makes the import a plain copy. + if (!isDbtModules(parsed.modules)) { + _moduleScriptPaths.add(base); + } } } catch {} } @@ -926,7 +1218,7 @@ function ZipFSElement( let finalPath = transformPath(); // Redirect content files for scripts with modules into __mod/ folder - if (kind == "other" && exts.some((ext) => p.endsWith(ext))) { + if (kind == "other" && hasScriptExt(p)) { const normalizedP = p.replace(/^\.[\\/]/, ""); const moduleScripts = await getModuleScriptPaths(); for (const basePath of moduleScripts) { @@ -934,7 +1226,11 @@ function ZipFSElement( const ext = normalizedP.slice(basePath.length); // e.g., ".ts", ".py" const dir = path.dirname(finalPath); const base = path.basename(basePath); - finalPath = path.join(dir, base + getModuleFolderSuffix(), "script" + ext); + finalPath = path.join( + dir, + base + getModuleFolderSuffix(), + "script" + ext, + ); break; } } @@ -955,7 +1251,9 @@ function ZipFSElement( } let inlineScripts; try { - const assigner = newPathAssigner(defaultTs, { skipInlineScriptSuffix: getNonDottedPaths() }); + const assigner = newPathAssigner(defaultTs, { + skipInlineScriptSuffix: getNonDottedPaths(), + }); // Preserve original !inline filenames from the flow to avoid phantom renames const inlineMapping = extractCurrentMapping( flow.value.modules as any, @@ -969,27 +1267,40 @@ function ZipFSElement( SEP, defaultTs, assigner, - { skipInlineScriptSuffix: getNonDottedPaths(), failOnInlineDirective: true }, + { + skipInlineScriptSuffix: getNonDottedPaths(), + failOnInlineDirective: true, + }, ); if (flow.value.failure_module) { - inlineScripts.push(...extractInlineScriptsForFlows( - [flow.value.failure_module], - inlineMapping, - SEP, - defaultTs, - assigner, - { skipInlineScriptSuffix: getNonDottedPaths(), failOnInlineDirective: true }, - )); + inlineScripts.push( + ...extractInlineScriptsForFlows( + [flow.value.failure_module], + inlineMapping, + SEP, + defaultTs, + assigner, + { + skipInlineScriptSuffix: getNonDottedPaths(), + failOnInlineDirective: true, + }, + ), + ); } if (flow.value.preprocessor_module) { - inlineScripts.push(...extractInlineScriptsForFlows( - [flow.value.preprocessor_module], - inlineMapping, - SEP, - defaultTs, - assigner, - { skipInlineScriptSuffix: getNonDottedPaths(), failOnInlineDirective: true }, - )); + inlineScripts.push( + ...extractInlineScriptsForFlows( + [flow.value.preprocessor_module], + inlineMapping, + SEP, + defaultTs, + assigner, + { + skipInlineScriptSuffix: getNonDottedPaths(), + failOnInlineDirective: true, + }, + ), + ); } } catch (error) { log.error( @@ -1038,7 +1349,9 @@ function ZipFSElement( inlineScripts = extractInlineScriptsForApps( undefined, app?.["value"], - newPathAssigner(defaultTs, { skipInlineScriptSuffix: getNonDottedPaths() }), + newPathAssigner(defaultTs, { + skipInlineScriptSuffix: getNonDottedPaths(), + }), (_, val) => val["name"], false, ); @@ -1120,11 +1433,19 @@ function ZipFSElement( ) { continue; } + // Strip only a leading `/` (keys are app-root-relative), so the + // relative path handed to the guard matches what the backend's + // `strip_prefix('/')` validates — the two must not disagree on a + // non-`/` key, or a deploy the backend allows would abort the pull. + const filePathInApp = rawAppPathWithinFolder( + finalPath, + filePath.replace(/^\//, ""), + ); yield { isDirectory: false, - path: path.join(finalPath, filePath.substring(1)), + path: filePathInApp, async *getChildren() {}, - async getContentText() { + async getContentText() { if (typeof content !== "string") { throw new Error( `Content of raw app file ${filePath} is not a string`, @@ -1213,14 +1534,17 @@ function ZipFSElement( // Simplify fields for cleaner YAML output if (simplifiedRunnable.fields) { - simplifiedRunnable.fields = simplifyFields(simplifiedRunnable.fields); + simplifiedRunnable.fields = simplifyFields( + simplifiedRunnable.fields, + ); } yield { isDirectory: false, - path: path.join( - finalPath, - APP_BACKEND_FOLDER, + // The runnable id is app-author-controlled and names its file, so + // keep it inside the backend folder the same way `files` keys are. + path: rawAppPathWithinFolder( + path.join(finalPath, APP_BACKEND_FOLDER), `${runnableId}.yaml`, ), async *getChildren() {}, @@ -1272,17 +1596,28 @@ function ZipFSElement( log.error(`Failed to parse script.yaml at path: ${p}`); throw error; } - const hasModules = parsed["modules"] && Object.keys(parsed["modules"]).length > 0; + const hasModules = + parsed["modules"] && Object.keys(parsed["modules"]).length > 0; + // A dbt script's module folder holds its dbt project and dbt's own + // files, so its lock stays outside like a plain script's — only the + // descriptor lives in there. + const isDbtScript = isDbtModules(parsed["modules"]); if ( parsed["lock"] && parsed["lock"] != "" && parsed["codebase"] == undefined ) { - if (hasModules) { + if (hasModules && !isDbtScript) { // Lock lives inside __mod/ folder as script.lock - const scriptBase = removeSuffix(removeSuffix(p.replaceAll(SEP, "/"), ".json"), ".script"); + const scriptBase = removeSuffix( + removeSuffix(p.replaceAll(SEP, "/"), ".json"), + ".script", + ); parsed["lock"] = - "!inline " + scriptBase + getModuleFolderSuffix() + "/script.lock"; + "!inline " + + scriptBase + + getModuleFolderSuffix() + + "/script.lock"; } else { parsed["lock"] = "!inline " + @@ -1322,8 +1657,7 @@ function ZipFSElement( throw error; } const resourceType = parsed["resource_type"]; - const formatExtension = - resourceTypeToFormatExtension[resourceType]; + const formatExtension = resourceTypeToFormatExtension[resourceType]; const isFileset = resourceTypeToIsFileset[resourceType] ?? false; if (isFileset) { @@ -1388,18 +1722,24 @@ function ZipFSElement( throw error; } const lock = parsed["lock"]; - const scriptModules: Record | undefined = parsed["modules"]; + const scriptModules: Record | undefined = + parsed["modules"]; const hasModules = scriptModules && Object.keys(scriptModules).length > 0; + // A dbt script's module folder is its dbt project, so the metadata and + // lock stay beside it — the descriptor is the one Windmill file that goes + // in, because it is the script's content. + const isDbt = isDbtModules(scriptModules); // Compute base path and module folder const metaExt = useYaml ? ".yaml" : ".json"; const scriptBasePath = removeSuffix( removeSuffix(finalPath, metaExt), - ".script" + ".script", ); - const moduleFolderPath = scriptBasePath + getModuleFolderSuffix(); + const moduleFolderPath = + scriptBasePath + getModuleFolderSuffix(isDbt ? "dbt" : undefined); - if (hasModules) { + if (hasModules && !isDbt) { // Redirect metadata into __mod/script.yaml r[0].path = path.join(moduleFolderPath, "script" + metaExt); } @@ -1407,9 +1747,10 @@ function ZipFSElement( if (lock && lock != "") { r.push({ isDirectory: false, - path: hasModules - ? path.join(moduleFolderPath, "script.lock") - : removeSuffix(finalPath, metaExt) + ".lock", + path: + hasModules && !isDbt + ? path.join(moduleFolderPath, "script.lock") + : removeSuffix(finalPath, metaExt) + ".lock", async *getChildren() {}, async getContentText() { return lock; @@ -1436,7 +1777,7 @@ function ZipFSElement( // Yield the module lock file if present if (mod.lock) { - const baseName = relPath.replace(/\.[^.]+$/, ''); + const baseName = relPath.replace(/\.[^.]+$/, ""); yield { isDirectory: false, path: path.join(moduleFolderPath, baseName + ".lock"), @@ -1464,11 +1805,14 @@ function ZipFSElement( throw error; } const resourceType = parsed["resource_type"]; - const formatExtension = - resourceTypeToFormatExtension[resourceType]; + const formatExtension = resourceTypeToFormatExtension[resourceType]; const isFileset = resourceTypeToIsFileset[resourceType] ?? false; - if (isFileset && typeof parsed["value"] === "object" && parsed["value"] !== null) { + if ( + isFileset && + typeof parsed["value"] === "object" && + parsed["value"] !== null + ) { const filesetBasePath = removeSuffix(finalPath, ".resource.json") + ".fileset"; // Push directory entry for the fileset @@ -1476,7 +1820,9 @@ function ZipFSElement( isDirectory: true, path: filesetBasePath, async *getChildren() { - for (const [relPath, fileContent] of Object.entries(parsed["value"])) { + for (const [relPath, fileContent] of Object.entries( + parsed["value"], + )) { if (typeof fileContent === "string") { yield { isDirectory: false, @@ -1539,6 +1885,18 @@ function ZipFSElement( return _internal_folder("." + SEP, zip); } +/** + * Directories no walk over a workspace ever descends, whatever the sync scope: + * dependency trees, and the dot-directories that hold tooling state and + * fixtures. Exported because a second walk that disagrees with this one reads + * files sync will never see, and draws conclusions from them. + */ +export function isNeverWalkedDir(dirName: string | undefined): boolean { + return ( + dirName === "node_modules" || (dirName !== undefined && dirName.startsWith(".")) + ); +} + export async function* readDirRecursiveWithIgnore( ignore: (path: string, isDirectory: boolean) => boolean, root: DynFSElement, @@ -1575,15 +1933,8 @@ export async function* readDirRecursiveWithIgnore( const e = stack.pop()!; yield e; for await (const e2 of e.c()) { - if (e2.isDirectory) { - const dirName = e2.path.split(SEP).pop(); - if ( - dirName == "node_modules" || - dirName == ".claude" || - dirName?.startsWith(".") - ) { - continue; - } + if (e2.isDirectory && isNeverWalkedDir(e2.path.split(SEP).pop())) { + continue; } stack.push({ path: e2.path, @@ -1652,9 +2003,15 @@ export async function elementsToMap( } const path = entry.path; // Include module files in the map so they're compared for changes, - // but they're pushed as part of their parent script via handleFile + // but they're pushed as part of their parent script via handleFile. + // `--skip-scripts` therefore covers them, and has to be applied here: the + // filters below are past this shortcut, so a changed module would push the + // parent script the flag asked to leave alone — every file of a dbt project + // is one of these. if (isScriptModulePath(path)) { - map[path] = await entry.getContentText(); + if (!skips.skipScripts) { + map[path] = await entry.getContentText(); + } continue; } if ( @@ -1663,6 +2020,10 @@ export async function elementsToMap( !isRawAppFile(path) && !isWorkspaceDependencies(path) ) { + // The metadata format decides which of the two metadata twins is read, + // and drops the other. A dbt descriptor is not metadata and is not + // reached here: it lives inside the project folder, so the module branch + // above already took it, in both modes. if (json && path.endsWith(".yaml")) continue; if (!json && path.endsWith(".json")) continue; @@ -1695,9 +2056,16 @@ export async function elementsToMap( } if (isRawAppFile(path)) { - const suffix = path.split(getFolderSuffix("raw_app") + SEP).pop(); + // FSFSElement builds paths with the platform separator, while the checks + // below are written with "/": without normalizing, none of them match on + // Windows and the push collector's own exclusions become perpetual diffs. + const suffix = path + .split(getFolderSuffix("raw_app") + SEP) + .pop() + ?.replaceAll(SEP, "/"); if ( suffix?.startsWith("dist/") || + suffix?.startsWith(RECORDINGS_FOLDER + "/") || suffix == "wmill.d.ts" || suffix == "package-lock.json" || suffix == "DATATABLES.md" @@ -1706,7 +2074,11 @@ export async function elementsToMap( } } - if (skips.skipResources && (isFileResource(path) || isFilesetResource(path))) continue; + if ( + skips.skipResources && + (isFileResource(path) || isFilesetResource(path)) + ) + continue; const ext = json ? ".json" : ".yaml"; if (!skips.includeSchedules && path.endsWith(".schedule" + ext)) continue; @@ -1740,7 +2112,13 @@ export async function elementsToMap( try { const fileType = getTypeStrFromPath(path); if (skips.skipVariables && fileType === "variable") continue; - if (skips.skipScripts && fileType === "script") continue; + // A shared lockfile is part of the scripts that reference it. + if ( + skips.skipScripts && + (fileType === "script" || fileType === "shared_lock") + ) { + continue; + } if (skips.skipFlows && fileType === "flow") continue; if (skips.skipApps && fileType === "app") continue; if (skips.skipFolders && fileType === "folder") continue; @@ -1749,6 +2127,8 @@ export async function elementsToMap( fileType === "workspace_dependencies" ) continue; + if (skips.skipDatatableMigrations && fileType === "datatable_migration") + continue; } catch { // If getTypeStrFromPath can't determine the type, continue processing the file } @@ -1827,17 +2207,38 @@ export async function elementsToMap( if (wrongFormatPaths.length > 0) { const isNonDotted = getNonDottedPaths(); - const foundFormat = isNonDotted ? ".flow/.app/.raw_app" : "__flow/__app/__raw_app"; - const expectedFormat = isNonDotted ? "__flow/__app/__raw_app" : ".flow/.app/.raw_app"; + const foundFormat = isNonDotted + ? ".flow/.app/.raw_app" + : "__flow/__app/__raw_app"; + const expectedFormat = isNonDotted + ? "__flow/__app/__raw_app" + : ".flow/.app/.raw_app"; const configHint = isNonDotted ? "Either remove 'nonDottedPaths: true' from wmill.yaml, or rename these directories to use __flow/__app/__raw_app format." : "Either add 'nonDottedPaths: true' to wmill.yaml, or rename these directories to use .flow/.app/.raw_app format."; const pathList = wrongFormatPaths.map((p) => ` ${p}`).join("\n"); throw new Error( - `Found ${wrongFormatPaths.length} directory(ies) using ${foundFormat} format, but wmill.yaml expects ${expectedFormat}:\n${pathList}\n${configHint}` + `Found ${wrongFormatPaths.length} directory(ies) using ${foundFormat} format, but wmill.yaml expects ${expectedFormat}:\n${pathList}\n${configHint}`, ); } + // A dbt project's descriptor is optional, and the two sides spell "absent" + // differently: nothing on disk, and nothing in the export (which omits an + // empty one so a project that never named a descriptor never grows one). + // Left alone that reads as an addition on every push and a deletion on every + // pull, forever. Both sides are given the empty descriptor the absence means, + // so a descriptor-less project reaches a clean sync state. + for (const key of Object.keys(map)) { + // Normalized first: the local map's keys are built with `path.join`, so on + // Windows this reads `__dbt\\dbt_project.yml` and an unnormalized match + // would synthesize nothing — leaving exactly the perpetual push/pull diff + // above unguarded, on that platform only. + if (!key.replaceAll("\\", "/").endsWith("__dbt/dbt_project.yml")) continue; + const descriptor = + key.slice(0, -"dbt_project.yml".length) + DBT_DESCRIPTOR_NAME; + if (!(descriptor in map)) map[descriptor] = ""; + } + return map; } @@ -1851,6 +2252,7 @@ export interface Skips { skipApps?: boolean | undefined; skipFolders?: boolean | undefined; skipWorkspaceDependencies?: boolean | undefined; + skipDatatableMigrations?: boolean | undefined; skipScriptsMetadata?: boolean | undefined; includeSchedules?: boolean | undefined; includeTriggers?: boolean | undefined; @@ -1967,7 +2369,11 @@ export function canonicalizeCaseInsensitiveKeys( const lk = seg.toLowerCase(); let entry = node.children.get(lk); if (!entry) { - entry = { canonical: seg, ambiguous: false, node: { children: new Map() } }; + entry = { + canonical: seg, + ambiguous: false, + node: { children: new Map() }, + }; node.children.set(lk, entry); } else if (entry.canonical !== seg) { entry.ambiguous = true; @@ -2141,7 +2547,9 @@ export function preservePendingScriptLocks( remoteParsed = isYaml ? yamlParseContent(metaKey, remote[metaKey]) : JSON.parse(remote[metaKey]); - localParsed = isYaml ? yamlParseContent(metaKey, localMeta) : JSON.parse(localMeta); + localParsed = isYaml + ? yamlParseContent(metaKey, localMeta) + : JSON.parse(localMeta); } catch { continue; } @@ -2155,7 +2563,8 @@ export function preservePendingScriptLocks( // The local side must reference an inline lock backed by a committed file. const localLock = localParsed["lock"]; - if (typeof localLock !== "string" || !localLock.startsWith("!inline ")) continue; + if (typeof localLock !== "string" || !localLock.startsWith("!inline ")) + continue; // Derive the lock-file key from the `!inline` reference itself, not from the // metadata path: a multi-module script keeps its lock at `…__mod/script.lock`, @@ -2177,7 +2586,7 @@ async function compareDynFSElement( els2: DynFSElement | undefined, ignore: (path: string, isDirectory: boolean) => boolean, json: boolean, - skips: Skips, + skips: Skips & LockDedupOptions, ignoreMetadataDeletion: boolean, codebases: SyncCodebase[], ignoreCodebaseChanges: boolean, @@ -2188,10 +2597,37 @@ async function compareDynFSElement( ): Promise<{ changes: Change[]; localMap: Record }> { let [m1, m2] = els2 ? await Promise.all([ - elementsToMap(els1, ignore, json, skips, specificItems, branchOverride, isEls1Remote), - elementsToMap(els2, ignore, json, skips, specificItems, branchOverride, !isEls1Remote), + elementsToMap( + els1, + ignore, + json, + skips, + specificItems, + branchOverride, + isEls1Remote, + ), + elementsToMap( + els2, + ignore, + json, + skips, + specificItems, + branchOverride, + !isEls1Remote, + ), ]) - : [await elementsToMap(els1, ignore, json, skips, specificItems, branchOverride, isEls1Remote), {}]; + : [ + await elementsToMap( + els1, + ignore, + json, + skips, + specificItems, + branchOverride, + isEls1Remote, + ), + {}, + ]; // Reconcile letter-case differences between the local tree and the // authoritative server casing. Only meaningful for an actual two-sided diff @@ -2239,6 +2675,36 @@ async function compareDynFSElement( preservePendingScriptLocks(m1, m2); } + // The remote serializes one lock per script; `dedupeLockfiles` is how the repo + // represents them. Collapsing the remote side (in both directions) is what + // makes the two sides comparable: a pull then writes the shared file instead + // of thousands of copies, and a push sees no diff for the copies it does not + // keep. + if (skips.dedupeLockfiles) { + const remoteMap = isEls1Remote === true ? m1 : m2; + const localMapForLocks = isEls1Remote === true ? m2 : m1; + // The local side supplies what the remote never serializes: the shared + // lockfiles already on disk, so one whose scripts are out of this sync's + // scope is carried forward rather than read as a deletion. + const present: Record = {}; + for (const [key, content] of Object.entries(localMapForLocks)) { + if (isSharedLockPath(key)) present[key.replaceAll(SEP, "/")] = content; + } + applySharedLockPlanToMap( + remoteMap, + computeSharedLockPlan(remoteMap, { + defaultTs: skips.defaultTs, + present, + // Only when the map cannot speak for them: with dependency files in the + // map, its absences are real deletions, and reading disk here would keep + // a lockfile alive one sync past the file it is named after. + depFiles: skips.skipWorkspaceDependencies + ? Object.keys(await getRawWorkspaceDependencies(false)) + : undefined, + }), + ); + } + const changes: Change[] = []; function parseYaml(k: string, v: string) { @@ -2502,13 +2968,16 @@ const isNotWmillFile = (p: string, isDirectory: boolean) => { !p.startsWith("users" + SEP) && !p.startsWith("groups" + SEP) && !p.startsWith("dependencies" + SEP) && + !p.startsWith(SHARED_LOCK_DIR + SEP) && !p.startsWith("migrations" + SEP) ); } - // Files inside __mod/ folders are script module files — always valid wmill files + // Files inside a module folder belong to their parent script, so they are + // always valid wmill files — except the ones dbt generates, which are not + // part of the bundle and must not surface as items of their own. if (isScriptModulePath(p)) { - return false; + return isDbtGeneratedPath(p); } try { @@ -2524,6 +2993,8 @@ const isNotWmillFile = (p: string, isDirectory: boolean) => { typ == "encryption_key" ) { return p.includes(SEP); + } else if (typ == "shared_lock") { + return false; } else { return ( !p.startsWith("u" + SEP) && @@ -2550,6 +3021,7 @@ export const isWhitelisted = (p: string) => { p == "users" || p == "groups" || p == "dependencies" || + p == SHARED_LOCK_DIR || p == "migrations" ); }; @@ -2558,8 +3030,10 @@ export async function ignoreF(wmillconf: { includes?: string[]; excludes?: string[]; extraIncludes?: string[]; + dedupeLockfiles?: boolean; skipResourceTypes?: boolean; skipWorkspaceDependencies?: boolean; + skipDatatableMigrations?: boolean; json?: boolean; includeUsers?: boolean; includeGroups?: boolean; @@ -2597,6 +3071,15 @@ export async function ignoreF(wmillconf: { // new Gitignore.default({ initialRules: ignoreContent.split("\n")}).ignoreContent).compile(); return (p: string, isDirectory: boolean) => { + // Without the option, `locks/` is not Windmill's: a repo that keeps its own + // lockfiles there would otherwise see them pulled into the diff and deleted + // as absent from a remote that never serializes shared locks. + if ( + !wmillconf.dedupeLockfiles && + (p === SHARED_LOCK_DIR || p.startsWith(SHARED_LOCK_DIR + SEP)) + ) { + return true; + } const ext = wmillconf.json ? ".json" : ".yaml"; if (!isDirectory && p.endsWith(".resource-type" + ext)) { return wmillconf.skipResourceTypes ?? false; @@ -2624,6 +3107,20 @@ export async function ignoreF(wmillconf: { ) { return false; // Don't ignore workspace dependencies (they are always included unless explicitly skipped) } + // A shared lockfile lives outside the u/f/g namespaces the include + // patterns are written against, and dropping it from the diff would + // leave every script that references it pointing at nothing. + if (fileType === "shared_lock") { + return false; + } + // `migrations/datatable/**` is outside the u/f/g namespaces the path + // filters are written against, so the skip flag is its only control. + if ( + !wmillconf.skipDatatableMigrations && + fileType === "datatable_migration" + ) { + return false; + } } catch { // If getTypeStrFromPath can't determine the type, fall through to normal logic } @@ -2637,6 +3134,46 @@ export async function ignoreF(wmillconf: { }; } +/** + * How many migration *records* a set of changed files covers. One migration is two + * files (`.up.sql` + optional `.down.sql`) for a single `(datatable, timestamp)`, + * so counting paths would overstate what a prompt is about to delete. + */ +export function countDatatableMigrationRecords( + changes: { path: string }[], +): number { + const records = new Set(); + for (const c of changes) { + const parsed = parseDatatableMigrationPath(c.path); + if (parsed) records.add(`${parsed.datatable}\0${parsed.timestamp}`); + } + return records.size; +} + +/** + * The `deleted` changes for data table migrations that a push cannot safely trust. + * + * Migrations bypass the repo's path filters (they live outside `f/`/`u/`), so a clone + * made before they were synced sees every server-side migration as remote-only — and + * `pushMigrationFromDisk` reads a locally absent `.up.sql` as an instruction to delete + * it. `recorded` is what this repository's own history says (see + * `gitRecordedDatatableMigrationPaths`): a recorded path was genuinely tracked, so its + * absence now is a real deletion; a path missing from a `known` set is one this branch + * has never had, and deleting it is a guess. `unknown` history is not evidence of + * anything, so nothing is trusted. The caller confirms whatever comes back explicitly + * and never deletes it unattended. + */ +export function untrackedDatatableMigrationDeletions< + T extends { name: string; path: string }, +>(changes: T[], recorded: RecordedMigrationPaths): T[] { + return changes.filter( + (c) => + c.name === "deleted" && + isDatatableMigrationPath(c.path) && + !(recorded.kind === "known" && recorded.paths.has(c.path)), + ); +} + interface ChangeTracker { scripts: string[]; flows: string[]; @@ -2644,13 +3181,78 @@ interface ChangeTracker { rawApps: string[]; } +/// The script a module file belongs to, added to the tracker so its top hash is +/// refreshed. Derived with `getScriptBasePathFromModulePath`, which normalizes +/// separators: searching the raw path for `__dbt/` found nothing on Windows, +/// where the folder is spelled `__dbt\`, so every model edit there was skipped. +async function addModuleParentToChanged(p: string, tracker: ChangeTracker) { + // A folder layout's METADATA — `__mod/script.yaml` — is an entry-point + // path too, and it is not a content file: pushed as one, the metadata pass + // asks `inferContentTypeFromFilePath` for the language of `.yaml` and aborts + // the whole command. It resolves to its content file like any other metadata. + if (isModuleEntryMetadata(p)) { + try { + const contentPath = await findContentFile(p); + if (contentPath && !tracker.scripts.includes(contentPath)) { + tracker.scripts.push(contentPath); + } + } catch { + // ignore — content file not found + } + return; + } + if (isModuleEntryPoint(p)) { + // Entry point (e.g. __mod/script.ts) IS the parent script content file. + if (!tracker.scripts.includes(p)) { + tracker.scripts.push(p); + } + return; + } + const scriptBasePath = getScriptBasePathFromModulePath(p); + if (scriptBasePath === undefined) { + return; + } + const push = async (candidate: string): Promise => { + try { + const contentPath = await findContentFile(candidate); + if (contentPath && !tracker.scripts.includes(contentPath)) { + tracker.scripts.push(contentPath); + } + return contentPath != undefined; + } catch { + return false; + } + }; + // A dbt script's metadata sits beside its folder; the descriptor is inside. + if (isDbtModulePath(p)) { + await push(scriptBasePath + ".script.yaml"); + return; + } + // Folder layout first (`__mod/script.{ext}`), then flat. + if (!(await push(scriptBasePath + getModuleFolderSuffix() + "/script.yaml"))) { + await push(scriptBasePath + ".script.yaml"); + } +} + async function addToChangedIfNotExists(p: string, tracker: ChangeTracker) { // Datatable migration .sql files are not scripts; they're synced via the // dedicated datatable_migration handler in the push loop. if (isDatatableMigrationPath(p)) { return; } - const isScript = exts.some((e) => p.endsWith(e)) && !isFileResource(p) && !isFilesetResource(p); + // Module files first, and whatever their extension: a dbt project authors + // `dbt_project.yml`, `packages.yml`, schema YAML and seed CSVs, none of which + // are Windmill script extensions — gated behind that test they never reached + // the tracker, so the top hash in `wmill-lock.yaml` (which covers the modules) + // stayed stale for exactly the files a dbt project is mostly made of. + if (isScriptModulePath(p)) { + await addModuleParentToChanged(p, tracker); + return; + } + const isScript = + hasScriptExt(p) && + !isFileResource(p) && + !isFilesetResource(p); if (isScript) { if (isFlowPath(p)) { const folder = extractFolderPath(p, "flow")!; @@ -2667,37 +3269,6 @@ async function addToChangedIfNotExists(p: string, tracker: ChangeTracker) { if (!tracker.rawApps.includes(folder)) { tracker.rawApps.push(folder); } - } else if (isScriptModulePath(p)) { - if (isModuleEntryPoint(p)) { - // Entry point (e.g. __mod/script.ts) IS the parent script content file - if (!tracker.scripts.includes(p)) { - tracker.scripts.push(p); - } - } else { - // Module file changed — find the parent script content file - const moduleSuffix = getModuleFolderSuffix() + "/"; - const idx = p.indexOf(moduleSuffix); - if (idx !== -1) { - const scriptBasePath = p.substring(0, idx); - // Try folder layout first: __mod/script.{ext} - try { - const contentPath = await findContentFile(scriptBasePath + getModuleFolderSuffix() + "/script.yaml"); - if (contentPath && !tracker.scripts.includes(contentPath)) { - tracker.scripts.push(contentPath); - } - } catch { - // Fall back to flat layout: scriptBasePath.script.yaml - try { - const contentPath = await findContentFile(scriptBasePath + ".script.yaml"); - if (contentPath && !tracker.scripts.includes(contentPath)) { - tracker.scripts.push(contentPath); - } - } catch { - // ignore — content file not found - } - } - } - } } else { if (!tracker.scripts.includes(p)) { tracker.scripts.push(p); @@ -2715,7 +3286,7 @@ async function addToChangedIfNotExists(p: string, tracker: ChangeTracker) { } } -async function buildTracker(changes: Change[]) { +export async function buildTracker(changes: Change[]) { const tracker: ChangeTracker = { scripts: [], flows: [], @@ -2734,7 +3305,7 @@ async function buildTracker(changes: Change[]) { * When a module file changes, find and push the parent script. * The parent script's handleFile will read the __mod/ folder and include all modules. */ -async function pushParentScriptForModule( +export async function pushParentScriptForModule( modulePath: string, workspace: Workspace, alreadySynced: string[], @@ -2743,11 +3314,82 @@ async function pushParentScriptForModule( rawWorkspaceDependencies: Record, codebases: SyncCodebase[], ): Promise { - const moduleSuffix = getModuleFolderSuffix() + "/"; - const idx = modulePath.indexOf(moduleSuffix); - if (idx === -1) return; - const scriptBasePath = modulePath.substring(0, idx); - const moduleFolderPath = scriptBasePath + getModuleFolderSuffix(); + const isDbt = isDbtModulePath(modulePath); + // Via the shared helper, which normalizes separators: a Windows path spells + // the folder `__dbt\\`, and searching the raw path for `__dbt/` would find + // nothing and silently return without deploying the parent — while the caller + // still records the file as synced. + const scriptBasePath = getScriptBasePathFromModulePath(modulePath); + if (scriptBasePath === undefined) return; + const moduleFolderPath = + scriptBasePath + getModuleFolderSuffix(isDbt ? "dbt" : undefined); + + // A dbt project's descriptor sits INSIDE its folder (`__dbt/wm_dbt.yaml`) and + // is optional, so the project itself is what identifies the script. + if (isDbt) { + // Only the LOOKUP is tolerated: a module under no script's project is a + // stray file, not an error. Deploying it is not — swallowing that would let + // a module-only push report success while the remote project is unchanged. + // BEFORE the lookup, because the lookup succeeds whenever a descriptor is + // there: `dbt_project.yml` is what makes the bundle a project, and pushing + // without it replaces a healthy deployment with one whose dependency job + // fails for having no project at all. + const hasMetadata = + existsSync(scriptBasePath + ".script.yaml") || + existsSync(scriptBasePath + ".script.json"); + if (!existsSync(moduleFolderPath + "/dbt_project.yml")) { + if (hasMetadata) { + throw new Error( + `${moduleFolderPath} has no dbt_project.yml but ${scriptBasePath}.script.yaml ` + + `remains, so there is no dbt project left to push. Delete the metadata too to ` + + `archive the script, or restore the project.` + ); + } + // Nothing local claims this script any more — neither project nor + // metadata — so it is archived like any other locally deleted item. The + // deletions arrive one file at a time, hence `alreadySynced`. + const remote = scriptBasePath.replaceAll(SEP, "/"); + if (!alreadySynced.includes(remote)) { + alreadySynced.push(remote); + log.info(`Archiving script ${remote}`); + await wmill + .archiveScriptByPath({ workspace: workspace.workspaceId, path: remote }) + .catch((e: any) => { + // Only "already gone" is the state we wanted. An auth, network or + // server failure must fail the push: swallowing it reports success + // while the project stays deployed, which is the thing this branch + // exists to prevent. + if (e?.status !== 404) throw e; + log.debug(`${remote} was already gone remotely`); + }); + } + return; + } + let contentPath: string | undefined; + try { + contentPath = await findContentFile(scriptBasePath + ".script.yaml"); + } catch (e) { + // A path claimed by two scripts is not a parent that cannot be found: + // swallowed here, `wmill sync push` reports success on a model edit that + // deployed nothing, and the collision stays invisible until the ordinary + // script overwrites the project. + if (e instanceof DbtPathCollisionError) throw e; + log.debug(`Could not find parent script for dbt module: ${modulePath}`); + return; + } + if (contentPath) { + await handleFile( + contentPath, + workspace, + alreadySynced, + message, + opts, + rawWorkspaceDependencies, + codebases, + ); + } + return; + } // Try folder layout first: look for script.{ext} inside __mod/ try { @@ -2958,11 +3600,16 @@ export async function pull( let specificItems = getSpecificItemsForCurrentBranch(opts, wsNameForConfig); // Compute the workspace name for file naming (default to workspaceId) - let wsNameForFiles = wsNameForConfig ? resolveWsNameForFiles(opts, wsNameForConfig) : workspace.workspaceId; + let wsNameForFiles = wsNameForConfig + ? resolveWsNameForFiles(opts, wsNameForConfig) + : workspace.workspaceId; // Augment specificItems with server-side ws_specific entries const localSpecificItems = specificItems; - const wsSpecificMerge = await mergeWsSpecificFromServer(workspace.workspaceId, specificItems); + const wsSpecificMerge = await mergeWsSpecificFromServer( + workspace.workspaceId, + specificItems, + ); specificItems = wsSpecificMerge.merged; // Merge CLI flags with resolved settings (CLI flags take precedence only for explicit overrides) @@ -3003,7 +3650,9 @@ export async function pull( opts.includeSettings, opts.includeKey, opts.skipWorkspaceDependencies, + opts.skipDatatableMigrations, opts.defaultTs, + opts.syncBehavior, ); const remote = ZipFSElement( @@ -3105,10 +3754,18 @@ export async function pull( return; } + // Script lockfile deletions, held back until every metadata edit has been + // applied or refused — see the deletion branch below. + const deferredLockDeletions: { + path: string; + target: string; + stateTarget: string; + }[] = []; const conflicts = []; log.info(colors.gray(`Applying changes to files ...`)); - for await (const change of changes) { + for await (const rawChange of changes) { + const change: Change = rawChange; // Determine if this file should be written to a workspace-specific path let targetPath = change.path; if (specificItems && isSpecificItem(change.path, specificItems)) { @@ -3124,6 +3781,28 @@ export async function pull( const target = path.join(process.cwd(), targetPath); const stateTarget = path.join(process.cwd(), ".wmill", targetPath); + // An empty dbt descriptor is not a file: the remote spells "this project + // named no descriptor" as empty content, and writing that would put a + // Windmill file inside a project that has none. ABSENCE is the state to + // reach, so both copies are removed if present and their being missing — + // a project pulled for the first time — is the goal, not an error. The + // `.wmill` copy goes too, or the same change is reported on every pull. + // + // `force` covers the missing file and NOTHING else: a permission or + // read-only-filesystem failure has to surface, or the pull reports + // success while the old descriptor — its warehouse, its command, its + // arguments — is still what runs locally. + if ( + isDbtDescriptorPath(change.path) && + ((change.name === "added" && change.content === "") || + (change.name === "edited" && change.after === "")) + ) { + await rm(target, { force: true }); + if (opts.stateful) { + await rm(stateTarget, { force: true }); + } + continue; + } if (change.name === "edited") { if (opts.stateful) { try { @@ -3165,11 +3844,13 @@ export async function pull( // ignore } } - if (exts.some((e) => change.path.endsWith(e))) { + if (hasScriptExt(change.path)) { log.info( `Editing script content of ${targetPath}${ targetPath !== change.path - ? colors.gray(` (workspace-specific override for ${change.path})`) + ? colors.gray( + ` (workspace-specific override for ${change.path})`, + ) : "" }`, ); @@ -3180,7 +3861,9 @@ export async function pull( log.info( `Editing ${changeTypeLabel(change.path)}${targetPath}${ targetPath !== change.path - ? colors.gray(` (workspace-specific override for ${change.path})`) + ? colors.gray( + ` (workspace-specific override for ${change.path})`, + ) : "" }`, ); @@ -3198,7 +3881,9 @@ export async function pull( log.info( `Adding ${changeTypeLabel(change.path)}${targetPath}${ targetPath !== change.path - ? colors.gray(` (workspace-specific override for ${change.path})`) + ? colors.gray( + ` (workspace-specific override for ${change.path})`, + ) : "" }`, ); @@ -3215,19 +3900,47 @@ export async function pull( await copyFile(target, stateTarget); } } else if (change.name === "deleted") { - try { - log.info( - `Deleting ${changeTypeLabel(change.path)}${change.path}`, - ); - await rm(target); - if (opts.stateful) { - await rm(stateTarget); - } - } catch { - if (opts.stateful) { - await rm(stateTarget); - } + // A script's lockfile goes last, once the metadata around it has + // settled: `dedupeLockfiles` deletes the per-script locks it collapses, + // and a conflict resolved as "preserve local" keeps metadata that still + // reads one. Deleted here, that reference would dangle and the script + // would deploy with an empty lock. + if (isScriptLockPath(change.path) || isSharedLockPath(change.path)) { + deferredLockDeletions.push({ path: change.path, target, stateTarget }); + continue; } + log.info(`Deleting ${changeTypeLabel(change.path)}${change.path}`); + // `force` on both: the goal is that neither copy exists, and a file + // already absent — a dbt project's optional descriptor is never written + // — is that goal, not an error. Anything else (permissions, a read-only + // mount) surfaces rather than leaving a file the sync believes is gone. + // The state copy goes too, or the same deletion replays on every sync. + await rm(target, { force: true }); + if (opts.stateful) { + await rm(stateTarget, { force: true }); + } + } + } + + const sharedReaders = deferredLockDeletions.some((d) => + isSharedLockPath(d.path), + ) + ? await collectSharedLockReaders(opts.json ?? false) + : { byRef: new Map(), unreadable: [] }; + for (const deferred of deferredLockDeletions) { + const keptBecause = await lockStillReadBecause( + deferred.path, + opts.json ?? false, + sharedReaders, + ); + if (keptBecause !== undefined) { + log.info(colors.yellow(`Keeping ${deferred.path}: ${keptBecause}.`)); + continue; + } + log.info(`Deleting ${changeTypeLabel(deferred.path)}${deferred.path}`); + await rm(deferred.target, { force: true }); + if (opts.stateful) { + await rm(deferred.stateTarget, { force: true }); } } if (opts.failConflicts) { @@ -3362,7 +4075,8 @@ export async function pull( try { // Dynamic import to avoid a circular dep between sync.ts and // generate-metadata.ts. Don't "clean up" to a static import. - const { rehashOnly } = await import("../generate-metadata/generate-metadata.ts"); + const { rehashOnly } = + await import("../generate-metadata/generate-metadata.ts"); // Reuse the local-side file list from the change-tracker so we don't // re-walk the filesystem. Apply the just-applied changes to derive the // post-pull state: localMap is pre-pull, but auto-fill needs to see @@ -3381,7 +4095,7 @@ export async function pull( log.info( colors.gray( `Auto-filled ${total} missing lockfile entr${total === 1 ? "y" : "ies"} ` + - `(${filled.scripts} script, ${filled.flows} flow, ${filled.apps} app) from disk.`, + `(${filled.scripts} script, ${filled.flows} flow, ${filled.apps} app) from disk.`, ), ); } @@ -3419,6 +4133,92 @@ export async function pull( // stays exported for callers that want the same commit/push behavior. } +/** + * Fold the lockfile that the scripts of a language share back into the one + * shared file (`dedupeLockfiles`), and give the scripts that ended up with a + * lock of their own theirs back. + * + * A lock is regenerated one script at a time, so only a pass over the whole + * tree can tell a dependency bump every script took (the shared file moves) + * from one script drifting away from the rest (it gets a lock of its own). + * + * Rewriting a script's metadata invalidates the hash the generation above just + * recorded, so every rewritten script is re-hashed from disk — the same + * lock-untouching pass `sync pull` runs, no dependency job involved. + */ +export async function dedupeLockfilesOnDisk(args: { + opts: GlobalOptions & SyncOptions; + workspace: Workspace; + codebases: SyncCodebase[]; + ignore: (p: string, isD: boolean) => boolean; + rawWorkspaceDependencies: Record; + tree: DoubleLinkedDependencyTree; + /** Content paths whose generation failed this run. Their metadata may be + * rewritten, but never re-hashed: recording a hash for a script whose lock + * never regenerated marks it up-to-date, and it is never retried. */ + failed?: string[]; + dryRun?: boolean; +}): Promise { + const { + opts, + workspace, + codebases, + ignore, + rawWorkspaceDependencies, + tree, + failed = [], + dryRun, + } = args; + const map = await elementsToMap( + await FSFSElement(process.cwd(), codebases, false), + ignore, + opts.json ?? false, + opts, + ); + const plan = computeSharedLockPlan(map, { + defaultTs: opts.defaultTs, + depFiles: opts.skipWorkspaceDependencies + ? Object.keys(await getRawWorkspaceDependencies(false)) + : undefined, + }); + if (isEmptySharedLockPlan(plan)) return; + + const summary = `${Object.keys(plan.writes).length} file(s) written, ${plan.deletes.length} removed`; + if (dryRun) { + log.info(`Would deduplicate lockfiles: ${summary}`); + return; + } + + await applySharedLockPlanToDisk(plan); + log.info(`Deduplicated lockfiles: ${summary}`); + + for (const rewritten of Object.keys(plan.writes)) { + // Metadata only — the lockfiles the plan also writes are not hashed. + if (!rewritten.endsWith(".yaml") && !rewritten.endsWith(".json")) continue; + let contentPath: string | undefined; + try { + contentPath = await findContentFile(rewritten); + } catch { + continue; + } + if (!contentPath || failed.includes(contentPath)) continue; + await generateScriptMetadataInternal( + contentPath, + workspace, + opts, + false, // dryRun + true, // noStaleMessage + rawWorkspaceDependencies, + codebases, + true, // justUpdateMetadataLock: re-hash from disk, no lock generation + // The same tree the generation above ran with: it is what decides whether + // the workspace dependencies are part of the hash, and a hash written the + // other way would read as stale on every later run. + tree, + ); + } +} + // Internal git-sync deployment-callback entrypoint. Invoked only by the // git-sync hub script (not user-facing — see the hidden `git-deploy` // subcommand). Runs inside an existing clone of the repo: switches to the @@ -3469,17 +4269,14 @@ export async function gitDeploy( // the wm_deploy branch). Mirrors the hub script's `--promotion `. const promotion = useIndividualBranch && !opts.promotion - ? getCurrentGitBranch() ?? undefined + ? (getCurrentGitBranch() ?? undefined) : opts.promotion; await pull({ ...opts, yes: true, skipBranchValidation: true, - extraIncludes: [ - ...(opts.extraIncludes ?? []), - ...includes.extraIncludes, - ], + extraIncludes: [...(opts.extraIncludes ?? []), ...includes.extraIncludes], // Workspace-wide mode force-includes the deployed default-excluded kinds // (full mirror). Individual-branch/promotion mode forces nothing — these // keys stay ABSENT so pull resolves them from the promotion target's @@ -3527,7 +4324,9 @@ function prettyChanges( const folderNote = folderDefaultAnnotations?.get(change.path); const extraNote = folderNote - ? colors.cyan(` (will be permissioned as ${folderNote} via folder default)`) + ? colors.cyan( + ` (will be permissioned as ${folderNote} via folder default)`, + ) : ""; if (change.name === "added") { @@ -3672,14 +4471,12 @@ async function checkServerLockJobs( const pending = (queued as { script_path?: string }[]).filter((j) => belongsToPush(j.script_path), ).length; - const failed = ( - completed as { script_path?: string; result?: unknown }[] - ) + const failed = (completed as { script_path?: string; result?: unknown }[]) .filter((j) => belongsToPush(j.script_path)) .map((j) => ({ path: j.script_path!, - error: (j.result as { error?: { message?: string } } | undefined) - ?.error?.message, + error: (j.result as { error?: { message?: string } } | undefined)?.error + ?.message, })); return { pending, failed }; } catch { @@ -3689,9 +4486,15 @@ async function checkServerLockJobs( } export async function push( - opts: GlobalOptions & SyncOptions & { repository?: string; branch?: string; acceptOverridingPermissionedAsWithSelf?: boolean }, + opts: GlobalOptions & + SyncOptions & { + repository?: string; + branch?: string; + acceptOverridingPermissionedAsWithSelf?: boolean; + }, ) { if ((opts as any).jsonOutput) log.setSilent(true); + markRequestsAsSyncOrigin(); // Save original CLI options before merging with config file const originalCliOpts = { ...opts }; @@ -3752,7 +4555,9 @@ export async function push( let specificItems = getSpecificItemsForCurrentBranch(opts, wsNameForConfig); // Compute the workspace name for file naming (default to workspaceId) - let wsNameForFiles = wsNameForConfig ? resolveWsNameForFiles(opts, wsNameForConfig) : workspace.workspaceId; + let wsNameForFiles = wsNameForConfig + ? resolveWsNameForFiles(opts, wsNameForConfig) + : workspace.workspaceId; // Keep the pre-merge specificItems so we can detect entries that are // flagged locally but not yet ws_specific on the server (post-merge would @@ -3760,7 +4565,10 @@ export async function push( const localSpecificItems = specificItems; // Augment specificItems with server-side ws_specific entries - const wsSpecificMerge = await mergeWsSpecificFromServer(workspace.workspaceId, specificItems); + const wsSpecificMerge = await mergeWsSpecificFromServer( + workspace.workspaceId, + specificItems, + ); specificItems = wsSpecificMerge.merged; const serverWsSpecificItems = wsSpecificMerge.serverItems; @@ -3844,7 +4652,9 @@ export async function push( opts.includeSettings, opts.includeKey, opts.skipWorkspaceDependencies, + opts.skipDatatableMigrations, opts.defaultTs, + opts.syncBehavior, ))!, !opts.json, opts.defaultTs ?? "bun", @@ -3854,7 +4664,11 @@ export async function push( parseSyncBehavior(opts.syncBehavior) >= 1, ); - const local = await FSFSElement(path.join(process.cwd(), ""), codebases, false); + const local = await FSFSElement( + path.join(process.cwd(), ""), + codebases, + false, + ); const { changes, localMap } = await compareDynFSElement( local, remote, @@ -3897,6 +4711,86 @@ export async function push( const tracker: ChangeTracker = await buildTracker(changes); + // A shared lockfile (`dedupeLockfiles`) has no object of its own on the + // remote: it IS the lock of every script that references it, and those + // scripts are what carries its new content over. Nothing else queues them — + // their own metadata is byte-identical on both sides. + // + // After the tracker on purpose: these scripts need no metadata regeneration + // (their lock is on disk already, in the shared file), and `--auto-metadata` + // would otherwise run one dependency job per script sharing the lock. + const changedPaths = new Set(changes.map((c) => c.path)); + let unconvertedTree = false; + // The whole tree, not `localMap`: `includes`/`excludes` have already filtered + // that, and a shared lockfile's readers are exactly what the filter hides. + // One metadata pass, on the first shared-lock change and never otherwise, so + // it is a dependency bump that pays for it and not an ordinary push. + let treeReaders: SharedLockReaders | undefined; + for (let i = changes.length - 1; i >= 0; i--) { + const change = changes[i]; + if (!isSharedLockPath(change.path)) continue; + if (change.name === "deleted") { + // The remote view is deduplicated whether or not the tree is: a shared + // lockfile missing from the tree reads as a deletion to push, and there is + // no such object to delete. + unconvertedTree = true; + changes.splice(i, 1); + continue; + } + const referrers = scriptsReferencingSharedLock(localMap, change.path); + if (treeReaders === undefined) { + try { + treeReaders = await collectSharedLockReaders(opts.json ?? false); + } catch (e) { + // The walk is fail-loud because a deletion hangs on it. Nothing hangs + // on an advisory, so an unreadable directory costs the advisory, not + // the push. + log.debug(`Could not scan for shared-lock readers: ${e}`); + treeReaders = { byRef: new Map(), unreadable: [] }; + } + } + const outOfScope = + (treeReaders?.byRef.get(change.path.replaceAll(SEP, "/"))?.length ?? 0) - + referrers.length; + // Out of `changes` either way: a shared lockfile has no object on the + // remote, so the apply loop skips it. Left in, the preview and the "N + // changes" count would report something no push ever applies as such. + changes.splice(i, 1); + // A scoped push deploys what it was scoped to, so the readers the filter + // excluded keep the previous lock on the remote. Silence there is the + // trap: the changed file has no object of its own, so nothing else in the + // output would account for it. + if (outOfScope > 0) { + log.warn( + colors.yellow( + `${change.path} changed, but ${outOfScope} of the script(s) sharing it are outside this push's scope and keep the previous lock on the remote. Widen --includes/--excludes to deploy the new lock to all of them.`, + ), + ); + } + if (referrers.length === 0) continue; + log.info( + colors.gray( + `${change.path} changed: re-pushing the ${referrers.length} script(s) sharing it`, + ), + ); + for (const metaPath of referrers) { + if (changedPaths.has(metaPath)) continue; + changes.push({ + name: "edited", + path: metaPath, + before: localMap[metaPath], + after: localMap[metaPath], + }); + } + } + if (unconvertedTree) { + log.warn( + colors.yellow( + `dedupeLockfiles is on but this checkout still holds one lockfile per script. Run 'wmill generate-metadata' (or pull) to convert it — until then every script reads as changed.`, + ), + ); + } + const autoRegenerate = !!(opts as any).autoMetadata; const staleScripts: string[] = []; const staleFlows: string[] = []; @@ -4054,6 +4948,26 @@ export async function push( staleApps.push(generated as string); } } + + if (opts.dedupeLockfiles) { + // Batched: the pass re-hashes every metadata file it rewrites, and one + // wmill-lock.yaml write per script is what a workspace-wide conversion + // would otherwise cost. + await beginLockfileBatch(); + try { + await dedupeLockfilesOnDisk({ + opts, + workspace, + codebases, + ignore: await ignoreF(opts), + rawWorkspaceDependencies, + tree, + dryRun: opts.dryRun, + }); + } finally { + await flushLockfileBatch(); + } + } } if (staleScripts.length > 0) { @@ -4120,14 +5034,20 @@ export async function push( let triggerCount = 0; for await (const entry of readDirRecursiveWithIgnore(() => false, local)) { if (entry.isDirectory) continue; - if (!opts.includeSchedules && entry.path.endsWith(".schedule.yaml")) scheduleCount++; - if (!opts.includeTriggers && entry.path.endsWith("_trigger.yaml")) triggerCount++; + if (!opts.includeSchedules && entry.path.endsWith(".schedule.yaml")) + scheduleCount++; + if (!opts.includeTriggers && entry.path.endsWith("_trigger.yaml")) + triggerCount++; } if (scheduleCount > 0) { - skippedWarnings.push(`Skipping ${scheduleCount} schedule file(s). Use --include-schedules or set includeSchedules: true in wmill.yaml`); + skippedWarnings.push( + `Skipping ${scheduleCount} schedule file(s). Use --include-schedules or set includeSchedules: true in wmill.yaml`, + ); } if (triggerCount > 0) { - skippedWarnings.push(`Skipping ${triggerCount} trigger file(s). Use --include-triggers or set includeTriggers: true in wmill.yaml`); + skippedWarnings.push( + `Skipping ${triggerCount} trigger file(s). Use --include-triggers or set includeTriggers: true in wmill.yaml`, + ); } for (const warning of skippedWarnings) { log.warn(warning); @@ -4137,6 +5057,44 @@ export async function push( await fetchRemoteVersion(workspace); + const recordedMigrationPaths: RecordedMigrationPaths = changes.some( + (c) => c.name === "deleted" && isDatatableMigrationPath(c.path), + ) + ? gitRecordedDatatableMigrationPaths() + : { kind: "known", paths: new Set() }; + const ambiguousMigrationDeletions = untrackedDatatableMigrationDeletions( + changes, + recordedMigrationPaths, + ); + const keepAmbiguousMigrationsOnRemote = () => { + log.info( + colors.yellow( + `Keeping ${countDatatableMigrationRecords(ambiguousMigrationDeletions)} data table migration(s) on the remote: ` + + (recordedMigrationPaths.kind === "known" + ? `this branch has never tracked them. Run 'wmill sync pull' to track them in git, or delete them from the workspace.` + : `${recordedMigrationPaths.reason}, so whether it ever tracked them cannot be established. ` + + `${recordedMigrationPaths.remedy} so a real deletion can be told apart, or delete them from the workspace.`), + ), + ); + const kept = changes.filter( + (c) => !ambiguousMigrationDeletions.includes(c), + ); + changes.length = 0; + changes.push(...kept); + }; + // An unattended run never resolves this ambiguity destructively, and a dry-run + // preview has to show what a push would really do — settle both before the + // change list is printed or serialized. A TTY push asks instead, after the + // user has seen the list. + let ambiguousMigrationsResolved = false; + if ( + ambiguousMigrationDeletions.length > 0 && + (opts.dryRun || opts.yes || !process.stdin.isTTY) + ) { + keepAmbiguousMigrationsOnRemote(); + ambiguousMigrationsResolved = true; + } + // Shared UI (the ui/ folder) is pushed out-of-band via pushSharedUi on apply // and is excluded from the file diff (isNotWmillFile), so surface its diff in // the dry-run preview. Without this the "Pull from repo" preview reads "no @@ -4218,7 +5176,18 @@ export async function push( `Run 'wmill folder add-missing' to create them locally, then push again.`; if (!userIsAdmin) { if (opts.jsonOutput) { - console.log(JSON.stringify({ success: false, error: "missing_folders", missing_folders: missingFolders, message: msg }, null, 2)); + console.log( + JSON.stringify( + { + success: false, + error: "missing_folders", + missing_folders: missingFolders, + message: msg, + }, + null, + 2, + ), + ); } else { log.error(msg); } @@ -4229,6 +5198,61 @@ export async function push( } } + // Non-canonical fileset pointers abort here — before the dry-run output and + // before any change is applied (deletes run first in the apply loop, so a + // mid-apply rejection would leave a partial deploy). All violations are + // reported at once. + { + const wsNameForPointerCheck = + wsNameForFiles || (isGitRepository() ? getCurrentGitBranch() : null); + const pointerErrors: string[] = []; + for (const change of changes) { + if (change.name !== "added" && change.name !== "edited") { + continue; + } + const normalizedPath = change.path.replaceAll(SEP, "/"); + if ( + !normalizedPath.endsWith(".resource.yaml") && + !normalizedPath.endsWith(".resource.json") + ) { + continue; + } + // Fileset content is arbitrary: a child may itself be named + // `*.resource.yaml`, and its body is not this resource's metadata. + if (isFilesetResource(change.path)) { + continue; + } + const content = change.name === "added" ? change.content : change.after; + let parsed: any; + try { + parsed = parseFromPath(change.path, content); + } catch { + // Malformed files surface their own error in the apply loop. + continue; + } + if ( + typeof parsed?.value === "string" && + parsed.value.startsWith("!inline_fileset ") + ) { + const serverPath = + wsNameForPointerCheck && isWorkspaceSpecificFile(change.path) + ? fromWorkspaceSpecificPath(change.path, wsNameForPointerCheck) + : change.path; + try { + validateFilesetPointer( + parsed.value.split(" ")[1], + removeType(serverPath, "resource"), + ); + } catch (e) { + pointerErrors.push(e instanceof Error ? e.message : String(e)); + } + } + } + if (pointerErrors.length > 0) { + throw new Error(pointerErrors.join("\n")); + } + } + // Handle JSON output for dry-run if (opts.dryRun && opts.jsonOutput) { const result = { @@ -4251,9 +5275,7 @@ export async function push( : {}), })), total: changes.length, - ...(changes.length > 0 - ? { warning: SYNC_PUSH_DESTRUCTIVE_WARNING } - : {}), + ...(changes.length > 0 ? { warning: SYNC_PUSH_DESTRUCTIVE_WARNING } : {}), }; console.log(JSON.stringify(result, null, 2)); return; @@ -4264,7 +5286,10 @@ export async function push( let folderDefaultAnnotations: Map | undefined; if (parseSyncBehavior(opts.syncBehavior) >= 1) { folderDefaultAnnotations = new Map(); - const folderRulesCache = new Map>(); + const folderRulesCache = new Map< + string, + Array<{ path_glob: string; permissioned_as: string }> + >(); for (const change of changes) { if (change.name !== "added") continue; const match = change.path.match(/^f\/([^/]+)\//); @@ -4272,14 +5297,26 @@ export async function push( const folderName = match[1]; if (!folderRulesCache.has(folderName)) { try { - const folder = await wmill.getFolder({ workspace: workspace.workspaceId, name: folderName }); - folderRulesCache.set(folderName, (folder as any).default_permissioned_as ?? []); + const folder = await wmill.getFolder({ + workspace: workspace.workspaceId, + name: folderName, + }); + folderRulesCache.set( + folderName, + (folder as any).default_permissioned_as ?? [], + ); } catch { folderRulesCache.set(folderName, []); } } const rules = folderRulesCache.get(folderName)!; - const remotePath = change.path.replace(/\.(script|schedule|http_trigger|websocket_trigger|kafka_trigger|nats_trigger|postgres_trigger|mqtt_trigger|amqp_trigger|sqs_trigger|gcp_trigger|azure_trigger|email_trigger)\.(yaml|json)$/, "").replace(/(\.flow|__flow)\/flow\.(yaml|json)$/, "").replace(/\.(app|raw_app)(\/app\.(yaml|json))?$/, ""); + const remotePath = change.path + .replace( + /\.(script|schedule|http_trigger|websocket_trigger|kafka_trigger|nats_trigger|postgres_trigger|mqtt_trigger|amqp_trigger|sqs_trigger|gcp_trigger|azure_trigger|email_trigger)\.(yaml|json)$/, + "", + ) + .replace(/(\.flow|__flow)\/flow\.(yaml|json)$/, "") + .replace(/\.(app|raw_app)(\/app\.(yaml|json))?$/, ""); const relative = remotePath.slice(`f/${folderName}/`.length); if (!relative) continue; for (const rule of rules) { @@ -4292,7 +5329,12 @@ export async function push( } if (!opts.jsonOutput) { - prettyChanges(changes, specificItems, wsNameForFiles, folderDefaultAnnotations); + prettyChanges( + changes, + specificItems, + wsNameForFiles, + folderDefaultAnnotations, + ); } if (opts.dryRun) { @@ -4306,7 +5348,9 @@ export async function push( const user = await wmill.whoami({ workspace: workspace.workspaceId }); const userIsAdminOrDeployer = user.is_admin || (user.groups ?? []).includes("wm_deployers"); - log.debug(`permissioned_as: user=${user.email}, is_admin=${user.is_admin}, groups=${JSON.stringify(user.groups)}, isAdminOrDeployer=${userIsAdminOrDeployer}`); + log.debug( + `permissioned_as: user=${user.email}, is_admin=${user.is_admin}, groups=${JSON.stringify(user.groups)}, isAdminOrDeployer=${userIsAdminOrDeployer}`, + ); permissionedAsContext = { userCache: new Map(), userIsAdminOrDeployer, @@ -4324,10 +5368,12 @@ export async function push( !!process.stdin.isTTY, ); } else if (folderDefaultAnnotations && folderDefaultAnnotations.size > 0) { - log.warn(colors.yellow( - `This workspace has folder default_permissioned_as rules that affect ${folderDefaultAnnotations.size} item(s) being pushed, ` + - `but syncBehavior is not set in wmill.yaml. Add 'syncBehavior: v1' to enable ownership preservation on update and on_behalf_of stripping on pull.` - )); + log.warn( + colors.yellow( + `This workspace has folder default_permissioned_as rules that affect ${folderDefaultAnnotations.size} item(s) being pushed, ` + + `but syncBehavior is not set in wmill.yaml. Add 'syncBehavior: v1' to enable ownership preservation on update and on_behalf_of stripping on pull.`, + ), + ); } // Reject malformed datatable migrations (duplicate timestamps, orphan downs) @@ -4358,6 +5404,18 @@ export async function push( return; } + if (ambiguousMigrationDeletions.length > 0 && !ambiguousMigrationsResolved) { + const deleteThem = await Confirm.prompt({ + message: + `Nothing in this repository's history accounts for ${countDatatableMigrationRecords(ambiguousMigrationDeletions)} migration definition(s), so it may simply never have synced them. ` + + `Delete them from the workspace anyway?`, + default: false, + }); + if (!deleteThem) { + keepAmbiguousMigrationsOnRemote(); + } + } + const start = performance.now(); const pushStartedAt = new Date().toISOString(); log.info(colors.gray(`Applying changes to files ...`)); @@ -4374,7 +5432,14 @@ export async function push( // Group changes by base path (before first dot) const groupedChanges = new Map(); for (const change of changes) { - const basePath = change.path.split(".")[0]; + // A module file is pushed by pushing its parent script, so it belongs in + // that script's group. Left in a group of its own it gets its own + // `alreadySynced`, and a push touching several files of one bundle then + // deploys the script once per file: several versions in a row, of which + // only the last is the one the asset graph ends up describing. + const basePath = + getScriptBasePathFromModulePath(change.path) ?? + change.path.split(".")[0]; if (!groupedChanges.has(basePath)) { groupedChanges.set(basePath, []); } @@ -4422,7 +5487,8 @@ export async function push( const effectiveParallelism = () => folderPhaseRemaining > 0 ? 1 : parallelizationFactor; // Cache git branch at the start to avoid repeated execSync calls per change - const cachedWsNameForPush = wsNameForFiles || (isGitRepository() ? getCurrentGitBranch() : null); + const cachedWsNameForPush = + wsNameForFiles || (isGitRepository() ? getCurrentGitBranch() : null); // Datatable migrations are two files (.up.sql/.down.sql) for one record, so // dedupe upsert/delete by (datatable, version) across the whole push. @@ -4461,11 +5527,20 @@ export async function push( if (deleteRawApp) { changes = [deleteRawApp]; } else { + // The app is one bundle, so a single change re-pushes all of it. + // That leaves the loop exactly one change: any skip it takes for + // a raw-app path drops the whole app from the push, and nothing + // downstream records that as a failure. changes.splice(1, changes.length - 1); } } for await (const change of changes) { + // A shared lockfile is a repo-side artifact: the scripts queued + // above are what deploys its content. + if (isSharedLockPath(change.path)) { + continue; + } // A datatable migration is one record across two files; upsert/delete // it from disk once (deduped), regardless of which file changed. if (isDatatableMigrationPath(change.path)) { @@ -4491,6 +5566,93 @@ export async function push( } if (change.name === "edited") { + // A file/fileset resource's content file can carry a script + // extension (.sql, .ts, …), so it must be routed to its parent + // resource before the script handlers get a chance to treat it + // as a standalone script. + if ( + isFileResource(change.path) || + isFilesetResource(change.path) + ) { + if (stateTarget) { + await mkdir(path.dirname(stateTarget), { recursive: true }); + log.info( + `Editing ${getTypeStrFromPath(change.path)} ${change.path}`, + ); + } + } + // Fileset routing must precede the single-file check (as it does + // in the added/deleted branches): a fileset accepts arbitrary + // child names, so a child like `.fileset/q.resource.file.sql` + // matches both predicates and belongs to its fileset parent. + if (isFilesetResource(change.path)) { + const result = await pushFilesetParentResource( + change.path, + workspace.workspaceId, + alreadySynced, + cachedWsNameForPush, + specificItems, + ); + if (result.status === "parent-missing") { + throw new Error( + `No resource metadata file found for fileset resource: ${change.path}`, + ); + } + // Pushed or already-synced: the parent resource carries the + // whole fileset, so this child's content is on the remote. + if (stateTarget) { + await writeFile(stateTarget, change.after, "utf-8"); + } + continue; + } + if (isFileResource(change.path)) { + const resourceFilePath = await findResourceFile(change.path); + if (!alreadySynced.includes(resourceFilePath)) { + alreadySynced.push(resourceFilePath); + + const newObj = parseFromPath( + resourceFilePath, + await readTextFile(resourceFilePath), + ); + + // For branch-specific resources, push to the base path on the workspace server + // This ensures workspace-specific files are stored with their base names in the workspace + let serverPath = resourceFilePath; + const currentBranch = cachedWsNameForPush; + let isFileResWsSpecific = false; + + if ( + currentBranch && + isWorkspaceSpecificFile(resourceFilePath) + ) { + serverPath = fromWorkspaceSpecificPath( + resourceFilePath, + currentBranch, + ); + isFileResWsSpecific = true; + } else if ( + specificItems && + isSpecificItem(change.path, specificItems) + ) { + isFileResWsSpecific = true; + } + + await pushResource( + workspace.workspaceId, + serverPath, + undefined, + newObj, + resourceFilePath, + isFileResWsSpecific ? true : undefined, + true, + ); + } + // Already-synced parents got the full content this run. + if (stateTarget) { + await writeFile(stateTarget, change.after, "utf-8"); + } + continue; + } if ( await handleScriptMetadata( change.path, @@ -4545,74 +5707,13 @@ export async function push( `Editing ${getTypeStrFromPath(change.path)} ${change.path}`, ); } - - if (isFileResource(change.path)) { - const resourceFilePath = await findResourceFile(change.path); - if (!alreadySynced.includes(resourceFilePath)) { - alreadySynced.push(resourceFilePath); - - const newObj = parseFromPath( - resourceFilePath, - await readTextFile(resourceFilePath), - ); - - // For branch-specific resources, push to the base path on the workspace server - // This ensures workspace-specific files are stored with their base names in the workspace - let serverPath = resourceFilePath; - const currentBranch = cachedWsNameForPush; - let isFileResWsSpecific = false; - - if (currentBranch && isWorkspaceSpecificFile(resourceFilePath)) { - serverPath = fromWorkspaceSpecificPath( - resourceFilePath, - currentBranch, - ); - isFileResWsSpecific = true; - } else if (specificItems && isSpecificItem(change.path, specificItems)) { - isFileResWsSpecific = true; - } - - await pushResource( - workspace.workspaceId, - serverPath, - undefined, - newObj, - resourceFilePath, - isFileResWsSpecific ? true : undefined, - ); - if (stateTarget) { - await writeFile(stateTarget, change.after, "utf-8"); - } - continue; - } - } - if (isFilesetResource(change.path)) { - const result = await pushFilesetParentResource( - change.path, - workspace.workspaceId, - alreadySynced, - cachedWsNameForPush, - specificItems, - ); - if (result.status === "parent-missing") { - throw new Error( - `No resource metadata file found for fileset resource: ${change.path}`, - ); - } - if (result.status === "pushed") { - if (stateTarget) { - await writeFile(stateTarget, change.after, "utf-8"); - } - continue; - } - // "already-synced": fall through (pre-existing behavior). - } const oldObj = parseFromPath(change.path, change.before); const newObj = parseFromPath(change.path, change.after); // Check if this is a branch-specific item and get the original workspace-specific path let originalWorkspaceSpecificPath: string | undefined; - const isWsSpecific = specificItems && isSpecificItem(change.path, specificItems); + const isWsSpecific = + specificItems && isSpecificItem(change.path, specificItems); if (isWsSpecific) { originalWorkspaceSpecificPath = getWorkspaceSpecificPath( change.path, @@ -4628,13 +5729,17 @@ export async function push( newObj, opts.plainSecrets ?? false, alreadySynced, - opts.message, - originalWorkspaceSpecificPath, - permissionedAsContext, - isWsSpecific ? true : undefined, { - noninteractive: (opts.yes ?? false) || !process.stdin.isTTY, - skipReencrypt: opts.skipReencryptOnKeyChange, + message: opts.message, + originalLocalPath: originalWorkspaceSpecificPath, + permissionedAsContext, + wsSpecific: isWsSpecific ? true : undefined, + keyPushOpts: { + noninteractive: + (opts.yes ?? false) || !process.stdin.isTTY, + skipReencrypt: opts.skipReencryptOnKeyChange, + }, + defaultTs: opts.defaultTs, }, ); @@ -4695,7 +5800,11 @@ export async function push( !isRawAppFile(change.path) && (change.path.endsWith(".script.json") || change.path.endsWith(".script.yaml") || - change.path.endsWith(".lock") || + // A `.lock` is the script's generated lockfile — except inside + // a dbt bundle, where the project may author one (`uv.lock`). + // Skipping it there would report the add on every push and + // never apply it, because no state file is written either. + (change.path.endsWith(".lock") && !isDbtModulePath(change.path)) || isFileResource(change.path)) ) { continue; @@ -4735,7 +5844,8 @@ export async function push( // Determine the actual local file path for this change // For branch-specific items, we read from workspace-specific files but push to base server paths let localFilePath = change.path; - const isAddedWsSpecific = specificItems && isSpecificItem(change.path, specificItems); + const isAddedWsSpecific = + specificItems && isSpecificItem(change.path, specificItems); if (isAddedWsSpecific) { const workspaceSpecificPath = getWorkspaceSpecificPath( change.path, @@ -4754,13 +5864,17 @@ export async function push( obj, opts.plainSecrets ?? false, [], - opts.message, - localFilePath, // Pass the actual local file path - permissionedAsContext, - isAddedWsSpecific ? true : undefined, { - noninteractive: (opts.yes ?? false) || !process.stdin.isTTY, - skipReencrypt: opts.skipReencryptOnKeyChange, + message: opts.message, + originalLocalPath: localFilePath, + permissionedAsContext, + wsSpecific: isAddedWsSpecific ? true : undefined, + keyPushOpts: { + noninteractive: + (opts.yes ?? false) || !process.stdin.isTTY, + skipReencrypt: opts.skipReencryptOnKeyChange, + }, + defaultTs: opts.defaultTs, }, ); @@ -4768,7 +5882,15 @@ export async function push( await writeFile(stateTarget, change.content, "utf-8"); } } else if (change.name === "deleted") { - if (change.path.endsWith(".lock")) { + // Same as the added branch: a dbt project's own `.lock` is one of + // its files, so deleting it has to reach the parent script. A raw + // app's `.lock` is part of its bundle, and a raw-app group is + // collapsed to one change, so skipping it drops the whole app. + if ( + !isRawAppFile(change.path) && + change.path.endsWith(".lock") && + !isDbtModulePath(change.path) + ) { continue; } if (isScriptModulePath(change.path)) { @@ -4821,15 +5943,20 @@ export async function push( }); break; case "resource": { - const resourcePath = removeSuffix(target, ".resource.json"); + const resourcePath = removeResourceSuffix(target); try { await wmill.deleteResource({ workspace: workspaceId, path: resourcePath, }); } catch (e: any) { - if (e?.status === 404 && deletedVarsResPaths.includes(resourcePath)) { - log.debug(`Resource ${resourcePath} already deleted by linked variable`); + if ( + e?.status === 404 && + deletedVarsResPaths.includes(resourcePath) + ) { + log.debug( + `Resource ${resourcePath} already deleted by linked variable`, + ); } else { throw e; } @@ -4848,7 +5975,10 @@ export async function push( // Metadata file deleted — delete the entire flow await wmill.deleteFlowByPath({ workspace: workspaceId, - path: removeSuffix(target, getDeleteSuffix("flow", "json")), + path: removeSuffix( + target, + getDeleteSuffix("flow", "json"), + ), }); } else { // Inline script file deleted within flow folder @@ -4871,7 +6001,7 @@ export async function push( undefined, opts.plainSecrets ?? false, alreadySynced, - opts.message, + { message: opts.message }, ); } else { // Flow folder doesn't exist locally — delete on server @@ -4890,7 +6020,10 @@ export async function push( // Metadata file deleted — delete the entire app await wmill.deleteApp({ workspace: workspaceId, - path: removeSuffix(target, getDeleteSuffix("app", "json")), + path: removeSuffix( + target, + getDeleteSuffix("app", "json"), + ), }); } else { // Inline script file deleted within app folder @@ -4913,7 +6046,7 @@ export async function push( undefined, opts.plainSecrets ?? false, alreadySynced, - opts.message, + { message: opts.message }, ); } else { // App folder doesn't exist locally — delete on server @@ -4932,7 +6065,10 @@ export async function push( // Delete the entire raw app await wmill.deleteApp({ workspace: workspaceId, - path: removeSuffix(target, getDeleteSuffix("raw_app", "json")), + path: removeSuffix( + target, + getDeleteSuffix("raw_app", "json"), + ), }); } else { const rawAppFolder = extractFolderPath(target, "raw_app"); @@ -4956,7 +6092,7 @@ export async function push( undefined, opts.plainSecrets ?? false, alreadySynced, - opts.message, + { message: opts.message, defaultTs: opts.defaultTs }, ); } else { // The entire raw app folder was deleted locally, @@ -5047,7 +6183,7 @@ export async function push( const triggerInfo = extractNativeTriggerInfo(change.path); if (!triggerInfo) { throw new Error( - `Invalid native trigger path: ${change.path}` + `Invalid native trigger path: ${change.path}`, ); } await wmill.deleteNativeTrigger({ @@ -5065,8 +6201,13 @@ export async function push( path: variablePath, }); } catch (e: any) { - if (e?.status === 404 && deletedVarsResPaths.includes(variablePath)) { - log.debug(`Variable ${variablePath} already deleted by linked resource`); + if ( + e?.status === 404 && + deletedVarsResPaths.includes(variablePath) + ) { + log.debug( + `Variable ${variablePath} already deleted by linked resource`, + ); } else { throw e; } @@ -5177,10 +6318,14 @@ export async function push( log.warn(`Failed to push shared UI folder: ${e}`); } try { - await offerToRunNewMigrations(workspace.workspaceId, newDatatableMigrations, { - yes: opts.yes, - jsonOutput: opts.jsonOutput, - }); + await offerToRunNewMigrations( + workspace.workspaceId, + newDatatableMigrations, + { + yes: opts.yes, + jsonOutput: opts.jsonOutput, + }, + ); } catch (e: any) { log.warn( `Failed to run new datatable migrations: ${e?.body ?? e?.message ?? e}`, @@ -5314,7 +6459,10 @@ const command = new Command() .option("--json", "Use JSON instead of YAML") .option("--skip-variables", "Skip syncing variables (including secrets)") .option("--skip-secrets", "Skip syncing only secrets variables") - .option("--include-secrets", "Include secrets in sync (overrides skipSecrets in wmill.yaml)") + .option( + "--include-secrets", + "Include secrets in sync (overrides skipSecrets in wmill.yaml)", + ) .option("--skip-resources", "Skip syncing resources") .option("--skip-resource-types", "Skip syncing resource types") .option("--skip-scripts", "Skip syncing scripts") @@ -5370,7 +6518,10 @@ const command = new Command() .option("--json", "Use JSON instead of YAML") .option("--skip-variables", "Skip syncing variables (including secrets)") .option("--skip-secrets", "Skip syncing only secrets variables") - .option("--include-secrets", "Include secrets in sync (overrides skipSecrets in wmill.yaml)") + .option( + "--include-secrets", + "Include secrets in sync (overrides skipSecrets in wmill.yaml)", + ) .option("--skip-resources", "Skip syncing resources") .option("--skip-resource-types", "Skip syncing resource types") .option("--skip-scripts", "Skip syncing scripts") @@ -5424,7 +6575,10 @@ const command = new Command() "--locks-required", "Fail if scripts or flow inline scripts that need locks have no locks", ) - .option("--auto-metadata", "Automatically regenerate stale metadata (locks and schemas) before pushing") + .option( + "--auto-metadata", + "Automatically regenerate stale metadata (locks and schemas) before pushing", + ) .option( "--accept-overriding-permissioned-as-with-self", "Accept that items with a different permissioned_as will be updated with your own user", @@ -5463,7 +6617,7 @@ const command = new Command() ) .option( "--dev-workspace-label ", - "Environment label of a dev workspace (dev/staging); its deploys go to that branch", + "Environment label of a dev workspace (dev, staging, uat, ...); its deploys go to that branch", ) .option( "--parent-dev-workspace-label ", diff --git a/cli/src/commands/workspace/merge.ts b/cli/src/commands/workspace/merge.ts index 585772a9d0..9f39bec52c 100644 --- a/cli/src/commands/workspace/merge.ts +++ b/cli/src/commands/workspace/merge.ts @@ -2,7 +2,7 @@ import { GlobalOptions } from "../../types.ts"; import { colors } from "@cliffy/ansi/colors"; import { Table } from "@cliffy/table"; import * as log from "../../core/log.ts"; -import { setClient } from "../../core/client.ts"; +import { markRequestsAsSyncOrigin, setClient } from "../../core/client.ts"; import { tryResolveBranchWorkspace } from "../../core/context.ts"; import * as wmill from "../../../gen/services.gen.ts"; import { OpenAPI } from "../../../gen/core/OpenAPI.ts"; @@ -534,6 +534,12 @@ async function mergeWorkspaces( direction === "to-parent" ? parentWorkspaceId : forkWorkspaceId; // 10. Deploy + // Updating the fork copies the parent's state in, so nothing it writes there — + // least of all a deletion — may be read as the fork's own decision. Merging the + // other way stays authored: someone chose those changes for the target. + if (direction === "to-fork") { + markRequestsAsSyncOrigin(); + } let successCount = 0; let failCount = 0; // Datatable migrations deployed (not deleted) into the target. Deploying a diff --git a/cli/src/core/client.ts b/cli/src/core/client.ts index 4b0b98e30f..32776bd498 100644 --- a/cli/src/core/client.ts +++ b/cli/src/core/client.ts @@ -1,5 +1,30 @@ import { OpenAPI } from "../../gen/index.ts"; +/** + * Mark every subsequent request as applying a state computed elsewhere rather + * than authoring one in the target workspace (the fork tally reads the header to + * decide whether a removal was deliberate). + * + * Belongs to the commands that apply and nothing that authors: `sync push` + * (including the git-sync auto-pull, which runs it inside a job) and the + * parent-to-fork half of `workspace merge`. + */ +export function markRequestsAsSyncOrigin() { + const existing = typeof OpenAPI.HEADERS === "object" ? OpenAPI.HEADERS : {}; + OpenAPI.HEADERS = { ...existing, "X-Windmill-Deploy-Origin": "sync" }; +} + +/** + * Name this process as the CLI on every subsequent request, so a trigger the + * CLI created or disabled is attributed to `cli` rather than to a bare API + * call in `trigger_history`. Attribution only — nothing on the server grants + * anything on the strength of it. + */ +export function markRequestsAsCliClient() { + const existing = typeof OpenAPI.HEADERS === "object" ? OpenAPI.HEADERS : {}; + OpenAPI.HEADERS = { ...existing, "X-Windmill-Client": "cli" }; +} + export function setClient(token?: string, baseUrl?: string) { if (baseUrl === undefined) { baseUrl = process.env["BASE_INTERNAL_URL"] ?? diff --git a/cli/src/core/conf.ts b/cli/src/core/conf.ts index 0129a4fd18..1fcce7a43f 100644 --- a/cli/src/core/conf.ts +++ b/cli/src/core/conf.ts @@ -78,6 +78,7 @@ export interface SyncOptions { skipResourceTypes?: boolean; skipSecrets?: boolean; skipWorkspaceDependencies?: boolean; + skipDatatableMigrations?: boolean; skipScripts?: boolean; skipFlows?: boolean; skipApps?: boolean; @@ -108,6 +109,7 @@ export interface SyncOptions { promotion?: string; lint?: boolean; locksRequired?: boolean; + dedupeLockfiles?: boolean; syncBehavior?: string; } @@ -343,6 +345,7 @@ export const DEFAULT_SYNC_OPTIONS: Readonly< | "includeSchedules" | "includeTriggers" | "skipWorkspaceDependencies" + | "skipDatatableMigrations" | "skipScripts" | "skipFlows" | "skipApps" @@ -375,6 +378,7 @@ export const DEFAULT_SYNC_OPTIONS: Readonly< includeSettings: false, includeKey: false, skipWorkspaceDependencies: false, + skipDatatableMigrations: false, nonDottedPaths: false, syncBehavior: "v1", } as const; diff --git a/cli/src/core/constants.ts b/cli/src/core/constants.ts index 59de21e2bf..64b77a690d 100644 --- a/cli/src/core/constants.ts +++ b/cli/src/core/constants.ts @@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork"; // (e.g. utils.ts) can read it without importing main.ts and creating a circular // dependency (main → workspace → utils → main) that triggers a TDZ. // Re-exported from main.ts for backwards compatibility. -export const VERSION = "1.775.2"; +export const VERSION = "1.795.0"; diff --git a/cli/src/core/context.ts b/cli/src/core/context.ts index 6440213d5c..502157f0a7 100644 --- a/cli/src/core/context.ts +++ b/cli/src/core/context.ts @@ -22,8 +22,11 @@ import { readConfigFile, findWorkspaceByGitBranch, getEffectiveWorkspaceId, + getWmillYamlPath, WorkspaceEntryConfig, } from "./conf.ts"; +import { existsSync, realpathSync } from "node:fs"; +import { basename, dirname, isAbsolute, join, relative, resolve } from "node:path"; import { getCurrentGitBranch, getOriginalBranchForWorkspaceForks, @@ -743,6 +746,92 @@ export async function tryResolveVersion( } } +/** + * Directory the local tree mirrors the workspace from: the one holding + * wmill.yaml. Not `process.cwd()` — that only lands there once a config read + * has chdir'd into it, which `--remote` and fully-flagged invocations skip. + */ +function syncRoot(): string { + const wmillYaml = getWmillYamlPath(); + return wmillYaml ? dirname(wmillYaml) : process.cwd(); +} + +/** + * Re-express a user-supplied file or folder argument as a path relative to the + * sync root, so the Windmill path derived from it is the same whatever shape + * the argument had (`./f/a/b.ts`, `/abs/repo/f/a/b.ts`, `b.ts` from inside + * `f/a`). + * + * `cwdBeforeConfig` must be the working directory as it was *before* the + * command read wmill.yaml: reading it chdirs into the directory holding it, and + * a relative argument was written against the directory the user was in. + * Arguments are resolved against that directory first and the sync root second, + * so both readings work. + * + * Resolving the sync root leaves the process in it, which is what makes the + * returned path readable by the caller — keep that in step if this ever stops + * going through `getWmillYamlPath`. + */ +export function toSyncRootRelativePath( + arg: string, + cwdBeforeConfig: string +): string { + const root = syncRoot(); + const candidates = isAbsolute(arg) + ? [arg] + : [resolve(cwdBeforeConfig, arg), resolve(root, arg)]; + // A descriptor-less dbt project is named by a file that is deliberately not + // there, so an argument existing under neither reading is still a real one if + // the directory holding it is; only a path whose directory is missing too + // falls through to the sync-root reading for the caller to reject. + const abs = + candidates.find((c) => existsSync(c)) ?? + candidates.find((c) => existsSync(dirname(c))) ?? + candidates.at(-1)!; + const rel = relative(root, abs); + if (rel === "") return "."; + if (!rel.startsWith("..")) return rel; + // `relative` is purely lexical, so a root reached through a symlink (macOS' + // /var -> /private/var, a symlinked checkout) makes an absolute argument look + // like it escapes the tree. Resolve links only then, so a symlinked file + // *inside* the tree keeps the path it is filed under. + try { + const resolved = relative(realpathSync(root), realpathOfNamed(abs)); + return resolved === "" ? "." : resolved; + } catch { + return rel; + } +} + +/** + * `realpathSync` needs its target to exist, and a dbt descriptor deliberately + * does not. The directory naming it does, so resolve that and reattach. + */ +function realpathOfNamed(p: string): string { + return existsSync(p) + ? realpathSync(p) + : join(realpathSync(dirname(p)), basename(p)); +} + +/** Windmill workspace path: `u|f|g` followed by at least a folder and a name. */ +const REMOTE_PATH_RE = /^[ufg](\/[^/]+){2,}$/; + +/** + * Guard the Windmill path a preview run is pushed under. A preview job carries + * no runnable of its own, so this path is the only identity it has: it is what + * `WM_JOB_PATH` reports, what the runs page links to, and what relative imports + * inside the previewed code resolve against. + */ +export function assertRemotePath(remotePath: string, arg: string): void { + if (REMOTE_PATH_RE.test(remotePath)) return; + throw new Error( + `Cannot derive a Windmill path from '${arg}'` + + (remotePath ? ` (it maps to '${remotePath}')` : "") + + `: a preview runs under the path of the file it previews, which must sit inside the ` + + `wmill.yaml root and be of the form //.` + ); +} + export function validatePath(path: string): boolean { if (!(path.startsWith("g") || path.startsWith("u") || path.startsWith("f"))) { log.infoStderr( diff --git a/cli/src/core/specific_items.ts b/cli/src/core/specific_items.ts index 6e64381876..cbfd52530e 100644 --- a/cli/src/core/specific_items.ts +++ b/cli/src/core/specific_items.ts @@ -429,6 +429,15 @@ const workspacePatternCache = new Map(); * workspaceNameOverride is the effective git branch name (for file naming on disk). */ export function isCurrentWorkspaceFile(path: string, workspaceNameOverride?: string): boolean { + // Fileset children are never workspace-specific: only the parent's metadata + // file carries the suffix, children stay in the shared canonical directory. + // Their names are arbitrary, so one can look like typed metadata — and since + // the patterns match a workspace-name segment with `[^.]` (which spans `/`), + // an unguarded child is dropped from the diff or, worse, remapped onto a + // sibling's key and deployed over it. + if (isFilesetResource(path)) { + return false; + } let currentWorkspace: string | null = null; if (workspaceNameOverride) { currentWorkspace = workspaceNameOverride; @@ -464,6 +473,10 @@ export function isCurrentWorkspaceFile(path: string, workspaceNameOverride?: str * Used to identify and skip files from other branches during sync operations */ export function isWorkspaceSpecificFile(path: string): boolean { + // Same fileset exemption as isCurrentWorkspaceFile. + if (isFilesetResource(path)) { + return false; + } const typePattern = buildItemTypePattern(); return new RegExp( `\\.[^.]+\\.${typePattern}\\.(yaml|json)$|` + diff --git a/cli/src/guidance/skills.gen.ts b/cli/src/guidance/skills.gen.ts index 50ac0abc9f..0f57c21463 100644 --- a/cli/src/guidance/skills.gen.ts +++ b/cli/src/guidance/skills.gen.ts @@ -547,6 +547,22 @@ const result: wmill.S3Object = await wmill.writeS3File( Import: import * as wmill from 'windmill-client' +The client configures itself from the job's environment — base URL, token and credentials mode +are all set before your code runs, so there is nothing to initialize and no reason to read +WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only +reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw +HTTP for third-party APIs. + +The helpers below are the surface to prefer. For an endpoint none of them covers, import the +generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not +listed here but they do exist. What does not exist is a helper name you guessed at: if it is +neither listed below nor a service method, do not call it. + +To know who is running the script, read the contextual variables rather than calling the API: +\`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL\`. WM_END_USER_EMAIL is the app viewer when +the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL +is the user the job is permissioned as. WM_USERNAME is the matching username. + workerHasInternalServer(): boolean /** @@ -577,11 +593,6 @@ async getResource(path?: string, undefinedIfEmpty?: boolean): Promise */ async getRootJobId(jobId?: string): Promise -/** - * @deprecated Use runScriptByPath or runScriptByHash instead - */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise - /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill @@ -647,9 +658,12 @@ async getResult(jobId: string): Promise async getResultMaybe(jobId: string): Promise /** - * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead + * Cancel a queued or running job by ID. + * @param jobId - UUID of the job to cancel + * @param reason - Optional reason for cancellation + * @returns Response message from the cancel endpoint */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise +async cancelJob(jobId: string, reason: string | undefined = undefined): Promise /** * Run a script asynchronously by its path @@ -703,13 +717,6 @@ getStatePath(): string */ async setResource(value: any, path?: string, initializeToTypeIfNotExist?: string): Promise -/** - * Set the state - * @param state state to set - * @deprecated use setState instead - */ -async setInternalState(state: any): Promise - /** * Set the state * @param state state to set @@ -745,12 +752,6 @@ async setFlowUserState(key: string, value: any, errorIfNotPossible?: boolean): P */ async getFlowUserState(key: string, errorIfNotPossible?: boolean): Promise -/** - * Get the internal state - * @deprecated use getState instead - */ -async getInternalState(): Promise - /** * Get the state shared across executions * @param path Optional state resource path override. Defaults to \`getStatePath()\`. @@ -886,15 +887,6 @@ async getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{ cancel: string; }> -/** - * @deprecated use getResumeUrls instead - */ -getResumeEndpoints(approver?: string): Promise<{ - approvalPage: string; - resume: string; - cancel: string; -}> - /** * Get an OIDC jwt token for auth to external services (e.g: Vault, AWS) (ee only) * @param audience audience of the token @@ -917,15 +909,6 @@ base64ToUint8Array(data: string): Uint8Array */ uint8ArrayToBase64(arrayBuffer: Uint8Array): string -/** - * Get email from workspace username - * This method is particularly useful for apps that require the email address of the viewer. - * Indeed, in the viewer context, WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app. - * @param username - * @returns email address - */ -async usernameToEmail(username: string): Promise - /** * Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields. * @@ -1347,6 +1330,22 @@ const result: wmill.S3Object = await wmill.writeS3File( Import: import * as wmill from 'windmill-client' +The client configures itself from the job's environment — base URL, token and credentials mode +are all set before your code runs, so there is nothing to initialize and no reason to read +WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only +reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw +HTTP for third-party APIs. + +The helpers below are the surface to prefer. For an endpoint none of them covers, import the +generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not +listed here but they do exist. What does not exist is a helper name you guessed at: if it is +neither listed below nor a service method, do not call it. + +To know who is running the script, read the contextual variables rather than calling the API: +\`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL\`. WM_END_USER_EMAIL is the app viewer when +the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL +is the user the job is permissioned as. WM_USERNAME is the matching username. + workerHasInternalServer(): boolean /** @@ -1377,11 +1376,6 @@ async getResource(path?: string, undefinedIfEmpty?: boolean): Promise */ async getRootJobId(jobId?: string): Promise -/** - * @deprecated Use runScriptByPath or runScriptByHash instead - */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise - /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill @@ -1447,9 +1441,12 @@ async getResult(jobId: string): Promise async getResultMaybe(jobId: string): Promise /** - * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead + * Cancel a queued or running job by ID. + * @param jobId - UUID of the job to cancel + * @param reason - Optional reason for cancellation + * @returns Response message from the cancel endpoint */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise +async cancelJob(jobId: string, reason: string | undefined = undefined): Promise /** * Run a script asynchronously by its path @@ -1503,13 +1500,6 @@ getStatePath(): string */ async setResource(value: any, path?: string, initializeToTypeIfNotExist?: string): Promise -/** - * Set the state - * @param state state to set - * @deprecated use setState instead - */ -async setInternalState(state: any): Promise - /** * Set the state * @param state state to set @@ -1545,12 +1535,6 @@ async setFlowUserState(key: string, value: any, errorIfNotPossible?: boolean): P */ async getFlowUserState(key: string, errorIfNotPossible?: boolean): Promise -/** - * Get the internal state - * @deprecated use getState instead - */ -async getInternalState(): Promise - /** * Get the state shared across executions * @param path Optional state resource path override. Defaults to \`getStatePath()\`. @@ -1686,15 +1670,6 @@ async getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{ cancel: string; }> -/** - * @deprecated use getResumeUrls instead - */ -getResumeEndpoints(approver?: string): Promise<{ - approvalPage: string; - resume: string; - cancel: string; -}> - /** * Get an OIDC jwt token for auth to external services (e.g: Vault, AWS) (ee only) * @param audience audience of the token @@ -1717,15 +1692,6 @@ base64ToUint8Array(data: string): Uint8Array */ uint8ArrayToBase64(arrayBuffer: Uint8Array): string -/** - * Get email from workspace username - * This method is particularly useful for apps that require the email address of the viewer. - * Indeed, in the viewer context, WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app. - * @param username - * @returns email address - */ -async usernameToEmail(username: string): Promise - /** * Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields. * @@ -2241,6 +2207,22 @@ const result: wmill.S3Object = await wmill.writeS3File( Import: import * as wmill from 'windmill-client' +The client configures itself from the job's environment — base URL, token and credentials mode +are all set before your code runs, so there is nothing to initialize and no reason to read +WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only +reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw +HTTP for third-party APIs. + +The helpers below are the surface to prefer. For an endpoint none of them covers, import the +generated service classes (JobService, ScriptService, ...) from 'windmill-client' — they are not +listed here but they do exist. What does not exist is a helper name you guessed at: if it is +neither listed below nor a service method, do not call it. + +To know who is running the script, read the contextual variables rather than calling the API: +\`process.env.WM_END_USER_EMAIL || process.env.WM_EMAIL\`. WM_END_USER_EMAIL is the app viewer when +the run was triggered from an app and empty otherwise (both variables are always defined), WM_EMAIL +is the user the job is permissioned as. WM_USERNAME is the matching username. + workerHasInternalServer(): boolean /** @@ -2271,11 +2253,6 @@ async getResource(path?: string, undefinedIfEmpty?: boolean): Promise */ async getRootJobId(jobId?: string): Promise -/** - * @deprecated Use runScriptByPath or runScriptByHash instead - */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise - /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill @@ -2341,9 +2318,12 @@ async getResult(jobId: string): Promise async getResultMaybe(jobId: string): Promise /** - * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead + * Cancel a queued or running job by ID. + * @param jobId - UUID of the job to cancel + * @param reason - Optional reason for cancellation + * @returns Response message from the cancel endpoint */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise +async cancelJob(jobId: string, reason: string | undefined = undefined): Promise /** * Run a script asynchronously by its path @@ -2397,13 +2377,6 @@ getStatePath(): string */ async setResource(value: any, path?: string, initializeToTypeIfNotExist?: string): Promise -/** - * Set the state - * @param state state to set - * @deprecated use setState instead - */ -async setInternalState(state: any): Promise - /** * Set the state * @param state state to set @@ -2439,12 +2412,6 @@ async setFlowUserState(key: string, value: any, errorIfNotPossible?: boolean): P */ async getFlowUserState(key: string, errorIfNotPossible?: boolean): Promise -/** - * Get the internal state - * @deprecated use getState instead - */ -async getInternalState(): Promise - /** * Get the state shared across executions * @param path Optional state resource path override. Defaults to \`getStatePath()\`. @@ -2580,15 +2547,6 @@ async getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{ cancel: string; }> -/** - * @deprecated use getResumeUrls instead - */ -getResumeEndpoints(approver?: string): Promise<{ - approvalPage: string; - resume: string; - cancel: string; -}> - /** * Get an OIDC jwt token for auth to external services (e.g: Vault, AWS) (ee only) * @param audience audience of the token @@ -2611,15 +2569,6 @@ base64ToUint8Array(data: string): Uint8Array */ uint8ArrayToBase64(arrayBuffer: Uint8Array): string -/** - * Get email from workspace username - * This method is particularly useful for apps that require the email address of the viewer. - * Indeed, in the viewer context, WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app. - * @param username - * @returns email address - */ -async usernameToEmail(username: string): Promise - /** * Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields. * @@ -3965,6 +3914,22 @@ result: S3Object = wmill.write_s3_file( Import: import wmill +The client configures itself from the job's environment — base URL, token and credentials mode +are all set before your code runs, so there is nothing to initialize and no reason to read +WM_TOKEN or BASE_INTERNAL_URL and build an API URL yourself. Reconstructing that by hand only +reintroduces details the client already handles. Call the SDK for anything Windmill, and use raw +HTTP for third-party APIs. + +The functions below are the surface to prefer. For an endpoint none of them covers, +wmill.Windmill().get(endpoint) and .post(endpoint) issue an authenticated request against this +instance. What does not exist is a function name you guessed at: if it is not listed below, do +not call it. + +To know who is running the script, read the contextual variables rather than calling the API: +\`os.environ.get("WM_END_USER_EMAIL") or os.environ.get("WM_EMAIL")\`. WM_END_USER_EMAIL is the app +viewer when the run was triggered from an app and empty otherwise (both variables are always +defined), WM_EMAIL is the user the job is permissioned as. WM_USERNAME is the matching username. + def worker_has_internal_server() -> bool def get_mocked_api() -> Optional[dict] @@ -4006,11 +3971,6 @@ def post(endpoint, raise_for_status = True, **kwargs) -> httpx.Response # New authentication token string def create_token(duration = dt.timedelta(days=1)) -> str -# Create a script job and return its job id. -# -# .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. -def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str - # Create a script job by path and return its job id. def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str @@ -4020,11 +3980,6 @@ def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: i # Create a flow job and return its job id. def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str -# Run script synchronously and return its result. -# -# .. deprecated:: Use run_script_by_path or run_script_by_hash instead. -def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any - # Run script by path synchronously and return its result. def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any @@ -4411,11 +4366,6 @@ def get_approval_urls(step_key: str = 'approval', approver: str = None) -> dict # - The function checks for required environment variables (\`WM_FLOW_JOB_ID\`, \`WM_FLOW_STEP_ID\`) to ensure it is run in the appropriate context. def request_interactive_slack_approval(slack_resource_path: str, channel_id: str, message: str = None, approver: str = None, default_args_json: dict = None, dynamic_enums_json: dict = None) -> None -# Get email from workspace username -# This method is particularly useful for apps that require the email address of the viewer. -# Indeed, in the viewer context WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app. -def username_to_email(username: str) -> str - # Send a message to a Microsoft Teams conversation with conversation_id, where success is used to style the message def send_teams_message(conversation_id: str, text: str, success: bool = True, card_block: dict = None) @@ -4449,6 +4399,18 @@ def get_workspace() -> str def get_version() -> str +# Create a script job and return its job ID. +# +# Args: +# hash_or_path: Script hash or path (determined by presence of '/') +# args: Script arguments +# scheduled_in_secs: Delay before execution in seconds +# tag: Override the worker tag the job runs on +# +# Returns: +# Job ID string +def run_script_async(hash_or_path: str, args: Dict[str, Any] = None, scheduled_in_secs: int = None, tag: str = None) -> str + # Run a script synchronously by hash and return its result. # # Args: @@ -5206,11 +5168,75 @@ value: - Use underscores, not spaces (e.g., \`fetch_data\` not \`fetch data\`) - Use descriptive names that reflect the step's purpose +## AI Agent Modules + +An \`aiagent\` module runs an LLM that can call tools. Each entry of \`value.tools\` is a module-shaped +object with an extra \`value.tool_type\`: \`flowmodule\` for a script/flow tool, \`mcp\` for an MCP server +tool, \`websearch\` for web search. + +\`\`\`json +{ + "id": "support_agent", + "summary": "AI agent for customer support", + "value": { + "type": "aiagent", + "input_transforms": { + "provider": { + "type": "static", + "value": { "kind": "openai", "resource": "$res:f/ai_providers/openai", "model": "gpt-4o" } + }, + "output_type": { "type": "static", "value": "text" }, + "user_message": { "type": "javascript", "expr": "flow_input.query" }, + "system_prompt": { "type": "static", "value": "You are a helpful assistant." } + }, + "tools": [ + { + "id": "search_docs", + "summary": "search_documentation", + "description": "Search the product documentation. Use it whenever the user asks how a feature works.", + "value": { + "tool_type": "flowmodule", + "type": "rawscript", + "language": "bun", + "content": "export async function main(query: string) { return ['doc1', 'doc2']; }", + "input_transforms": { "query": { "type": "static", "value": "" } } + } + } + ] + } +} +\`\`\` + +- \`provider\` is a static object, not a bare resource string: \`{ "kind": , + "resource": "$res:", "model": }\`. Required unless the module links to a saved + agent through \`value.agent\` + +### Tool Naming Rules + +These rules cover \`flowmodule\` tools, the ones the agent calls by name. A \`websearch\` tool's +\`summary\` is a plain label (\`Web Search\`), and an \`mcp\` tool exposes the MCP server's own tool +names, so neither is name-checked at all — leave those summaries as they are. + +- A flowmodule tool's \`summary\` is the **name the agent calls it by**, not a human label. Put the + human-readable explanation in \`description\` +- \`summary\` must match \`^[a-zA-Z0-9_]+$\`: letters, numbers and underscores only. No spaces, dashes, + dots or accents — \`search_documentation\`, never \`Search documentation\` +- Always set \`summary\`. It must be unique among that agent's tools, and must not be one of the + reserved ids (\`do\`, \`bg\`, \`ctx\`, \`state\`, \`if\`, \`else\`, \`for\`, \`delete\`, \`while\`, \`new\`, \`in\`, + \`failure\`, \`preprocessor\`, \`as\`, \`Input\`, \`Result\`, \`Trigger\`) +- A tool name outside that character set is rejected: flow write tools refuse it, and a flow that + reaches the worker with one fails every run with \`Invalid tool name\` +- Tool \`id\` follows the same rules as any module ID — unique across the flow, underscores not spaces +- \`description\` is optional free text telling the agent when and how to call the tool. Set it + whenever the name alone does not make that obvious; it overrides the description derived from the + underlying script + ## Common Mistakes to Avoid - Missing \`input_transforms\` - Rawscript parameters won't receive values without them - Referencing future steps - \`results.step_id\` only works for steps that execute before the current one - Duplicate module IDs - Each module ID must be unique in the flow +- AI agent flowmodule tool names with spaces - \`summary\` is the tool name and only accepts letters, numbers and underscores ## Data Flow Between Steps @@ -5429,6 +5455,7 @@ Before finalizing a flow, verify: - any failure handler is in \`value.failure_module\` - any approval step has module-level \`suspend\` - no downstream step references inner branch step ids from outside the branch +- every AI agent flowmodule tool has a unique \`summary\` made only of letters, numbers and underscores ## S3 Object Operations @@ -5486,7 +5513,7 @@ Reference a specific resource using \`$res:\` prefix: ## OpenFlow Schema -{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \\"yellow\\", \\"#ffff00\\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"description":{"type":"string","description":"Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script."},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\\nRange: 1-1000.\\n"}},"required":["user_message"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default \`flow\`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"agent":{"type":"string","description":"Path of a reusable \`ai_agent\` resource (hybrid linking). When set, the agent brain\\nconfig (provider/model/system prompt/etc.) and tool set are resolved at runtime from\\nthat resource; the module's input_transforms then only carry the flow-local inputs\\n(user_message/user_attachments).\\n"},"tool_inputs":{"type":"object","description":"Host-local wiring for an agent's tool inputs, keyed by tool id then input key. Binds the\\nreferenced agent's tools to this flow's context (flow_input/results) without mutating the\\nshared resource; overlaid onto the tools' input_transforms at runtime \\u2014 including when\\n\`agent\` is unset, since a step forked for editing keeps these overrides until it is saved\\nback or unlinked.\\n","additionalProperties":{"type":"object","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}}},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}`, +{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"Address of the account the flow runs on behalf of. Derived from on_behalf_of on read; accepted on write, where it is resolved to the account it names."},"on_behalf_of":{"type":"string","description":"The flow runs with the permissions of this identity: u/{username}, g/{group}, or a bare email when the username is itself email-shaped. The only stored half of the identity; on_behalf_of_email is derived from it. Omit it when writing and it is resolved from that address instead."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"preserve_step_tags":{"type":"boolean","description":"If true and the flow runs on a custom worker tag, steps that declare their own non-empty tag run on it instead of inheriting the flow tag. Steps without their own tag still inherit the flow tag."},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"delete_after_secs":{"type":"integer","description":"If set, delete the flow job's args, result and logs after this many seconds following job completion"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \\"yellow\\", \\"#ffff00\\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."},"error_include_result":{"type":"boolean","description":"When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_secs":{"type":"integer","description":"If set, delete the step's args, result and logs after this many seconds following job completion"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside for loops, use 'flow_input.iter.value' for the current iteration value (in while loops it equals 'flow_input.iter.index')","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","azure_foundry","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"},"reasoning_effort":{"type":"string","description":"Provider-native reasoning effort token (e.g. 'low', 'high', 'none') for models that support extended thinking. Optional; unset leaves the provider default."}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","bunnative","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","rlang","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume","dbt"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly until stopped. The implicit iterator is the iteration counter, so 'flow_input.iter.value' equals 'flow_input.iter.index' (0, 1, 2, ...) and never carries state. To carry state across iterations, a step reads its own previous-iteration result via 'results.' with a first-iteration fallback - the loop's stop_after_if must then be on that inner step (a plain single-step body with stop_after_if on the loop module does not resolve 'results' across iterations and never terminates); plain counters can instead be derived from 'flow_input.iter.index', which works in every configuration. stop_after_if is evaluated after each iteration - on the loop module 'result' is the last iteration's result","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"The name the AI agent calls this tool by, not a human label. On a flowmodule tool it must match ^[a-zA-Z0-9_]+$ - letters, numbers and underscores only (e.g. 'search_documentation', not 'Search documentation') - and always be set; on an mcp or websearch tool it is a plain label. Put the human-readable explanation in 'description'."},"description":{"type":"string","description":"Free-text description of the tool given to the AI to decide when and how to call it. Overrides the description auto-derived from the underlying script."},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, reasoning_token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n"},"max_iterations":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Number. Limits how many times the agent can loop through reasoning and tool use.\\nRange: 1-1000.\\n"}},"required":["user_message"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"tag":{"type":"string","description":"Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default \`flow\`)"},"omit_output_from_conversation":{"type":"boolean","default":false,"description":"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled."},"agent":{"type":"string","description":"Path of a reusable \`ai_agent\` resource (hybrid linking). When set, the agent brain\\nconfig (provider/model/system prompt/etc.) and tool set are resolved at runtime from\\nthat resource; the module's input_transforms then only carry the flow-local inputs\\n(user_message/user_attachments).\\n"},"tool_inputs":{"type":"object","description":"Host-local wiring for an agent's tool inputs, keyed by tool id then input key. Binds the\\nreferenced agent's tools to this flow's context (flow_input/results) without mutating the\\nshared resource; overlaid onto the tools' input_transforms at runtime \\u2014 including when\\n\`agent\` is unset, since a step forked for editing keeps these overrides until it is saved\\nback or unlinked.\\n","additionalProperties":{"type":"object","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}}},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}`, "raw-app": `--- name: raw-app description: MUST use when creating raw apps. @@ -5762,16 +5789,50 @@ The frontend imports a generated module that mirrors the backend runnables. **Ne ### Calling backend runnables -Import the generated bindings and call the runnable like a function: +Import the generated bindings and call the runnable like a function. \`./wmill\` is the **only** way the frontend reaches anything server-side — datatables, workspace items, external services. Never \`fetch\` the Windmill API from frontend code: the bundle holds no token and builds no API URL. -\`\`\`typescript +| Export | Resolves to | Use it for | +|---|---|---| +| \`backend.(args)\` | the runnable's result | the default — run and wait | +| \`backendAsync.(args)\` | the **job id** (a string) | long-running work you want to track | +| \`waitJob(jobId)\` | the job's **result** (rejects if the job failed) | awaiting a \`backendAsync\` job | +| \`getJob(jobId)\` | a \`Job\` (\`{ type, success, result, duration_ms, ... }\`) | polling status without blocking | +| \`streamJob(jobId, onUpdate?)\` | the final result, calling \`onUpdate\` per chunk | showing output as it is produced | + +Run and wait — the common case: + +\`\`\`tsx import { backend } from './wmill'; -// Call a backend runnable const user = await backend.get_user({ user_id: '123' }); \`\`\` -The frontend cannot reach datatables, workspace items, or external services on its own — it goes through \`backend.(args)\` for everything server-side. +Start a long job, then await it: + +\`\`\`tsx +import { backendAsync, waitJob } from './wmill'; + +const jobId = await backendAsync.run_report({ month: '2026-08' }); // a string +const report = await waitJob(jobId); // the result itself +\`\`\` + +Or poll it without blocking, to render progress: + +\`\`\`tsx +import { getJob } from './wmill'; + +const job = await getJob(jobId); +if (job.type === 'CompletedJob') setReport(job.result); +\`\`\` + +\`backendAsync\` resolves a job id and nothing else — guard on it before storing or polling. A poll loop started on an \`undefined\` id never completes and shows as a row stuck "running" forever: + +\`\`\`tsx +const jobId = await backendAsync.run_report(args); +if (!jobId) throw new Error('run_report did not start a job'); +\`\`\` + +**Never hand-write a job-polling runnable.** A backend runnable that calls \`jobs/list\`, or that returns \`getResultMaybe(...)\` for the frontend to poll, reimplements \`backendAsync\` + \`waitJob\` / \`getJob\` / \`streamJob\` — and it is what leads to guessing at base URLs and tokens. ### Keeping data out of recorded demos @@ -5823,10 +5884,33 @@ def main(user_id: str): return user \`\`\` +#### The \`wmill\` client is already authenticated + +An inline runnable runs as an ordinary Windmill job. \`import * as wmill from 'windmill-client'\` (TypeScript) and \`import wmill\` (Python) are already pointed at this instance and this workspace — there is nothing to configure. + +**Don't read \`WM_TOKEN\` or \`BASE_INTERNAL_URL\` and build an API URL to \`fetch\`.** The client's own \`setClient\` already reads exactly those, and it also sets the credentials mode a raw app needs (\`WM_RAW_APP\` suppresses credentials, because a sandboxed bundle calls the API from an opaque origin that can never pair with \`Access-Control-Allow-Origin: *\`). Rebuilding that by hand drops the parts you can't see. Use \`wmill.*\` for everything Windmill, and \`fetch\` only for third-party APIs. + +Prefer the \`wmill\` functions that appear in the SDK reference; for an endpoint none of them covers, the generated service classes (\`JobService\`, \`ScriptService\`, ...) are importable from \`windmill-client\`. What is not available is a name you guessed at: \`getBaseUrl\` and \`getWorkspaceToken\` are inventions, not API. + ### Path runnables (script / flow / hubscript) When \`type\` is \`script\`, \`flow\`, or \`hubscript\`, the runnable just stores a \`path\` to an existing workspace or hub item — no inline code. The referenced item's input/output schema becomes the runnable's surface. +### Draft code vs deployed code + +This decides whether an app works before anything is deployed: + +- **Inline runnables run the app's current code.** The editor sends the runnable's source with each request, so an inline runnable works in the preview with nothing deployed. +- **Path runnables (\`script\` / \`flow\` / \`hubscript\`) run the DEPLOYED item at that path.** So do \`wmill.runFlow\`, \`wmill.runFlowAsync\` and \`wmill.runScriptByPath\` called from inside a runnable. A draft — including a draft you just created — does not exist for them. + +So an app wired to a flow you just wrote does nothing until **that flow is deployed**. The app itself does NOT have to be deployed for this: the preview runs the app's draft, so the referenced flow is the only thing that has to exist deployed. + +That makes the fix a one-item deploy, not a release. Offer to deploy exactly the referenced flow or script and leave the app a draft the user keeps testing in the preview — do not push the whole change set through the review-and-deploy page, and do not ask the user to deploy the app, unless they said they want to ship it. + +Do NOT quietly reimplement the flow inside an inline runnable to dodge the deployment: that leaves the user with two copies of the same logic and an app that ignores the flow they asked for. Inline the logic only when the user actually wants it inline. + +Prefer a **path runnable of type \`flow\`** over an inline runnable that calls \`wmill.runFlowAsync\`. The path runnable gives the frontend the flow's real input schema and works with \`backend\` / \`backendAsync\` / \`waitJob\` like any other runnable; a hand-written wrapper gives up all of that. + ### Static inputs \`staticInputs\` is an optional \`Record\` for arguments not overridable from the frontend. Useful with path runnables to pre-fill some args while leaving the rest to the frontend caller. @@ -5888,6 +5972,8 @@ def main(user_id: str): 4. **Use descriptive keys** — \`get_user\`, not \`a\`. 5. **Always whitelist tables** — adding a runnable that queries a new table requires the table to be in \`data.tables\` first. 6. **Mark sensitive UI with \`data-wm-no-record\`** — it is what keeps that data out of a recorded demo; passwords are handled for you. +7. **Reach for \`backendAsync\` + \`waitJob\`** for long work — never a hand-written job-polling runnable. +8. **Deploy what a path runnable points at** — a path runnable aimed at a draft fails at runtime; tell the user what needs deploying. `, "triggers": `--- name: triggers @@ -6840,8 +6926,12 @@ app related commands - \`--host \` - Host to bind the dev server to - \`--entry \` - Entry point file (default: index.ts for Svelte/Vue, index.tsx otherwise) - \`--no-open\` - Don't automatically open the browser + - \`--recording\` - Frame the app in a shell with a Record button, to capture a replayable session recording of the app under development - \`app lint [app_folder:string]\` - Lint a raw app folder to validate structure and buildability - \`--fix\` - Attempt to fix common issues (not implemented yet) +- \`app bundle [app_folder:string]\` - Bundle a raw app folder to js/css without deploying it + - \`--out \` - Directory to write bundle.js and bundle.css into (default: /dist) + - \`--no-minify\` - Skip minification - \`app new\` - create a new raw app from a template - \`--summary \` - App summary (short description). Skips the prompt when provided. Triggers non-interactive mode. - \`--path \` - App path (e.g., f/folder/my_app or u/username/my_app). Skips the prompt when provided. Triggers non-interactive mode. @@ -8087,13 +8177,21 @@ properties: gcp_resource_path: type: string description: Path to the GCP resource containing service account credentials for - authentication. + authentication. Omit to authenticate with the instance's application default + credentials. + project_id: + type: string + description: GCP project the client operates in. Defaults to the project of the + credentials. Topics and subscriptions given as fully qualified names are reached + whatever it is. topic_id: type: string - description: Google Cloud Pub/Sub topic ID to subscribe to. + description: Google Cloud Pub/Sub topic ID to subscribe to. Accepts a bare ID + or a fully qualified name (projects//topics/). subscription_id: type: string - description: Google Cloud Pub/Sub subscription ID. + description: Google Cloud Pub/Sub subscription ID. Accepts a bare ID or a fully + qualified name (projects//subscriptions/). delivery_type: type: string enum: @@ -8163,7 +8261,6 @@ properties: required: - script_path - is_flow -- gcp_resource_path - topic_id - subscription_id - delivery_type @@ -8365,18 +8462,62 @@ properties: filters: type: array items: - type: object - properties: - key: - type: string - value: {} + oneOf: + - type: object + properties: + key: + type: string + value: {} + required: + - key + - value + - type: object + properties: + path: + type: string + description: Dotted path into nested objects, e.g. \`a.b.c\`. Does not traverse + arrays. + value: {} + required: + - path + - value + - type: object + properties: + any_of: + type: array + items: + type: object + required: + - any_of + - type: object + properties: + all_of: + type: array + items: + type: object + required: + - all_of + - type: object + properties: + none_of: + type: array + items: + type: object + required: + - none_of + description: 'Filters to match incoming messages (only matching messages trigger + the script). Each entry is either a leaf \`{key, value}\` (top-level field) or + \`{path, value}\` (dotted path into nested objects), or a group \`{any_of: [...]}\` + / \`{all_of: [...]}\` / \`{none_of: [...]}\` nesting more entries. Entries at the + top level are combined with \`filter_logic\`.' filter_logic: type: string enum: - and - or - description: Logic to apply when evaluating filters. 'and' requires all filters - to match, 'or' requires any filter to match. + description: Logic to apply when evaluating the top-level filters. 'and' requires + all of them to match, 'or' requires any of them to match. Nested \`any_of\`/\`all_of\`/\`none_of\` + groups carry their own logic. auto_offset_reset: type: string enum: @@ -8477,6 +8618,18 @@ properties: type: array items: type: object + properties: + qos: + type: string + enum: + - qos0 + - qos1 + - qos2 + topic: + type: string + required: + - qos + - topic description: Array of MQTT topics to subscribe to, each with topic name and QoS level v3_config: @@ -9021,24 +9174,91 @@ properties: filters: type: array items: - type: object - properties: - key: - type: string - value: {} - description: Array of key-value filters to match incoming messages (only matching - messages trigger the script) + oneOf: + - type: object + properties: + key: + type: string + value: {} + required: + - key + - value + - type: object + properties: + path: + type: string + description: Dotted path into nested objects, e.g. \`a.b.c\`. Does not traverse + arrays. + value: {} + required: + - path + - value + - type: object + properties: + any_of: + type: array + items: + type: object + required: + - any_of + - type: object + properties: + all_of: + type: array + items: + type: object + required: + - all_of + - type: object + properties: + none_of: + type: array + items: + type: object + required: + - none_of + description: 'Filters to match incoming messages (only matching messages trigger + the script). Each entry is either a leaf \`{key, value}\` (top-level field) or + \`{path, value}\` (dotted path into nested objects), or a group \`{any_of: [...]}\` + / \`{all_of: [...]}\` / \`{none_of: [...]}\` nesting more entries. Entries at the + top level are combined with \`filter_logic\`.' filter_logic: type: string enum: - and - or - description: Logic to apply when evaluating filters. 'and' requires all filters - to match, 'or' requires any filter to match. + description: Logic to apply when evaluating the top-level filters. 'and' requires + all of them to match, 'or' requires any of them to match. Nested \`any_of\`/\`all_of\`/\`none_of\` + groups carry their own logic. initial_messages: type: array items: - type: object + oneOf: + - type: object + properties: + raw_message: + type: string + required: + - raw_message + - type: object + properties: + runnable_result: + type: object + properties: + path: + type: string + args: + type: object + description: The arguments to pass to the script or flow + additionalProperties: true + is_flow: + type: boolean + required: + - path + - args + - is_flow + required: + - runnable_result description: Messages to send immediately after connecting (can be raw strings or computed by runnables) url_runnable_args: diff --git a/cli/src/main.ts b/cli/src/main.ts index ff8c576ee9..ccce20da30 100755 --- a/cli/src/main.ts +++ b/cli/src/main.ts @@ -32,6 +32,7 @@ import { OpenAPI } from "../gen/index.ts"; import { getHeaders } from "./utils/utils.ts"; import { detectAuthGatewayChallenge } from "./utils/http_guards.ts"; import { setShowDiffs } from "./core/conf.ts"; +import { markRequestsAsCliClient } from "./core/client.ts"; import { NpmProvider } from "./utils/upgrade.ts"; import { pull as hubPull } from "./commands/hub/hub.ts"; import { pull, push } from "./commands/sync/sync.ts"; @@ -300,6 +301,7 @@ async function main() { if (extraHeaders) { OpenAPI.HEADERS = extraHeaders; } + markRequestsAsCliClient(); OpenAPI.interceptors.response.use(async (response) => { await detectAuthGatewayChallenge(response); return response; diff --git a/cli/src/types.ts b/cli/src/types.ts index f0cd58abbb..c9244a8f4c 100644 --- a/cli/src/types.ts +++ b/cli/src/types.ts @@ -5,6 +5,7 @@ import * as path from "node:path"; import { sep as SEP } from "node:path"; import { stringify as yamlStringify } from "yaml"; import { yamlParseContent } from "./utils/yaml.ts"; +import { isDbtDescriptorPath } from "./utils/resource_folders.ts"; import { pushApp } from "./commands/app/app.ts"; import { pushFolder } from "./commands/folder/folder.ts"; import { pushFlow } from "./commands/flow/flow.ts"; @@ -34,6 +35,7 @@ import { buildFolderPath, isScriptModulePath, } from "./utils/resource_folders.ts"; +import { isSharedLockPath } from "./utils/script_common.ts"; export interface DifferenceCreate { type: "CREATE"; @@ -174,6 +176,21 @@ function redactString(s: string): string { return s.slice(0, 5) + "*".repeat(s.length - 5); } +export interface PushObjOptions { + /** Optional commit/update message */ + message?: string; + /** The original local file path (used for branch-specific resource file resolution) */ + originalLocalPath?: string; + /** Identity to attribute the push to, for the types that carry one */ + permissionedAsContext?: PermissionedAsContext; + /** Whether the item is workspace-specific */ + wsSpecific?: boolean; + /** encryption_key push: non-interactive flag and explicit re-encryption choice */ + keyPushOpts?: PushWorkspaceKeyOptions; + /** TypeScript runtime a bare `.ts` denotes, for raw-app runnables */ + defaultTs?: "bun" | "deno"; +} + /** * Pushes an object to the workspace server based on its type * @param workspace - The workspace ID to push to @@ -182,9 +199,7 @@ function redactString(s: string): string { * @param newObj - The new object state to push * @param plainSecrets - Whether to store secrets in plain text * @param alreadySynced - Array to track already synced items - * @param message - Optional commit/update message - * @param originalLocalPath - The original local file path (used for branch-specific resource file resolution) - * @param keyPushOpts - Options for the encryption_key push: non-interactive flag and explicit re-encryption choice + * @param opts - Per-type extras; see PushObjOptions */ export async function pushObj( workspace: string, @@ -193,12 +208,16 @@ export async function pushObj( newObj: any, plainSecrets: boolean, alreadySynced: string[], - message?: string, - originalLocalPath?: string, - permissionedAsContext?: PermissionedAsContext, - wsSpecific?: boolean, - keyPushOpts?: PushWorkspaceKeyOptions, + opts: PushObjOptions = {}, ) { + const { + message, + originalLocalPath, + permissionedAsContext, + wsSpecific, + keyPushOpts, + defaultTs, + } = opts; const typeEnding = getTypeStrFromPath(p); if (typeEnding === "app") { @@ -212,7 +231,7 @@ export async function pushObj( if (!rawAppName) { throw new Error(`Could not extract raw app name from path: ${p}`); } - await pushRawApp(workspace, rawAppName, buildFolderPath(rawAppName, "raw_app"), message); + await pushRawApp(workspace, rawAppName, buildFolderPath(rawAppName, "raw_app"), message, defaultTs); } else if (typeEnding === "folder") { await pushFolder(workspace, p, befObj, newObj); } else if (typeEnding === "variable") { @@ -226,7 +245,7 @@ export async function pushObj( } else if (typeEnding === "resource") { if (!alreadySynced.includes(p)) { alreadySynced.push(p); - await pushResource(workspace, p, befObj, newObj, originalLocalPath || p, wsSpecific); + await pushResource(workspace, p, befObj, newObj, originalLocalPath || p, wsSpecific, true); } } else if (typeEnding === "resource-type") { await pushResourceType(workspace, p, befObj, newObj); @@ -348,6 +367,7 @@ export function getTypeStrFromPath( | "group" | "settings" | "encryption_key" + | "shared_lock" | "workspace_dependencies" { if (isDatatableMigrationPath(p)) { return "datatable_migration"; @@ -364,6 +384,10 @@ export function getTypeStrFromPath( if (isRawAppPath(p)) { return "raw_app"; } + // A repo-side artifact of `dedupeLockfiles`: it has no object on the server. + if (isSharedLockPath(p)) { + return "shared_lock"; + } if (p.startsWith("dependencies" + SEP)) { return "workspace_dependencies"; } @@ -388,7 +412,11 @@ export function getTypeStrFromPath( parsed.ext == ".rb" || parsed.ext == ".r" || // for related places search: ADD_NEW_LANG - (parsed.ext == ".yml" && parsed.name.split(".").pop() == "playbook") + (parsed.ext == ".yml" && parsed.name.split(".").pop() == "playbook") || + // A dbt descriptor is `__dbt/wm_dbt.yaml`. Without this it reads + // as one of the CLI's own `.yaml` metadata files and a pull writes the + // script's metadata and lock but never its content. + isDbtDescriptorPath(p) ) { return "script"; } diff --git a/cli/src/utils/git.ts b/cli/src/utils/git.ts index da1ac14414..5ca06f427a 100644 --- a/cli/src/utils/git.ts +++ b/cli/src/utils/git.ts @@ -75,6 +75,114 @@ export function getWorkspaceIdForWorkspaceForkFromBranchName(branchName: string) return `${WM_FORK_PREFIX}-${branchName.slice(start)}` } +/** + * Whether this checkout can vouch for what it once held under `migrations/datatable/**`. + * + * `known` lists every such path recorded on the current branch (paths stay listed after + * the commit that removed them, so a real deletion is still recognisable). `unknown` + * means a file's absence from the working tree proves nothing, either because the + * history can't be read — no repository, a shallow clone's truncated history, an + * unresolvable root, a failing git — or because the working tree deliberately doesn't + * mirror it, as in a sparse checkout. Both shapes carry the same obligation on the + * caller: trust nothing, rather than read absence as evidence. + */ +export type RecordedMigrationPaths = + | { kind: "known"; paths: Set } + | { kind: "unknown"; reason: string; remedy: string }; + +/** + * Resolve [`RecordedMigrationPaths`] for the checkout at the current directory. + * + * This is the durable answer to "did this checkout ever track that migration?", which + * the working tree cannot give: an absent `migrations/datatable/` is equally a clone + * that never pulled migrations and one where the last was deleted, and creating a + * migration locally (`wmill datatable migrate new`) makes the directory appear without + * anything having been tracked. + * + * Scoped to `HEAD`, not `--all`: a migration that only ever existed on some other + * branch is not evidence that *this* branch ever tracked it, and using it as such would + * authorize deleting it. Output paths are repo-root-relative, so the `--show-prefix` of + * the working directory is stripped to match the cwd-relative paths a sync diff uses. + */ +export function gitRecordedDatatableMigrationPaths(): RecordedMigrationPaths { + if (!isGitRepository()) { + return { + kind: "unknown", + reason: "this directory is not a git repository", + remedy: "Run the push from a git checkout of the synced repository", + }; + } + const shallow = spawnSync("git", ["rev-parse", "--is-shallow-repository"], { + encoding: "utf8", + stdio: "pipe", + }); + if ((shallow.stdout ?? "").trim() === "true") { + return { + kind: "unknown", + reason: "this is a shallow clone, so its history is truncated", + remedy: + "Fetch the full history (for actions/checkout, fetch-depth: 0)", + }; + } + // A sparse checkout can record migrations in history while never materialising + // them in the working tree, so their absence there says nothing about whether the + // user deleted them. Over-protective for a sparse cone that does include + // migrations/, which the interactive prompt can still override. + // `--type=bool` normalises git's booleans (1, yes, on, …) to true/false; a raw + // `--get` would let `core.sparseCheckout = 1` walk straight past this. + const sparse = spawnSync( + "git", + ["config", "--type=bool", "--get", "core.sparseCheckout"], + { encoding: "utf8", stdio: "pipe" }, + ); + if ((sparse.stdout ?? "").trim() === "true") { + return { + kind: "unknown", + reason: + "this is a sparse checkout, so its working tree may not hold every tracked file", + remedy: "Run the push from a full (non-sparse) checkout", + }; + } + const prefixOut = spawnSync("git", ["rev-parse", "--show-prefix"], { + encoding: "utf8", + stdio: "pipe", + }); + if ((prefixOut.status ?? 1) !== 0) { + return { + kind: "unknown", + reason: "the repository root could not be resolved", + remedy: "Check that git runs correctly in this directory", + }; + } + const prefix = (prefixOut.stdout ?? "").trim(); + + const r = spawnSync( + "git", + ["log", "HEAD", "--format=", "--name-only", "--", "migrations/datatable"], + { encoding: "utf8", stdio: "pipe", maxBuffer: 64 * 1024 * 1024 }, + ); + if ((r.status ?? 1) !== 0) { + log.debug(`Could not read git history for migrations: ${r.stderr ?? ""}`); + return { + kind: "unknown", + reason: "its history could not be read", + remedy: "Check that git runs correctly in this directory", + }; + } + const paths = new Set(); + for (const line of (r.stdout ?? "").split("\n")) { + const p = line.trim(); + if (p.length === 0) continue; + if (prefix.length > 0) { + if (!p.startsWith(prefix)) continue; + paths.add(p.slice(prefix.length)); + } else { + paths.add(p); + } + } + return { kind: "known", paths }; +} + export function isGitRepository(): boolean { try { execSync("git rev-parse --git-dir", { @@ -319,6 +427,10 @@ export function gitSyncIncludePattern( return `${path}.azure_trigger.*`; case "emailtrigger": return `${path}.email_trigger.*`; + case "datatable_migration": + // One migration is two files under `migrations/datatable/
/`; the + // backend already sends the repo-relative base path. + return `${path}.up.sql,${path}.down.sql`; default: // Scripts: `${path}.*` matches the dotted layout // (`${path}.script.yaml` etc.), `${path}__mod/**` matches the folder diff --git a/cli/src/utils/lock_dedup.ts b/cli/src/utils/lock_dedup.ts new file mode 100644 index 0000000000..88778bc569 --- /dev/null +++ b/cli/src/utils/lock_dedup.ts @@ -0,0 +1,556 @@ +import { stringify as yamlStringify } from "yaml"; +import { mkdir, rm, writeFile } from "node:fs/promises"; +import { existsSync } from "node:fs"; +import * as path from "node:path"; +import { yamlOptions } from "../commands/sync/sync.ts"; +import { yamlParseContent } from "./yaml.ts"; +import { + depFileOfSharedLock, + extractWorkspaceDepsAnnotation, + hasLockAffectingAnnotation, + inferContentTypeFromFilePath, + isSharedLockPath, + languageNeedsLock, + sharedLockPathFor, + workspaceDependenciesPathToLanguageAndFilename, + type ScriptLanguage, +} from "./script_common.ts"; + +/** + * Lockfile deduplication (`dedupeLockfiles` in wmill.yaml). + * + * A workspace whose dependencies come from `dependencies/` resolves to the + * very same lock for every script of that language, so the repo ends up holding + * thousands of byte-identical `.script.lock` files: one dependency bump rewrites + * all of them, and every open branch conflicts on all of them. + * + * With dedup on, the scripts that resolve against a workspace dependency file + * reference ONE lockfile named after it — `dependencies/requirements.in` -> + * `locks/requirements.in.lock` — through the `!inline` indirection their + * metadata already uses. A dependency bump is a one-file diff. + * + * Identity comes from the dependency file, never from the content or from which + * scripts happen to be in view. That is what makes the pass stateless: a sync + * narrowed to a single script (the git-sync deploy callback) computes the same + * NAME as a full one. + * + * The CONTENT follows, because a group only ever holds scripts whose lock IS + * that file's lock: one carrying an annotation the worker acts on — a pinned + * interpreter, `npm`, `nobundling` — never joins, so there is no variant inside + * a group to tell apart from a bump. What is left is a script whose committed + * lock is simply behind, and the many outvote it. + * + * Two cases are not shared at all and keep a `.script.lock` of their own: a + * script whose annotation is `extra_` or carries inline dependencies (its lock + * folds in its own imports), and one naming several dependency files at once + * (its lock is no single file's). + */ + +const INLINE_PREFIX = "!inline "; + +/** Sync maps are keyed with the platform separator, while an `!inline` + * reference is always forward-slash. */ +const toMapKey = (refPath: string) => refPath.replaceAll("/", path.sep); +const toRefPath = (mapKey: string) => mapKey.replaceAll("\\", "/"); + +/** What the sync layer needs to know to dedup: whether to, and how to read a + * `.ts` script's language. Both ride on the same `wmill.yaml` options object. */ +export type LockDedupOptions = { + dedupeLockfiles?: boolean | undefined; + defaultTs?: "bun" | "deno" | undefined; +}; + +/** The files a dedup pass writes and removes, as paths relative to the sync + * root — applicable to an in-memory sync map or to the working tree. */ +export type SharedLockPlan = { + writes: Record; + deletes: string[]; +}; + +export function isEmptySharedLockPlan(plan: SharedLockPlan): boolean { + return Object.keys(plan.writes).length === 0 && plan.deletes.length === 0; +} + +type ScriptEntry = { + metaKey: string; + isJson: boolean; + compactJson: boolean; + parsed: Record; + /** The lockfile the metadata references today, as a map key. */ + lockKey: string; + /** The lockfile this script owns when it is not sharing one. */ + ownLockKey: string; + lock: string; + /** The workspace dependency file whose lock this is, when it is one. */ + depFile: string | undefined; +}; + +/** `f/foo.script.yaml` -> base `f/foo`, lock `f/foo.script.lock`; + * `f/foo__mod/script.yaml` -> base `f/foo__mod/script`, lock `…/script.lock`. + * The base is what the script's content file is named after. All returned + * forward-slashed, whatever the map's separator. */ +function scriptMetaBase( + key: string, +): { base: string; ownLockKey: string; isJson: boolean } | undefined { + const ref = toRefPath(key); + for (const [suffix, isJson] of [ + [".script.yaml", false], + [".script.json", true], + ["/script.yaml", false], + ["/script.json", true], + ] as const) { + if (!ref.endsWith(suffix)) continue; + const stripped = ref.slice(0, ref.length - suffix.length); + if (suffix.startsWith("/")) { + // `/script.yaml` is only script metadata inside a module folder; anywhere + // else it is an ordinary file that happens to be called `script.yaml`. + if (!stripped.endsWith("__mod")) return undefined; + return { + base: stripped + "/script", + ownLockKey: stripped + "/script.lock", + isJson, + }; + } + return { base: stripped, ownLockKey: stripped + ".script.lock", isJson }; + } + return undefined; +} + +/** Content-file extensions of the languages that carry a lock, longest first. + * A language absent here is simply never deduplicated. + * for related places search: ADD_NEW_LANG */ +const LOCKABLE_EXTS = [ + ".fetch.ts", + ".deno.ts", + ".bun.ts", + ".playbook.yml", + ".ts", + ".py", + ".go", + ".php", + ".rs", +]; + +/** Map keys grouped by directory, so a script's content file is looked up among + * its own directory's entries: splitting a name at its first dot would put + * `f/a.b.py` under `f/a` and leave every dotted script path undeduplicated. */ +function indexByDirectory( + map: Record, +): Map> { + const index = new Map>(); + for (const key of Object.keys(map)) { + const ref = toRefPath(key); + const dir = ref.slice(0, ref.lastIndexOf("/") + 1); + const bucket = index.get(dir); + if (bucket) { + bucket.add(ref); + } else { + index.set(dir, new Set([ref])); + } + } + return index; +} + +/** A script's content file and its language: `` and nothing looser, + * since `f/a.b.py` is the content file of `f/a.b`, not of `f/a`. */ +function contentOfScript( + map: Record, + base: string, + byDirectory: Map>, + defaultTs: "bun" | "deno" | undefined, +): { content: string; language: ScriptLanguage } | undefined { + const siblings = byDirectory.get(base.slice(0, base.lastIndexOf("/") + 1)); + if (!siblings) return undefined; + for (const ext of LOCKABLE_EXTS) { + const candidate = base + ext; + if (!siblings.has(candidate)) continue; + const content = map[toMapKey(candidate)] ?? map[candidate]; + if (content === undefined) continue; + try { + return { + content, + language: inferContentTypeFromFilePath(candidate, defaultTs), + }; + } catch { + // not a language this CLI knows + } + } + return undefined; +} + +/** A map entry addressed by a forward-slashed path, whatever separator the map + * was keyed with. */ +function lookup( + map: Record, + refPath: string, +): { key: string; content: string } | undefined { + for (const key of [toMapKey(refPath), refPath]) { + const content = map[key]; + if (content !== undefined) return { key, content }; + } + return undefined; +} + +/** + * Both parsers, declared format first. YAML is a superset of JSON, and + * flow-style YAML (`{summary: x, lock: '!inline …'}`) starts with `{` while + * failing `JSON.parse` — deciding by the first character loses the reference. + */ +function parseMetadata( + metaPath: string, + metaContent: string, + isJson: boolean, +): Record | undefined { + for (const asJson of isJson ? [true, false] : [false, true]) { + try { + const parsed = asJson + ? JSON.parse(metaContent) + : yamlParseContent(metaPath, metaContent); + if (typeof parsed === "object" && parsed !== null) return parsed; + } catch { + // try the other one + } + } + return undefined; +} + +/** + * The shared lockfile a metadata file's `lock` field names, if any. The raw text + * is only a prefilter: a summary or a comment can carry the same words, and + * `!inline` decides where a lock is written, so it is read from the parsed field + * and nowhere else. + */ +export function sharedLockRefOf( + metaPath: string, + metaContent: string, + isJson: boolean, +): string | undefined { + // Without the trailing space: the YAML serializer folds a long `lock:` line + // at a space, so `!inline locks/…` can reach disk as `!inline\n locks/…` + // and a prefilter looking for the space would call it a non-reader. + if (!metaContent.includes(INLINE_PREFIX.trimEnd())) return undefined; + const lock = parseMetadata(metaPath, metaContent, isJson)?.["lock"]; + if (typeof lock !== "string" || !lock.startsWith(INLINE_PREFIX)) { + return undefined; + } + const ref = lock.slice(INLINE_PREFIX.length); + return isSharedLockPath(ref) ? ref : undefined; +} + +/** + * Whether a metadata file may reference a shared lockfile but cannot say which. + * + * A `.script.yaml` carrying git conflict markers is a file whose `lock` cannot + * be read, not one that reads nothing, and deleting a lockfile it may point at + * is the unrecoverable half of that guess. + */ +export function metadataLockUnreadable( + metaPath: string, + metaContent: string, + isJson: boolean, +): boolean { + if (!metaContent.includes(INLINE_PREFIX.trimEnd())) return false; + return parseMetadata(metaPath, metaContent, isJson) === undefined; +} + +/** + * The shared lockfile a metadata FILE reads, when it reads one that is there. + * `parseMetadataFile` resolves `lock` to the lockfile's content, so the + * reference itself survives only in the raw text. + */ +export function sharedLockRefIn( + metadataContent: string, + isJson: boolean, + root: string = ".", +): string | undefined { + const ref = sharedLockRefOf("metadata", metadataContent, isJson); + return ref && existsSync(path.resolve(root, ref)) ? ref : undefined; +} + +/** The key a dependency file answers to, i.e. what a script names it by. */ +function depKeyOf(depFilePath: string): string | undefined { + const info = workspaceDependenciesPathToLanguageAndFilename(depFilePath); + return info && languageNeedsLock(info.language) + ? `${info.language} ${info.name ?? "default"}` + : undefined; +} + +/** Workspace dependency files keyed by the language and name a script names. */ +function depFilesByKey(paths: Iterable): Map { + const byKey = new Map(); + for (const key of paths) { + const ref = toRefPath(key); + if (!ref.startsWith("dependencies/")) continue; + // A set named `team/python` exports as `dependencies/team/python.`, + // which has no distinct name under `locks/`: flattened it collides with the + // top-level file, and the sweep would then retire a lockfile whose scripts + // still read it. Such a set shares nothing and its scripts keep own locks. + if (ref.slice("dependencies/".length).includes("/")) continue; + const depKey = depKeyOf(ref); + if (depKey) byKey.set(depKey, ref); + } + return byKey; +} + +/** + * The workspace dependency file whose lock a script's lock IS — undefined when + * the script's lock is its own (see the header for the two cases). + */ +function shareableDepFile( + scriptContent: string, + language: ScriptLanguage, + depFiles: Map, +): string | undefined { + // A script the worker locks differently for reasons of its own — a pinned + // interpreter, `//npm`, `//nobundling` — cannot stand for its dependency + // file's lock, so it never joins a group and its lock stays its own. + if (hasLockAffectingAnnotation(scriptContent, language)) return undefined; + const annotation = extractWorkspaceDepsAnnotation(scriptContent, language); + if (annotation && (annotation.mode === "extra" || annotation.inline)) { + return undefined; + } + const names = annotation ? annotation.external : ["default"]; + if (names.length !== 1) return undefined; + return depFiles.get(`${language} ${names[0]}`); +} + +/** + * The shared lockfile a script belongs in, given the workspace dependency files + * available — the one place that decides it, for both the sync planner and the + * per-script regeneration in `updateScriptLock`. + */ +export function sharedLockTargetFor( + scriptContent: string, + language: ScriptLanguage, + depPaths: Iterable, +): string | undefined { + const depFile = shareableDepFile( + scriptContent, + language, + depFilesByKey(depPaths), + ); + return depFile === undefined ? undefined : sharedLockPathFor(depFile); +} + +function collectScripts( + map: Record, + defaultTs: "bun" | "deno" | undefined, + depFiles: Map, +): ScriptEntry[] { + const byDirectory = indexByDirectory(map); + const entries: ScriptEntry[] = []; + for (const [metaKey, metaContent] of Object.entries(map)) { + const meta = scriptMetaBase(metaKey); + if (!meta) continue; + + const parsed = parseMetadata(metaKey, metaContent, meta.isJson); + if (parsed === undefined) continue; + + const lockRef = parsed["lock"]; + if (typeof lockRef !== "string" || !lockRef.startsWith(INLINE_PREFIX)) { + continue; + } + const lockFile = lookup(map, lockRef.slice(INLINE_PREFIX.length)); + // An absent or empty lock is not a lock to share: a script with no + // dependencies carries `lock: ''` and no file at all. + if (lockFile === undefined || lockFile.content === "") continue; + + const script = contentOfScript(map, meta.base, byDirectory, defaultTs); + if (script === undefined || !languageNeedsLock(script.language)) continue; + + entries.push({ + metaKey, + isJson: meta.isJson, + compactJson: !metaContent.includes("\n"), + parsed, + lockKey: lockFile.key, + ownLockKey: toMapKey(meta.ownLockKey), + lock: lockFile.content, + depFile: shareableDepFile(script.content, script.language, depFiles), + }); + } + return entries; +} + +function serializeMetadata(entry: ScriptEntry): string { + if (!entry.isJson) return yamlStringify(entry.parsed, yamlOptions); + // Indented or compact as it was found: `sync` writes JSON metadata indented + // and `generate-metadata` writes it compact, so imposing either one here + // reformats files this feature exists to keep quiet. + return entry.compactJson + ? JSON.stringify(entry.parsed) + : JSON.stringify(entry.parsed, null, 2); +} + +/** + * What a sync map (path -> content) has to change for the scripts of a workspace + * dependency file to share one lockfile. Pure: the map is not touched. + */ +export type SharedLockPlanContext = { + defaultTs?: "bun" | "deno" | undefined; + /** + * Workspace dependency files to consider beyond the ones in `map`, for the + * one caller whose map cannot hold them: `--skip-workspace-dependencies`. + * Pass nothing otherwise — with dependency files in the map, an absence there + * is a deletion, and adding disk's copy would keep a lockfile alive one sync + * past the file it is named after. + */ + depFiles?: Iterable; + /** + * Shared lockfiles the working tree already holds. The remote never + * serializes one, so without this a sync that has no script for a dependency + * file reads its lockfile as deleted — and every script still pointing at it + * is left with an `!inline` that resolves to nothing. + */ + present?: Record; +}; + +export function computeSharedLockPlan( + map: Record, + ctx: SharedLockPlanContext = {}, +): SharedLockPlan { + const plan: SharedLockPlan = { writes: {}, deletes: [] }; + const depFiles = depFilesByKey([...Object.keys(map), ...(ctx.depFiles ?? [])]); + // Every shared lockfile this sync can see, from either side. + const present: Record = { ...ctx.present }; + for (const [key, content] of Object.entries(map)) { + if (isSharedLockPath(toRefPath(key))) present[toRefPath(key)] = content; + } + + const byDepFile = new Map(); + const ownLock: ScriptEntry[] = []; + for (const entry of collectScripts(map, ctx.defaultTs, depFiles)) { + if (entry.depFile === undefined) { + ownLock.push(entry); + continue; + } + // Push into the existing array rather than rebuild it: a workspace where + // every script shares one dependency file is the case this exists for, and + // copying the group per insert makes that quadratic. + const group = byDepFile.get(entry.depFile); + if (group) group.push(entry); + else byDepFile.set(entry.depFile, [entry]); + } + + const point = (entry: ScriptEntry, targetKey: string) => { + if (entry.lockKey === targetKey) return; + // A shared lockfile is dropped by the sweep below, which knows whether its + // dependency file is still there; only a private one goes with its script. + if (!isSharedLockPath(entry.lockKey)) plan.deletes.push(entry.lockKey); + entry.parsed["lock"] = INLINE_PREFIX + toRefPath(targetKey); + plan.writes[entry.metaKey] = serializeMetadata(entry); + }; + + const takeOwnLock = (entry: ScriptEntry) => { + if (map[entry.ownLockKey] !== entry.lock) { + plan.writes[entry.ownLockKey] = entry.lock; + } + point(entry, entry.ownLockKey); + }; + + for (const [depFile, group] of byDepFile) { + // Every script here resolves against the same file and carries nothing the + // worker locks separately, so their locks agree — unless one's committed + // lock is simply behind. The many outvote the one; ties break on the content + // itself so the outcome never depends on map ordering. + const byContent = new Map(); + for (const entry of group) { + const sameLock = byContent.get(entry.lock); + if (sameLock) sameLock.push(entry); + else byContent.set(entry.lock, [entry]); + } + let content = ""; + let count = 0; + for (const [lock, members] of byContent) { + if ( + members.length > count || + (members.length === count && lock < content) + ) { + content = lock; + count = members.length; + } + } + + const sharedKey = toMapKey(sharedLockPathFor(depFile)); + if (map[sharedKey] !== content) plan.writes[sharedKey] = content; + for (const entry of group) { + if (entry.lock === content) point(entry, sharedKey); + else takeOwnLock(entry); + } + } + + // A script that stopped resolving against a dependency file takes its lock + // back with it. + for (const entry of ownLock) { + if (entry.lockKey !== entry.ownLockKey) takeOwnLock(entry); + } + + // A shared lockfile lives exactly as long as the dependency file it is named + // after. Asking that, rather than "does any script still read it", is what + // lets a sync narrowed to one item leave the rest of the workspace alone: a + // lockfile with no script in view is carried forward, not deleted. + for (const [sharedRef, content] of Object.entries(present)) { + const depFile = depFileOfSharedLock(sharedRef); + if (depFile === undefined) continue; + const key = toMapKey(sharedRef); + if (plan.writes[key] !== undefined) continue; + if (depFiles.has(depKeyOf(depFile) ?? "")) { + if (map[key] === undefined) plan.writes[key] = content; + } else { + plan.deletes.push(key); + } + } + + // Deletes are applied after writes, so a path some script still writes must + // not also be dropped — two metadata files pointing at one lock file would + // otherwise cancel each other out and leave the survivor without a lock. + plan.deletes = plan.deletes.filter((key) => plan.writes[key] === undefined); + + return plan; +} + +export function applySharedLockPlanToMap( + map: Record, + plan: SharedLockPlan, +): void { + for (const [key, content] of Object.entries(plan.writes)) { + map[key] = content; + } + for (const key of plan.deletes) { + delete map[key]; + } +} + +export async function applySharedLockPlanToDisk( + plan: SharedLockPlan, +): Promise { + for (const [key, content] of Object.entries(plan.writes)) { + // Per write, so `locks/` comes into existence only when there is a shared + // lockfile to put in it. + await mkdir(path.dirname(key), { recursive: true }); + await writeFile(key, content, "utf-8"); + } + for (const key of plan.deletes) { + await rm(key, { force: true }); + } +} + +/** + * The script metadata files that read a given shared lockfile. A change to that + * file is a change to their lock, and they are what carries it to the remote. + */ +export function scriptsReferencingSharedLock( + map: Record, + sharedKey: string, +): string[] { + const reference = toRefPath(sharedKey); + const referrers: string[] = []; + for (const [metaKey, metaContent] of Object.entries(map)) { + const meta = scriptMetaBase(metaKey); + if (!meta) continue; + if (sharedLockRefOf(metaKey, metaContent, meta.isJson) === reference) { + referrers.push(metaKey); + } + } + return referrers; +} diff --git a/cli/src/utils/metadata.ts b/cli/src/utils/metadata.ts index ebb35e9d33..1f2ef86769 100644 --- a/cli/src/utils/metadata.ts +++ b/cli/src/utils/metadata.ts @@ -4,7 +4,7 @@ import { colors } from "@cliffy/ansi/colors"; import * as log from "../core/log.ts"; import { stringify as yamlStringify } from "yaml"; import { yamlParseFile } from "./yaml.ts"; -import { writeFile, stat, rm, readdir } from "node:fs/promises"; +import { writeFile, stat, rm, readdir, mkdir } from "node:fs/promises"; import { readFileSync, existsSync, readdirSync, statSync, mkdirSync, writeFileSync } from "node:fs"; import * as path from "node:path"; import { createRequire } from "node:module"; @@ -17,11 +17,24 @@ import { ScriptLanguage, workspaceDependenciesLanguages, languageNeedsLock, + LANG_COMMENT_LIT, } from "./script_common.ts"; import { inferContentTypeFromFilePath } from "./script_common.ts"; -import { getModuleFolderSuffix, isModuleEntryPoint, scriptPathToRemotePath } from "./resource_folders.ts"; +// Workspace-dependency vocabulary lives with the languages it describes; these +// re-exports keep the CLI's existing import sites working. +export { + workspaceDependenciesPathToLanguageAndFilename, + extractWorkspaceDepsAnnotation, + type WorkspaceDepsAnnotation, +} from "./script_common.ts"; +import { + workspaceDependenciesPathToLanguageAndFilename, + extractWorkspaceDepsAnnotation, +} from "./script_common.ts"; +import { dbtGeneratedDirs, isUnderGeneratedDir, isBundledModuleFile, getModuleFolderSuffix, isModuleEntryPoint, scriptPathToRemotePath } from "./resource_folders.ts"; import { findCodebase, yamlOptions } from "../commands/sync/sync.ts"; import { generateHash, readInlinePathSync, getHeaders, readTextFile, readTextFileSync } from "./utils.ts"; +import { DBT_DESCRIPTOR_NAME, isMissingDbtDescriptor } from "./resource_folders.ts"; import { detectAuthGatewayChallenge } from "./http_guards.ts"; import { SyncCodebase } from "./codebase.ts"; @@ -30,6 +43,7 @@ import { getIsWin } from "./utils.ts"; import { extractRelativeImports } from "./relative_imports.ts"; import { DoubleLinkedDependencyTree } from "./dependency_tree.ts"; import { pollJobWithQueueLogging } from "./job_polling.ts"; +import { sharedLockRefIn, sharedLockTargetFor } from "./lock_dedup.ts"; const _require = createRequire(import.meta.url); const _parserCache = new Map>(); @@ -105,17 +119,6 @@ export async function getRawWorkspaceDependencies(legacyBehaviour: boolean): Pro return rawWorkspaceDeps; } -export function workspaceDependenciesPathToLanguageAndFilename(path: string): { name: string | undefined, language: ScriptLanguage } | undefined { - const relativePath = path.replace("dependencies/", ""); - for (const { filename, language } of workspaceDependenciesLanguages) { - if (relativePath.endsWith(filename)) { - return { - name: relativePath === filename ? undefined : relativePath.replace("." + filename, ""), - language - }; - } - } -} /** * Filters raw workspace dependencies to only include those that: @@ -202,6 +205,7 @@ export async function generateScriptMetadataInternal( schemaOnly?: boolean | undefined; defaultTs?: "bun" | "deno"; rehashOnly?: boolean | undefined; + dedupeLockfiles?: boolean | undefined; }, dryRun: boolean, noStaleMessage: boolean, @@ -218,6 +222,13 @@ export async function generateScriptMetadataInternal( const language = inferContentTypeFromFilePath(scriptPath, opts.defaultTs); + // Whether the metadata and lock live INSIDE that folder. They do for a `__mod` + // bundle, whose folder is Windmill's. A dbt project's folder is dbt's, taken + // verbatim, so its companions stay beside it — writing them in would leave + // stray `script.yaml`/`script.lock` files in the deployed project and leave + // the metadata sync actually reads untouched. + const metadataInFolder = isFolderLayout && language !== "dbt"; + // For folder layout, parseMetadataFile is called with remotePath which // will find __mod/script.yaml via the folder layout fallback const metadataWithType = await parseMetadataFile( @@ -225,8 +236,12 @@ export async function generateScriptMetadataInternal( undefined, ); - // read script content - const scriptContent = await readTextFile(scriptPath); + // read script content — a dbt project's descriptor is optional, and absent + // means an empty descriptor rather than a script that cannot be pushed. + const scriptContent = await readTextFile(scriptPath).catch((e) => { + if (isMissingDbtDescriptor(scriptPath, e)) return ""; + throw e; + }); const metadataContent = await readTextFile(metadataWithType.path); const filteredRawWorkspaceDependencies = filterWorkspaceDependencies( @@ -238,7 +253,8 @@ export async function generateScriptMetadataInternal( // Compute the module folder path early so we can include module hashes in stale check const moduleFolderPath = isFolderLayout ? path.dirname(scriptPath) - : scriptPath.substring(0, scriptPath.indexOf(".")) + getModuleFolderSuffix(); + : scriptPath.substring(0, scriptPath.indexOf(".")) + + getModuleFolderSuffix(language); const hasModules = existsSync(moduleFolderPath) && statSync(moduleFolderPath).isDirectory(); @@ -250,7 +266,8 @@ export async function generateScriptMetadataInternal( let moduleHashes: Record = {}; if (hasModules) { moduleHashes = await computeModuleHashes( - moduleFolderPath, opts.defaultTs, tree ? {} : rawWorkspaceDependencies, isFolderLayout + moduleFolderPath, opts.defaultTs, tree ? {} : rawWorkspaceDependencies, isFolderLayout, + language === "dbt", ); } const hasModuleHashes = Object.keys(moduleHashes).length > 0; @@ -349,7 +366,7 @@ export async function generateScriptMetadataInternal( if (!hasCodebase) { const tempScriptRefs = tree?.getTempScriptRefs(remotePath); - const lockPathOverride = isFolderLayout + const lockPathOverride = metadataInFolder ? path.dirname(scriptPath) + "/script.lock" : undefined; await updateScriptLock( @@ -361,13 +378,27 @@ export async function generateScriptMetadataInternal( filteredRawWorkspaceDependencies, tempScriptRefs, lockPathOverride, + // The lockfile of the workspace dependency file this script resolves + // against, if any: joining it is a matter of resolving to its content. + opts.dedupeLockfiles + ? sharedLockTargetFor( + scriptContent, + language, + Object.keys(rawWorkspaceDependencies), + ) + : undefined, ); } else { metadataParsedContent.lock = ""; } - // Generate locks for modules in __mod/ folder - if (hasModules) { + // Generate locks for modules in __mod/ folder. + // + // Never for a dbt bundle: its files are the project's own SQL and YAML, none + // of which is a Windmill script needing a lockfile, and writing `foo.lock` + // beside `foo.sql` puts our artifacts inside a tree we promise to round-trip + // byte-for-byte. + if (hasModules && language !== "dbt") { // Identify which modules changed by comparing per-module hashes let changedModules: string[] | undefined; if (hasModuleHashes) { @@ -387,7 +418,14 @@ export async function generateScriptMetadataInternal( ); } } else { - if (isFolderLayout) { + // `parseMetadataFile` resolved `lock` to the lockfile's CONTENT, so the + // reference has to be restored from the raw text — including a shared one + // (`dedupeLockfiles`), which `--schema-only` would otherwise replace with a + // per-script path whose file deduplication removed. + const sharedRef = sharedLockRefIn(metadataContent, metadataWithType.isJson); + if (sharedRef) { + metadataParsedContent.lock = "!inline " + sharedRef; + } else if (metadataInFolder) { metadataParsedContent.lock = "!inline " + remotePath.replaceAll(SEP, "/") + getModuleFolderSuffix() + "/script.lock"; } else { @@ -399,7 +437,7 @@ export async function generateScriptMetadataInternal( // Write metadata back to the correct path let metaPath: string; let newMetadataContent: string; - if (isFolderLayout) { + if (metadataInFolder) { if (metadataWithType.isJson) { metaPath = path.dirname(scriptPath) + "/script.json"; newMetadataContent = JSON.stringify(metadataParsedContent); @@ -474,119 +512,6 @@ export async function updateScriptSchema( delete metadataContent.no_main_func; } -// --------------------------------------------------------------------------- -// Annotation parser — mirrors backend's WorkspaceDependenciesAnnotatedRefs::parse -// (windmill-common/src/workspace_dependencies.rs) so the cache key captures -// exactly the parts of scriptContent that affect lockfile generation. -// --------------------------------------------------------------------------- - -type AnnotationMode = "manual" | "extra"; - -interface WorkspaceDepsAnnotation { - mode: AnnotationMode; - external: string[]; - inline: string | null; -} - -const LANG_ANNOTATION_CONFIG: Partial< - Record -> = { - python3: { comment: "#", keyword: "requirements", validityRe: /^#\s?(\S+)\s*$/ }, - bun: { comment: "//", keyword: "package_json" }, - nativets: { comment: "//", keyword: "package_json" }, - go: { comment: "//", keyword: "go_mod" }, - php: { comment: "//", keyword: "composer_json" }, - powershell: { comment: "#", keyword: "modules_json" }, -}; - -export function extractWorkspaceDepsAnnotation( - scriptContent: string, - language: ScriptLanguage, -): WorkspaceDepsAnnotation | null { - const config = LANG_ANNOTATION_CONFIG[language]; - if (!config) return null; - - const { comment, keyword, validityRe } = config; - const extraMarkerUnderscore = `extra_${keyword}:`; - const extraMarkerHyphen = `extra-${keyword}:`; - const manualMarker = `${keyword}:`; - - const stripComment = (l: string): string | null => { - if (!l.startsWith(comment)) return null; - return l.substring(comment.length).trimStart(); - }; - const isExtra = (l: string): boolean => { - const s = stripComment(l); - return s !== null && (s.startsWith(extraMarkerUnderscore) || s.startsWith(extraMarkerHyphen)); - }; - const isManual = (l: string): boolean => { - const s = stripComment(l); - return s !== null && s.startsWith(manualMarker); - }; - - const lines = scriptContent.split("\n"); - - // Find first annotation line (mirrors Rust find_position) - let pos = -1; - for (let i = 0; i < lines.length; i++) { - if (isExtra(lines[i]) || isManual(lines[i])) { - pos = i; - break; - } - } - if (pos === -1) return null; - - const annotationLine = lines[pos]; - const mode: AnnotationMode = isExtra(annotationLine) ? "extra" : "manual"; - - // Parse external references from the annotation line - const marker = mode === "extra" - ? (annotationLine.includes(extraMarkerUnderscore) ? extraMarkerUnderscore : extraMarkerHyphen) - : manualMarker; - const unparsed = annotationLine.replaceAll(marker, "").replaceAll(comment, ""); - const external = unparsed - .split(",") - .map((s) => s.trim()) - .filter((s) => s.length > 0); - - // Parse inline deps from subsequent lines - const inlineParts: string[] = []; - for (let i = pos + 1; i < lines.length; i++) { - const l = lines[i]; - if (validityRe) { - const match = validityRe.exec(l); - if (match && match[1]) { - inlineParts.push(match[1]); - } else { - break; - } - } else { - if (!l.startsWith(comment)) { - break; - } - inlineParts.push(l.substring(comment.length)); - } - } - - const inlineStr = inlineParts.join("\n"); - const inline = inlineStr.trim().length > 0 ? inlineStr : null; - - return { mode, external, inline }; -} - -// Mirrors backend ScriptLang::as_comment_lit (windmill-types/src/scripts.rs) -// for the languages that can reach the lock cache. -const LANG_COMMENT_LIT: Partial> = { - python3: "#", - ansible: "#", - powershell: "#", - bun: "//", - nativets: "//", - deno: "//", - go: "//", - php: "//", - rust: "//!", -}; /** * Returns the leading comment/blank-line block of the script, verbatim. @@ -764,16 +689,20 @@ async function updateScriptLock( rawWorkspaceDependencies: Record, tempScriptRefs?: Record, lockPathOverride?: string, + sharedLockRef?: string, ): Promise { - if ( - !( - (workspaceDependenciesLanguages.some((l) => l.language == language) && - language !== "powershell") || - language == "deno" || - language == "rust" || - language == "ansible" - ) - ) { + if (!languageNeedsLock(language)) { + // A dbt lock is written by the dependency job on a worker, from a real + // `dbt deps`/`dbt parse`, so there is nothing to generate here. Restore the + // reference to the file `wmill sync pull` wrote: the caller has already + // resolved it into the metadata, and leaving it resolved inlines the lock + // into the yaml on every run. + if (language === "dbt") { + const lockPath = lockPathOverride ?? remotePath + ".script.lock"; + if (existsSync(lockPath)) { + metadataContent.lock = "!inline " + lockPath.replaceAll(SEP, "/"); + } + } return; } @@ -794,6 +723,23 @@ async function updateScriptLock( const lockPath = lockPathOverride ?? remotePath + ".script.lock"; if (lock != "") { + // Joins an agreeing shared lockfile, and never creates or moves one: this + // runs per script and in parallel, so two scripts that resolve differently + // — a pinned Python version, an `//npm` annotation — would both find the + // file absent, both write it, and one would lose its lock with no copy left + // anywhere. Deciding a shared lockfile's content is the whole-tree pass's + // job, which sees every script at once. + if ( + sharedLockRef && + existsSync(sharedLockRef) && + readTextFileSync(sharedLockRef) === lock + ) { + if (existsSync(lockPath)) { + await rm(lockPath); + } + metadataContent.lock = "!inline " + sharedLockRef; + return; + } await writeFile(lockPath, lock, "utf-8"); metadataContent.lock = "!inline " + lockPath.replaceAll(SEP, "/"); } else { @@ -995,6 +941,9 @@ export async function inferSchema( } else if (language === "rlang") { const { parse_r } = await loadParser("windmill-parser-wasm-r"); inferedSchema = JSON.parse(parse_r(content)); + } else if (language === "dbt") { + const { parse_dbt } = await loadParser("windmill-parser-wasm-yaml"); + inferedSchema = JSON.parse(parse_dbt(content)); // for related places search: ADD_NEW_LANG } else { throw new Error("Invalid language: " + language); @@ -1368,8 +1317,15 @@ async function computeModuleHashes( defaultTs: "bun" | "deno" | undefined, rawWorkspaceDependencies: Record, isFolderLayout: boolean, + // A dbt project's files are taken verbatim, so hash them the same way the + // push reads them: a `.sql` model has no inferable language, and dropping it + // here would leave an edited model looking up to date. + verbatim: boolean = false, ): Promise> { const hashes: Record = {}; + const skipDirs = verbatim + ? dbtGeneratedDirs(moduleFolderPath) + : new Set(); async function readDir(dirPath: string, relPrefix: string) { const entries = readdirSync(dirPath, { withFileTypes: true }); @@ -1379,15 +1335,31 @@ async function computeModuleHashes( const isTopLevel = relPrefix === ""; if (entry.isDirectory()) { + // A configured `target-path` may be nested (`build/target`), so the + // comparison is on the project-relative path, not the entry name. + if (skipDirs.size > 0 && isUnderGeneratedDir(relPath, skipDirs)) continue; await readDir(fullPath, relPath); + // See the bundle builder: a verbatim (dbt) bundle carries a `.lock` the + // project authored, so the hash has to see it or a change to it would + // never be detected as a change. } else if ( entry.isFile() && - !entry.name.endsWith(".lock") && - !(isFolderLayout && isTopLevel && entry.name.startsWith("script.")) + (verbatim || !entry.name.endsWith(".lock")) && + !(isFolderLayout && isTopLevel && entry.name.startsWith("script.")) && + // The descriptor is the script's CONTENT, hashed as such: counting it + // here too would make one edit look like two changes. + !(verbatim && isTopLevel && entry.name === DBT_DESCRIPTOR_NAME) ) { - try { - inferContentTypeFromFilePath(entry.name, defaultTs); - } catch { + if (!verbatim) { + try { + inferContentTypeFromFilePath(entry.name, defaultTs); + } catch { + continue; + } + } else if (!isBundledModuleFile(fullPath)) { + // Hash only what the push actually sends. Hashing a file the bundle + // drops would make the script permanently stale: every check would + // see a change no push can ever resolve. continue; } const content = readTextFileSync(fullPath); diff --git a/cli/src/utils/resource_folders.ts b/cli/src/utils/resource_folders.ts index 653c443935..04ed260f07 100644 --- a/cli/src/utils/resource_folders.ts +++ b/cli/src/utils/resource_folders.ts @@ -8,6 +8,7 @@ * (.flow, .app, .raw_app) or dunder-prefixed names (__flow, __app, __raw_app). */ +import { existsSync } from "node:fs"; import * as log from "../core/log.ts"; import { sep as SEP } from "node:path"; import { yamlParseFile } from "./yaml.ts"; @@ -503,28 +504,323 @@ export function isFlowFolderMetadataFile(p: string): boolean { * to avoid confusion with file extensions. */ const MODULE_SUFFIX = "__mod"; +/** dbt scripts carry a whole dbt project, not helper code. The folder says so, + * and it is what a dbt developer points `--project-dir` at. */ +export const DBT_MODULE_SUFFIX = "__dbt"; +const MODULE_SUFFIXES = [MODULE_SUFFIX, DBT_MODULE_SUFFIX]; + +/** A dbt project's descriptor, inside the project it configures and OPTIONAL: + * an unmodified dbt project is already a complete Windmill script, and this + * file only appears when one needs something Windmill-specific (run arguments, + * a named warehouse, an engine pin). Its absence is an empty descriptor, never + * a missing script. */ +export const DBT_DESCRIPTOR_NAME = "wm_dbt.yaml"; + +/** Where a dbt script's descriptor lives, given its base path. */ +export function dbtDescriptorPath(scriptBasePath: string): string { + return scriptBasePath + DBT_MODULE_SUFFIX + "/" + DBT_DESCRIPTOR_NAME; +} + +/** Whether an error is a dbt descriptor that simply is not there. */ +export function isMissingDbtDescriptor(filePath: string, e: unknown): boolean { + if ((e as { code?: string })?.code !== "ENOENT") return false; + const norm = filePath.replaceAll("\\", "/"); + if (!isDbtDescriptorPath(norm)) return false; + // And the PROJECT is there. Absent both, this is not a descriptor-less + // project but a path that does not exist — a typo, or a project someone + // deleted — and treating it as an empty descriptor pushes `modules: + // undefined` over a deployed bundle, dropping it while reporting success. + return existsSync( + norm.slice(0, -DBT_DESCRIPTOR_NAME.length) + "dbt_project.yml" + ); +} + +/** Whether a path is a dbt descriptor, i.e. a dbt script's content file. */ +export function isDbtDescriptorPath(p: string): boolean { + const norm = normalizeSep(p); + const base = getScriptBasePathFromModulePath(norm); + return base !== undefined && norm === dbtDescriptorPath(base); +} /** - * Get the module folder suffix (always "__mod") + * Module folder suffix for a script: `__dbt` for a dbt project, `__mod` + * otherwise. */ -export function getModuleFolderSuffix(): string { - return MODULE_SUFFIX; +export function getModuleFolderSuffix(language?: string): string { + return language === "dbt" ? DBT_MODULE_SUFFIX : MODULE_SUFFIX; } /** * Check if a path is inside a script module folder. - * Matches patterns like: .../my_script__mod/... + * Matches patterns like: .../my_script__mod/... or .../my_project__dbt/... */ export function isScriptModulePath(p: string): boolean { - return normalizeSep(p).includes(MODULE_SUFFIX + "/"); + const n = normalizeSep(p); + return MODULE_SUFFIXES.some((suffix) => n.includes(suffix + "/")); +} + +/** Per-file ceiling for a dbt project's bundle. Real dbt code is small (about + * 500 bytes median, 1.9 KB at p90 measured across dbt_utils), so this only + * ever catches a committed dataset, which belongs in the warehouse rather than + * in every version of the script. */ +export const MAX_MODULE_BYTES = 5 * 1024 * 1024; + +/** + * Whether a dbt project file is one the bundle carries. + * + * A dbt project's authored files are text. A binary one -- an image under + * `docs/`, a `.DS_Store`, a parquet seed -- would be read as mojibake and, if + * it carries a NUL, rejected by Postgres with an opaque `unsupported Unicode + * escape sequence`. Binary is detected the way `git` does it, by a NUL in the + * first 8000 bytes, rather than by extension, which `docs/` and stray dotfiles + * do not follow. + * + * The push, the staleness hash and the sync diff all ask this same question: a + * file one drops and another keeps is a change no push can ever resolve. + */ +export function isBundledModuleFile(fullPath: string): boolean { + return moduleFileExclusion(fullPath) === undefined; +} + +/** + * WHY the bundle does not carry a file, when it does not. + * + * The two reasons are not interchangeable. `binary` is dbt's own leftovers and + * stray archives: nothing to say, so sync hides them. `oversized` is a file the + * project authored and dbt WOULD read — a large seed CSV — so it has to stay + * visible in the diff, or the push that reports the actionable size error never + * runs and the remote project is silently left incomplete. + */ +export function moduleFileExclusion( + fullPath: string, +): "binary" | "oversized" | undefined { + // Size from `stat` and only the first 8 KB read: a project may sit next to a + // multi-gigabyte parquet seed or a stray archive, and reading one whole just + // to classify it would stall the sync or exhaust the CLI. + let size: number; + let fd: number; + try { + size = fs.statSync(fullPath).size; + fd = fs.openSync(fullPath, "r"); + } catch { + // Unreadable is not the same as excluded. A pull asks this about files that + // do not exist locally yet, and answering "not carried" there would make + // sync ignore the whole incoming project and write nothing. + return undefined; + } + let binary: boolean; + try { + const head = Buffer.alloc(8000); + const read = fs.readSync(fd, head, 0, 8000, 0); + binary = head.subarray(0, read).includes(0); + } catch { + return undefined; + } finally { + fs.closeSync(fd); + } + if (binary) return "binary"; + return size > MAX_MODULE_BYTES ? "oversized" : undefined; +} + +/** + * The refusal an oversized dbt project file earns, raised WITHOUT reading it. + * + * dbt would have read the file, so deploying the project without it ships + * something that compiles here and fails at run time with a missing relation — + * hence an error rather than a skip. Every path that would otherwise load the + * body (the sync map, the push) asks first: a multi-gigabyte seed must not be + * buffered just to be refused. + */ +export function oversizedModuleFileError(relPath: string, size: number): Error { + return new Error( + `${relPath} is ${Math.ceil(size / 1024 / 1024)} MB, over the ` + + `${MAX_MODULE_BYTES / 1024 / 1024} MB per-file limit for a dbt project file. ` + + `Deploying without it would leave the project incomplete — shrink the file, or ` + + `keep it out of the project folder.`, + ); +} + +/** + * Refuse an oversized dbt project file before its content is read. `undefined` + * for everything else, including binary files the bundle merely drops. + */ +export function oversizedDbtFileError( + fullPath: string, + relPath: string, +): Error | undefined { + if (!isDbtModulePath(relPath)) return undefined; + if (moduleFileExclusion(fullPath) !== "oversized") return undefined; + let size = 0; + try { + size = fs.statSync(fullPath).size; + } catch { + return undefined; + } + return oversizedModuleFileError(relPath, size); +} + +/** Whether a path is inside a dbt project's module folder specifically: those + * files are taken verbatim, with no language inference. */ +export function isDbtModulePath(p: string): boolean { + // The OUTERMOST boundary decides, like `getScriptBasePathFromModulePath`. + // Scanning anywhere in the path would call `foo__mod/vendor/x__dbt/a.ts` a dbt + // project file, and the push would then look for `foo.script.yaml` instead of + // the ordinary module entry point and skip the edit. + const norm = normalizeSep(p); + const base = getScriptBasePathFromModulePath(norm); + return base !== undefined && norm.startsWith(base + DBT_MODULE_SUFFIX + "/"); +} + +/** dbt writes these; a project authors them nowhere. Importing a stale + * `target/` would ship a manifest this runtime then reads as the graph, and + * `dbt_packages/` is a vendored copy the worker restores from its own cache. */ +const DBT_GENERATED_DIRS = ["target", "dbt_packages", "logs", ".git", ".venv", "__pycache__"]; + +const dbtGeneratedDirsCache = new Map< + string, + { stamp: string; dirs: Set } +>(); + +/** `{{ env_var('NAME') }}` / `{{ env_var("NAME", "default") }}`. */ +const DBT_ENV_VAR_CALL = + /\{\{\s*env_var\(\s*['"]([^'"]+)['"]\s*(?:,\s*['"]([^'"]*)['"]\s*)?\)\s*\}\}/g; + +/** + * Render `dbt_project.yml`'s own `env_var()` calls, which dbt allows there too. + * A directory setting left as its template names no directory on disk, so the + * generated tree it points at would be bundled as project source. + * + * Against `process.env`, because the CLI runs where the project was built: that + * is the environment dbt used to produce the tree being read. + */ +export function renderDbtEnvVars(value: string): string { + return value.replace( + DBT_ENV_VAR_CALL, + (whole, name: string, fallback: string | undefined) => + process.env[name] ?? fallback ?? whole, + ); +} + +/** + * Files that keep a project's secrets next to it rather than in it: dbt reads + * none of them (`env_var()` takes the process environment), and the documented + * way into a bundle is `cp -r my-project/.`, which copies whatever the checkout + * holds — including the `.env` a `.gitignore` was keeping out of the repo. + */ +export function isLocalSecretFile(name: string): boolean { + return name === ".env" || name.startsWith(".env.") || name === ".envrc"; +} + +/** + * Directories to leave out of a dbt project's module bundle, as project-relative + * paths — `target-path` and friends may be nested (`build/target`). + * + * `target-path`, `packages-install-path` and `clean-targets` are configurable, + * so they are read from the project rather than assumed. Cached per project + * folder: this is called once per file of a sync. + */ +export function dbtGeneratedDirs(moduleFolderPath: string): Set { + const projectFile = path.join(moduleFolderPath, "dbt_project.yml"); + // Cached against the project file's identity, not merely its folder: `wmill + // dev` is a long-running process, so a `target-path` edited mid-session would + // otherwise keep excluding the old directory and start bundling the new one + // as project source. One entry per folder, replaced when the file changes. + let stamp = ""; + try { + const st = fs.statSync(projectFile); + stamp = `${st.mtimeMs}:${st.size}`; + } catch { + // No project file yet: the defaults apply, and "absent" is its own stamp. + } + const cached = dbtGeneratedDirsCache.get(moduleFolderPath); + if (cached && cached.stamp === stamp) return cached.dirs; + const dirs = new Set(DBT_GENERATED_DIRS); + const add = (raw: string) => { + const v = normalizeSep(renderDbtEnvVars(raw).trim().replace(/^["']|["']$/g, "")) + .replace(/^\.\//, "") + .replace(/\/+$/, ""); + // A configured path that escapes the project is dbt's problem, not ours; + // ignoring it here just means those files stay in the bundle. + if (v && !v.startsWith("/") && !v.split("/").includes("..")) dirs.add(v); + }; + try { + const projectYml = fs.readFileSync(projectFile, "utf-8"); + for (const m of projectYml.matchAll( + /^\s*(?:target-path|packages-install-path)\s*:\s*([^\n#]+)/gm, + )) { + add(m[1]); + } + // `clean-targets` in either of dbt's two spellings: inline `[a, b]`, and the + // block form, whose entries are on the lines that follow. + const lines = projectYml.split(/\r?\n/); + for (let i = 0; i < lines.length; i++) { + const head = lines[i].match(/^\s*clean-targets\s*:\s*(.*)$/); + if (!head) continue; + const inline = head[1].match(/^\[([^\]]*)\]/); + if (inline) { + inline[1].split(",").forEach(add); + continue; + } + for (let j = i + 1; j < lines.length; j++) { + const item = lines[j].match(/^\s+-\s*([^\n#]+)$/); + if (!item) break; + add(item[1]); + } + } + } catch { + // No dbt_project.yml yet (a descriptor pushed before its project): the + // defaults still apply. + } + dbtGeneratedDirsCache.set(moduleFolderPath, { stamp, dirs }); + return dirs; +} + +/** + * Whether a project-relative path sits inside one of `dirs`. Compared segment + * by segment: `targetx/a` must not match a configured `target`. + */ +export function isUnderGeneratedDir(rel: string, dirs: Set): boolean { + const n = normalizeSep(rel); + for (const d of dirs) { + if (n === d || n.startsWith(d + "/")) return true; + } + return false; +} + +/** + * Whether a path under a `__dbt/` folder is one dbt generated rather than one + * the project authors. Those never belong to the bundle, so sync must not offer + * them as items of their own either. + */ +export function isDbtGeneratedPath(p: string): boolean { + const n = normalizeSep(p); + // Anchored on the outermost boundary, like every other helper here. Matching + // `__dbt/` anywhere would call `foo__mod/vendor/x__dbt/target/a.ts` generated + // dbt output, and `ignoreF` would then exclude an ordinary module file so a + // module-only edit never deploys its parent script. + if (!isDbtModulePath(n)) return false; + const base = getScriptBasePathFromModulePath(n)!; + const projectRoot = base + DBT_MODULE_SUFFIX; + const rel = n.slice(projectRoot.length + 1); + if (isUnderGeneratedDir(rel, dbtGeneratedDirs(projectRoot))) { + return true; + } + // Not generated, but not carried either: the bundle drops it, so the diff + // must not keep offering it as a pending change. An OVERSIZED one is the + // exception — see `moduleFileExclusion`: hiding it here is what would make an + // edit to a large seed report no change at all. + return ( + isLocalSecretFile(n.slice(n.lastIndexOf("/") + 1)) || + moduleFileExclusion(p) === "binary" + ); } /** * Build the module folder path from a script's base path (without extension). - * e.g., "f/my_script" -> "f/my_script__mod" + * e.g., "f/my_script" -> "f/my_script__mod", or "__dbt" for a dbt project. */ -export function buildModuleFolderPath(scriptBasePath: string): string { - return scriptBasePath + MODULE_SUFFIX; +export function buildModuleFolderPath(scriptBasePath: string, language?: string): string { + return scriptBasePath + getModuleFolderSuffix(language); } /** @@ -533,10 +829,21 @@ export function buildModuleFolderPath(scriptBasePath: string): string { */ export function isModuleEntryPoint(p: string): boolean { const norm = normalizeSep(p); + // Anchored on the OUTERMOST module boundary, like + // `getScriptBasePathFromModulePath`. Scanning for `__mod/` alone would match a + // `legacy__mod/` directory nested inside a dbt project — dbt owns those names + // verbatim — and call its `script.ts` this script's entry point. + const base = getScriptBasePathFromModulePath(norm); + if (base === undefined) return false; + // A dbt project's entry point is its descriptor, which sits INSIDE the + // project so that an author writes nothing outside the directory dbt itself + // reads. + if (norm.startsWith(base + DBT_MODULE_SUFFIX + "/")) { + return norm === dbtDescriptorPath(base); + } const suffix = MODULE_SUFFIX + "/"; - const idx = norm.indexOf(suffix); - if (idx === -1) return false; - const rest = norm.slice(idx + suffix.length); + if (!norm.startsWith(base + suffix)) return false; + const rest = norm.slice(base.length + suffix.length); return rest.startsWith("script.") && !rest.includes("/"); } @@ -544,13 +851,21 @@ export function isModuleEntryPoint(p: string): boolean { * Extract the script base path from a module folder entry. * e.g., "u/admin/my_script__mod/script.ts" -> "u/admin/my_script" * e.g., "u/admin/my_script__mod/helper.ts" -> "u/admin/my_script" + * e.g., "f/x/proj__dbt/models/a.sql" -> "f/x/proj" */ export function getScriptBasePathFromModulePath(p: string): string | undefined { const norm = normalizeSep(p); - const suffix = MODULE_SUFFIX + "/"; - const idx = norm.indexOf(suffix); - if (idx === -1) return undefined; - return norm.slice(0, idx); + // The OUTERMOST boundary, not the first suffix that happens to match. A dbt + // project's directories are the author's verbatim, so `foo__dbt/models/ + // legacy__mod/a.sql` is legal — taking `__mod` first would call + // `foo__dbt/models/legacy` the script and look for a descriptor that is not + // there, silently skipping the deploy. + let best: number | undefined; + for (const suffix of MODULE_SUFFIXES) { + const idx = norm.indexOf(suffix + "/"); + if (idx !== -1 && (best === undefined || idx < best)) best = idx; + } + return best === undefined ? undefined : norm.slice(0, best); } /** diff --git a/cli/src/utils/resource_types.ts b/cli/src/utils/resource_types.ts index 5bd56c9a8e..8317ee7ab5 100644 --- a/cli/src/utils/resource_types.ts +++ b/cli/src/utils/resource_types.ts @@ -4,18 +4,29 @@ function quotePropName(name: string): string { return /^[a-zA-Z_$][a-zA-Z0-9_$]*$/.test(name) ? name : JSON.stringify(name); } -export function compileResourceTypeToTsType(schema: Schema) { - function rec(x: { [name: string]: SchemaProperty }, root = false) { - let res = "{\n"; +function isPropertyMap(x: unknown): x is { [name: string]: SchemaProperty } { + return typeof x === "object" && x !== null && !Array.isArray(x); +} + +// Schemas are free-form jsonb: the column is nullable and hub types such as +// `record` or `dbt_profile` carry `{}` / `{"type":"object"}` with no +// `properties`. Anything that is not a property map compiles to `any`, since a +// throw here aborts the whole rt.d.ts generation. +export function compileResourceTypeToTsType(schema: Schema | undefined | null) { + function rec(x: unknown): string { + if (!isPropertyMap(x)) { + return "any"; + } const entries = Object.entries(x); if (entries.length == 0) { return "any"; } + let res = "{\n"; let i = 0; for (let [name, prop] of entries) { - if (prop.type == "object") { - res += ` ${quotePropName(name)}: ${rec(prop.properties ?? {})}`; - } else if (prop.type == "array") { + if (prop?.type == "object") { + res += ` ${quotePropName(name)}: ${rec(prop.properties)}`; + } else if (prop?.type == "array") { res += ` ${quotePropName(name)}: ${prop?.items?.type ?? "any"}[]`; } else { let typ = prop?.type ?? "any"; @@ -33,5 +44,5 @@ export function compileResourceTypeToTsType(schema: Schema) { return res; } - return rec(schema.properties, true); + return rec(schema?.properties); } diff --git a/cli/src/utils/script_common.ts b/cli/src/utils/script_common.ts index a713e41782..d804318491 100644 --- a/cli/src/utils/script_common.ts +++ b/cli/src/utils/script_common.ts @@ -1,3 +1,4 @@ +import { isDbtDescriptorPath } from "./resource_folders.ts"; export type ScriptLanguage = | "python3" | "deno" @@ -21,6 +22,7 @@ export type ScriptLanguage = | "ansible" | "ruby" | "rlang" + | "dbt" | "java"; // for related places search: ADD_NEW_LANG @@ -42,6 +44,222 @@ export const workspaceDependenciesLanguages: WorkspaceDependenciesLanguage[] = [ { language: "powershell", filename: "modules.json" }, ] as const; +export function workspaceDependenciesPathToLanguageAndFilename(path: string): { name: string | undefined, language: ScriptLanguage } | undefined { + const relativePath = path.replace("dependencies/", ""); + for (const { filename, language } of workspaceDependenciesLanguages) { + if (relativePath.endsWith(filename)) { + return { + name: relativePath === filename ? undefined : relativePath.replace("." + filename, ""), + language + }; + } + } +} + +// --------------------------------------------------------------------------- +// Annotation parser — mirrors backend's WorkspaceDependenciesAnnotatedRefs::parse +// (windmill-common/src/workspace_dependencies.rs), so the CLI can tell which +// workspace dependency file a script resolves against without asking a worker. +// --------------------------------------------------------------------------- + +export type AnnotationMode = "manual" | "extra"; + +export interface WorkspaceDepsAnnotation { + mode: AnnotationMode; + external: string[]; + inline: string | null; +} + +const LANG_ANNOTATION_CONFIG: Partial< + Record +> = { + python3: { comment: "#", keyword: "requirements", validityRe: /^#\s?(\S+)\s*$/ }, + bun: { comment: "//", keyword: "package_json" }, + nativets: { comment: "//", keyword: "package_json" }, + go: { comment: "//", keyword: "go_mod" }, + php: { comment: "//", keyword: "composer_json" }, + powershell: { comment: "#", keyword: "modules_json" }, +}; + +export function extractWorkspaceDepsAnnotation( + scriptContent: string, + language: ScriptLanguage, +): WorkspaceDepsAnnotation | null { + const config = LANG_ANNOTATION_CONFIG[language]; + if (!config) return null; + + const { comment, keyword, validityRe } = config; + const extraMarkerUnderscore = `extra_${keyword}:`; + const extraMarkerHyphen = `extra-${keyword}:`; + const manualMarker = `${keyword}:`; + + const stripComment = (l: string): string | null => { + if (!l.startsWith(comment)) return null; + return l.substring(comment.length).trimStart(); + }; + const isExtra = (l: string): boolean => { + const s = stripComment(l); + return s !== null && (s.startsWith(extraMarkerUnderscore) || s.startsWith(extraMarkerHyphen)); + }; + const isManual = (l: string): boolean => { + const s = stripComment(l); + return s !== null && s.startsWith(manualMarker); + }; + + const lines = scriptContent.split("\n"); + + // Find first annotation line (mirrors Rust find_position) + let pos = -1; + for (let i = 0; i < lines.length; i++) { + if (isExtra(lines[i]) || isManual(lines[i])) { + pos = i; + break; + } + } + if (pos === -1) return null; + + const annotationLine = lines[pos]; + const mode: AnnotationMode = isExtra(annotationLine) ? "extra" : "manual"; + + // Parse external references from the annotation line + const marker = mode === "extra" + ? (annotationLine.includes(extraMarkerUnderscore) ? extraMarkerUnderscore : extraMarkerHyphen) + : manualMarker; + const unparsed = annotationLine.replaceAll(marker, "").replaceAll(comment, ""); + const external = unparsed + .split(",") + .map((s) => s.trim()) + .filter((s) => s.length > 0); + + // Parse inline deps from subsequent lines + const inlineParts: string[] = []; + for (let i = pos + 1; i < lines.length; i++) { + const l = lines[i]; + if (validityRe) { + const match = validityRe.exec(l); + if (match && match[1]) { + inlineParts.push(match[1]); + } else { + break; + } + } else { + if (!l.startsWith(comment)) { + break; + } + inlineParts.push(l.substring(comment.length)); + } + } + + const inlineStr = inlineParts.join("\n"); + const inline = inlineStr.trim().length > 0 ? inlineStr : null; + + return { mode, external, inline }; +} + +/** The comment marker each language's annotations are written behind. */ +export const LANG_COMMENT_LIT: Partial> = { + python3: "#", + ansible: "#", + powershell: "#", + bun: "//", + nativets: "//", + deno: "//", + go: "//", + php: "//", + rust: "//!", +}; + +/** + * The annotations each language recognises, by the exact names the worker + * matches (`#[annotations(..)]` structs in windmill-common/src/worker.rs). + * Several change what it locks — a pinned interpreter, `npm`, `nobundling` — + * and the rest are cheap to treat the same way, since the only cost is that + * such a script keeps a lockfile of its own. + * for related places search: ADD_NEW_LANG + */ +const LANG_ANNOTATIONS: Partial> = { + python3: [ + "no_cache", + "no_postinstall", + "py_select_latest", + "skip_result_postprocessing", + "py310", + "py311", + "py312", + "py313", + "sandbox", + ], + bun: ["npm", "nodejs", "native", "nobundling", "sandbox"], + nativets: ["npm", "nodejs", "native", "nobundling", "sandbox"], + deno: ["npm", "nodejs", "native", "nobundling", "sandbox"], + go: ["go1_22_compat"], +}; + +/** + * Whether a script's leading comment block carries an annotation the worker + * acts on, which means its lock may not be its dependency file's. + * + * Matched the way the worker matches: the key is the line, or what precedes the + * first `=`, and it has to BE one of the names above. Unknown keys are ignored + * there and so here — which is what keeps `# TODO:` or `# type: ignore` from + * quietly dropping an ordinary documented script out of deduplication. + * + * `# py: ` is the exception the macro does not cover: the python + * import parser reads it directly (`windmill-parser-py-imports`, alongside the + * `py310`..`py313` flags) to pick the interpreter, which changes what resolves. + */ +export function hasLockAffectingAnnotation( + scriptContent: string, + language: ScriptLanguage, +): boolean { + const comment = LANG_COMMENT_LIT[language]; + const names = LANG_ANNOTATIONS[language]; + if (!comment || !names) return false; + for (const line of scriptContent.split("\n")) { + const trimmed = line.trim(); + if (trimmed === "") continue; + if (!trimmed.startsWith(comment)) break; // past the header block + // Matched on the raw line: the parser tests `# py:`/`#py:` before trimming. + if (language === "python3" && /^#\s?py:/.test(line)) return true; + const body = trimmed.slice(comment.length).trim(); + const key = body.split("=")[0].trim(); + if (names.includes(key)) return true; + } + return false; +} + +/** Where the lockfiles shared by several scripts live when `dedupeLockfiles` + * is on — see `utils/lock_dedup.ts`. A top-level directory of its own: what a + * group shares is a resolved lock, which needs no workspace dependency file + * behind it, and inline-script locks would belong here too. */ +export const SHARED_LOCK_DIR = "locks"; + +/** The lockfile shared by the scripts that resolve against a workspace + * dependency file: its own name, plus `.lock`. Appending rather than replacing + * the extension keeps the correspondence exact and reversible — + * `dependencies/team_a.requirements.in` <-> `locks/team_a.requirements.in.lock`. */ +export function sharedLockPathFor(depFilePath: string): string { + const name = depFilePath.replaceAll("\\", "/").split("/").pop()!; + return `${SHARED_LOCK_DIR}/${name}.lock`; +} + +/** The workspace dependency file a shared lockfile belongs to, if it is one. */ +export function depFileOfSharedLock(p: string): string | undefined { + const normalized = p.replaceAll("\\", "/"); + if (!normalized.startsWith(SHARED_LOCK_DIR + "/")) return undefined; + const name = normalized.slice(SHARED_LOCK_DIR.length + 1); + if (name.includes("/") || !name.endsWith(".lock")) return undefined; + const depFile = "dependencies/" + name.slice(0, -".lock".length); + const info = workspaceDependenciesPathToLanguageAndFilename(depFile); + // `locks/vendor.lock` names no dependency file, so it is not Windmill's: a + // repo that already keeps lockfiles here keeps them. + return info && languageNeedsLock(info.language) ? depFile : undefined; +} + +export function isSharedLockPath(p: string): boolean { + return depFileOfSharedLock(p) !== undefined; +} + /** * Returns true if a script in the given language requires a lock file. * Matches the condition in updateScriptLock (metadata.ts). @@ -106,6 +324,8 @@ export function inferContentTypeFromFilePath( return "java"; } else if (contentPath.endsWith(".rb")) { return "ruby"; + } else if (isDbtDescriptorPath(contentPath)) { + return "dbt"; } else if (contentPath.endsWith(".r")) { return "rlang"; // for related places search: ADD_NEW_LANG @@ -119,7 +339,7 @@ export function inferContentTypeFromFilePath( throw new Error( `Cannot infer script language from extension '${ext}' (file ${contentPath}).` + hint + - "\nSupported extensions: .ts (bun/deno), .py, .go, .sh, .ps1, .php, .rs, .cs, .nu, .java, .rb, .r, .gql, .playbook.yml, .pg.sql, .my.sql, .bq.sql, .sf.sql, .ms.sql, .odb.sql, .duckdb.sql" + "\nSupported extensions: .ts (bun/deno), .py, .go, .sh, .ps1, .php, .rs, .cs, .nu, .java, .rb, .r, .gql, .playbook.yml, .pg.sql, .my.sql, .bq.sql, .sf.sql, .ms.sql, .odb.sql, .duckdb.sql, and a dbt project folder `__dbt/`" ); } } diff --git a/cli/src/utils/utils.ts b/cli/src/utils/utils.ts index 4af76517fa..489a877af7 100644 --- a/cli/src/utils/utils.ts +++ b/cli/src/utils/utils.ts @@ -200,6 +200,20 @@ export function isFileResource(path: string): boolean { ); } +/** + * Local resource path -> the resource's path on the server. The suffix is + * `.resource.` on a metadata file but `.resource.file.` on a + * file resource, so its length depends on which of the two the path is. + */ +export function removeResourceSuffix(path: string): string { + if (isFileResource(path)) { + // isFileResource only matches a dotless extension, so the resource path is + // everything before the trailing `resource`, `file`, `` segments. + return path.split(".").slice(0, -3).join("."); + } + return path.replace(/\.resource\.(yaml|json)$/, ""); +} + /** Matches children inside a .fileset/ directory, not the directory itself. */ export function isFilesetResource(path: string): boolean { return path.includes(".fileset/") || path.includes(".fileset\\"); diff --git a/cli/test/app_policy_bundle_unit.test.ts b/cli/test/app_policy_bundle_unit.test.ts new file mode 100644 index 0000000000..4da05462ad --- /dev/null +++ b/cli/test/app_policy_bundle_unit.test.ts @@ -0,0 +1,69 @@ +/** + * The raw-app bundle job carries the frontend's policy derivation, vendored by + * cli/generate-app-policy.ts into backend/windmill-api/src/apps_raw_policy.gen.js + * and prepended to the job script. + * + * If that copy drifts from the frontend source, deployed apps get policy keys + * the app editor would not have written, and every runnable is refused at run + * time with "forbidden by policy" — an app that deploys and then does nothing. + * So rebuild the bundle here and fail when the committed one no longer matches. + * Fix by running `bun run gen:app-policy` from cli/. + * + * No backend required. + */ + +import { expect, test, describe } from "bun:test"; +import { readFileSync } from "node:fs"; +import { buildAppPolicyBundle, OUT_FILE } from "../generate-app-policy.ts"; + +describe("raw app policy bundle", () => { + test("the committed bundle matches the frontend source", async () => { + // Line endings normalized: a CRLF checkout is the same bundle, and must not + // read as drift (the committed file's header arrives as CRLF on Windows). + const lf = (s: string) => s.replace(/\r\n/g, "\n"); + expect(lf(readFileSync(OUT_FILE, "utf-8"))).toBe( + lf(await buildAppPolicyBundle()), + ); + }); + + test("derives the keys the app editor writes", async () => { + // Exercise the committed artifact itself, not the frontend module: it is + // what actually runs on the worker. + // A module's top-level `var` is not a global, and the job prepends this + // bundle into its own module, so reach the binding the same way it does. + const { updateRawAppPolicy } = new Function( + `${readFileSync(OUT_FILE, "utf-8")}\nreturn __wmillAppPolicy`, + )(); + + const content = "export async function main(a: string) { return a }\n"; + const sha = new Bun.CryptoHasher("sha256").update(content).digest("hex"); + + const policy = await updateRawAppPolicy( + { + inline: { + type: "inline", + inlineScript: { content, language: "bun" }, + fields: { + pinned: { type: "static", value: "by-the-publisher" }, + secret: { type: "static", value: "shh", sensitive: true }, + }, + }, + by_flow: { type: "path", runType: "flow", path: "u/admin/f", fields: {} }, + }, + undefined, + ); + + expect(Object.keys(policy.triggerables_v2).sort()).toEqual([ + "by_flow:flow/u/admin/f", + `inline:rawscript/${sha}`, + ]); + // `sensitive_inputs` is what makes the server encrypt the arg before it + // reaches the job, so losing it would silently store the value in plaintext. + const inline = policy.triggerables_v2[`inline:rawscript/${sha}`]; + expect(inline.static_inputs).toEqual({ + pinned: "by-the-publisher", + secret: "shh", + }); + expect(inline.sensitive_inputs).toEqual(["secret"]); + }); +}); diff --git a/cli/test/datatable_migrations_unit.test.ts b/cli/test/datatable_migrations_unit.test.ts index 09af49f890..2db9371629 100644 --- a/cli/test/datatable_migrations_unit.test.ts +++ b/cli/test/datatable_migrations_unit.test.ts @@ -15,6 +15,7 @@ import * as path from "node:path"; import * as os from "node:os"; import { parseDatatableMigrationPath } from "../src/types.ts"; import { validateLocalMigrations } from "../src/commands/datatable_migrations.ts"; +import { untrackedDatatableMigrationDeletions } from "../src/commands/sync/sync.ts"; describe("parseDatatableMigrationPath", () => { test("parses up and down files of the new layout", () => { @@ -120,3 +121,61 @@ describe("validateLocalMigrations", () => { expect(validateLocalMigrations()).toEqual([]); }); }); + +// ============================================================================= +// untrackedDatatableMigrationDeletions — the push-side safety net. +// +// Migrations bypass the repo's path filters, so a clone made before they were +// synced sees every server-side migration as remote-only, and +// `pushMigrationFromDisk` reads a missing `.up.sql` as "delete it". What the repo +// has ever committed under migrations/datatable/ is the durable answer to "did we +// track this?" — the working tree is not, because creating a migration locally +// makes the directory appear without anything having been tracked. +// ============================================================================= + +describe("untrackedDatatableMigrationDeletions", () => { + const A_UP = "migrations/datatable/mydb/20260101000000_a.up.sql"; + const A_DOWN = "migrations/datatable/mydb/20260101000000_a.down.sql"; + const changes = [ + { name: "deleted", path: A_UP }, + { name: "deleted", path: A_DOWN }, + { name: "deleted", path: "f/foo/bar.script.yaml" }, + { name: "added", path: "migrations/datatable/mydb/20260102000000_b.up.sql" }, + ]; + + test("trusts a deletion the repository has committed before", () => { + expect( + untrackedDatatableMigrationDeletions(changes, { kind: "known", paths: new Set([A_UP, A_DOWN]) }), + ).toEqual([]); + }); + + test("flags migrations this repository has never recorded", () => { + expect( + untrackedDatatableMigrationDeletions(changes, { kind: "known", paths: new Set() }).map((c) => c.path), + ).toEqual([A_UP, A_DOWN]); + }); + + test("a locally created migration does not vouch for unrelated ones", () => { + // `wmill datatable migrate new` makes migrations/datatable/ exist without the + // checkout having tracked anything, so only the recorded paths count. + const recorded = { + kind: "known" as const, + paths: new Set(["migrations/datatable/mydb/20260102000000_b.up.sql"]), + }; + expect( + untrackedDatatableMigrationDeletions(changes, recorded).map((c) => c.path), + ).toEqual([A_UP, A_DOWN]); + }); + + test("trusts nothing when the history cannot be consulted", () => { + // A shallow clone or a non-repository can't prove a path was never tracked, + // so absence is not read as permission to delete. + expect( + untrackedDatatableMigrationDeletions(changes, { + kind: "unknown", + reason: "this is a shallow clone, so its history is truncated", + remedy: "Fetch the full history (for actions/checkout, fetch-depth: 0)", + }).map((c) => c.path), + ).toEqual([A_UP, A_DOWN]); + }); +}); diff --git a/cli/test/dbt_module_tracker_unit.test.ts b/cli/test/dbt_module_tracker_unit.test.ts new file mode 100644 index 0000000000..90abee18ae --- /dev/null +++ b/cli/test/dbt_module_tracker_unit.test.ts @@ -0,0 +1,145 @@ +/** + * `buildTracker` decides whose top hash `wmill-lock.yaml` refreshes. A dbt + * project is mostly files that are not Windmill script extensions — the project + * file, `packages.yml`, schema YAML, seed CSVs — and its folder is spelled + * `__dbt\` on Windows, so both the extension gate and a raw-path search for + * `__dbt/` left the descriptor untracked and its hash stale. + */ +import { expect, test, describe, beforeEach, afterEach } from "bun:test"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import * as os from "node:os"; +import { buildTracker, elementsToMap } from "../src/commands/sync/sync.ts"; +import { isDbtGeneratedPath } from "../src/utils/resource_folders.ts"; +import { readModulesFromDisk } from "../src/commands/script/script.ts"; + +describe("buildTracker with a dbt project", () => { + let dir: string; + let cwd: string; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), "wmill-dbt-tracker-")); + cwd = process.cwd(); + process.chdir(dir); + fs.mkdirSync(path.join(dir, "f/analytics/analytics__dbt/models"), { + recursive: true, + }); + fs.mkdirSync(path.join(dir, "f/analytics/analytics__dbt/seeds"), { + recursive: true, + }); + // The descriptor sits INSIDE the project folder, and `findContentFile` + // resolves the metadata path to it. + fs.writeFileSync(path.join(dir, "f/analytics/analytics.script.yaml"), "{}"); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/dbt_project.yml"), + "name: analytics\n", + ); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/wm_dbt.yaml"), + "profile: {}\n", + ); + }); + + afterEach(() => { + process.chdir(cwd); + fs.rmSync(dir, { recursive: true, force: true }); + }); + + const tracked = async (p: string) => + (await buildTracker([{ name: "edited", path: p, before: "", after: "" }])) + .scripts; + + test("a model edit selects the descriptor", async () => { + expect( + await tracked("f/analytics/analytics__dbt/models/stg_orders.sql"), + ).toEqual(["f/analytics/analytics__dbt/wm_dbt.yaml"]); + }); + + test("so do the files that are not script extensions", async () => { + for (const p of [ + "f/analytics/analytics__dbt/dbt_project.yml", + "f/analytics/analytics__dbt/packages.yml", + "f/analytics/analytics__dbt/models/_models.yml", + "f/analytics/analytics__dbt/seeds/country_codes.csv", + ]) { + expect(await tracked(p)).toEqual( + ["f/analytics/analytics__dbt/wm_dbt.yaml"], + `${p} left the descriptor untracked`, + ); + } + }); + + // Regression: hoisting the module check above the extension gate made + // `__mod/script.yaml` — a folder-layout script's METADATA, which is an + // entry-point path — look like its own content file. Pushed as one, the + // metadata pass asks for the language of `.yaml` and aborts the command. Not a + // dbt shape at all; reached by editing the summary of any modular script. + test("a modular script's own metadata resolves to its content file", async () => { + fs.mkdirSync(path.join(dir, "f/helpers/util__mod"), { recursive: true }); + fs.writeFileSync(path.join(dir, "f/helpers/util__mod/script.yaml"), "{}"); + fs.writeFileSync( + path.join(dir, "f/helpers/util__mod/script.ts"), + "export function main() {}\n", + ); + expect(await tracked("f/helpers/util__mod/script.yaml")).toEqual([ + "f/helpers/util__mod/script.ts", + ]); + }); + + test("and a Windows-separated path", async () => { + expect( + await tracked("f\\analytics\\analytics__dbt\\models\\stg_orders.sql"), + ).toEqual(["f/analytics/analytics__dbt/wm_dbt.yaml"]); + }); + + // A dbt descriptor is the script's CONTENT and is a `.yaml` inside the + // project folder. `--json` drops every metadata `.yaml` as the twin it does + // not read — dropping this one too leaves a workspace whose dbt scripts have + // metadata, a lock and a project bundle, but nothing to run. + test("--json keeps the descriptor while dropping metadata yaml", async () => { + const file = (path: string) => ({ + path, + isDirectory: false, + getChildren: async function* () {}, + getContentText: async () => "x", + }); + const root = { + path: "", + isDirectory: true, + getChildren: async function* () { + yield file("f/analytics/analytics__dbt/wm_dbt.yaml"); + yield file("f/analytics/analytics.script.yaml"); + yield file("f/analytics/analytics.script.json"); + }, + getContentText: async () => "", + }; + const map = await elementsToMap(root as any, () => false, true, {}); + expect(Object.keys(map).sort()).toEqual([ + "f/analytics/analytics.script.json", + "f/analytics/analytics__dbt/wm_dbt.yaml", + ]); + }); + + // `cp -r my-dbt-project/.` copies whatever the checkout holds, and what a + // `.gitignore` was keeping out of the repo is exactly the file that must not + // become a script version. Both halves: bundled, it is uploaded; offered by + // the diff, every push asks to upload it again. + test("a local .env is neither bundled nor offered as a change", async () => { + const project = path.join(dir, "f/analytics/analytics__dbt"); + fs.writeFileSync(path.join(project, ".env"), "DBT_PASSWORD=hunter2\n"); + fs.writeFileSync(path.join(project, "models/stg.sql"), "select 1"); + + const modules = await readModulesFromDisk(project, undefined, false, true); + // Sorted: the bundle is a set of paths, and the walk follows `readdirSync`, + // whose order is the filesystem's. + expect(Object.keys(modules ?? {}).sort()).toEqual([ + "dbt_project.yml", + "models/stg.sql", + ]); + + // The predicate the sync's ignore filter asks, so the file is not offered + // as an item of its own either. + expect(isDbtGeneratedPath("f/analytics/analytics__dbt/.env")).toBe(true); + expect(isDbtGeneratedPath("f/analytics/analytics__dbt/models/stg.sql")).toBe(false); + }); +}); diff --git a/cli/test/dbt_optional_descriptor_unit.test.ts b/cli/test/dbt_optional_descriptor_unit.test.ts new file mode 100644 index 0000000000..f44bdc6103 --- /dev/null +++ b/cli/test/dbt_optional_descriptor_unit.test.ts @@ -0,0 +1,323 @@ +/** + * An unmodified dbt project is already a complete Windmill script: the + * descriptor is optional, and a project that never names one must push, diff + * and pull without ever growing a Windmill file inside it. + */ +import { expect, test, describe, beforeEach, afterEach } from "bun:test"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import * as os from "node:os"; +import { FSFSElement, elementsToMap } from "../src/commands/sync/sync.ts"; +import { listWorkspacePaths } from "../src/commands/dev/dev.ts"; +import { + DbtPathCollisionError, + findContentFile, + handleFile, + hasScriptExt, + removeExtensionToPath, +} from "../src/commands/script/script.ts"; +import { pushParentScriptForModule } from "../src/commands/sync/sync.ts"; + +/** The local map's keys are the walk's own — `path.join`, so `__dbt\\` on + * Windows — while a remote's are the API's. The synthesized descriptor follows + * the spelling of the `dbt_project.yml` it was derived from, like every other + * key in that map, so an assertion on one platform's separator tests the + * platform and not the synthesis. */ +const normalized = (m: Record) => + Object.fromEntries( + Object.entries(m).map(([k, v]) => [k.replaceAll("\\", "/"), v]), + ); + +describe("a dbt project without a descriptor", () => { + let dir: string; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), "wmill-dbt-nodesc-")); + fs.mkdirSync(path.join(dir, "f/analytics/analytics__dbt/models"), { + recursive: true, + }); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/dbt_project.yml"), + "name: analytics\n", + ); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/models/stg_orders.sql"), + "select 1", + ); + fs.writeFileSync(path.join(dir, "f/analytics/analytics.script.yaml"), "{}"); + }); + + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + // Without this the project has no content file, so nothing identifies it as a + // script and the whole project silently never deploys. + test("is still discovered, as an empty descriptor", async () => { + const root = await FSFSElement(dir, [], true); + const map = await elementsToMap(root, () => false, false, {}); + expect(normalized(map)["f/analytics/analytics__dbt/wm_dbt.yaml"]).toBe(""); + }); + + // The metadata has to resolve to a content path that is not on disk, or every + // caller that goes metadata -> content aborts the push. + test("resolves from its metadata to the absent descriptor", async () => { + const cwd = process.cwd(); + process.chdir(dir); + try { + expect(await findContentFile("f/analytics/analytics.script.yaml")).toBe( + "f/analytics/analytics__dbt/wm_dbt.yaml", + ); + } finally { + process.chdir(cwd); + } + }); + + // The descriptor is the one "extension" that contains a separator, so on + // Windows it is spelled `__dbt\wm_dbt.yaml` and a forward-slash suffix test + // matches nothing — every dbt project skipped, with no error. + test("is recognized when the path is spelled with backslashes", () => { + const win = "f\\analytics\\analytics__dbt\\wm_dbt.yaml"; + expect(hasScriptExt(win)).toBe(true); + expect(removeExtensionToPath(win)).toBe("f\\analytics\\analytics"); + }); + + // Both `.py` and `__dbt/` deploy to the SAME remote path, and the + // descriptor is optional — so the project is invisible to the candidate list + // while being perfectly real. Resolved to the ordinary file, a model edit + // deploys the Python script over the dbt one. + test("refuses to resolve when an ordinary script shares its path", async () => { + const cwd = process.cwd(); + process.chdir(dir); + try { + fs.writeFileSync(path.join(dir, "f/analytics/analytics.py"), "def main(): ..."); + const err = await findContentFile("f/analytics/analytics.script.yaml").then( + () => undefined, + (e) => e as Error, + ); + expect(err?.message).toContain("f/analytics/analytics__dbt/dbt_project.yml"); + expect(err?.message).toContain("f/analytics/analytics.py"); + } finally { + process.chdir(cwd); + } + }); + + // The guard has to sit on the push paths themselves, not only on the + // metadata->content resolution: an ordinary file goes straight to + // `handleFile`, and a module edit reaches its parent through a call whose + // errors were swallowed — so each path could still overwrite the other's + // script while reporting success. + test("both push paths refuse the collision", async () => { + const cwd = process.cwd(); + process.chdir(dir); + try { + fs.writeFileSync(path.join(dir, "f/analytics/analytics.py"), "def main(): ..."); + const ordinary = await handleFile( + "f/analytics/analytics.py", + { workspaceId: "w", remote: "http://localhost", name: "w", token: "t" } as any, + [], + undefined, + undefined, + {}, + [], + ).then( + () => undefined, + (e) => e as Error, + ); + expect(ordinary).toBeInstanceOf(DbtPathCollisionError); + expect(ordinary?.message).toContain("f/analytics/analytics.py"); + + const model = await pushParentScriptForModule( + "f/analytics/analytics__dbt/models/stg_orders.sql", + { workspaceId: "w", remote: "http://localhost", name: "w", token: "t" } as any, + [], + undefined, + undefined, + {}, + [], + ).then( + () => undefined, + (e) => e as Error, + ); + expect(model).toBeInstanceOf(DbtPathCollisionError); + } finally { + process.chdir(cwd); + } + }); + + // The guard above must not fire on the project's OWN descriptor: that file is + // the dbt script's content, and its base resolves to the same + // `__dbt/dbt_project.yml` — so a naive check finds the project + // colliding with itself and every dbt push fails before deploying anything. + test("a project does not collide with itself", async () => { + const cwd = process.cwd(); + process.chdir(dir); + const remote = { workspaceId: "w", remote: "http://127.0.0.1:1", name: "w", token: "t" }; + const push = (p: string) => + handleFile(p, remote as any, [], undefined, undefined, {}, []).then( + () => undefined, + (e) => e as Error, + ); + try { + // Descriptor-less: the fixture's project, pushed through the module path. + const nodesc = await pushParentScriptForModule( + "f/analytics/analytics__dbt/models/stg_orders.sql", + remote as any, + [], + undefined, + undefined, + {}, + [], + ).then( + () => undefined, + (e) => e as Error, + ); + expect(nodesc).not.toBeInstanceOf(DbtPathCollisionError); + + // Descriptor present, pushed directly. Both reach the network — which is + // unreachable here on purpose — so anything BUT the collision is a pass. + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/wm_dbt.yaml"), + "profile: {}\n", + ); + expect(await push("f/analytics/analytics__dbt/wm_dbt.yaml")).not.toBeInstanceOf( + DbtPathCollisionError, + ); + } finally { + process.chdir(cwd); + } + }); + + // The exemption above is only for the project's OWN marker. A descriptor + // pushed DIRECTLY never passes through the metadata resolution that catches + // the collision, so without this it would deploy over the ordinary script + // sitting at the same remote path. + test("but a descriptor still refuses an ordinary script at its path", async () => { + const cwd = process.cwd(); + process.chdir(dir); + try { + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/wm_dbt.yaml"), + "profile: {}\n", + ); + fs.writeFileSync(path.join(dir, "f/analytics/analytics.py"), "def main(): ..."); + const err = await handleFile( + "f/analytics/analytics__dbt/wm_dbt.yaml", + { workspaceId: "w", remote: "http://127.0.0.1:1", name: "w", token: "t" } as any, + [], + undefined, + undefined, + {}, + [], + ).then( + () => undefined, + (e) => e as Error, + ); + expect(err).toBeInstanceOf(DbtPathCollisionError); + expect(err?.message).toContain("f/analytics/analytics.py"); + } finally { + process.chdir(cwd); + } + }); + + // Both layouts deploy to the same remote path, and the folder layout is the + // one whose base is NOT its filename: `__mod/script.ts` deploys to + // ``, exactly where the dbt project goes. + test("the collision holds for a folder-layout script too", async () => { + const cwd = process.cwd(); + process.chdir(dir); + const remote = { workspaceId: "w", remote: "http://127.0.0.1:1", name: "w", token: "t" }; + const push = (p: string) => + handleFile(p, remote as any, [], undefined, undefined, {}, []).then( + () => undefined, + (e) => e as Error, + ); + try { + fs.mkdirSync(path.join(dir, "f/analytics/analytics__mod"), { recursive: true }); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__mod/script.ts"), + "export async function main() {}", + ); + fs.writeFileSync( + path.join(dir, "f/analytics/analytics__dbt/wm_dbt.yaml"), + "profile: {}\n", + ); + + // From the dbt side: the descriptor must find the `__mod` entry point. + const fromDbt = await push("f/analytics/analytics__dbt/wm_dbt.yaml"); + expect(fromDbt).toBeInstanceOf(DbtPathCollisionError); + expect(fromDbt?.message).toContain("analytics__mod/script.ts"); + + // And from the ordinary side, whose base is not its filename. + const fromMod = await push("f/analytics/analytics__mod/script.ts"); + expect(fromMod).toBeInstanceOf(DbtPathCollisionError); + expect(fromMod?.message).toContain("analytics__dbt/dbt_project.yml"); + } finally { + process.chdir(cwd); + } + }); + + // The two sides spell "absent" differently — nothing on disk, nothing in the + // export — so without one normalization a descriptor-less project reads as an + // addition on every push and a deletion on every pull, forever. + test("compares equal to a remote that carries no descriptor either", async () => { + const remote = { + path: "", + isDirectory: true, + getChildren: async function* () { + for (const p of [ + "f/analytics/analytics__dbt/dbt_project.yml", + "f/analytics/analytics__dbt/models/stg_orders.sql", + "f/analytics/analytics.script.yaml", + ]) { + yield { + path: p, + isDirectory: false, + getChildren: async function* () {}, + getContentText: async () => "x", + }; + } + }, + getContentText: async () => "", + }; + const local = await elementsToMap( + await FSFSElement(dir, [], true), + () => false, + false, + {}, + ); + const remoteMap = await elementsToMap(remote as any, () => false, false, {}); + const key = "f/analytics/analytics__dbt/wm_dbt.yaml"; + expect(normalized(local)[key]).toBe(""); + expect(normalized(remoteMap)[key]).toBe(""); + }); +}); + +// `wmill dev` walks basenames, and a dbt script is a DIRECTORY whose descriptor +// may not exist — so it is recognized by the project folder or not at all. The +// walk must also stop there: the project's own `.sql` models match the script +// extensions and would each be listed as a script of their own. +describe("dev-mode discovery of a dbt project", () => { + let dir: string + let cwd: string + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'wmill-dbt-dev-')) + fs.mkdirSync(path.join(dir, 'f/analytics/analytics__dbt/models'), { recursive: true }) + fs.writeFileSync(path.join(dir, 'f/analytics/analytics__dbt/dbt_project.yml'), 'name: a\n') + fs.writeFileSync(path.join(dir, 'f/analytics/analytics__dbt/models/stg.sql'), 'select 1') + cwd = process.cwd() + process.chdir(dir) + }) + + afterEach(() => { + process.chdir(cwd) + fs.rmSync(dir, { recursive: true, force: true }) + }) + + test('lists the project itself and nothing inside it', async () => { + const items = await listWorkspacePaths() + const paths = items.map((i) => i.path).sort() + expect(paths).toEqual(['f/analytics/analytics']) + }) +}) diff --git a/cli/test/deploy_on_behalf_of_unit.test.ts b/cli/test/deploy_on_behalf_of_unit.test.ts new file mode 100644 index 0000000000..2214c0dc37 --- /dev/null +++ b/cli/test/deploy_on_behalf_of_unit.test.ts @@ -0,0 +1,83 @@ +import { expect, test } from "bun:test"; +import { deployItem } from "../windmill-utils-internal/src/deploy.ts"; + +// `deployItem` spreads the source item into the request body, and a script's/flow's +// on_behalf_of names a username that only exists in the source +// workspace. Sending it to the target pairs one workspace's principal with the other's +// email, which the backend rejects. Deleting the spread is an easy regression, so pin +// that the key never reaches the wire. +function recordingProvider(captured: [string, any][], flowExists: boolean) { + const source = { + on_behalf_of_email: "alice@corp", + on_behalf_of: "u/alice", + }; + return { + existsFlowByPath: async () => flowExists, + existsScriptByPath: async () => true, + getFlowByPath: async () => ({ + path: "f/x/f", + summary: "", + value: { modules: [] }, + ...source, + }), + createFlow: async (p: any) => void captured.push(["createFlow", p.requestBody]), + updateFlow: async (p: any) => void captured.push(["updateFlow", p.requestBody]), + getScriptByPath: async () => ({ + path: "f/x/s", + summary: "", + content: "x", + language: "bun", + hash: "abc", + ...source, + }), + createScript: async (p: any) => + void captured.push(["createScript", p.requestBody]), + } as any; +} + +test("deployItem: never sends the source workspace's on_behalf_of", async () => { + const captured: [string, any][] = []; + + // The clear is written out once per branch, so exercise all three: a flow that + // does not exist in the target (create), one that does (update — the branch + // `wmill workspace merge` takes for anything already deployed), and a script. + await deployItem( + recordingProvider(captured, false), + "flow" as any, + "f/x/f", + "src", + "dst", + "alice@corp", + ); + await deployItem( + recordingProvider(captured, true), + "flow" as any, + "f/x/f", + "src", + "dst", + "alice@corp", + ); + await deployItem( + recordingProvider(captured, false), + "script" as any, + "f/x/s", + "src", + "dst", + "alice@corp", + ); + + expect(captured.map(([fn]) => fn)).toEqual([ + "createFlow", + "updateFlow", + "createScript", + ]); + for (const [, body] of captured) { + // The email is still overridden with the caller's choice... + expect(body.on_behalf_of_email).toBe("alice@corp"); + expect(body.preserve_on_behalf_of).toBe(true); + // ...while the principal is dropped, so the backend derives the target's own. + expect( + "on_behalf_of" in JSON.parse(JSON.stringify(body)), + ).toBe(false); + } +}); diff --git a/cli/test/dev_recorder_bundle_unit.test.ts b/cli/test/dev_recorder_bundle_unit.test.ts new file mode 100644 index 0000000000..2c7a35f696 --- /dev/null +++ b/cli/test/dev_recorder_bundle_unit.test.ts @@ -0,0 +1,42 @@ +/** + * The recorder `wmill app dev --recording` serves is generated from the + * frontend's raw-app recorder, not written here, so it can silently ship a stale + * event model after that recorder changes. The committed bundle records the + * sources it was built from and their hash; this fails when they no longer + * agree. + * + * Fix a failure with `bun run gen:dev-recorder` from cli/. + */ + +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import { hashRecorderSources } from "../generate-dev-recorder.ts"; +import { + DEV_RECORDER_BUNDLE, + DEV_RECORDER_SOURCE_HASH, + DEV_RECORDER_SOURCES, +} from "../src/commands/app/devRecorderBundle.gen.ts"; + +const REPO_ROOT = path.join(import.meta.dir, "..", ".."); + +describe("dev recorder bundle", () => { + test("exposes the recorder factory as a global", () => { + expect(DEV_RECORDER_BUNDLE).toContain("__wmillRecorder"); + expect(DEV_RECORDER_BUNDLE).toContain("createRawAppRecording"); + // Runes are stripped at generation; one left in would throw at load time. + expect(DEV_RECORDER_BUNDLE).not.toContain("$state"); + }); + + test("is up to date with the frontend recorder", () => { + expect(DEV_RECORDER_SOURCES.length).toBeGreaterThan(0); + const present = DEV_RECORDER_SOURCES.every((f) => + fs.existsSync(path.join(REPO_ROOT, f)) + ); + // The published CLI package ships without the frontend sources. + if (!present) return; + expect(hashRecorderSources(DEV_RECORDER_SOURCES, REPO_ROOT)).toBe( + DEV_RECORDER_SOURCE_HASH, + ); + }); +}); diff --git a/cli/test/dev_recorder_routes_unit.test.ts b/cli/test/dev_recorder_routes_unit.test.ts new file mode 100644 index 0000000000..29ad71c983 --- /dev/null +++ b/cli/test/dev_recorder_routes_unit.test.ts @@ -0,0 +1,43 @@ +/** + * Guards on the routes `wmill app dev --recording` adds: what may write a + * recording, what a recording may be named, and that two saves never collide. + */ + +import { expect, test } from "bun:test"; +import { + isOwnOrigin, + isRecordingFileName, + recordingFileName, +} from "../src/commands/app/devRecorder.ts"; + +test("only the shell's own origin may save a recording", () => { + expect(isOwnOrigin("http://localhost:4000", "localhost:4000")).toBe(true); + expect(isOwnOrigin("http://127.0.0.1:4000", "127.0.0.1:4000")).toBe(true); + // A cross-site POST carrying JSON under a simple content type needs no + // preflight, so a foreign origin sharing the port must still be refused. + expect(isOwnOrigin("http://attacker.example:4000", "localhost:4000")).toBe( + false, + ); + expect(isOwnOrigin("null", "localhost:4000")).toBe(false); + // No Origin at all is a non-browser client, not a cross-site page. + expect(isOwnOrigin(undefined, "localhost:4000")).toBe(true); +}); + +test("recording names stay inside the recordings folder", () => { + expect(isRecordingFileName("recording-2026-01-01-00-00-00-000.json")).toBe( + true, + ); + expect(isRecordingFileName("../../../etc/passwd")).toBe(false); + expect(isRecordingFileName("..%2Fx.json")).toBe(false); + expect(isRecordingFileName("sub/dir.json")).toBe(false); + expect(isRecordingFileName("recording.txt")).toBe(false); +}); + +test("two saves in the same millisecond get distinct names", () => { + const now = new Date("2026-01-01T00:00:00.123Z"); + const first = recordingFileName(now, 0); + const second = recordingFileName(now, 1); + expect(first).toBe("recording-2026-01-01-00-00-00-123.json"); + expect(second).not.toBe(first); + expect(isRecordingFileName(second)).toBe(true); +}); diff --git a/cli/test/elements_to_map_branch_specific_unit.test.ts b/cli/test/elements_to_map_branch_specific_unit.test.ts index b9abaa8dd1..ebab7ef929 100644 --- a/cli/test/elements_to_map_branch_specific_unit.test.ts +++ b/cli/test/elements_to_map_branch_specific_unit.test.ts @@ -423,3 +423,45 @@ test("elementsToMap: isRemote undefined behaves like local (backward compatible) // Base file should be skipped (same behavior as isRemote=false) expect(Object.keys(result).includes("f/test.variable.yaml")).toEqual(false); }); + +// ============================================================================= +// REGRESSION TEST: --skip-scripts covers a script's module files +// A module is deployed as part of its parent script, and the module shortcut +// runs before every skip filter — so a changed model under a dbt project's +// `__dbt/` folder pushed the script `--skip-scripts` asked to leave alone. +// ============================================================================= + +test("elementsToMap: script modules are excluded when skipScripts is set", async () => { + const files: MockFile[] = [ + { path: "f/analytics/analytics.dbt.yaml", content: "profile: {}\n" }, + { + path: "f/analytics/analytics__dbt/models/stg_orders.sql", + content: "select 1", + }, + { path: "f/Shared/Variable/TestVar.variable.yaml", content: "value: test" }, + ]; + + const kept = await elementsToMap( + createMockDynFSElement(files), + noIgnore, + false, + defaultSkips, + ); + expect( + Object.keys(kept).includes("f/analytics/analytics__dbt/models/stg_orders.sql"), + ).toEqual(true); + + const skipped = await elementsToMap( + createMockDynFSElement(files), + noIgnore, + false, + { skipScripts: true }, + ); + expect( + Object.keys(skipped).includes("f/analytics/analytics__dbt/models/stg_orders.sql"), + ).toEqual(false); + // Everything else is unaffected. + expect( + Object.keys(skipped).includes("f/Shared/Variable/TestVar.variable.yaml"), + ).toEqual(true); +}); diff --git a/cli/test/fileset_sync_unit.test.ts b/cli/test/fileset_sync_unit.test.ts new file mode 100644 index 0000000000..6c8a93ce1a --- /dev/null +++ b/cli/test/fileset_sync_unit.test.ts @@ -0,0 +1,155 @@ +/** + * Unit tests for fileset resource sync routing and pointer validation. + * These tests require no backend — they test standalone logic. + */ + +import { expect, test, describe } from "bun:test"; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, sep as SEP } from "node:path"; +import { handleFile } from "../src/commands/script/script.ts"; +import { validateFilesetPointer } from "../src/commands/resource/resource.ts"; +import { findFilesetResourceFile } from "../src/commands/sync/sync.ts"; +import { + isCurrentWorkspaceFile, + isWorkspaceSpecificFile, +} from "../src/core/specific_items.ts"; + +describe("handleFile routing", () => { + test("returns false for fileset children with script extensions", async () => { + // A fileset child belongs to its parent resource's value, never to a + // standalone script: bare `.sql` has no script language (only `.pg.sql` + // etc. do), so routing it to the script pusher aborts the whole push. + for (const p of [ + "f/resources/data.fileset/energy/queries/report.sql", + "f/resources/data.fileset/scripts/main.ts", + "f/resources/data.fileset/scripts/main.py", + ]) { + expect( + await handleFile(p, {} as any, [], undefined, undefined, {}, []), + ).toBe(false); + } + }); + + test("returns false for single-file resource content files", async () => { + expect( + await handleFile( + "f/resources/query.resource.file.sql", + {} as any, + [], + undefined, + undefined, + {}, + [], + ), + ).toBe(false); + }); +}); + +describe("workspace-specific classification", () => { + test("fileset children are never workspace-specific", () => { + // The workspace-name segment of the pattern spans `/`, so a child whose + // own name looks like a typed metadata file would otherwise be read as + // belonging to another workspace and dropped from every diff — i.e. + // silently never deployed. + for (const p of [ + "f/resources/data.fileset/edge/q.resource.file.sql", + "f/resources/data.fileset/edge/inner.resource.yaml", + "f/resources/data.fileset/queries/report.sql", + ]) { + expect(isWorkspaceSpecificFile(p)).toBe(false); + } + // A child carrying the active workspace's own suffix must not be remapped + // onto the unsuffixed sibling key, which would deploy over that sibling. + expect( + isCurrentWorkspaceFile( + "f/resources/data.fileset/edge/inner.ws_main.resource.yaml", + "ws_main", + ), + ).toBe(false); + // The parent's own metadata still carries the suffix. + expect(isWorkspaceSpecificFile("f/resources/data.ws_main.resource.yaml")).toBe( + true, + ); + expect( + isCurrentWorkspaceFile("f/resources/data.ws_main.resource.yaml", "ws_main"), + ).toBe(true); + }); +}); + +describe("validateFilesetPointer", () => { + test("accepts the server-canonical pointer", () => { + expect(() => + validateFilesetPointer("f/resources/data.fileset", "f/resources/data"), + ).not.toThrow(); + }); + + test("normalizes trailing slashes", () => { + expect(() => + validateFilesetPointer("f/resources/data.fileset/", "f/resources/data"), + ).not.toThrow(); + }); + + test("rejects a custom pointer with an actionable message", () => { + expect(() => + validateFilesetPointer( + "f/queries.fileset", + "f/resources/analytics_data", + ), + ).toThrow(/must live next to its resource file, at 'f\/resources\/analytics_data\.fileset'/); + }); + + test("resolves workspace-specific metadata for canonical children", async () => { + // The metadata file carries the workspace suffix while children stay at + // the server-canonical `.fileset/` directory. + const dir = mkdtempSync(join(tmpdir(), "fileset-ws-")); + const cwd = process.cwd(); + try { + mkdirSync(join(dir, "f/res"), { recursive: true }); + writeFileSync( + join(dir, "f/res/data.ws_main.resource.yaml"), + "resource_type: c_files\nvalue: '!inline_fileset f/res/data.fileset'\n", + ); + process.chdir(dir); + // findFilesetResourceFile derives the metadata path from the child + // path, so both use the platform separator. + const childPath = ["f", "res", "data.fileset", "q.sql"].join(SEP); + const wsMetadataPath = ["f", "res", "data.ws_main.resource.yaml"].join( + SEP, + ); + const baseMetadataPath = ["f", "res", "data.resource.yaml"].join(SEP); + expect(await findFilesetResourceFile(childPath, "ws_main")).toBe( + wsMetadataPath, + ); + await expect(findFilesetResourceFile(childPath, null)).rejects.toThrow( + /No resource metadata file found/, + ); + // The suffixed file is the workspace's authoritative metadata: it wins + // even when a base metadata file coexists with it. + writeFileSync( + join(dir, "f/res/data.resource.yaml"), + "resource_type: c_files\nvalue: '!inline_fileset f/res/data.fileset'\n", + ); + expect(await findFilesetResourceFile(childPath, "ws_main")).toBe( + wsMetadataPath, + ); + expect(await findFilesetResourceFile(childPath, null)).toBe( + baseMetadataPath, + ); + } finally { + process.chdir(cwd); + rmSync(dir, { recursive: true, force: true }); + } + }); + + test("rejects a branch-suffixed directory for a workspace-specific resource", () => { + // The remote exporter always renders children at the server-canonical + // location, so a `.ws_.fileset` directory would never round-trip. + expect(() => + validateFilesetPointer( + "f/resources/data.ws_main.fileset", + "f/resources/data", + ), + ).toThrow(/at 'f\/resources\/data\.fileset'/); + }); +}); diff --git a/cli/test/git_unit.test.ts b/cli/test/git_unit.test.ts index 9665b8581a..92d44e09d9 100644 --- a/cli/test/git_unit.test.ts +++ b/cli/test/git_unit.test.ts @@ -182,6 +182,30 @@ describe("computeGitSyncDeployBranch", () => { ).toBe("wm_deploy/prod/f__team_a"); }); + test("datatable migration branches off its repo-relative migrations/ path", () => { + const items = [ + { + path_type: "datatable_migration", + path: "migrations/datatable/mydb/20260101000000_add_users", + }, + ]; + expect( + computeGitSyncDeployBranch({ ...base, useIndividualBranch: true, items }) + ).toBe( + "wm_deploy/prod/datatable_migration/migrations__datatable__mydb__20260101000000_add_users" + ); + // group_by_folder collapses every data table's migrations onto one branch — + // the backend's debounce key takes the same two segments. + expect( + computeGitSyncDeployBranch({ + ...base, + useIndividualBranch: true, + groupByFolder: true, + items, + }) + ).toBe("wm_deploy/prod/migrations__datatable"); + }); + test("falls back to parent_path when path is absent", () => { expect( computeGitSyncDeployBranch({ @@ -357,6 +381,17 @@ describe("gitSyncIncludePattern", () => { "f/t.amqp_trigger.*" ); }); + test("datatable migration expands to its two repo-relative .sql files", () => { + expect( + gitSyncIncludePattern( + "datatable_migration", + "migrations/datatable/mydb/20260101000000_add_users" + ) + ).toBe( + "migrations/datatable/mydb/20260101000000_add_users.up.sql," + + "migrations/datatable/mydb/20260101000000_add_users.down.sql" + ); + }); }); describe("deriveGitSyncDeployIncludes", () => { diff --git a/cli/test/gitsync_converter_unit.test.ts b/cli/test/gitsync_converter_unit.test.ts index e84c2ffa11..628b6dd161 100644 --- a/cli/test/gitsync_converter_unit.test.ts +++ b/cli/test/gitsync_converter_unit.test.ts @@ -21,6 +21,24 @@ describe("GitSyncSettingsConverter.fromBackendFormat", () => { expect(result.skipWorkspaceDependencies).toBe(false); }); + test("converts datatablemigration in include_type to skipDatatableMigrations: false", () => { + const backend = { + include_path: ["f/**"], + include_type: ["script", "flow", "datatablemigration"], + }; + const result = GitSyncSettingsConverter.fromBackendFormat(backend); + expect(result.skipDatatableMigrations).toBe(false); + }); + + test("sets skipDatatableMigrations: true when datatablemigration is absent", () => { + const backend = { + include_path: ["f/**"], + include_type: ["script", "flow"], + }; + const result = GitSyncSettingsConverter.fromBackendFormat(backend); + expect(result.skipDatatableMigrations).toBe(true); + }); + test("sets skipWorkspaceDependencies: true when workspacedependencies is absent", () => { const backend = { include_path: ["f/**"], @@ -71,6 +89,21 @@ describe("GitSyncSettingsConverter.fromBackendFormat", () => { // ============================================================================= describe("GitSyncSettingsConverter.toBackendFormat", () => { + test("adds datatablemigration when skipDatatableMigrations is false", () => { + expect( + GitSyncSettingsConverter.toBackendFormat({ + includes: ["f/**"], + skipDatatableMigrations: false, + }).include_type, + ).toContain("datatablemigration"); + expect( + GitSyncSettingsConverter.toBackendFormat({ + includes: ["f/**"], + skipDatatableMigrations: true, + }).include_type, + ).not.toContain("datatablemigration"); + }); + test("adds workspacedependencies when skipWorkspaceDependencies is false", () => { const opts = { includes: ["f/**"], diff --git a/cli/test/lock_dedup_unit.test.ts b/cli/test/lock_dedup_unit.test.ts new file mode 100644 index 0000000000..7ff413b231 --- /dev/null +++ b/cli/test/lock_dedup_unit.test.ts @@ -0,0 +1,500 @@ +/** + * Lockfile deduplication (`dedupeLockfiles`) — WIN-1756. + * + * A workspace with one `dependencies/requirements.in` resolves the same lock for + * every Python script, so the repo carries thousands of identical + * `.script.lock` files: a dependency bump rewrites all of them and every open + * branch conflicts on all of them. Dedup keeps one lockfile per dependency file, + * named after it, and points the metadata at it. + * + * What these pin, per the invariants in `lock_dedup.ts`: + * - the name comes from the dependency file, so a bump is a one-file diff and a + * sync narrowed to one script reaches the same answer as a full one + * - a script whose lock is its own (an `extra_`/inline annotation, or several + * dependency files at once) keeps a `.script.lock` + * - the pass is idempotent, which is what keeps `sync pull`/`push` from seeing + * a diff on every run + */ + +import { expect, test, describe } from "bun:test"; +import { stringify as yamlStringify } from "yaml"; +import * as path from "node:path"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import { + applySharedLockPlanToMap, + computeSharedLockPlan, + isEmptySharedLockPlan, + metadataLockUnreadable, + scriptsReferencingSharedLock, + sharedLockRefOf, + sharedLockRefIn, +} from "../src/utils/lock_dedup.ts"; +import { isSharedLockPath } from "../src/utils/script_common.ts"; +import { yamlOptions } from "../src/commands/sync/sync.ts"; + +const PY_LOCK = "requests==2.32.0\nurllib3==2.2.1\n"; +const PY_LOCK_BUMPED = "requests==2.32.3\nurllib3==2.2.1\n"; +const OTHER_LOCK = "requests==2.32.0\nurllib3==2.2.1\npandas==2.2.0\n"; +const PY_DEPS = "dependencies/requirements.in"; +const TEAM_DEPS = "dependencies/team_a.requirements.in"; +const BUN_DEPS = "dependencies/package.json"; +const SHARED_PY = "locks/requirements.in.lock"; +const SHARED_TEAM = "locks/team_a.requirements.in.lock"; +const SHARED_BUN = "locks/package.json.lock"; + +/** A sync map is keyed with the platform separator — `sync.ts` walks the tree + * with `path.join` — while an `!inline` reference is always forward-slash. + * The fixtures speak both, or on Windows they would pin a map shape the CLI + * never builds. */ +const k = (p: string) => p.replaceAll("/", path.sep); + +function meta(lockRef: string, summary = ""): string { + return yamlStringify({ summary, lock: lockRef }, yamlOptions); +} + +/** A workspace holding the given dependency files. */ +function workspace(...depFiles: string[]): Record { + const map: Record = {}; + for (const dep of depFiles) map[k(dep)] = "requests\n"; + return map; +} + +/** A script with its own lockfile, as `sync pull` writes it without dedup. */ +function ownLock( + map: Record, + base: string, + ext: string, + lock: string, + body = "def main(): ...", +) { + map[k(`${base}${ext}`)] = body; + map[k(`${base}.script.yaml`)] = meta(`!inline ${base}.script.lock`); + map[k(`${base}.script.lock`)] = lock; +} + +/** A script already reading a shared lockfile. */ +function sharedLock( + map: Record, + base: string, + ext: string, + shared: string, + body = "def main(): ...", +) { + map[k(`${base}${ext}`)] = body; + map[k(`${base}.script.yaml`)] = meta(`!inline ${shared}`); +} + +function lockRefOf(metaContent: string): string { + return metaContent.match(/lock: '(.*)'/)![1]; +} + +const plan = (map: Record) => + computeSharedLockPlan(map, { defaultTs: "bun" }); + +describe("computeSharedLockPlan", () => { + test("collapses the scripts of a dependency file into its lockfile", () => { + const map = workspace(PY_DEPS, BUN_DEPS); + ownLock(map, "f/a", ".py", PY_LOCK); + ownLock(map, "f/b", ".py", PY_LOCK); + ownLock(map, "f/c", ".py", PY_LOCK); + ownLock(map, "f/ts", ".ts", "bun-lock", "export async function main() {}"); + ownLock(map, "f/ts2", ".ts", "bun-lock", "export async function main() {}"); + + applySharedLockPlanToMap(map, plan(map)); + + expect(map[k(SHARED_PY)]).toEqual(PY_LOCK); + expect(map[k(SHARED_BUN)]).toEqual("bun-lock"); + for (const base of ["f/a", "f/b", "f/c"]) { + expect(map[k(`${base}.script.lock`)]).toBeUndefined(); + expect(lockRefOf(map[k(`${base}.script.yaml`)])).toEqual( + `!inline ${SHARED_PY}`, + ); + } + expect(lockRefOf(map[k("f/ts.script.yaml")])).toEqual( + `!inline ${SHARED_BUN}`, + ); + }); + + test("is idempotent — a deduplicated tree yields no further changes", () => { + const map = workspace(PY_DEPS); + ownLock(map, "f/a", ".py", PY_LOCK); + ownLock(map, "f/b", ".py", PY_LOCK); + applySharedLockPlanToMap(map, plan(map)); + + expect(isEmptySharedLockPlan(plan(map))).toBe(true); + }); + + test("a dependency bump moves only the shared file", () => { + const committed = workspace(PY_DEPS); + sharedLock(committed, "f/a", ".py", SHARED_PY); + sharedLock(committed, "f/b", ".py", SHARED_PY); + committed[k(SHARED_PY)] = PY_LOCK; + + // What the remote sends after the bump: one lock per script, all bumped. + const remote = workspace(PY_DEPS); + ownLock(remote, "f/a", ".py", PY_LOCK_BUMPED); + ownLock(remote, "f/b", ".py", PY_LOCK_BUMPED); + applySharedLockPlanToMap(remote, plan(remote)); + + expect( + Object.keys(remote).filter((key) => remote[key] !== committed[key]), + ).toEqual([k(SHARED_PY)]); + }); + + test("each named dependency file gets a lockfile of its own", () => { + const map = workspace(PY_DEPS, TEAM_DEPS); + ownLock(map, "f/a", ".py", PY_LOCK); + ownLock(map, "f/b", ".py", PY_LOCK); + const team = "# requirements: team_a\ndef main(): ..."; + ownLock(map, "f/t1", ".py", OTHER_LOCK, team); + ownLock(map, "f/t2", ".py", OTHER_LOCK, team); + + applySharedLockPlanToMap(map, plan(map)); + + expect(map[k(SHARED_PY)]).toEqual(PY_LOCK); + expect(map[k(SHARED_TEAM)]).toEqual(OTHER_LOCK); + expect(lockRefOf(map[k("f/t1.script.yaml")])).toEqual( + `!inline ${SHARED_TEAM}`, + ); + }); + + // The git-sync deploy callback narrows the sync to a single item, so this is + // the normal case rather than an edge one. + test("one script in view reaches the same answer as the whole workspace", () => { + const full = workspace(PY_DEPS); + for (const base of ["f/a", "f/b", "f/c"]) { + ownLock(full, base, ".py", PY_LOCK); + } + applySharedLockPlanToMap(full, plan(full)); + + const narrow = workspace(PY_DEPS); + ownLock(narrow, "f/a", ".py", PY_LOCK); + applySharedLockPlanToMap(narrow, plan(narrow)); + + expect(narrow[k(SHARED_PY)]).toEqual(full[k(SHARED_PY)]); + expect(lockRefOf(narrow[k("f/a.script.yaml")])).toEqual( + lockRefOf(full[k("f/a.script.yaml")]), + ); + }); + + test("a script the worker locks differently keeps a private lockfile", () => { + // The worker reads these from the leading comment block and several of them + // change what it locks, so such a script cannot stand for the file's lock. + const map = workspace(PY_DEPS, BUN_DEPS); + ownLock(map, "f/pinned", ".py", OTHER_LOCK, "# py311\nimport requests"); + + ownLock(map, "f/plain", ".py", PY_LOCK); + ownLock(map, "f/plain2", ".py", PY_LOCK); + // Only the names the worker matches count, so a documented script — or one + // with a `# TODO:` or a `# type: ignore` — still deduplicates. + ownLock(map, "f/doc", ".py", PY_LOCK, "# TODO: clean up\n# type: ignore\nimport requests"); + // Both annotation forms the worker accepts: a bare name and `name=value`. + ownLock(map, "f/npm", ".ts", "npm-lock", "//npm\nexport async function main() {}"); + ownLock(map, "f/nb", ".ts", "nb-lock", "//nobundling=true\nexport async function main() {}"); + ownLock(map, "f/ts", ".ts", "bun-lock", "export async function main() {}"); + ownLock(map, "f/ts2", ".ts", "bun-lock", "export async function main() {}"); + + applySharedLockPlanToMap(map, plan(map)); + + expect(map[k(SHARED_PY)]).toEqual(PY_LOCK); + expect(map[k("f/pinned.script.lock")]).toEqual(OTHER_LOCK); + expect(lockRefOf(map[k("f/doc.script.yaml")])).toEqual( + `!inline ${SHARED_PY}`, + ); + expect(map[k(SHARED_BUN)]).toEqual("bun-lock"); + expect(map[k("f/npm.script.lock")]).toEqual("npm-lock"); + expect(map[k("f/nb.script.lock")]).toEqual("nb-lock"); + }); + + test("an annotated script alone in its group creates no shared lockfile", () => { + // Alone, so nothing outvotes it: without the gate its variant would BECOME + // the dependency file's lock. `# py: ` is the interpreter pin the + // python import parser reads, which the annotations macro does not cover. + for (const header of ["# py: 3.11", "#py:3.11.4", "# py311"]) { + const map = workspace(PY_DEPS); + ownLock(map, "f/only", ".py", OTHER_LOCK, `${header}\nimport requests`); + + applySharedLockPlanToMap(map, plan(map)); + + expect(map[k(SHARED_PY)]).toBeUndefined(); + expect(map[k("f/only.script.lock")]).toEqual(OTHER_LOCK); + expect(lockRefOf(map[k("f/only.script.yaml")])).toEqual( + "!inline f/only.script.lock", + ); + } + }); + + test("a script whose lock is its own keeps a private lockfile", () => { + const map = workspace(PY_DEPS, TEAM_DEPS); + // `extra_` folds the script's own imports into the lock… + const extra = "# extra_requirements: default\ndef main(): ..."; + ownLock(map, "f/extra", ".py", OTHER_LOCK, extra); + // …and naming two files makes it no single file's lock. + const both = "# requirements: default, team_a\ndef main(): ..."; + ownLock(map, "f/both", ".py", OTHER_LOCK, both); + ownLock(map, "f/plain", ".py", PY_LOCK); + ownLock(map, "f/plain2", ".py", PY_LOCK); + + applySharedLockPlanToMap(map, plan(map)); + + expect(map[k("f/extra.script.lock")]).toEqual(OTHER_LOCK); + expect(map[k("f/both.script.lock")]).toEqual(OTHER_LOCK); + expect(lockRefOf(map[k("f/extra.script.yaml")])).toEqual( + "!inline f/extra.script.lock", + ); + expect(map[k(SHARED_PY)]).toEqual(PY_LOCK); + }); + + test("a script with no dependency file behind it is left alone", () => { + const map: Record = {}; + ownLock(map, "f/a", ".py", PY_LOCK); + ownLock(map, "f/b", ".py", PY_LOCK); + + expect(isEmptySharedLockPlan(plan(map))).toBe(true); + }); + + test("a stale committed lock is outvoted, and keeps its own", () => { + const map = workspace(PY_DEPS); + ownLock(map, "f/a", ".py", PY_LOCK_BUMPED); + ownLock(map, "f/b", ".py", PY_LOCK_BUMPED); + ownLock(map, "f/stale", ".py", PY_LOCK); + + applySharedLockPlanToMap(map, plan(map)); + + expect(map[k(SHARED_PY)]).toEqual(PY_LOCK_BUMPED); + expect(map[k("f/stale.script.lock")]).toEqual(PY_LOCK); + }); + + test("a shared lockfile outlives its scripts but not its dependency file", () => { + // No script in view reads it — a narrowed sync must still leave it be. + const withDep = workspace(PY_DEPS); + withDep[k(SHARED_PY)] = PY_LOCK; + expect(plan(withDep).deletes).not.toContain(k(SHARED_PY)); + + const withoutDep: Record = { [k(SHARED_PY)]: PY_LOCK }; + expect(plan(withoutDep).deletes).toContain(k(SHARED_PY)); + }); + + // The git-sync deploy callback narrows to one item, so a dependency file with + // no script in view is routine — and its lockfile is read by scripts this sync + // cannot see. + test("a dependency file with no script in view keeps its lockfile", () => { + const map = workspace(PY_DEPS, BUN_DEPS); + ownLock(map, "f/a", ".py", PY_LOCK); + ownLock(map, "f/b", ".py", PY_LOCK); + // `present` is forward-slashed whatever the map's separator: `sync.ts` + // normalizes it out of the local map before handing it over. + const present = { [SHARED_BUN]: "bun-lock" }; + + const result = computeSharedLockPlan(map, { defaultTs: "bun", present }); + applySharedLockPlanToMap(map, result); + + expect(result.deletes).not.toContain(k(SHARED_BUN)); + // Carried into the map, or the diff reads it as a local-only deletion. + expect(map[k(SHARED_BUN)]).toEqual("bun-lock"); + }); + + test("dependency files absent from the map are not gone", () => { + // `--skip-workspace-dependencies` keeps them out of both maps; taking that + // as "deleted" would un-deduplicate the tree and sweep the lockfiles. + const map: Record = {}; + ownLock(map, "f/a", ".py", PY_LOCK); + ownLock(map, "f/b", ".py", PY_LOCK); + + const result = computeSharedLockPlan(map, { + defaultTs: "bun", + depFiles: [PY_DEPS], + present: { [SHARED_PY]: PY_LOCK }, + }); + applySharedLockPlanToMap(map, result); + + expect(result.deletes).not.toContain(k(SHARED_PY)); + expect(lockRefOf(map[k("f/a.script.yaml")])).toEqual( + `!inline ${SHARED_PY}`, + ); + }); + + + + + + + test("a language that needs no lock is never deduplicated", () => { + const map = workspace("dependencies/modules.json"); + ownLock(map, "f/a", ".ps1", "some-lock", "echo hi"); + ownLock(map, "f/b", ".ps1", "some-lock", "echo hi"); + + expect(isEmptySharedLockPlan(plan(map))).toBe(true); + }); + + test("module-layout scripts share too, from their folder", () => { + const map = workspace(PY_DEPS); + for (const base of ["f/a__mod", "f/b__mod"]) { + map[k(`${base}/script.py`)] = "def main(): ..."; + map[k(`${base}/script.yaml`)] = meta(`!inline ${base}/script.lock`); + map[k(`${base}/script.lock`)] = PY_LOCK; + } + + applySharedLockPlanToMap(map, plan(map)); + + expect(map[k(SHARED_PY)]).toEqual(PY_LOCK); + expect(map[k("f/a__mod/script.lock")]).toBeUndefined(); + }); + + test("a script path containing dots reads its own content file", () => { + const map = workspace(PY_DEPS, BUN_DEPS); + ownLock(map, "f/a.b", ".py", PY_LOCK); + ownLock(map, "f/c.d", ".py", PY_LOCK); + // `f/a` is a bun script whose name is a prefix of `f/a.b`: its language and + // its annotation must come from its own file, not its neighbour's. + ownLock(map, "f/a", ".ts", "bun-lock", "export async function main() {}"); + ownLock(map, "f/e", ".ts", "bun-lock", "export async function main() {}"); + + applySharedLockPlanToMap(map, plan(map)); + + expect(lockRefOf(map[k("f/a.b.script.yaml")])).toEqual( + `!inline ${SHARED_PY}`, + ); + expect(lockRefOf(map[k("f/a.script.yaml")])).toEqual( + `!inline ${SHARED_BUN}`, + ); + }); + + test("only the lock line of the metadata changes", () => { + const map = workspace(PY_DEPS); + ownLock(map, "f/a", ".py", PY_LOCK); + ownLock(map, "f/b", ".py", PY_LOCK); + map[k("f/a.script.yaml")] = meta("!inline f/a.script.lock", "does a thing"); + const before = map[k("f/a.script.yaml")]; + + applySharedLockPlanToMap(map, plan(map)); + + expect(map[k("f/a.script.yaml")]).toEqual( + before.replace("f/a.script.lock", SHARED_PY), + ); + }); + + test("a dependency set named with a slash shares nothing", () => { + // `dependencies/team/python.requirements.in` has no distinct name under + // `locks/`: flattened to `locks/python.requirements.in.lock` it names a + // different (top-level) dependency file, and every later sweep would then + // read the lockfile as orphaned and retire it. + const map = workspace("dependencies/team/python.requirements.in"); + const body = "# requirements: team/python\ndef main(): ..."; + ownLock(map, "f/a", ".py", PY_LOCK, body); + ownLock(map, "f/b", ".py", PY_LOCK, body); + + const p = plan(map); + expect(p.writes).toEqual({}); + expect(p.deletes).toEqual([]); + }); +}); + +describe("isSharedLockPath", () => { + test("claims only the names a dependency file gives", () => { + for (const p of [SHARED_PY, SHARED_TEAM, SHARED_BUN]) { + expect(isSharedLockPath(p)).toBe(true); + } + // Not a dependency file's name, so a repo that already keeps lockfiles here + // keeps them; `modules.json` is powershell, which takes no lock. + for (const p of [ + "locks/vendor.lock", + "locks/Cargo.lock", + "locks/modules.json.lock", + "locks/sub/requirements.in.lock", + "locks/../../escape.lock", + ]) { + expect(isSharedLockPath(p)).toBe(false); + } + }); +}); + +describe("scriptsReferencingSharedLock", () => { + test("finds every script on the shared lock and nothing else", () => { + const map = workspace(PY_DEPS); + ownLock(map, "f/a", ".py", PY_LOCK); + ownLock(map, "f/b", ".py", PY_LOCK); + const extra = "# extra_requirements: default\ndef main(): ..."; + ownLock(map, "f/extra", ".py", OTHER_LOCK, extra); + applySharedLockPlanToMap(map, plan(map)); + + expect(scriptsReferencingSharedLock(map, k(SHARED_PY)).sort()).toEqual([ + k("f/a.script.yaml"), + k("f/b.script.yaml"), + ]); + }); + + test("prose naming the file is not a reference", () => { + const map = { + [k("f/a.py")]: "def main(): ...", + [k("f/a.script.yaml")]: yamlStringify( + { summary: `see !inline ${SHARED_PY}`, lock: "!inline f/a.script.lock" }, + yamlOptions, + ), + }; + expect(scriptsReferencingSharedLock(map, k(SHARED_PY))).toEqual([]); + }); +}); + +describe("sharedLockRefIn", () => { + test("returns a reference only when it is one, and it exists", async () => { + const root = await mkdtemp(path.join(os.tmpdir(), "wmill-dedup-ref-")); + try { + await mkdir(path.join(root, "locks"), { recursive: true }); + await writeFile(path.join(root, SHARED_PY), PY_LOCK, "utf-8"); + + expect(sharedLockRefIn(meta(`!inline ${SHARED_PY}`), false, root)).toEqual( + SHARED_PY, + ); + // Flow-style YAML starts with `{` and is not JSON: deciding the format by + // the first character would drop the reference and repoint the script. + expect( + sharedLockRefIn( + `{summary: x, lock: '!inline ${SHARED_PY}'}`, + false, + root, + ), + ).toEqual(SHARED_PY); + // A regenerated script falls back to its own lock rather than point at a + // shared file that is not there. + expect( + sharedLockRefIn(meta(`!inline ${SHARED_BUN}`), false, root), + ).toBeUndefined(); + expect( + sharedLockRefIn(meta("!inline f/a.script.lock"), false, root), + ).toBeUndefined(); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); +}); + +describe("reading the lock field off disk", () => { + test("a folded reference is still a reference", () => { + // A long enough dependency-set name pushes `lock:` past the serializer's + // 80-column default, which breaks the line at the space inside the value. + const ref = + "locks/" + "very_long_set_name_".repeat(4) + "requirements.in.lock"; + const folded = yamlStringify( + { lock: `!inline ${ref}`, summary: "" }, + yamlOptions, + ); + expect(folded).not.toContain("!inline " + ref); + + expect(sharedLockRefOf("f/a.script.yaml", folded, false)).toEqual(ref); + expect(metadataLockUnreadable("f/a.script.yaml", folded, false)).toBe(false); + }); + + test("metadata that cannot be parsed is flagged rather than read as empty", () => { + const conflicted = `summary: ''\n<<<<<<< HEAD\nlock: '!inline ${SHARED_PY}'\n=======\nlock: '!inline ${SHARED_BUN}'\n>>>>>>> other\n`; + expect(sharedLockRefOf("f/a.script.yaml", conflicted, false)).toBeUndefined(); + expect(metadataLockUnreadable("f/a.script.yaml", conflicted, false)).toBe( + true, + ); + // Nothing to read: no reference of any kind in the file. + expect(metadataLockUnreadable("f/a.script.yaml", "summary: ''\n", false)).toBe( + false, + ); + }); +}); diff --git a/cli/test/pipeline_local_graph_unit.test.ts b/cli/test/pipeline_local_graph_unit.test.ts index 7b8b8b0dc5..29d016857c 100644 --- a/cli/test/pipeline_local_graph_unit.test.ts +++ b/cli/test/pipeline_local_graph_unit.test.ts @@ -5,6 +5,7 @@ import { join } from "node:path"; import { buildLocalPipelineGraph, + hideDbtRunnables, parseMuteAnnotations, } from "../src/commands/pipeline/localGraph.ts"; @@ -915,3 +916,96 @@ test("pipeline docs renders a `Macro libraries` section (call + `// use`)", asyn }, ); }); + +test("a dbt script is not a pipeline node — a dbt-only folder has no graph", async () => { + await withFolder( + { + "warehouse.dbt.yaml": `engine: dbt-core-1x\nprofile:\n resource: $res:u/admin/wh\n`, + }, + async (root, folder) => { + const { graph, scripts } = await buildLocalPipelineGraph({ + root, + folder, + defaultTs: "bun", + }); + // A dbt project is authored and run as itself, never as a pipeline member; + // the deploy leaves its `auto_kind` unset, so the local graph must agree. + expect(graph.runnables).toEqual([]); + expect(scripts).toEqual([]); + }, + ); +}); + +test("a folder mixing dbt and a pipeline keeps only the pipeline member", async () => { + await withFolder( + { + "warehouse.dbt.yaml": `engine: dbt-core-1x\nprofile:\n resource: $res:u/admin/wh\n`, + "report.bun.ts": `// pipeline\nexport async function main() {}\n`, + }, + async (root, folder) => { + const { graph } = await buildLocalPipelineGraph({ + root, + folder, + defaultTs: "bun", + }); + expect(graph.runnables.map((r) => r.path)).toEqual(["f/mypipe/report"]); + }, + ); +}); + +test("hideDbtRunnables drops the dbt node from a deployed graph, keeping its relations", () => { + // `/assets/graph` is asset-usage driven, so it returns the dbt script as a + // producer. The CLI's deployed views must not render it as a pipeline script. + const deployed = { + runnables: [ + { path: "f/x/dbtproj", usage_kind: "script" as const, dbt: { model_count: 2 } }, + { path: "f/x/report", usage_kind: "script" as const, in_pipeline: true }, + ], + assets: [ + { kind: "table", path: "u/a/wh/s/stg_orders" }, + { kind: "table", path: "u/a/wh/s/fct_orders" }, + ], + edges: [ + { runnable_kind: "script", runnable_path: "f/x/dbtproj", asset_kind: "table", asset_path: "u/a/wh/s/stg_orders", access_type: "w" as const }, + { runnable_kind: "script", runnable_path: "f/x/dbtproj", asset_kind: "table", asset_path: "u/a/wh/s/fct_orders", access_type: "w" as const }, + { runnable_kind: "script", runnable_path: "f/x/report", asset_kind: "table", asset_path: "u/a/wh/s/fct_orders", access_type: "r" as const }, + ], + triggers: [ + { trigger_kind: "asset" as const, asset_kind: "table", asset_path: "u/a/wh/s/fct_orders", runnable_kind: "script", runnable_path: "f/x/report" }, + ], + }; + const g = hideDbtRunnables(deployed); + expect(g.runnables.map((r) => r.path)).toEqual(["f/x/report"]); + expect(g.edges.map((e) => e.runnable_path)).toEqual(["f/x/report"]); + // The relations stay: they are what the downstream pipeline script reads. + expect(g.assets).toHaveLength(2); + expect(g.triggers).toHaveLength(1); +}); + +test("hideDbtRunnables keeps a flow sharing a path with the dbt script", () => { + // Runnable identity in the graph is `(usage_kind, path)`; a script and a flow + // may share a path, so only the dbt script may be removed. + const g = hideDbtRunnables({ + runnables: [ + { path: "f/x/proj", usage_kind: "script", dbt: { model_count: 1 } }, + { path: "f/x/proj", usage_kind: "flow" }, + ], + edges: [ + { runnable_kind: "flow", runnable_path: "f/x/proj" }, + { runnable_kind: "script", runnable_path: "f/x/proj" }, + ], + triggers: [{ runnable_kind: "flow", runnable_path: "f/x/proj" }], + }); + expect(g.runnables.map((r) => r.usage_kind)).toEqual(["flow"]); + expect(g.edges.map((e) => e.runnable_kind)).toEqual(["flow"]); + expect(g.triggers).toHaveLength(1); +}); + +test("hideDbtRunnables is a no-op when the folder has no dbt project", () => { + const g = { + runnables: [{ path: "f/x/a", usage_kind: "script" }], + edges: [], + triggers: [], + }; + expect(hideDbtRunnables(g)).toBe(g); +}); diff --git a/cli/test/preview.test.ts b/cli/test/preview.test.ts index 4ead0a06c9..ffa900c8f7 100644 --- a/cli/test/preview.test.ts +++ b/cli/test/preview.test.ts @@ -185,6 +185,60 @@ test("script preview: regular script (non-codebase)", async () => { }); }); +test("script preview: job path is the script's Windmill path for every argument shape", async () => { + await withTestBackend(async (backend, tempDir) => { + await createWmillConfig(tempDir, { defaultTs: "bun" }); + await createScript( + tempDir, + "f/test/job_path_script.ts", + `export function main() { + return process.env.WM_JOB_PATH; +}` + ); + + const invocations: Array<[string, string]> = [ + ["f/test/job_path_script.ts", tempDir], + ["./f/test/job_path_script.ts", tempDir], + [`${tempDir}/f/test/job_path_script.ts`, tempDir], + ["job_path_script.ts", `${tempDir}/f/test`], + ]; + + for (const [arg, workingDir] of invocations) { + const result = await backend.runCLICommand( + ["script", "preview", arg, "--silent"], + workingDir + ); + + expect(result.code).toEqual(0); + expect(result.stdout.trim()).toEqual(`"f/test/job_path_script"`); + } + + // Folder layout: the job runs under the script's path, not the entry file's. + await createScript( + tempDir, + "f/test/job_path_module__mod/script.ts", + `export function main() { + return process.env.WM_JOB_PATH; +}` + ); + const moduleResult = await backend.runCLICommand( + ["script", "preview", "f/test/job_path_module__mod/script.ts", "--silent"], + tempDir + ); + expect(moduleResult.code).toEqual(0); + expect(moduleResult.stdout.trim()).toEqual(`"f/test/job_path_module"`); + + // A file outside the workspace tree has no Windmill path to run under, so + // the run is refused rather than pushed with a made-up one. + await writeFile(`${tempDir}/stray.ts`, `export function main() {}`, "utf-8"); + const strayResult = await backend.runCLICommand( + ["script", "preview", "stray.ts", "--silent"], + tempDir + ); + expect(strayResult.code).toEqual(1); + }); +}); + test("script preview: codebase script (CJS)", async () => { await withTestBackend(async (backend, tempDir) => { await createWmillConfig(tempDir, { @@ -529,6 +583,37 @@ test("flow preview: simple flow", async () => { }); }); +test("flow preview: step job path is anchored on the flow's Windmill path", async () => { + await withTestBackend(async (backend, tempDir) => { + await createWmillConfig(tempDir, { defaultTs: "bun" }); + await createFlow(tempDir, "f/test/job_path_flow.flow", { + summary: "Test flow", + scriptContent: `export function main() { return process.env.WM_JOB_PATH; }`, + }); + + // The last one is `--remote` from a subdirectory: that combination reads + // no config of its own, so it is the one shape where nothing but the path + // resolution can put the process in the sync root. + const invocations: Array<[string[], string]> = [ + [["f/test/job_path_flow.flow"], tempDir], + [["./f/test/job_path_flow.flow"], tempDir], + [[`${tempDir}/f/test/job_path_flow.flow`], tempDir], + [["job_path_flow.flow"], `${tempDir}/f/test`], + [["--remote", "job_path_flow.flow"], `${tempDir}/f/test`], + ]; + + for (const [args, workingDir] of invocations) { + const result = await backend.runCLICommand( + ["flow", "preview", ...args, "--silent"], + workingDir + ); + + expect(result.code).toEqual(0); + expect(result.stdout.trim()).toEqual(`"f/test/job_path_flow/a"`); + } + }); +}); + test("flow preview: uses local PathScript by default and remote PathScript with --remote", async () => { await withTestBackend(async (backend, tempDir) => { await createWmillConfig(tempDir, { defaultTs: "bun" }); diff --git a/cli/test/preview_path_unit.test.ts b/cli/test/preview_path_unit.test.ts new file mode 100644 index 0000000000..4bf9e8632c --- /dev/null +++ b/cli/test/preview_path_unit.test.ts @@ -0,0 +1,112 @@ +/** + * A preview job carries no runnable, so the path derived from the file + * argument is the whole of its identity — it has to come out the same + * whatever shape the argument had, and be refused rather than guessed when + * the file has no place in the workspace tree. + */ +import { expect, test, describe, beforeEach, afterEach } from "bun:test"; +import * as fs from "node:fs"; +import * as path from "node:path"; +import * as os from "node:os"; +import { + assertRemotePath, + toSyncRootRelativePath, +} from "../src/core/context.ts"; + +describe("toSyncRootRelativePath", () => { + let root: string; + let previousCwd: string; + let temps: string[]; + + beforeEach(() => { + previousCwd = process.cwd(); + temps = []; + // realpath: macOS' tmpdir is /var -> /private/var, and the assertions + // compare against what the process reports as its own directory. + root = fs.realpathSync( + fs.mkdtempSync(path.join(os.tmpdir(), "wmill_preview_path_")), + ); + temps.push(root); + fs.writeFileSync(path.join(root, "wmill.yaml"), "defaultTs: bun\n"); + fs.mkdirSync(path.join(root, "f", "test"), { recursive: true }); + fs.writeFileSync(path.join(root, "f", "test", "script.ts"), ""); + process.chdir(root); + }); + + afterEach(() => { + process.chdir(previousCwd); + for (const dir of temps) fs.rmSync(dir, { recursive: true, force: true }); + }); + + const normalized = (p: string) => p.replaceAll("\\", "/"); + + /** A dbt project whose descriptor is deliberately not written. */ + function dbtProject(): string { + const project = path.join(root, "f", "test", "proj__dbt"); + fs.mkdirSync(project, { recursive: true }); + fs.writeFileSync(path.join(project, "dbt_project.yml"), "name: proj\n"); + return project; + } + + test("every spelling of the same file lands on the same path", () => { + const fromRoot = ["f/test/script.ts", "./f/test/script.ts"].map((arg) => + normalized(toSyncRootRelativePath(arg, root)), + ); + const absolute = normalized( + toSyncRootRelativePath(path.join(root, "f", "test", "script.ts"), root), + ); + // As typed from the directory the file is in: the config read has already + // moved the process to the root by the time the argument is resolved. + const fromSubdir = normalized( + toSyncRootRelativePath("script.ts", path.join(root, "f", "test")), + ); + + expect(fromRoot).toEqual(["f/test/script.ts", "f/test/script.ts"]); + expect(absolute).toEqual("f/test/script.ts"); + expect(fromSubdir).toEqual("f/test/script.ts"); + }); + + test("a file that is deliberately absent keeps the directory it was named in", () => { + // A dbt project's descriptor is optional; `wmill script preview + // wm_dbt.yaml` from inside the project must still resolve to the project. + const project = dbtProject(); + + expect(normalized(toSyncRootRelativePath("wm_dbt.yaml", project))).toEqual( + "f/test/proj__dbt/wm_dbt.yaml", + ); + }); + + // Windows only creates symlinks for a privileged process. + test.skipIf(process.platform === "win32")( + "an absent file reached through a symlinked root still lands in the tree", + () => { + dbtProject(); + const aliasDir = fs.mkdtempSync(path.join(os.tmpdir(), "wmill_alias_")); + temps.push(aliasDir); + const alias = path.join(aliasDir, "link"); + fs.symlinkSync(root, alias, "dir"); + + const arg = path.join(alias, "f", "test", "proj__dbt", "wm_dbt.yaml"); + expect(normalized(toSyncRootRelativePath(arg, root))).toEqual( + "f/test/proj__dbt/wm_dbt.yaml", + ); + }, + ); + + test("a file outside the tree stays outside it", () => { + const outside = path.join(root, "..", "elsewhere.ts"); + expect(toSyncRootRelativePath(outside, root).startsWith("..")).toBe(true); + }); +}); + +describe("assertRemotePath", () => { + test("accepts a workspace path and refuses anything else", () => { + expect(() => assertRemotePath("f/test/script", "f/test/script.ts")).not.toThrow(); + expect(() => assertRemotePath("u/admin/script", "script.ts")).not.toThrow(); + for (const bad of ["", "script", "f/script", "../elsewhere"]) { + expect(() => assertRemotePath(bad, "arg.ts")).toThrow( + /Cannot derive a Windmill path/, + ); + } + }); +}); diff --git a/cli/test/raw_app_path_traversal_unit.test.ts b/cli/test/raw_app_path_traversal_unit.test.ts new file mode 100644 index 0000000000..7040c852db --- /dev/null +++ b/cli/test/raw_app_path_traversal_unit.test.ts @@ -0,0 +1,33 @@ +import { expect, test } from "bun:test"; +import { join as joinPath } from "node:path"; +import { rawAppPathWithinFolder } from "../src/commands/sync/sync.ts"; + +const APP = joinPath("u", "admin", "myapp.raw_app"); +const BACKEND = joinPath(APP, "wm_backend"); + +test("keys that stay inside the folder resolve to a path within it", () => { + // `value.files` keys arrive with a leading slash the caller strips. + expect(rawAppPathWithinFolder(APP, "index.tsx")).toBe( + joinPath(APP, "index.tsx"), + ); + expect(rawAppPathWithinFolder(APP, "src/util.ts")).toBe( + joinPath(APP, "src", "util.ts"), + ); + // A runnable id names its yaml under the backend folder. + expect(rawAppPathWithinFolder(BACKEND, "a.yaml")).toBe( + joinPath(BACKEND, "a.yaml"), + ); +}); + +test("keys that resolve outside the folder are rejected", () => { + for (const [base, rel] of [ + [APP, "../sibling.ts"], // a files key into the app's parent folder + [APP, "../../../f/other/outside.ts"], // into an unrelated folder tree + [APP, "../../../../../../elsewhere.txt"], // above the app folder entirely + [BACKEND, "../../../../etc/evil.yaml"], // a runnable id escaping the backend folder + ] as const) { + expect(() => rawAppPathWithinFolder(base, rel)).toThrow( + /escapes the app folder/, + ); + } +}); diff --git a/cli/test/raw_app_recordings_skip_unit.test.ts b/cli/test/raw_app_recordings_skip_unit.test.ts new file mode 100644 index 0000000000..408671cfe3 --- /dev/null +++ b/cli/test/raw_app_recordings_skip_unit.test.ts @@ -0,0 +1,63 @@ +/** + * `wmill app dev --recording` writes multi-MB session recordings into + * `.raw_app/recordings/`. They are local artifacts: the sync differ must + * not offer them as app source (the push itself drops them in + * `collectAppFiles`, so a differ that still sees them reports a change that + * pushing can never settle). + */ + +import { expect, test } from "bun:test"; +import { sep as SEP } from "node:path"; +import { elementsToMap } from "../src/commands/sync/sync.ts"; + +type MockFile = { path: string; content: string }; + +// FSFSElement joins with the platform separator, and the exclusion has to hold +// on Windows too. +const p = (...parts: string[]) => parts.join(SEP); + +function mockElement(files: MockFile[]) { + return { + isDirectory: true, + path: "", + async getContentText() { + return ""; + }, + async *getChildren() { + for (const file of files) { + yield { + isDirectory: false, + path: file.path, + async getContentText() { + return file.content; + }, + async *getChildren() {}, + }; + } + }, + }; +} + +test("elementsToMap skips recordings/ at the root of a raw app folder only", async () => { + const app = p("f", "demo", "myapp.raw_app"); + const files: MockFile[] = [ + { path: p(app, "index.tsx"), content: "export {}" }, + { + path: p(app, "recordings", "recording-2026-01-01-00-00-00.json"), + content: '{"version":1}', + }, + // The dev server never writes here, so this is the app's own source. + { path: p(app, "src", "recordings", "fixture.json"), content: "{}" }, + ]; + + const result = await elementsToMap( + mockElement(files) as any, + () => false, + false, + {}, + ); + + expect(Object.keys(result).sort()).toEqual( + [p(app, "index.tsx"), p(app, "src", "recordings", "fixture.json")].sort(), + ); +}); diff --git a/cli/test/raw_app_svelte_compiler_resolution_unit.test.ts b/cli/test/raw_app_svelte_compiler_resolution_unit.test.ts index 5fbf05384e..a81c19c2af 100644 --- a/cli/test/raw_app_svelte_compiler_resolution_unit.test.ts +++ b/cli/test/raw_app_svelte_compiler_resolution_unit.test.ts @@ -42,6 +42,50 @@ function installStubSvelte(dir: string, version: string) { ); } +/** + * A stand-in shaped like the real svelte: `./compiler` maps `require` at a UMD + * bundle and `default` at the ESM sources, and only the ESM half survives an + * `import()` with its named exports intact. + */ +function installDualStubSvelte(dir: string) { + const pkgDir = path.join(dir, "node_modules", "svelte"); + fs.mkdirSync(pkgDir, { recursive: true }); + fs.writeFileSync( + path.join(pkgDir, "package.json"), + JSON.stringify({ + name: "svelte", + version: "0.0.0-dual", + type: "module", + exports: { + "./package.json": "./package.json", + "./compiler": { + types: "./types/index.d.ts", + require: "./compiler/index.cjs", + default: "./src/compiler/index.js", + }, + }, + }), + "utf-8", + ); + fs.mkdirSync(path.join(pkgDir, "src", "compiler"), { recursive: true }); + fs.writeFileSync( + path.join(pkgDir, "src", "compiler", "index.js"), + `export const VERSION = "0.0.0-esm";\n` + + `export function compile() { return { js: { code: "", map: null }, warnings: [] }; }\n`, + "utf-8", + ); + fs.mkdirSync(path.join(pkgDir, "compiler"), { recursive: true }); + fs.writeFileSync( + path.join(pkgDir, "compiler", "index.cjs"), + // The UMD wrapper the published bundle uses: no static `exports.x = ...` for + // a lexer to find, so an `import()` of this file sees only `default`. + `!function(e,t){"object"==typeof exports&&"undefined"!=typeof module?t(exports):e(t)}` + + `(0,function(e){e.VERSION="0.0.0-cjs";` + + `e.compile=function(){return{js:{code:"",map:null},warnings:[]}}});\n`, + "utf-8", + ); +} + beforeEach(() => { tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "raw-app-svelte-compiler-")); fs.writeFileSync( @@ -64,6 +108,15 @@ describe("loadSvelteCompiler", () => { expect(compiler.VERSION).toBe("0.0.0-app-local"); }); + test("takes the ESM entry, not the `require` one, off a dual exports map", async () => { + installDualStubSvelte(tempDir); + + const compiler = await loadSvelteCompiler(tempDir); + + expect(compiler.VERSION).toBe("0.0.0-esm"); + expect(typeof compiler.compile).toBe("function"); + }); + test("resolves against the app even when given a relative dir", async () => { installStubSvelte(tempDir, "0.0.0-relative"); const cwd = process.cwd(); diff --git a/cli/test/raw_app_sync.test.ts b/cli/test/raw_app_sync.test.ts index 3cc7026430..315b274d80 100644 --- a/cli/test/raw_app_sync.test.ts +++ b/cli/test/raw_app_sync.test.ts @@ -464,6 +464,74 @@ excludes: []`, "utf-8"); }); }); +test("Raw App: deleted .lock sorting first does not short-circuit the app push", async () => { + // Regression: raw-app changes collapse to one representative change and + // deletes sort first, so removing a backend runnable makes its `.lock` the + // representative. Skipping a `.lock` there drops the whole app while the + // push still reports success. + await withTestBackend(async (backend, tempDir) => { + const testWorkspace = { + remote: backend.baseUrl, + workspaceId: backend.workspace, + name: "raw_app_lock_first_test", + token: backend.token + }; + await addWorkspace(testWorkspace, { force: true, configDir: backend.testConfigDir }); + + await writeFile(`${tempDir}/wmill.yaml`, `defaultTs: bun +includes: + - "**" +excludes: []`, "utf-8"); + + const appDir = path.join(tempDir, "f", "test", "lock_first_app.raw_app"); + await mkdir(path.join(tempDir, "f", "test"), { recursive: true }); + await createRawAppOnDisk(appDir, true); + + // A backend runnable's lockfile. Among the app's files it sorts first, + // so once deleted it becomes changes[0] for the whole group. + const queryLockPath = path.join(appDir, "backend", "query.lock"); + await writeFile(queryLockPath, INLINE_SCRIPT_A_LOCK, "utf-8"); + + const pushResult1 = await backend.runCLICommand( + ['sync', 'push', '--yes'], + tempDir, "raw_app_lock_first_test" + ); + expect(pushResult1.code).toEqual(0); + await waitForDeploymentJobs(backend); + + // Remove the backend runnable (lock included) and edit a frontend file. + await rm(queryLockPath); + await rm(path.join(appDir, "backend", "query.ts")); + await rm(path.join(appDir, "backend", "query.yaml")); + const appTsxPath = path.join(appDir, "App.tsx"); + const appTsxContent = await readFileContent(appTsxPath); + await writeFile( + appTsxPath, + appTsxContent.replace("hello world", "REGRESSION MARKER"), + "utf-8" + ); + + const pushResult2 = await backend.runCLICommand( + ['sync', 'push', '--yes'], + tempDir, "raw_app_lock_first_test" + ); + expect(pushResult2.code).toEqual(0); + await waitForDeploymentJobs(backend); + + // The edit must have reached the remote bundle, and the removed runnable + // must be gone from it. + const appResp = await backend.apiRequest!( + `/api/w/${backend.workspace}/apps/get/p/f/test/lock_first_app` + ); + expect(appResp.status).toEqual(200); + const appJson = await appResp.json(); + const files = appJson?.value?.files ?? {}; + expect(files["/App.tsx"]).toContain("REGRESSION MARKER"); + // Backend runnables live in value.runnables, not in the bundled files. + expect(appJson?.value?.runnables?.query).toBeUndefined(); + }); +}); + test("Raw App: delete file and push", async () => { await withTestBackend(async (backend, tempDir) => { // Set up workspace diff --git a/cli/test/raw_app_workspace_deps_unit.test.ts b/cli/test/raw_app_workspace_deps_unit.test.ts new file mode 100644 index 0000000000..a76064e8b9 --- /dev/null +++ b/cli/test/raw_app_workspace_deps_unit.test.ts @@ -0,0 +1,84 @@ +/** + * Raw app workspace dependencies + * + * A raw app keeps its runnables in `backend/`, not in `raw_app.yaml`. The + * workspace dependency filtering must resolve those files, otherwise the + * default `dependencies/package.json` is dropped and locks are regenerated + * against unpinned versions. + * + * Exercised through the legacy (tree-less) path: tree mode sources its deps + * from `getMismatchedWorkspaceDeps()`, which is only populated by an + * `uploadScripts` round-trip, so it cannot run offline. Both paths filter the + * same `appValue`, so resolving it correctly is what this pins. + */ + +import { expect, test } from "bun:test"; +import * as path from "node:path"; +import os from "node:os"; +import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises"; +import { generateAppLocksInternal } from "../src/commands/app/app_metadata.ts"; +import { Workspace } from "../src/commands/workspace/workspace.ts"; + +const stubWorkspace: Workspace = { + remote: "http://localhost:0/", + workspaceId: "test", + name: "test", + token: "test", +}; + +const APP_FOLDER = path.join("f", "example.raw_app"); + +async function withTempDir(fn: (tempDir: string) => Promise): Promise { + const tempDir = await mkdtemp(path.join(os.tmpdir(), "wmill_raw_app_deps_")); + const originalCwd = process.cwd(); + try { + process.chdir(tempDir); + await fn(tempDir); + } finally { + process.chdir(originalCwd); + await rm(tempDir, { recursive: true, force: true }); + } +} + +test("raw app: default workspace deps are picked up from backend runnables", async () => { + await withTempDir(async () => { + await mkdir(path.join(APP_FOLDER, "backend"), { recursive: true }); + await writeFile( + path.join(APP_FOLDER, "raw_app.yaml"), + `summary: "example raw app"\npolicy:\n execution_mode: publisher\n triggerables: {}\n`, + "utf-8", + ); + await writeFile( + path.join(APP_FOLDER, "backend", "test.ts"), + `import * as wmill from "windmill-client"\n\nexport async function main() {\n return wmill.getVariable("example")\n}\n`, + "utf-8", + ); + + await generateAppLocksInternal( + APP_FOLDER, + true, + false, + stubWorkspace, + { defaultTs: "bun" }, + true, // justUpdateMetadataLock — no backend round-trip + true, + ); + + expect( + await generateAppLocksInternal(APP_FOLDER, true, true, stubWorkspace, { defaultTs: "bun" }, false, true), + ).toBeUndefined(); + + // The runnable has no `package_json` annotation, so it uses the default + // manifest — adding it must invalidate the app. + await mkdir("dependencies", { recursive: true }); + await writeFile( + path.join("dependencies", "package.json"), + `{"dependencies": {"windmill-client": "1.742.0"}}`, + "utf-8", + ); + + expect( + await generateAppLocksInternal(APP_FOLDER, true, true, stubWorkspace, { defaultTs: "bun" }, false, true), + ).toEqual("f/example.raw_app"); + }); +}); diff --git a/cli/test/repro_diffname.test.ts b/cli/test/repro_diffname.test.ts index e096c52031..8eef88c357 100644 --- a/cli/test/repro_diffname.test.ts +++ b/cli/test/repro_diffname.test.ts @@ -1,7 +1,8 @@ import { test, expect } from "bun:test"; import { writeFile, readdir } from "node:fs/promises"; import path from "node:path"; -import { withTestBackend } from "./test_backend.ts"; +import { withTestBackend, type TestBackend } from "./test_backend.ts"; +import { waitForDeploymentJobs } from "./new_commands_helpers.ts"; import { addWorkspace } from "../src/commands/workspace/workspace.ts"; /** @@ -13,6 +14,21 @@ import { addWorkspace } from "../src/commands/workspace/workspace.ts"; * generate-metadata created duplicate content files (e.g., fetch_users.ts alongside a.ts). * On next push, loadRunnablesFromBackend auto-detected the orphans as new runnables. */ + +/** Fails with the CLI's own output, which is otherwise swallowed. */ +async function runCLI( + backend: TestBackend, + args: string[], + tempDir: string, +): Promise { + const result = await backend.runCLICommand(args, tempDir); + if (result.code !== 0) { + throw new Error( + `wmill ${args.join(" ")} exited with ${result.code}\n${result.stdout}\n${result.stderr}`, + ); + } +} + test("Raw app: generate-metadata must not create duplicate files when runnable key != name", async () => { await withTestBackend(async (backend, tempDir) => { const testWorkspace = { @@ -70,23 +86,25 @@ test("Raw app: generate-metadata must not create duplicate files when runnable k { method: "POST", headers: { Authorization: `Bearer ${backend.token}` }, body: formData }, ); expect(createResp.ok).toBeTruthy(); + // Deploying queues a dependency job that writes the generated locks back into + // the app value. Pulling mid-flight leaves the locks out locally, which turns + // the push below into a real change — and rebuilding a raw app bundle needs a + // package.json this fixture has no reason to carry. + await waitForDeploymentJobs(backend); // Pull the app - const pullResult = await backend.runCLICommand(["sync", "pull", "--yes"], tempDir); - expect(pullResult.code).toEqual(0); + await runCLI(backend, ["sync", "pull", "--yes"], tempDir); const backendDir = path.join(tempDir, "f/test/diffname_app.raw_app", "backend"); let files = await readdir(backendDir); - // After pull: files named by KEY (a, b). - // Lock files are generated asynchronously by the backend worker and may not - // have landed yet — exclude them from this precondition to avoid a Windows race. - const nonLockFiles = files.filter((f) => !f.endsWith(".lock")).sort(); - expect(nonLockFiles).toEqual(["a.ts", "a.yaml", "b.ts", "b.yaml"]); + // After pull: files named by KEY (a, b), locks included. + expect(files.sort()).toEqual([ + "a.lock", "a.ts", "a.yaml", "b.lock", "b.ts", "b.yaml", + ]); // Run generate-metadata — this previously created duplicate files named by NAME - const metaResult = await backend.runCLICommand(["generate-metadata", "--yes"], tempDir); - expect(metaResult.code).toEqual(0); + await runCLI(backend, ["generate-metadata", "--yes"], tempDir); // After generate-metadata: must still only have KEY-based files, no NAME-based duplicates files = await readdir(backendDir); @@ -102,8 +120,7 @@ test("Raw app: generate-metadata must not create duplicate files when runnable k expect(files).not.toContain("update_record.lock"); // Push should be a no-op (no phantom changes) - const pushResult = await backend.runCLICommand(["sync", "push", "--yes"], tempDir); - expect(pushResult.code).toEqual(0); + await runCLI(backend, ["sync", "push", "--yes"], tempDir); // Backend should still have exactly 2 runnables const getResp = await fetch( diff --git a/cli/test/resource_folders_unit.test.ts b/cli/test/resource_folders_unit.test.ts index 0d07812124..de3daa18cd 100644 --- a/cli/test/resource_folders_unit.test.ts +++ b/cli/test/resource_folders_unit.test.ts @@ -3,7 +3,7 @@ * Tests both dotted (.flow, .app, .raw_app) and non-dotted (__flow, __app, __raw_app) modes. */ -import { expect, test, describe, beforeEach } from "bun:test"; +import { expect, test, describe, beforeEach, afterEach } from "bun:test"; import { setNonDottedPaths, getNonDottedPaths, @@ -37,6 +37,13 @@ import { transformJsonPathToDir, isModuleEntryPoint, getScriptBasePathFromModulePath, + isDbtGeneratedPath, + dbtGeneratedDirs, + isUnderGeneratedDir, + isDbtModulePath, + isBundledModuleFile, + moduleFileExclusion, + MAX_MODULE_BYTES, } from "../src/utils/resource_folders.ts"; import { removeWorkerPrefix } from "../src/commands/worker-groups/worker-groups.ts"; @@ -657,3 +664,194 @@ describe("removeWorkerPrefix", () => { expect(removeWorkerPrefix("worker__")).toBe(""); }); }); + +// A dbt project's generated directories never belong to the bundle: hashing and +// uploading a local `target/` would make every local `dbt run` look like a +// project change, and a stale manifest in it is what the runtime reads as the +// graph. They are configurable, so they are read from the project. +describe("dbtGeneratedDirs", () => { + const fs = require("node:fs"); + const os = require("node:os"); + const nodePath = require("node:path"); + let dir: string; + // Every temp directory this block makes, so none outlives the run: the helper + // below mints one per call by design. + let made: string[] = []; + + beforeEach(() => { + dir = fs.mkdtempSync(nodePath.join(os.tmpdir(), "dbtgen-")); + made = [dir]; + }); + + afterEach(() => { + for (const d of made) fs.rmSync(d, { recursive: true, force: true }); + }); + + // A fresh directory per call: `dbtGeneratedDirs` memoizes per project folder, + // since within one sync the project file does not change under it. + const write = (yml: string) => { + const d = fs.mkdtempSync(nodePath.join(os.tmpdir(), "dbtgen-")); + made.push(d); + fs.writeFileSync(nodePath.join(d, "dbt_project.yml"), yml); + return dbtGeneratedDirs(d); + }; + + test("defaults apply with no project file", () => { + const dirs = dbtGeneratedDirs(dir); + expect(dirs.has("target")).toBe(true); + expect(dirs.has("dbt_packages")).toBe(true); + }); + + test("reads nested target-path and packages-install-path", () => { + const dirs = write( + 'name: p\ntarget-path: "build/target"\npackages-install-path: ./vendor/pkgs\n', + ); + expect(dirs.has("build/target")).toBe(true); + expect(dirs.has("vendor/pkgs")).toBe(true); + }); + + test("reads clean-targets in both of dbt's spellings", () => { + expect(write('name: p\nclean-targets: ["a", b]\n').has("a")).toBe(true); + const block = write("name: p\nclean-targets:\n - out/one\n - two\nmodels: {}\n"); + expect(block.has("out/one")).toBe(true); + expect(block.has("two")).toBe(true); + expect(block.has("models")).toBe(false); + }); + + test("ignores a configured path that escapes the project", () => { + const dirs = write('name: p\ntarget-path: "../../etc"\n'); + expect([...dirs].some((d) => d.includes(".."))).toBe(false); + }); +}); + +describe("isUnderGeneratedDir", () => { + const dirs = new Set(["target", "build/target"]); + + test("matches the directory and everything under it", () => { + expect(isUnderGeneratedDir("target", dirs)).toBe(true); + expect(isUnderGeneratedDir("target/manifest.json", dirs)).toBe(true); + expect(isUnderGeneratedDir("build/target/run_results.json", dirs)).toBe(true); + }); + + test("does not match a sibling sharing the prefix", () => { + expect(isUnderGeneratedDir("targetx/a.sql", dirs)).toBe(false); + expect(isUnderGeneratedDir("models/target_helper.sql", dirs)).toBe(false); + expect(isUnderGeneratedDir("build/targeted/a.sql", dirs)).toBe(false); + }); +}); + +// A Windows path spells the folder `__dbt\\`. A lookup that searched the raw +// path for `__dbt/` would find nothing, so a module-only edit would return +// without deploying its parent while the file was still recorded as synced. +describe("dbt module paths on either separator", () => { + test("isDbtModulePath and getScriptBasePathFromModulePath accept backslashes", () => { + expect(isDbtModulePath("f\\x\\proj__dbt\\models\\a.sql")).toBe(true); + expect(getScriptBasePathFromModulePath("f\\x\\proj__dbt\\models\\a.sql")).toBe( + "f/x/proj", + ); + expect(getScriptBasePathFromModulePath("f/x/proj__dbt/models/a.sql")).toBe( + "f/x/proj", + ); + }); + + test("a path with no module folder has no base path", () => { + expect(getScriptBasePathFromModulePath("f/x/proj.dbt.yaml")).toBeUndefined(); + }); +}); + +// The push, the staleness hash and the sync diff all ask this question. They +// must agree: a file one drops and another keeps is a change no push resolves. +describe("isBundledModuleFile", () => { + const fs = require("node:fs"); + const os = require("node:os"); + const nodePath = require("node:path"); + let dir: string; + + beforeEach(() => { + dir = fs.mkdtempSync(nodePath.join(os.tmpdir(), "bundled-")); + }); + + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + const write = (name: string, data: Buffer | string) => { + const p = nodePath.join(dir, name); + fs.writeFileSync(p, data); + return p; + }; + + test("keeps text, including empty and unicode", () => { + expect(isBundledModuleFile(write("a.sql", "select 1\n"))).toBe(true); + expect(isBundledModuleFile(write("empty.sql", ""))).toBe(true); + expect(isBundledModuleFile(write("u.sql", "select 'café'\n"))).toBe(true); + }); + + test("drops a binary file, which a NUL identifies", () => { + expect(isBundledModuleFile(write("x.png", Buffer.from([0x89, 0x50, 0x00, 0x1a])))).toBe( + false, + ); + }); + + test("drops a file over the per-file limit", () => { + expect(isBundledModuleFile(write("huge.csv", "x".repeat(MAX_MODULE_BYTES + 1)))).toBe( + false, + ); + }); + + // The two reasons a file is not carried are not interchangeable: sync hides + // dbt's binary leftovers, but an oversized seed the project authored has to + // stay in the diff, or the push that reports the size error never runs and the + // remote project is left incomplete without saying so. + test("says WHY a file is not carried", () => { + expect(moduleFileExclusion(write("a.sql", "select 1\n"))).toBe(undefined); + expect(moduleFileExclusion(write("x.png", Buffer.from([0x89, 0x50, 0x00, 0x1a])))).toBe( + "binary", + ); + expect(moduleFileExclusion(write("seed.csv", "x".repeat(MAX_MODULE_BYTES + 1)))).toBe( + "oversized", + ); + }); + + // A pull asks this about files that do not exist locally yet. Answering "not + // carried" there made sync ignore the whole incoming project and write + // nothing — the bundle silently vanished on every fresh checkout. + test("treats a missing file as carried, not as excluded", () => { + expect(isBundledModuleFile(nodePath.join(dir, "does-not-exist.sql"))).toBe(true); + }); +}); + +test("a nested __mod inside a dbt project does not steal the script boundary", () => { + // dbt owns its directory names verbatim, so a folder ending `__mod` is legal + // inside a project. The script is the OUTER module boundary. + expect( + getScriptBasePathFromModulePath("f/x/proj__dbt/models/legacy__mod/a.sql"), + ).toBe("f/x/proj"); + expect(getScriptBasePathFromModulePath("f/x/proj__dbt/models/a.sql")).toBe( + "f/x/proj", + ); + expect(getScriptBasePathFromModulePath("f/x/s__mod/inner.ts")).toBe("f/x/s"); +}); + +test("a nested __mod inside a dbt project is not a module entry point", () => { + expect(isModuleEntryPoint("f/x/s__mod/script.ts")).toBe(true); + // `legacy__mod` is a legal dbt directory; its script.ts belongs to the dbt + // project, not to a module folder of its own. + expect(isModuleEntryPoint("f/x/proj__dbt/models/legacy__mod/script.ts")).toBe( + false, + ); + expect(isModuleEntryPoint("f/x/proj__dbt/models/a.sql")).toBe(false); +}); + +test("a __dbt directory nested inside an ordinary module is not a dbt project file", () => { + expect(isDbtModulePath("f/x/proj__dbt/models/a.sql")).toBe(true); + // `vendor/x__dbt/` belongs to the `foo__mod` script, not to a dbt project. + expect(isDbtModulePath("f/x/foo__mod/vendor/x__dbt/a.ts")).toBe(false); + expect(isDbtModulePath("f/x/foo__mod/helper.ts")).toBe(false); +}); + +test("a __dbt/target nested in an ordinary module is not generated dbt output", () => { + expect(isDbtGeneratedPath("f/x/proj__dbt/target/manifest.json")).toBe(true); + // Belongs to the `foo__mod` script; excluding it would drop a real edit. + expect(isDbtGeneratedPath("f/x/foo__mod/vendor/x__dbt/target/a.ts")).toBe(false); +}); diff --git a/cli/test/resource_types_unit.test.ts b/cli/test/resource_types_unit.test.ts index 0966ae25bf..c8bebb1baa 100644 --- a/cli/test/resource_types_unit.test.ts +++ b/cli/test/resource_types_unit.test.ts @@ -52,6 +52,26 @@ test("non-identifier property names are double-quoted", () => { expect(out).toContain(' "3leading": boolean'); }); +// WIN-2392: hub types like `record` (schema `{}`) or `dbt_profile` +// (`{"type":"object"}`) have no `properties`, and the schema column itself is +// nullable. A type whose property map is missing must still compile, otherwise +// it aborts the whole rt.d.ts generation. +test("schemas without a usable property map compile to any", () => { + expect(compileResourceTypeToTsType(undefined)).toBe("any"); + expect(compileResourceTypeToTsType(null)).toBe("any"); + expect(compileResourceTypeToTsType({ type: "object" } as any)).toBe("any"); + expect(compileResourceTypeToTsType({ properties: null } as any)).toBe("any"); + expect(compileResourceTypeToTsType(schema({}))).toBe("any"); +}); + +test("a null property compiles to any instead of throwing", () => { + const out = compileResourceTypeToTsType( + schema({ host: null, port: { type: "integer" } } as any) + ); + expect(out).toContain(" host: any"); + expect(out).toContain(" port: number"); +}); + test("nested object and array property names are quoted too", () => { const out = compileResourceTypeToTsType( schema({ diff --git a/cli/test/script_push_up_to_date.test.ts b/cli/test/script_push_up_to_date.test.ts new file mode 100644 index 0000000000..e4a3e91268 --- /dev/null +++ b/cli/test/script_push_up_to_date.test.ts @@ -0,0 +1,145 @@ +/** + * `wmill script push` short-circuits when the local script already matches the + * remote. The comparison has to hold in both directions: an untouched script + * deploys nothing, and every field the push body carries (labels and the language + * inferred from defaultTs included) still counts as a change. + */ + +import { expect, test } from "bun:test"; +import { writeFile, readFile, mkdir } from "node:fs/promises"; +import { withTestBackend } from "./test_backend.ts"; +import { waitForDeploymentJobs } from "./new_commands_helpers.ts"; + +test("Integration: script push skips an unchanged script and deploys a changed one", async () => { + await withTestBackend(async (backend, tempDir) => { + const uniqueId = Date.now(); + const scriptPath = `f/test/uptodate_${uniqueId}`; + const getScript = async () => + await ( + await backend.apiRequest!( + `/api/w/${backend.workspace}/scripts/get/p/${scriptPath}`, + ) + ).json(); + const push = async () => + await backend.runCLICommand(["script", "push", `${scriptPath}.ts`], tempDir); + const wmillYaml = (defaultTs: string) => + `defaultTs: ${defaultTs}\nincludes:\n - "${scriptPath}**"\nexcludes: []\n`; + + await mkdir(`${tempDir}/f/test`, { recursive: true }); + await backend.apiRequest!(`/api/w/${backend.workspace}/folders/create`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ name: "test" }), + }); + const createResp = await backend.apiRequest!( + `/api/w/${backend.workspace}/scripts/create`, + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + path: scriptPath, + content: `export async function main() {\n return "Hello world";\n}`, + summary: "Test up to date", + description: "", + language: "bun", + kind: "script", + schema: { + $schema: "https://json-schema.org/draft/2020-12/schema", + type: "object", + properties: {}, + required: [], + }, + labels: ["l1"], + }), + }, + ); + expect(createResp.ok).toEqual(true); + + await writeFile(`${tempDir}/wmill.yaml`, wmillYaml("bun"), "utf-8"); + // The lock a deploy's dependency job writes is part of the comparison, so every + // pull has to happen after that job has landed or the skip races it. + await waitForDeploymentJobs(backend); + expect((await backend.runCLICommand(["sync", "pull", "--yes"], tempDir)).code).toEqual(0); + + const hashBefore = (await getScript()).hash; + expect((await push()).stdout).toContain("is up to date"); + expect((await getScript()).hash).toEqual(hashBefore); + + const metadataPath = `${tempDir}/${scriptPath}.script.yaml`; + await writeFile( + metadataPath, + (await readFile(metadataPath, "utf-8")).replace("- l1", "- l2"), + "utf-8", + ); + expect((await push()).stdout).not.toContain("is up to date"); + expect((await getScript()).labels).toEqual(["l2"]); + + // 2, not 0 or 1: those two are the values a truthiness comparison would also + // call equal, so they cannot pin that priority is compared by value. + await waitForDeploymentJobs(backend); + expect((await backend.runCLICommand(["sync", "pull", "--yes"], tempDir)).code).toEqual(0); + await writeFile(metadataPath, (await readFile(metadataPath, "utf-8")) + "priority: 2\n", "utf-8"); + expect((await push()).stdout).not.toContain("is up to date"); + expect((await getScript()).priority).toEqual(2); + await waitForDeploymentJobs(backend); + expect((await backend.runCLICommand(["sync", "pull", "--yes"], tempDir)).code).toEqual(0); + expect((await push()).stdout).toContain("is up to date"); + + await writeFile(`${tempDir}/wmill.yaml`, wmillYaml("deno"), "utf-8"); + expect((await push()).stdout).not.toContain("is up to date"); + expect((await getScript()).language).toEqual("deno"); + }); +}); + +test("Integration: an unchanged bunnative script is not redeployed", async () => { + await withTestBackend(async (backend, tempDir) => { + const uniqueId = Date.now(); + const scriptPath = `f/test/native_${uniqueId}`; + + await mkdir(`${tempDir}/f/test`, { recursive: true }); + await backend.apiRequest!(`/api/w/${backend.workspace}/folders/create`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ name: "test" }), + }); + const createResp = await backend.apiRequest!( + `/api/w/${backend.workspace}/scripts/create`, + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + path: scriptPath, + // The server stores this as bunnative: the language is derived from the + // annotation, and no file extension can express it back. + content: `//native\nexport async function main() {\n return "Hello world";\n}`, + summary: "Test bunnative", + description: "", + language: "bun", + kind: "script", + }), + }, + ); + expect(createResp.ok).toEqual(true); + + await writeFile( + `${tempDir}/wmill.yaml`, + `defaultTs: bun\nincludes:\n - "${scriptPath}**"\nexcludes: []\n`, + "utf-8", + ); + await waitForDeploymentJobs(backend); + expect((await backend.runCLICommand(["sync", "pull", "--yes"], tempDir)).code).toEqual(0); + + const remote = await ( + await backend.apiRequest!( + `/api/w/${backend.workspace}/scripts/get/p/${scriptPath}`, + ) + ).json(); + expect(remote.language).toEqual("bunnative"); + + const result = await backend.runCLICommand( + ["script", "push", `${scriptPath}.ts`], + tempDir, + ); + expect(result.stdout).toContain("is up to date"); + }); +}); diff --git a/cli/test/script_runtime_settings_sync.test.ts b/cli/test/script_runtime_settings_sync.test.ts new file mode 100644 index 0000000000..2b3272b20a --- /dev/null +++ b/cli/test/script_runtime_settings_sync.test.ts @@ -0,0 +1,126 @@ +/** + * Runtime settings that live only in the script metadata file (the retention + * delay, the debouncing bounds, the cache s3-path flag) must survive a sync + * pull/push cycle. A field missing from the create_script body the CLI builds + * lands as NULL on the deployed version; one missing from its up-to-date + * comparison makes a change to it alone report as up to date and never deploy. + */ + +import { expect, test } from "bun:test"; +import { writeFile, readFile, mkdir } from "node:fs/promises"; +import { withTestBackend } from "./test_backend.ts"; +import { waitForDeploymentJobs } from "./new_commands_helpers.ts"; + +// The debounce bounds this PR also restores cannot be asserted here: a build without +// git tags reports a bare commit as its version, GIT_SEM_VERSION then falls back to +// 0.1.0, and every version-gated feature (debouncing wants 1.566.0) is refused. That is +// what CI builds, so a fixture that sets any debounce field fails at create. +const SETTINGS = { + delete_after_secs: 900, + cache_ignore_s3_path: true, +}; + +test("Integration: script runtime settings survive a sync pull/push cycle", async () => { + await withTestBackend(async (backend, tempDir) => { + const uniqueId = Date.now(); + const scriptPath = `f/test/settings_${uniqueId}`; + const getScript = async () => { + const resp = await backend.apiRequest!( + `/api/w/${backend.workspace}/scripts/get/p/${scriptPath}`, + ); + expect(resp.ok).toEqual(true); + return await resp.json(); + }; + + await mkdir(`${tempDir}/f/test`, { recursive: true }); + const folderResp = await backend.apiRequest!( + `/api/w/${backend.workspace}/folders/create`, + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ name: "test" }), + }, + ); + const folderStatus = `${folderResp.status} ${await folderResp.text()}`; + + const createResp = await backend.apiRequest!( + `/api/w/${backend.workspace}/scripts/create`, + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + path: scriptPath, + content: `export async function main() {\n return "Hello world";\n}`, + summary: "Test runtime settings", + description: "", + language: "bun", + kind: "script", + schema: { + $schema: "https://json-schema.org/draft/2020-12/schema", + type: "object", + properties: {}, + required: [], + }, + ...SETTINGS, + }), + }, + ); + if (!createResp.ok) { + throw new Error( + `scripts/create failed: ${createResp.status} ${await createResp.text()} ` + + `(folders/create: ${folderStatus})`, + ); + } + + await writeFile( + `${tempDir}/wmill.yaml`, + `defaultTs: bun\nincludes:\n - "${scriptPath}**"\nexcludes: []\n`, + "utf-8", + ); + + // The lock a deploy's dependency job writes is compared before the settings are, + // so a pull taken before that job lands makes the next push deploy over the lock + // instead of over the setting under test. + await waitForDeploymentJobs(backend); + const pullResult = await backend.runCLICommand(["sync", "pull", "--yes"], tempDir); + expect(pullResult.code).toEqual(0); + + const metadataPath = `${tempDir}/${scriptPath}.script.yaml`; + const pulledMetadata = await readFile(metadataPath, "utf-8"); + for (const key of Object.keys(SETTINGS)) { + expect(pulledMetadata).toContain(key); + } + + // A content-only edit must carry the settings through to the new version. + const scriptFilePath = `${tempDir}/${scriptPath}.ts`; + const originalContent = await readFile(scriptFilePath, "utf-8"); + await writeFile( + scriptFilePath, + originalContent.replace("Hello world", "Hello world modified"), + "utf-8", + ); + expect((await backend.runCLICommand(["sync", "push", "--yes"], tempDir)).code).toEqual(0); + + const afterContentPush = await getScript(); + expect(afterContentPush.content).toContain("Hello world modified"); + for (const [key, value] of Object.entries(SETTINGS)) { + expect(afterContentPush[key]).toEqual(value); + } + + // A settings-only edit must reach the remote rather than be skipped as up to + // date. 0 is "delete immediately after completion", not "unset". + await waitForDeploymentJobs(backend); + expect((await backend.runCLICommand(["sync", "pull", "--yes"], tempDir)).code).toEqual(0); + await writeFile( + metadataPath, + (await readFile(metadataPath, "utf-8")).replace( + `delete_after_secs: ${SETTINGS.delete_after_secs}`, + "delete_after_secs: 0", + ), + "utf-8", + ); + expect((await backend.runCLICommand(["sync", "push", "--yes"], tempDir)).code).toEqual(0); + + expect((await getScript()).delete_after_secs).toEqual(0); + }); +}); diff --git a/cli/test/utils_unit.test.ts b/cli/test/utils_unit.test.ts index de95848524..daba8c776e 100644 --- a/cli/test/utils_unit.test.ts +++ b/cli/test/utils_unit.test.ts @@ -4,7 +4,7 @@ */ import { expect, test, describe } from "bun:test"; -import { deepEqual, isFileResource, isFilesetResource, toCamel, capitalize, validateRequiredArgs, stripBom, readTextFile, readTextFileSync } from "../src/utils/utils.ts"; +import { deepEqual, isFileResource, isFilesetResource, removeResourceSuffix, toCamel, capitalize, validateRequiredArgs, stripBom, readTextFile, readTextFileSync } from "../src/utils/utils.ts"; import { mkdtempSync, writeFileSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -159,6 +159,30 @@ describe("isFileResource", () => { }); }); +// ============================================================================= +// removeResourceSuffix +// ============================================================================= + +describe("removeResourceSuffix", () => { + test("strips the metadata suffix", () => { + expect(removeResourceSuffix("f/test/my_resource.resource.yaml")).toBe( + "f/test/my_resource" + ); + expect(removeResourceSuffix("f/test/my_resource.resource.json")).toBe( + "f/test/my_resource" + ); + }); + + test("strips the file-resource suffix", () => { + expect(removeResourceSuffix("f/test/my_file.resource.file.txt")).toBe( + "f/test/my_file" + ); + expect(removeResourceSuffix("u/admin/config.resource.file.json")).toBe( + "u/admin/config" + ); + }); +}); + // ============================================================================= // isFilesetResource // ============================================================================= @@ -248,6 +272,21 @@ describe("getTypeStrFromPath", () => { expect(getTypeStrFromPath("f/test/my_script.rs")).toBe("script"); }); + test("a shared lockfile is its own type, not a workspace dependency", () => { + // A repo-side artifact with no object on the server: classified as a + // workspace dependency, `sync push` would try to deploy it as one. + // Sync paths carry the platform separator, so these do too. + expect(getTypeStrFromPath(join("locks", "requirements.in.lock"))).toBe( + "shared_lock", + ); + expect(getTypeStrFromPath(join("dependencies", "requirements.in"))).toBe( + "workspace_dependencies", + ); + // `locks/` is an ordinary word: only the names Windmill writes are claimed, + // so a repo that already keeps its own lockfiles there keeps them. + expect(() => getTypeStrFromPath(join("locks", "vendor.lock"))).toThrow(); + }); + test("detects metadata types by name suffix", () => { expect(getTypeStrFromPath("f/test/my_var.variable.yaml")).toBe("variable"); expect(getTypeStrFromPath("f/test/my_res.resource.yaml")).toBe("resource"); diff --git a/cli/windmill-utils-internal/src/deploy.ts b/cli/windmill-utils-internal/src/deploy.ts index 42874fda97..047d50ee78 100644 --- a/cli/windmill-utils-internal/src/deploy.ts +++ b/cli/windmill-utils-internal/src/deploy.ts @@ -460,6 +460,10 @@ export async function deployItem( ...flow, preserve_on_behalf_of: preserveOnBehalfOf, on_behalf_of_email: onBehalfOf, + // Usernames are per-workspace, so the source's principal names nobody in + // the target (or names a different person). Clearing it lets the backend + // derive the target's own principal from the email above. + on_behalf_of: undefined, }, }); } else { @@ -469,6 +473,10 @@ export async function deployItem( ...flow, preserve_on_behalf_of: preserveOnBehalfOf, on_behalf_of_email: onBehalfOf, + // Usernames are per-workspace, so the source's principal names nobody in + // the target (or names a different person). Clearing it lets the backend + // derive the target's own principal from the email above. + on_behalf_of: undefined, }, }); } @@ -493,6 +501,8 @@ export async function deployItem( parent_hash: parentHash, preserve_on_behalf_of: preserveOnBehalfOf, on_behalf_of_email: onBehalfOf, + // See the flow branch: a source-workspace principal is never valid here. + on_behalf_of: undefined, }, }); } else if (kind === "app" || kind === "raw_app") { diff --git a/cli/windmill-utils-internal/src/path-utils/path-assigner.ts b/cli/windmill-utils-internal/src/path-utils/path-assigner.ts index 94d9085b94..6aaeea4db1 100644 --- a/cli/windmill-utils-internal/src/path-utils/path-assigner.ts +++ b/cli/windmill-utils-internal/src/path-utils/path-assigner.ts @@ -36,6 +36,7 @@ export const LANGUAGE_EXTENSIONS: Record = { bunnative: "ts", ruby: "rb", rlang: "r", + dbt: "dbt.yaml", // for related places search: ADD_NEW_LANG }; @@ -84,6 +85,7 @@ export const EXTENSION_TO_LANGUAGE: Record = { "playbook.yml": "ansible", "java": "java", "duckdb.sql": "duckdb", + "dbt.yaml": "dbt", "rb": "ruby", // Plain .ts defaults to bun (will be overridden by defaultTs setting) "ts": "bun", diff --git a/cli/wmill.schema.json b/cli/wmill.schema.json index 6acbd4794b..b4b10087a8 100644 --- a/cli/wmill.schema.json +++ b/cli/wmill.schema.json @@ -24,7 +24,7 @@ "items": { "type": "string" }, - "description": "Additional glob patterns merged with includes (useful in branch overrides)" + "description": "Additional glob patterns merged with includes (useful in workspace overrides)" }, "excludes": { "type": "array", @@ -69,6 +69,10 @@ "type": "boolean", "description": "Skip syncing workspace dependencies" }, + "skipDatatableMigrations": { + "type": "boolean", + "description": "Skip syncing data table SQL migrations" + }, "includeSchedules": { "type": "boolean", "description": "Include schedules in sync" @@ -101,6 +105,10 @@ "type": "boolean", "description": "Require lock files for all scripts" }, + "dedupeLockfiles": { + "type": "boolean", + "description": "Share one lockfile per workspace dependency file (locks/.lock), instead of an identical .script.lock per script" + }, "lint": { "type": "boolean", "description": "Run linting before push" @@ -125,6 +133,10 @@ "type": "boolean", "description": "Use __flow/__app/__raw_app suffixes instead of .flow/.app/.raw_app" }, + "syncBehavior": { + "type": "string", + "description": "Sync behavior version — controls ownership handling during push/pull (v1: preserve permissioned_as on update, strip on_behalf_of_email on pull)" + }, "codebases": { "type": "array", "description": "Codebase bundling configurations for shared libraries", diff --git a/debugger/Dockerfile b/debugger/Dockerfile index 51e993746a..6b423920a0 100644 --- a/debugger/Dockerfile +++ b/debugger/Dockerfile @@ -19,7 +19,7 @@ FROM ghcr.io/windmill-labs/windmill-ee:main AS windmill-source # Stage 2: Build the debug service -FROM oven/bun:1 AS runtime +FROM oven/bun:1.4.0 AS runtime # Install Python and required system dependencies RUN apt-get update && apt-get install -y --no-install-recommends \ @@ -46,7 +46,9 @@ WORKDIR /app # Copy the debug service files COPY dap_debug_service.ts . COPY dap_websocket_server_bun.ts . +COPY env_passthrough.ts . COPY dap_websocket_server.py . +COPY registry_config.ts . # Expose the default port EXPOSE 5679 diff --git a/debugger/README.md b/debugger/README.md index dbeb213351..1dba49d977 100644 --- a/debugger/README.md +++ b/debugger/README.md @@ -75,6 +75,122 @@ Options: | `DAP_NSJAIL_PATH` | nsjail binary path | nsjail | | `DAP_NSJAIL_CONFIG` | nsjail config file path | - | +### Dependency preparation + +Before debugging a script, its imports are installed through `windmill prepare-deps`, which runs +`uv` (Python) or `bun install` (TypeScript) without a database connection. The install runs in the +service rather than in the session because the registry configuration usually embeds credentials +and a debug server executes the submitted script inside a process the script can read; the Python +server is handed only the resulting venv, with `--venv-path`, and a Bun session only the resulting +`node_modules`. + +`DAP_PREPARE_DEPS_TIMEOUT_MS` bounds the install (default 120000); past it the session starts +without its dependencies. When the install fails, the CLI answers `success: false` and carries the +installer's stderr in both `error` and `install_stderr`; the service reports it to the client as an +`output` event, so the reason (unreachable mirror, untrusted certificate, unknown package) reaches +the user instead of a bare `ModuleNotFoundError` at the first import. + +### Registry configuration + +Because `prepare-deps` has no database, the service reads the instance settings for it from +`GET /api/debug/registry_config` on `WINDMILL_BASE_URL` and passes them down over the CLI's stdin +request. It is authorized by the launch token of the session being started, and serves only the +settings that session's own installer runs on, so a TypeScript session's token cannot be used to +read the Python index credentials. + +The token also reaches the browser, so what it can fetch is what a workspace member can fetch. +Sessions started by an operator are refused outright, since an operator cannot run a preview job +either; for a member who can, the npm settings are already exposed by a preview (a worker leaves +the same `.npmrc` / `bunfig.toml` in the directory the previewed script runs in), while the Python +index URL, which otherwise only appears as uv's argv, becomes readable where it was not before. + +These settings are Enterprise-only, exactly as they are for jobs, and a CE instance reports that in +the session's output rather than applying them: + +| Setting | Applies to | +|---------|------------| +| `npm_config_registry` | `bun install` registry and its `:_authToken=` | +| `npmrc` | written verbatim as `.npmrc`, taking precedence over `npm_config_registry` | +| `bunfig_install_scopes` | `[install.scopes]` in the generated `bunfig.toml` | +| `pip_index_url` | `uv --index-url` | +| `pip_extra_index_url` | `uv --extra-index-url`, comma-separated | + +`uv_index_strategy` is served on any edition, like it is to a worker. An index URL holding the +`EPHEMERAL_TOKEN` placeholder is not served at all: only a worker can run the command that +substitutes it. + +The credential-bearing files (`.npmrc`, `bunfig.toml`) are written under +`/var/tmp/windmill-debug-registry`, not into the directory the install runs in, and are deleted +when the install ends. That directory is not private to the install: a session resolves its +`node_modules` symlink back into it, and `nsjail.debug.config.proto` bind-mounts the whole of +`/tmp` into every session, so credentials left there would be readable by a concurrent session. +`/var/tmp` is a tmpfs in that same config, one instance per jail, so a session sees an empty one +and a jailed install's credentials go away with the jail even when it is killed (the service kills +an install with SIGKILL, which no cleanup in the installer can survive). An install running +unjailed writes to the host's `/var/tmp` instead, where a directory a kill left behind is removed +by the next install; a session running unjailed is unconfined anyway and sees the whole filesystem, +as it already does the rest of the service's state. + +The rest of the registry configuration has no instance setting and is read from the environment of +the debug service. Where two names are listed the first wins; a worker reads the same names: + +| Variable | Description | Default | +|----------|-------------|---------| +| `PY_TRUSTED_HOST` / `PIP_TRUSTED_HOST` | Hosts to trust, whitespace-separated (`--trusted-host`) | - | +| `PY_INDEX_CERT` / `PIP_INDEX_CERT` | CA bundle for the index, passed to uv as `SSL_CERT_FILE`. Falls back to `SSL_CERT_FILE`, then `REQUESTS_CA_BUNDLE`, then `CURL_CA_BUNDLE`, so a host that configures its CA under any of those names is picked up. Whichever is used **replaces** uv's own roots rather than adding to them, so it has to be a complete bundle: one holding only a private CA leaves every public index untrusted. `bun install` gets the same bundle as `NODE_EXTRA_CA_CERTS`, the only spelling Bun reads | - | +| `SSL_CERT_DIR` | Directory of certificates, forwarded to uv as-is. Replaces uv's roots the same way the bundle does, so a directory holding only a private CA leaves public indexes untrusted | - | +| `PY_NATIVE_CERT` / `UV_NATIVE_TLS` | `true` to also trust the platform certificate store (`--native-tls`) | false | +| `UV_HTTP_TIMEOUT` | uv HTTP request timeout, in seconds | uv's own default | +| `DAP_REGISTRY_CONFIG_TIMEOUT_MS` | How long to wait on the settings fetch before installing without it | 10000 | + +`PY_INDEX_URL` / `PIP_INDEX_URL` and `PY_EXTRA_INDEX_URL` / `PIP_EXTRA_INDEX_URL`, along with +`UV_INDEX_STRATEGY`, are still read from the same environment whenever the fetch yields no index: +because the instance has none set, because this is a CE instance, or because the session was not +allowed the settings. A Python debug service configured that way therefore keeps working, but setting +them is an instance-wide decision to install Python dependencies from that index, independent of who +opened the session; leave them unset to let the instance settings alone decide. The npm settings have +no such fallback: the instance settings are the only source. + +Proxy variables (`HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY`, in either case) are forwarded from the +service into each session, since the debugged script needs them for its own outbound calls, exactly +as a job's script does on a worker. When a proxy is set without a bypass list, `NO_PROXY` defaults +to `localhost,127.0.0.1` so calls to `BASE_INTERNAL_URL` are not proxied. + +Trust roots are forwarded alongside them: `SSL_CERT_FILE`, `SSL_CERT_DIR`, `REQUESTS_CA_BUNDLE`, +`CURL_CA_BUNDLE` and `NODE_EXTRA_CA_CERTS`. Behind a TLS-intercepting proxy these are what let the +debugged script's own HTTPS calls verify, and installing the CA in the container's system store is +not enough on its own, since `requests` carries its own bundle and Node reads only +`NODE_EXTRA_CA_CERTS`. Registry settings are deliberately not forwarded: they carry credentials and +only the service needs them. + +Registering that CA in the container's system store happens on its own: mount it into +`/usr/local/share/ca-certificates/` **named `*.crt`**, the only extension `update-ca-certificates` +reads, and `windmill_extra` runs it before starting any service. `RUN_UPDATE_CA_CERTIFICATE_AT_START=true` forces the same thing whether or not +certificates are mounted there, and `RUN_UPDATE_CA_CERTIFICATE_PATH` overrides the tool, matching +the server and worker. Both are best-effort: a UID that cannot write `/etc/ssl/certs` logs a warning +and the container still boots. `INIT_SCRIPT` remains the hook for anything more involved, and unlike +the CA update it aborts startup when it fails. + +Note what the system store does *not* cover, which is most of what a debug session installs with: +uv trusts its own bundled roots unless `PY_NATIVE_CERT`/`UV_NATIVE_TLS` is `true`, Bun and Node read +only `NODE_EXTRA_CA_CERTS`, and `requests` carries certifi. Registering the CA fixes Python's stdlib +`ssl`, `curl` and `git`; the rest still needs the variables above. + +Keeping the settings out of the session's environment only bounds what the debugged script can read +from itself. An unsandboxed session runs under the same user as the service and can still read the +service's environment through `/proc`, the same way a job can read a worker's when the worker runs +unsandboxed. Isolating sessions from the service takes `--nsjail --nsjail-config +nsjail.debug.config.proto`: it is that config's PID namespace and `mount_proc` that put the service +out of reach, not the flag on its own. + +The installer is jailed on the same terms, in both languages: `uv pip install` builds source +distributions and `bun install` runs postinstall scripts, so a package's own code executes there +too. It keeps the service's environment across that boundary — the config sets `keep_env`, which is +how the settings above reach it — so replacing that with an allowlist would have to carry the +registry and CA variables in explicitly. It also runs in its own process group, because `uv` and +`bun` are grandchildren: signalling only the installer reparents them to init and they keep +downloading, which would make the timeout and the cancel-on-disconnect half-measures. + ### Frontend Integration ```svelte diff --git a/debugger/dap_debug_service.ts b/debugger/dap_debug_service.ts index 2b9ce46272..11616f7b11 100644 --- a/debugger/dap_debug_service.ts +++ b/debugger/dap_debug_service.ts @@ -40,7 +40,14 @@ import { tmpdir } from 'node:os' import { join } from 'node:path' // Import the working Bun debug session from the standalone server -import { DebugSession as BunDebugSessionWorking, type NsjailConfig } from './dap_websocket_server_bun' +import { + DebugSession as BunDebugSessionWorking, + killProcessTree, + nsjailWrap, + type NsjailConfig +} from './dap_websocket_server_bun' +import { sessionEnv } from './env_passthrough' +import { fetchRegistryConfig, type RegistryConfig } from './registry_config' // ============================================================================ // Configuration @@ -348,39 +355,35 @@ interface SpawnOptions { cmd: string[] cwd?: string env?: Record + stdin?: Blob + /** + * Hand the child this process's whole environment rather than the minimal set below. Only + * the dependency installer wants it: the registry credentials and CA settings it reads are + * precisely what the minimal set exists to keep away from a debugged script. + */ + inheritEnv?: boolean + /** + * Start the child in its own process group, so killProcessTree can signal what it spawns. + */ + detached?: boolean stdout?: 'pipe' | 'inherit' stderr?: 'pipe' | 'inherit' } +/** + * How long `windmill prepare-deps` may take before the session gives up on it and starts without + * the dependencies. Raise it for slow private mirrors, where a large install can outlast the default. + */ +const PREPARE_DEPS_TIMEOUT_MS = Number(process.env.DAP_PREPARE_DEPS_TIMEOUT_MS) || 120_000 + /** * Spawn a process, optionally wrapped with nsjail. * This is the key function for sandboxed execution. */ function spawnProcess(options: SpawnOptions): Subprocess { - let cmd = options.cmd + const cmd = nsjailWrap(options.cmd, config.nsjail, options.cwd) if (config.nsjail.enabled) { - // Build nsjail command - const nsjailCmd = [config.nsjail.binaryPath] - - // Add config file if specified - if (config.nsjail.configPath) { - nsjailCmd.push('--config', config.nsjail.configPath) - } - - // Add any extra nsjail arguments - nsjailCmd.push(...config.nsjail.extraArgs) - - // Add working directory if specified - if (options.cwd) { - nsjailCmd.push('--cwd', options.cwd) - } - - // Separator and actual command - nsjailCmd.push('--') - nsjailCmd.push(...cmd) - - cmd = nsjailCmd logger.info(`Spawning with nsjail: ${cmd.join(' ')}`) } else { logger.info(`Spawning: ${cmd.join(' ')}`) @@ -391,9 +394,12 @@ function spawnProcess(options: SpawnOptions): Subprocess { return spawn({ cmd, cwd: options.cwd || process.cwd(), + ...(options.stdin ? { stdin: options.stdin } : {}), + ...(options.detached ? { detached: true } : {}), stdout: options.stdout || 'pipe', stderr: options.stderr || 'pipe', env: { + ...(options.inheritEnv ? process.env : {}), // Essential system vars PATH: process.env.PATH || '/usr/bin:/bin', HOME: process.env.HOME, @@ -489,6 +495,15 @@ abstract class BaseDebugSession { // Python Debug Session // ============================================================================ +const DEFAULT_DEBUGPY_TIMEOUT_MS = 10_000 + +// `launch` waits on dependency preparation in the Python server, which allows `windmill +// prepare-deps` up to 120s; anything shorter here reports a timeout while the install is +// still legitimately running. +const DEBUGPY_TIMEOUT_MS_BY_COMMAND: Record = { + launch: 180_000 +} + class PythonDebugSession extends BaseDebugSession { private debugpyWs: WebSocket | null = null private debugpySeq = 1 @@ -502,6 +517,9 @@ class PythonDebugSession extends BaseDebugSession { private scriptResult: unknown = undefined private envVars: Record = {} private windmillPath?: string + private venvPath?: string + private prepareDepsProcess: Subprocess | null = null + private disposed = false private debugMode: boolean constructor(ws: { send: (data: string) => void; close: () => void }, windmillPath?: string, debugMode = false) { @@ -531,11 +549,13 @@ class PythonDebugSession extends BaseDebugSession { arguments: args } + const timeoutMs = DEBUGPY_TIMEOUT_MS_BY_COMMAND[command] ?? DEFAULT_DEBUGPY_TIMEOUT_MS + return new Promise((resolve, reject) => { const timeout = setTimeout(() => { this.pendingDebugpyRequests.delete(seq) - reject(new Error(`Debugpy command timeout: ${command}`)) - }, 10000) + reject(new Error(`Debugpy command timeout: ${command} (after ${timeoutMs}ms)`)) + }, timeoutMs) this.pendingDebugpyRequests.set(seq, { resolve: (value) => { @@ -627,6 +647,110 @@ class PythonDebugSession extends BaseDebugSession { } } + /** + * Install the script's imports through `windmill prepare-deps` and return the venv to add to + * the debugged script's sys.path. + * + * This runs here rather than in the Python server because the registry settings the CLI is + * given routinely embed private-registry credentials, and the Python server executes the + * submitted script inside its own interpreter: anything in that process is recoverable by the + * script. The service never executes user code, so the credentials stop here. + * + * It still goes through spawnProcess so nsjail confines it on the same terms as the debuggee: + * `uv pip install` builds source distributions, which executes their build backend's arbitrary + * Python. Those same credentials are what `inheritEnv` is for — the jail config keeps the + * environment across the boundary, so nothing else has to carry them in. + */ + private async prepareDependencies(code: string, registry: RegistryConfig): Promise { + if (!this.windmillPath) { + logger.info('No windmill binary path configured, skipping dependency preparation') + return null + } + + const warn = (reason: string): null => { + // cleanup() kills the installer, which ends the read with nothing to parse. Reporting + // that as an install failure blames the user for their own disconnect, on a websocket + // that is being torn down anyway. + if (this.disposed) { + return null + } + logger.error(`prepare-deps failed: ${reason}`) + this.sendEvent('output', { + category: 'stderr', + output: `Failed to prepare dependencies: ${reason}\n` + }) + return null + } + + try { + const proc = spawnProcess({ + cmd: [this.windmillPath, 'prepare-deps'], + // The venv has to be built against the interpreter that will run the script: its + // site-packages goes on that interpreter's sys.path, and uv otherwise picks its + // own, which silently leaves compiled extensions unimportable. + stdin: new Blob([ + JSON.stringify({ + code, + language: 'python3', + python_path: config.pythonPath, + registry + }) + '\n' + ]), + inheritEnv: true, + detached: true + }) + this.prepareDepsProcess = proc + + // The launch response is already sent, so an install that never returns would leave the + // client waiting on a session that never starts, with nothing on screen. The deadline + // races the read rather than only killing the child: a grandchild holding the pipe open + // keeps the read pending long after the child itself is gone. + let timer: ReturnType | undefined + // spawnProcess's return type does not carry the piped stdio through + const read = (async () => ({ + output: await new Response(proc.stdout as ReadableStream).text(), + stderr: await new Response(proc.stderr as ReadableStream).text() + }))() + const result = await Promise.race([ + read, + new Promise((resolve) => { + timer = setTimeout(() => resolve(null), PREPARE_DEPS_TIMEOUT_MS) + }) + ]) + clearTimeout(timer) + this.prepareDepsProcess = null + + if (!result) { + killProcessTree(proc) + return warn( + `dependency installation timed out after ${PREPARE_DEPS_TIMEOUT_MS / 1000}s` + ) + } + const { output, stderr } = result + + const lastLine = output.trim().split('\n').pop() || '' + if (!lastLine.startsWith('{')) { + return warn(stderr.trim() || 'windmill binary produced no response') + } + + const response = JSON.parse(lastLine) + if (!response.success) { + // install_stderr is the installer's raw output; `error` already contains it, so + // prefer whichever the CLI version at hand provides. + return warn(response.install_stderr || response.error || 'unknown error') + } + + if (response.venv_path) { + logger.info(`Dependencies installed at: ${response.venv_path}`) + } else { + logger.info('No external dependencies to install') + } + return response.venv_path || null + } catch (error) { + return warn(String(error)) + } + } + private async startPythonProcess(cwd: string): Promise { if (!this.scriptPath) { throw new Error('No script path') @@ -648,10 +772,11 @@ class PythonDebugSession extends BaseDebugSession { '--host', '127.0.0.1' ] - // Pass windmill path for dependency auto-installation if configured - if (this.windmillPath) { - cmd.push('--windmill', this.windmillPath) - logger.info(`Python session: autoinstall enabled with windmill at ${this.windmillPath}`) + // Dependencies are installed by the service (see prepareDependencies), so the server is + // handed the resulting venv instead of the windmill binary it would install with. + if (this.venvPath) { + cmd.push('--venv-path', this.venvPath) + logger.info(`Python session: using dependencies at ${this.venvPath}`) } // Pass debug flag to Python subprocess @@ -662,7 +787,7 @@ class PythonDebugSession extends BaseDebugSession { this.process = spawnProcess({ cmd, cwd, - env: { PYTHONUNBUFFERED: '1', ...this.envVars } + env: { PYTHONUNBUFFERED: '1', ...sessionEnv(), ...this.envVars } }) // Read stderr to capture startup messages @@ -794,6 +919,13 @@ class PythonDebugSession extends BaseDebugSession { this.debugpyWs.onclose = () => { logger.info('Debugpy WebSocket closed') this.debugpyWs = null + // A Python server that dies mid-request must fail it now; otherwise the caller + // waits out the command timeout, which for `launch` is minutes. + const aborted = Array.from(this.pendingDebugpyRequests.values()) + this.pendingDebugpyRequests.clear() + for (const pending of aborted) { + pending.reject(new Error('Debugpy connection closed')) + } } }) } @@ -905,6 +1037,10 @@ class PythonDebugSession extends BaseDebugSession { } private async handleLaunch(request: DAPMessage): Promise { + // Per launch, not per session: cleanup() also runs when a program finishes normally, and + // the flag must only mean "torn down while this launch was still preparing". + this.disposed = false + const args = request.arguments || {} let code = args.code as string | undefined this.scriptPath = args.program as string | undefined @@ -912,6 +1048,8 @@ class PythonDebugSession extends BaseDebugSession { this.callMain = (args.callMain as boolean) || false this.mainArgs = (args.args as Record) || {} this.envVars = (args.env as Record) || {} + // Also what authorizes the registry configuration fetch below. + const token = args.token as string | undefined // Enforce signing on every launch. The token is passed in the launch // arguments and is verified against the inline `code` (see windmill-api-debug). @@ -925,7 +1063,6 @@ class PythonDebugSession extends BaseDebugSession { return } - const token = args.token as string | undefined if (!token) { logger.error('No debug token provided but signed requests are required') this.sendResponse(request, false, {}, 'Debug token required. Ensure the debug session was signed by the backend.') @@ -987,6 +1124,32 @@ sys.stdout.flush() this.sendResponse(request) try { + if (code) { + const registry = await fetchRegistryConfig(token, logger) + // A round trip of its own, during which the client can give up: the installer runs + // a source distribution's build backend, so starting one for a session that is + // already gone executes package code nobody is waiting for. + if (this.disposed) { + logger.info('Session torn down during the registry configuration fetch, not installing') + await this.cleanup() + return + } + if (registry.message) { + this.sendEvent('output', { category: 'console', output: `${registry.message}\n` }) + } + this.venvPath = (await this.prepareDependencies(code, registry)) ?? undefined + } + + // Installing takes long enough for the client to give up meanwhile, and cleanup() has + // then already run: starting the debuggee now would leave a process nothing owns + // executing the script for a session that is gone. Clean up again on the way out, + // since a teardown that landed before the script was written left it behind. + if (this.disposed) { + logger.info('Session torn down during dependency preparation, not starting Python') + await this.cleanup() + return + } + await this.startPythonProcess(cwd) // Re-apply breakpoints to the Python server using the actual script path @@ -1012,7 +1175,13 @@ sys.stdout.flush() }) } catch (error) { this.sendEvent('output', { category: 'stderr', output: `Failed to start Python: ${error}\n` }) + // Claim the terminated event before cleanup kills the process, otherwise the + // `exited` handler sends a second one whose empty body erases this error. + this.terminatedSent = true this.sendEvent('terminated', { error: String(error) }) + // A Python server that refused the launch stays in its connection loop, so + // nothing else ever reaps it, its websocket or the temp dir. + await this.cleanup() } } @@ -1036,6 +1205,14 @@ sys.stdout.flush() } async cleanup(): Promise { + this.disposed = true + + // A client that gives up mid-install must not leave the package manager running + if (this.prepareDepsProcess) { + killProcessTree(this.prepareDepsProcess) + this.prepareDepsProcess = null + } + if (this.debugpyWs) { this.debugpyWs.close() this.debugpyWs = null diff --git a/debugger/dap_websocket_server.py b/debugger/dap_websocket_server.py index 624b339894..33937d9617 100644 --- a/debugger/dap_websocket_server.py +++ b/debugger/dap_websocket_server.py @@ -277,12 +277,55 @@ class WindmillDebugger(bdb.Bdb): return {} +PREPARE_DEPS_TIMEOUT_SECONDS = 120 +PREPARE_DEPS_PROGRESS_INTERVAL_SECONDS = 5 + + +@dataclass +class PrepareResult: + """ + Outcome of dependency preparation. + + `error` holds anything worth telling the user, including a problem reported by an + otherwise successful preparation. Only `fatal` means the packages are known to be + missing: failing to reach the CLI at all says nothing about the script's imports and + must not block a session that would otherwise run. + """ + + venv_path: str | None = None + error: str | None = None + fatal: bool = False + + +def _prepare_error_detail(response: dict) -> str: + """ + Build the failure reason from a prepare-deps response. + + `error` is the installer's own output prefixed with the step that failed, and + `install_stderr` is that same output unprefixed, so take one rather than both: joining + them prints the installer's output twice, and the prefix is what tells a reader whether + the venv or the install was what went wrong. + """ + for key in ("error", "install_stderr"): + detail = str(response.get(key) or "").strip() + if detail: + return detail + return "unknown error" + + +def _first_line(detail: str, limit: int = 300) -> str: + """Condense a multi-line failure into the single line a DAP response message allows.""" + line = next((s.strip() for s in detail.splitlines() if s.strip()), detail.strip()) + return line[:limit] + + class DebugSession: """Manages a single debug session.""" - def __init__(self, websocket, windmill_path: str | None = None): + def __init__(self, websocket, windmill_path: str | None = None, prepared_venv_path: str | None = None): self.websocket = websocket self.windmill_path = windmill_path + self._prepared_venv_path = prepared_venv_path self.seq = 1 self.initialized = False self.configured = False @@ -304,14 +347,22 @@ class DebugSession: self.seq += 1 return seq - def prepare_dependencies(self, code: str) -> str | None: + def prepare_dependencies(self, code: str) -> PrepareResult: """ Prepare Python dependencies by calling the windmill CLI. - Returns the path to the venv's site-packages directory, or None if no dependencies needed. + + Blocks for as long as the install takes, so it must run off the event loop; use + `_prepare_dependencies_with_progress` instead of calling this directly. """ + if self._prepared_venv_path: + # The debug service installs dependencies itself so that the registry credentials + # the CLI needs never enter this interpreter, which executes the debugged script. + logger.info(f"Using dependencies prepared by the debug service: {self._prepared_venv_path}") + return PrepareResult(venv_path=self._prepared_venv_path) + if not self.windmill_path: logger.info("No windmill binary path configured, skipping dependency preparation") - return None + return PrepareResult() logger.info(f"Preparing dependencies using {self.windmill_path}") @@ -328,7 +379,7 @@ class DebugSession: input=input_data, capture_output=True, text=True, - timeout=120, # 2 minute timeout for dependency installation + timeout=PREPARE_DEPS_TIMEOUT_SECONDS, ) elapsed = time.time() - start_time @@ -337,7 +388,11 @@ class DebugSession: if result.returncode != 0: logger.error(f"prepare-deps failed (stderr): {result.stderr}") logger.error(f"prepare-deps failed (stdout): {result.stdout}") - return None + detail = (result.stderr or "").strip() or (result.stdout or "").strip() + return PrepareResult( + error=detail or f"windmill prepare-deps exited with code {result.returncode}", + fatal=True, + ) # Log raw output for debugging logger.debug(f"prepare-deps stdout: {result.stdout[:500] if result.stdout else '(empty)'}") @@ -350,15 +405,18 @@ class DebugSession: json_start = output.find('{') if json_start == -1: logger.error(f"No JSON in prepare-deps output: {output}") - return None + return PrepareResult( + error=f"No JSON in prepare-deps output: {output[:500] or '(empty)'}" + ) json_str = output[json_start:] response = json.loads(json_str) logger.debug(f"prepare-deps response: {response}") if not response.get("success"): - logger.error(f"prepare-deps error: {response.get('error')}") - return None + detail = _prepare_error_detail(response) + logger.error(f"prepare-deps error: {detail}") + return PrepareResult(error=detail, fatal=True) venv_path = response.get("venv_path") cached = response.get("cached", False) @@ -371,18 +429,50 @@ class DebugSession: else: logger.info("No external dependencies detected in code") - return venv_path + return PrepareResult(venv_path=venv_path) except subprocess.TimeoutExpired: - logger.error("prepare-deps timed out after 120s") - return None + message = f"prepare-deps timed out after {PREPARE_DEPS_TIMEOUT_SECONDS}s" + logger.error(message) + return PrepareResult(error=message, fatal=True) except json.JSONDecodeError as e: + raw = output[:500] if 'output' in dir() else '(not available)' logger.error(f"Failed to parse prepare-deps JSON output: {e}") - logger.error(f"Raw output was: {output[:500] if 'output' in dir() else '(not available)'}") - return None + logger.error(f"Raw output was: {raw}") + return PrepareResult(error=f"Failed to parse prepare-deps output: {e}\n{raw}") except Exception as e: logger.exception(f"Error preparing dependencies: {e}") - return None + return PrepareResult(error=f"Error preparing dependencies: {e}") + + async def _prepare_dependencies_with_progress(self, code: str) -> PrepareResult: + """ + Run dependency preparation on a worker thread, reporting progress while it runs. + + The install can take minutes on a cold cache; on the event loop it would stall + websocket keepalive until it returns and block the progress events below. + """ + await self.send_event( + "output", {"category": "stdout", "output": "Preparing dependencies...\n"} + ) + + task = asyncio.create_task(asyncio.to_thread(self.prepare_dependencies, code)) + waited = 0 + while True: + done, _ = await asyncio.wait( + {task}, timeout=PREPARE_DEPS_PROGRESS_INTERVAL_SECONDS + ) + if done: + break + waited += PREPARE_DEPS_PROGRESS_INTERVAL_SECONDS + await self.send_event( + "output", + { + "category": "stdout", + "output": f"Still preparing dependencies... ({waited}s)\n", + }, + ) + + return task.result() def _next_var_ref(self) -> int: ref = self._variables_ref_counter @@ -521,7 +611,25 @@ class DebugSession: # Prepare dependencies before modifying the code if code: - self._venv_path = self.prepare_dependencies(code) + prepared = await self._prepare_dependencies_with_progress(code) + if prepared.error: + prefix = ( + "Failed to prepare dependencies" + if prepared.fatal + else "Warning: dependency preparation reported a problem, running anyway" + ) + await self.send_event( + "output", + {"category": "stderr", "output": f"{prefix}:\n{prepared.error}\n"}, + ) + if prepared.fatal: + await self.send_response( + request, + success=False, + message=f"Failed to prepare dependencies: {_first_line(prepared.error)}", + ) + return + self._venv_path = prepared.venv_path # If callMain is True, append a call to main() with the provided args if self._call_main and code: @@ -894,13 +1002,17 @@ class DebugSession: ) -# Module-level variable to store windmill binary path +# Module-level variables to store the windmill binary path and, when the debug service +# already installed the script's dependencies, the venv to use instead of installing here. _windmill_path: str | None = None +_prepared_venv_path: str | None = None async def handle_connection(websocket) -> None: """Handle a WebSocket connection.""" - session = DebugSession(websocket, windmill_path=_windmill_path) + session = DebugSession( + websocket, windmill_path=_windmill_path, prepared_venv_path=_prepared_venv_path + ) logger.info(f"New connection from {websocket.remote_address}") try: @@ -924,13 +1036,21 @@ async def handle_connection(websocket) -> None: session._cleanup_temp_file() -async def main(host: str = "localhost", port: int = 5679, windmill_path: str | None = None) -> None: +async def main( + host: str = "localhost", + port: int = 5679, + windmill_path: str | None = None, + prepared_venv_path: str | None = None, +) -> None: """Start the DAP WebSocket server.""" - global _windmill_path + global _windmill_path, _prepared_venv_path _windmill_path = windmill_path + _prepared_venv_path = prepared_venv_path if windmill_path: logger.info(f"Windmill binary path: {windmill_path}") + if prepared_venv_path: + logger.info(f"Dependencies prepared by the debug service: {prepared_venv_path}") logger.info(f"Starting DAP WebSocket server on ws://{host}:{port}") async with serve(handle_connection, host, port): @@ -944,6 +1064,7 @@ if __name__ == "__main__": parser.add_argument("--host", default="localhost", help="Host to bind to") parser.add_argument("--port", type=int, default=5679, help="Port to listen on") parser.add_argument("--windmill", help="Path to windmill binary for dependency preparation (or set WINDMILL_PATH env var)") + parser.add_argument("--venv-path", help="Site-packages directory of a venv the caller already prepared; skips dependency installation") parser.add_argument("--debug", action="store_true", help="Enable debug logging") args = parser.parse_args() @@ -957,6 +1078,6 @@ if __name__ == "__main__": windmill_path = args.windmill or os.environ.get("WINDMILL_PATH") try: - asyncio.run(main(args.host, args.port, windmill_path)) + asyncio.run(main(args.host, args.port, windmill_path, args.venv_path)) except KeyboardInterrupt: logger.info("Server stopped") diff --git a/debugger/dap_websocket_server_bun.ts b/debugger/dap_websocket_server_bun.ts index d6ed8967ad..3b5072b1e6 100644 --- a/debugger/dap_websocket_server_bun.ts +++ b/debugger/dap_websocket_server_bun.ts @@ -22,9 +22,12 @@ */ import { spawn, type Subprocess } from 'bun' +import { readFileSync } from 'node:fs' import { mkdtemp, writeFile, unlink, rmdir, symlink } from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' +import { sessionEnv } from './env_passthrough' +import { fetchRegistryConfig, type RegistryConfig } from './registry_config' // Types for V8 Inspector Protocol interface V8Message { @@ -222,6 +225,8 @@ function generateMainCallArgs(code: string, args: Record): stri const WINDMILL_BASE_URL = process.env.WINDMILL_BASE_URL || process.env.BASE_INTERNAL_URL // e.g., http://localhost:8000 const REQUIRE_SIGNED_REQUESTS = process.env.REQUIRE_SIGNED_DEBUG_REQUESTS !== 'false' +const PREPARE_DEPS_TIMEOUT_MS = 120_000 + // Opt-in cross-origin protection (CSWSH defense-in-depth); see // dap_debug_service.ts for the rationale. Only enforced for this file's // standalone Bun.serve entrypoint (the windmill-extra runtime imports the @@ -554,6 +559,63 @@ export interface NsjailConfig { extraArgs?: string[] } +/** + * Wrap a command so nsjail runs it, or return it unchanged when sandboxing is off. + * The environment is not filtered here: the config sets `keep_env`, so the jailed process + * receives whatever the spawning call gives it. + */ +export function nsjailWrap(cmd: string[], nsjail: NsjailConfig | undefined, cwd?: string): string[] { + if (!nsjail?.enabled) { + return cmd + } + const wrapped = [nsjail.binaryPath] + if (nsjail.configPath) { + wrapped.push('--config', nsjail.configPath) + } + if (nsjail.extraArgs) { + wrapped.push(...nsjail.extraArgs) + } + if (cwd) { + wrapped.push('--cwd', cwd) + } + wrapped.push('--', ...cmd) + return wrapped +} + +/** + * SIGKILL a subprocess along with everything it spawned. + * + * SIGKILL because `windmill prepare-deps` does not act on SIGTERM while uv is running. The + * whole group because uv is a grandchild: signalling the child alone reparents uv to init and + * it keeps downloading. The group id is read back from /proc instead of assumed, since a group + * kill aimed at this service's own group would take down every service in the container; a + * child spawned without `detached` therefore only gets the plain kill. + */ +export function killProcessTree(proc: Subprocess): void { + if (proc.exitCode !== null || proc.signalCode !== null) { + // Nothing left to signal, and the pid may already have been handed to someone else + return + } + + let ownsGroup = false + try { + const stat = readFileSync(`/proc/${proc.pid}/stat`, 'utf8') + // The comm field can hold spaces and parentheses, so read the fields after its closing one + ownsGroup = Number(stat.slice(stat.lastIndexOf(')') + 2).split(' ')[2]) === proc.pid + } catch { + // Already reaped, or not Linux: fall back to killing the process alone + } + try { + if (ownsGroup) { + process.kill(-proc.pid, 'SIGKILL') + } else { + proc.kill('SIGKILL') + } + } catch (error) { + logger.error('Failed to kill process:', error) + } +} + /** * VLQ (Variable-Length Quantity) decoder for source maps. * Returns array of decoded integers from VLQ string. @@ -751,6 +813,11 @@ export class DebugSession { // Path to installed node_modules (set after prepare-deps runs) private nodeModulesPath?: string + // Running dependency installer, so a teardown mid-install can stop it + private prepareDepsProcess: Subprocess | null = null + + private disposed = false + constructor(ws: WebSocket, options?: { nsjailConfig?: NsjailConfig; bunPath?: string; windmillPath?: string }) { this.ws = ws this.nsjailConfig = options?.nsjailConfig @@ -1435,6 +1502,10 @@ export class DebugSession { * Handle the 'launch' request. */ async handleLaunch(request: DAPMessage): Promise { + // Per launch, not per session: cleanup() also runs when a program finishes normally, and + // the flag must only mean "torn down while this launch was still preparing". + this.disposed = false + const args = request.arguments || {} let code = args.code as string | undefined this.scriptPath = args.program as string | undefined @@ -1442,6 +1513,8 @@ export class DebugSession { this.callMain = (args.callMain as boolean) || false this.mainArgs = (args.args as Record) || {} this.envVars = (args.env as Record) || {} + // Also what authorizes the registry configuration fetch below. + const token = args.token as string | undefined // Enforce signing on every launch. The token is passed in the launch // arguments and is verified against the inline `code` (see windmill-api-debug). @@ -1455,7 +1528,6 @@ export class DebugSession { return } - const token = args.token as string | undefined if (!token) { logger.error('No debug token provided but signed requests are required') this.sendResponse(request, false, {}, 'Debug token required. Ensure the debug session was signed by the backend.') @@ -1487,7 +1559,29 @@ export class DebugSession { // Prepare dependencies using the original code (before any modifications) // This analyzes imports and installs required npm packages if (code) { - this.nodeModulesPath = await this.prepareDependencies(code) || undefined + const registry = await fetchRegistryConfig(token, logger) + // A round trip of its own, during which the client can give up: the installer runs the + // packages' postinstall scripts, so starting one for a session that is already gone + // executes package code nobody is waiting for. + if (this.disposed) { + logger.info('Session was torn down during the registry configuration fetch, not installing') + this.sendResponse(request, false, {}, 'Session terminated during dependency preparation') + return + } + if (registry.message) { + this.sendEvent('output', { category: 'console', output: `${registry.message}\n` }) + } + this.nodeModulesPath = await this.prepareDependencies(code, registry) || undefined + + // Installing takes long enough for the client to give up meanwhile, and cleanup() has + // then already run: starting the debuggee now would leak a process nothing owns. + // The response still goes out, since a client that terminated without closing the + // socket is otherwise left waiting out its own launch timeout. + if (this.disposed) { + logger.info('Session was torn down during dependency preparation, not starting Bun') + this.sendResponse(request, false, {}, 'Session terminated during dependency preparation') + return + } // Remove version specifiers from imports (e.g., "lodash@4" -> "lodash") // This must happen AFTER prepareDependencies (which needs the versions) @@ -1570,8 +1664,21 @@ export class DebugSession { * Prepare dependencies by calling the windmill CLI's prepare-deps command. * This analyzes imports in the code and installs required npm packages. * Returns the path to node_modules if any were installed. + * + * Jailed on the same terms as the debuggee: `bun install` runs the packages' postinstall + * scripts, which is user-supplied code executing next to the other services in the container. + * Its environment is inherited rather than filtered, which is what carries the CA settings + * into the installer (the jail keeps the environment across the boundary). + * + * The CLI has no database, so `registry` carries the instance's registry settings down to it + * instead. They configure `bun install` and nothing else: the debugged script never gets + * them, since it could read them back out of the process it runs in. */ - private async prepareDependencies(code: string, language: string = 'bun'): Promise { + private async prepareDependencies( + code: string, + registry: RegistryConfig, + language: string = 'bun' + ): Promise { if (!this.windmillPath) { logger.info('No windmill binary path configured, skipping dependency preparation') return null @@ -1579,21 +1686,67 @@ export class DebugSession { logger.info(`Preparing dependencies using ${this.windmillPath}`) + // The launch response is only sent once this returns, so without progress a cold + // cache looks like a frozen debugger for as long as the install takes. + this.sendEvent('output', { category: 'console', output: 'Preparing dependencies...\n' }) + let waited = 0 + const progress = setInterval(() => { + waited += 5 + this.sendEvent('output', { + category: 'console', + output: `Still preparing dependencies... (${waited}s)\n` + }) + }, 5000) + let killTimer: ReturnType | undefined + let timedOut = false + try { - const input = JSON.stringify({ code, language }) + '\n' + const input = JSON.stringify({ code, language, registry }) + '\n' logger.info(`prepare-deps input length: ${input.length}`) - // Spawn the windmill binary with prepare-deps command + // Spawn the windmill binary with prepare-deps command. Its environment is inherited + // rather than filtered, which is what gives prepare-deps the container's index and + // certificate settings; the allowlist above is what keeps them from the debugged + // script, and the jail keeps them across its own boundary. + const cmd = nsjailWrap([this.windmillPath, 'prepare-deps'], this.nsjailConfig) + logger.info(`Spawning${this.nsjailConfig?.enabled ? ' with nsjail' : ''}: ${cmd.join(' ')}`) const proc = spawn({ - cmd: [this.windmillPath, 'prepare-deps'], + cmd, stdin: new Blob([input]), // Use Blob for complete stdin data stdout: 'pipe', - stderr: 'pipe' + stderr: 'pipe', + // So the installer and the bun it spawns can be killed as one group + detached: true }) + this.prepareDepsProcess = proc + + // Bound the wait: the only other ceiling is the DAP client's launch timeout, + // which is minutes, so a wedged installer would hang the session that long. + killTimer = setTimeout(() => { + timedOut = true + logger.error(`prepare-deps timed out after ${PREPARE_DEPS_TIMEOUT_MS}ms`) + killProcessTree(proc) + }, PREPARE_DEPS_TIMEOUT_MS) // Wait for completion const output = await new Response(proc.stdout).text() const stderr = await new Response(proc.stderr).text() + + // The read also ends when cleanup() kills the installer, which leaves no output to + // parse. Reporting that as an install failure blames the user for their own Stop. + if (this.disposed) { + return null + } + + if (timedOut) { + const errorMsg = `prepare-deps timed out after ${PREPARE_DEPS_TIMEOUT_MS / 1000}s` + this.sendEvent('output', { + category: 'console', + output: `Warning: Failed to prepare dependencies: ${errorMsg}\n` + }) + return null + } + logger.info(`prepare-deps output: ${output.substring(0, 200)}`) logger.info(`prepare-deps stderr: ${stderr.substring(0, 200)}`) @@ -1642,12 +1795,19 @@ export class DebugSession { logger.info('No external dependencies to install') return null } catch (error) { + if (this.disposed) { + return null + } logger.error(`Failed to prepare dependencies: ${error}`) this.sendEvent('output', { category: 'console', output: `Warning: Failed to prepare dependencies: ${error}\n` }) return null + } finally { + clearInterval(progress) + clearTimeout(killTimer) + this.prepareDepsProcess = null } } @@ -1663,24 +1823,13 @@ export class DebugSession { const inspectUrl = `127.0.0.1:${inspectPort}` // Build the command - optionally wrapped with nsjail - let cmd: string[] = [this.bunPath, `--inspect-wait=${inspectUrl}`, this.scriptPath] + const cmd = nsjailWrap( + [this.bunPath, `--inspect-wait=${inspectUrl}`, this.scriptPath], + this.nsjailConfig, + cwd + ) if (this.nsjailConfig?.enabled) { - const nsjailCmd = [this.nsjailConfig.binaryPath] - - if (this.nsjailConfig.configPath) { - nsjailCmd.push('--config', this.nsjailConfig.configPath) - } - - if (this.nsjailConfig.extraArgs) { - nsjailCmd.push(...this.nsjailConfig.extraArgs) - } - - nsjailCmd.push('--cwd', cwd) - nsjailCmd.push('--') - nsjailCmd.push(...cmd) - - cmd = nsjailCmd logger.info(`Starting Bun with nsjail: ${cmd.join(' ')}`) } else { logger.info(`Starting Bun with --inspect-wait=${inspectUrl}`) @@ -1698,12 +1847,16 @@ export class DebugSession { }, 10000) }) - // Only include essential env vars + client-provided ones + // Only include essential env vars + the network-config allowlist + client-provided ones. // Don't inherit all of process.env to keep debugger environment clean const envVars: Record = { // Essential system vars PATH: process.env.PATH || '/usr/bin:/bin', HOME: process.env.HOME, + // Proxy / TLS settings inherited from the container, before the client's env so an + // explicit override still wins. Package-index settings are deliberately absent: this + // runs user-supplied code and index URLs carry registry credentials. + ...sessionEnv(), // Client-provided env vars (WM_WORKSPACE, WM_TOKEN, etc.) // Note: WM_BASE_URL is already overridden by BASE_INTERNAL_URL if set ...this.envVars @@ -2453,15 +2606,23 @@ export class DebugSession { } /** - * Clean up resources. + * Clean up resources. Public because both servers call it when a client goes away. */ - private async cleanup(): Promise { + async cleanup(): Promise { + this.disposed = true + // Close inspector connection if (this.inspectorWs) { this.inspectorWs.close() this.inspectorWs = null } + // A disconnect during dependency installation must not leave bun install running + if (this.prepareDepsProcess) { + killProcessTree(this.prepareDepsProcess) + this.prepareDepsProcess = null + } + // Kill process if (this.process) { this.process.kill() @@ -2601,9 +2762,14 @@ if (import.meta.main) { logger.error('Error handling message:', error) } }, - close(ws) { + async close(ws) { logger.info('Client disconnected') - sessions.delete(ws) + const session = sessions.get(ws) + if (session) { + // Dropping the session without this leaves its installer and debuggee running + await session.cleanup() + sessions.delete(ws) + } } } }) diff --git a/debugger/env_passthrough.ts b/debugger/env_passthrough.ts new file mode 100644 index 0000000000..f713483d05 --- /dev/null +++ b/debugger/env_passthrough.ts @@ -0,0 +1,43 @@ +/** + * Container network configuration forwarded to a debug session, matching what a worker gives a + * job's script. The session environment is built from an allowlist rather than inherited, so an + * outbound proxy or a private CA is unreachable from a session unless these are passed + * explicitly. Registry settings are deliberately absent: they carry credentials and are consumed + * by the service itself (see PythonDebugSession.prepareDependencies). + * + * Lives in its own module because both session kinds build their own environment, and + * dap_debug_service.ts already imports from dap_websocket_server_bun.ts. + */ +export const SESSION_ENV_VARS = [ + 'HTTP_PROXY', + 'HTTPS_PROXY', + 'NO_PROXY', + // The lowercase spellings take precedence in the worker, so forward both. + 'http_proxy', + 'https_proxy', + 'no_proxy', + // Trust roots for a TLS-intercepting proxy. Installing the CA in the container's system + // store is not enough on its own: requests carries its own bundle and Node reads only + // NODE_EXTRA_CA_CERTS, so a debugged script's own HTTPS calls fail without these. + 'SSL_CERT_FILE', + 'SSL_CERT_DIR', + 'REQUESTS_CA_BUNDLE', + 'CURL_CA_BUNDLE', + 'NODE_EXTRA_CA_CERTS' +] + +export function sessionEnv(): Record { + const env: Record = {} + for (const key of SESSION_ENV_VARS) { + const value = process.env[key] + if (value) { + env[key] = value + } + } + // A proxy without a bypass list would send the script's calls to BASE_INTERNAL_URL through it; + // the worker defaults the same way (PROXY_ENVS in windmill-worker). + if (!env.NO_PROXY && !env.no_proxy && (env.HTTP_PROXY || env.http_proxy || env.HTTPS_PROXY || env.https_proxy)) { + env.NO_PROXY = 'localhost,127.0.0.1' + } + return env +} diff --git a/debugger/nsjail.debug.config.proto b/debugger/nsjail.debug.config.proto index 5e538e6002..65ea3456cf 100644 --- a/debugger/nsjail.debug.config.proto +++ b/debugger/nsjail.debug.config.proto @@ -63,6 +63,15 @@ mount { rw: true } +# Private scratch, one instance per jail. `windmill prepare-deps` writes the registry +# credentials here rather than into its install directory under the shared /tmp above, so no +# other session can read them, and they go away with the jail even when it is killed. +mount { + dst: "/var/tmp" + fstype: "tmpfs" + rw: true +} + # Debugger scripts directory (for Python debugger server) mount { src: "/debugger" diff --git a/debugger/registry_config.ts b/debugger/registry_config.ts new file mode 100644 index 0000000000..b7cb0451e5 --- /dev/null +++ b/debugger/registry_config.ts @@ -0,0 +1,84 @@ +/** + * Dependency-registry settings for a debug session's install. + * + * `windmill prepare-deps` installs a session's imports with no database connection, so the + * instance settings that point at a private npm or pip registry cannot be read there. They + * are fetched here instead, from the backend that signed the session's launch token, and + * passed down to the CLI over its stdin request. + * + * They stop at the installer. A registry URL usually embeds credentials and a debugged + * script can read whatever the process running it holds, so none of these values are ever + * put in a session's environment (see README.md, "Registry configuration"). + */ + +export interface RegistryConfig { + npm_config_registry?: string + npmrc?: string + bunfig_install_scopes?: string + pip_index_url?: string + pip_extra_index_url?: string + uv_index_strategy?: string + /** Why the instance's settings are not in this response, for the user to see. */ + message?: string +} + +const WINDMILL_BASE_URL = process.env.WINDMILL_BASE_URL || process.env.BASE_INTERNAL_URL + +/** + * Bounds how long a launch waits on the backend. The session can still start without the + * settings, it just installs from the public registries, so an unreachable backend must + * not hold it up for longer than the install itself would take. + */ +const FETCH_TIMEOUT_MS = Number(process.env.DAP_REGISTRY_CONFIG_TIMEOUT_MS) || 10_000 + +/** + * Fetch the registry settings for a session, authorized by its launch token. + * + * Never throws and never blocks a launch: on any failure it returns a config carrying only + * a `message`, so the session starts against the public registries and the user is told why + * instead of being left with an unexplained "package not found". + */ +export async function fetchRegistryConfig( + token: string | undefined, + logger: { info: (...args: unknown[]) => void; warn: (...args: unknown[]) => void } +): Promise { + if (!token || !WINDMILL_BASE_URL) { + return {} + } + + const url = `${WINDMILL_BASE_URL.replace(/\/$/, '')}/api/debug/registry_config` + try { + const response = await fetch(url, { + headers: { authorization: `Bearer ${token}` }, + signal: AbortSignal.timeout(FETCH_TIMEOUT_MS) + }) + if (response.status === 401 || response.status === 403 || response.status === 404) { + // Expected answers, not something the user can act on: a session that may not read + // the settings (an operator's) is refused, and a backend older than this image has + // no such route at all. Both install from the public registries. + logger.info(`Registry configuration not served for this session (${response.status})`) + return {} + } + if (!response.ok) { + const detail = (await response.text().catch(() => '')).trim() + return { + message: `Could not read the registry configuration (${response.status}): ${detail || response.statusText}` + } + } + + const config: RegistryConfig = await response.json() + // The values carry registry credentials, so only their names are logged. + const configured = Object.entries(config) + .filter(([key, value]) => key !== 'message' && value) + .map(([key]) => key) + logger.info( + configured.length > 0 + ? `Registry configuration from instance settings: ${configured.join(', ')}` + : 'No registry configuration set on the instance' + ) + return config + } catch (error) { + logger.warn(`Failed to fetch registry configuration: ${error}`) + return { message: `Could not read the registry configuration: ${error}` } + } +} diff --git a/debugger/test_dap_server.py b/debugger/test_dap_server.py index 45e138f2a3..17430ea25f 100644 --- a/debugger/test_dap_server.py +++ b/debugger/test_dap_server.py @@ -45,6 +45,10 @@ def main(x: str, count: int = 1): # Breakpoints for the main() test: lines 3 and 4 (inside main function) MAIN_BREAKPOINT_LINES = [3, 4] +# `launch` waits on dependency installation, so the import test below needs far more than +# the default budget on a cold cache. +REQUEST_TIMEOUTS = {"launch": 180.0} + class DAPTestClient: def __init__(self, url: str = "ws://localhost:5679"): @@ -103,7 +107,7 @@ class DAPTestClient: # Wait for response with timeout try: - response = await asyncio.wait_for(future, timeout=10.0) + response = await asyncio.wait_for(future, timeout=REQUEST_TIMEOUTS.get(command, 10.0)) return response except asyncio.TimeoutError: print(f"Timeout waiting for response to {command}") diff --git a/docker-compose.yml b/docker-compose.yml index 93fbf28e8d..a801a0ce7c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -187,6 +187,10 @@ services: # - DEBUG_ALLOWED_ORIGINS=https://your-windmill-host # Optional CSWSH hardening: comma-separated allowlist of browser Origins permitted to open debug WebSockets volumes: - lsp_cache:/pyls/.cache + # Behind a TLS-intercepting proxy, mount its CA here (as .crt) and it is registered in the + # system trust store before any service starts. That alone does not cover dependency + # installation — see debugger/README.md for the variables it also needs + # - ./corp-ca.crt:/usr/local/share/ca-certificates/corp-ca.crt:ro logging: *default-logging caddy: diff --git a/docker/DockerfileExtra b/docker/DockerfileExtra index 808bc6ed2a..9eb2a987f2 100644 --- a/docker/DockerfileExtra +++ b/docker/DockerfileExtra @@ -94,7 +94,9 @@ WORKDIR /debugger # Copy debugger files COPY debugger/dap_debug_service.ts . COPY debugger/dap_websocket_server_bun.ts . +COPY debugger/env_passthrough.ts . COPY debugger/dap_websocket_server.py . +COPY debugger/registry_config.ts . COPY debugger/nsjail.debug.config.proto . # Install Python debugger dependencies using uv diff --git a/docker/DockerfileFull b/docker/DockerfileFull index 6e3f218fd6..b827208c1d 100644 --- a/docker/DockerfileFull +++ b/docker/DockerfileFull @@ -45,6 +45,19 @@ RUN apt-get install -y ruby ruby-bundler RUN apt-get install -y r-base-dev \ && Rscript -e 'install.packages("renv", lib="/usr/lib/R/library", repos="https://cloud.r-project.org")' +# dbt +# NO dbt engine is baked in. Fusion may not be: its license grants only a +# non-transferable, non-sublicensable redistribution right. dbt-core 1.x cannot +# be, because its adapter is a Python package chosen per project. dbt-core 2.x +# could be — one adapter-agnostic binary — but shipping a pre-release nobody is +# defaulted onto costs a layer in every image and a version pinned in two places +# that nothing keeps in step. The worker fetches whichever engine a project asks +# for on first use and caches it (docs/dbt-runtime.md, decision 1). +# +# An operator who wants one pre-staged — an air-gapped instance, or a fleet that +# should not fetch per worker — populates `DBT_BUNDLED_DIR` (default +# /usr/local/dbt) with `core2x-/dbt-sa-cli` in their own image layer. + # Fix UV cache permissions for non-root user support (uid 1000, etc.) # The uv tool install ansible command populates the UV cache with root-owned files RUN chmod -R a+rw /tmp/windmill/cache/uv && \ diff --git a/docker/DockerfileFullEe b/docker/DockerfileFullEe index 14bfceca50..5b5347558a 100644 --- a/docker/DockerfileFullEe +++ b/docker/DockerfileFullEe @@ -78,6 +78,19 @@ RUN apt-get install -y iptables # Kerberos runtime RUN apt-get install -y libsasl2-modules-gssapi-mit krb5-user +# dbt +# NO dbt engine is baked in. Fusion may not be: its license grants only a +# non-transferable, non-sublicensable redistribution right. dbt-core 1.x cannot +# be, because its adapter is a Python package chosen per project. dbt-core 2.x +# could be — one adapter-agnostic binary — but shipping a pre-release nobody is +# defaulted onto costs a layer in every image and a version pinned in two places +# that nothing keeps in step. The worker fetches whichever engine a project asks +# for on first use and caches it (docs/dbt-runtime.md, decision 1). +# +# An operator who wants one pre-staged — an air-gapped instance, or a fleet that +# should not fetch per worker — populates `DBT_BUNDLED_DIR` (default +# /usr/local/dbt) with `core2x-/dbt-sa-cli` in their own image layer. + # Fix UV cache permissions for non-root user support (uid 1000, etc.) # The uv tool install ansible command populates the UV cache with root-owned files RUN chmod -R a+rw /tmp/windmill/cache/uv && \ diff --git a/docker/DockerfileSlim b/docker/DockerfileSlim index 915815f935..64fd1b735e 100644 --- a/docker/DockerfileSlim +++ b/docker/DockerfileSlim @@ -70,7 +70,7 @@ RUN mkdir -p /tmp/windmill/cache && \ rm -rf /tmp/build_cache && \ mkdir -p -m 777 /tmp/windmill/cache/uv -COPY --from=oven/bun:1.3.10 /usr/local/bin/bun /usr/bin/bun +COPY --from=oven/bun:1.4.0 /usr/local/bin/bun /usr/bin/bun # Install windmill CLI (node symlink needed for bun install) RUN ln -s /usr/bin/bun /usr/bin/node \ diff --git a/docker/DockerfileSlimEe b/docker/DockerfileSlimEe index d4ac4bd21c..de0d61a780 100644 --- a/docker/DockerfileSlimEe +++ b/docker/DockerfileSlimEe @@ -70,7 +70,7 @@ RUN mkdir -p /tmp/windmill/cache && \ rm -rf /tmp/build_cache && \ mkdir -p -m 777 /tmp/windmill/cache/uv -COPY --from=oven/bun:1.3.10 /usr/local/bin/bun /usr/bin/bun +COPY --from=oven/bun:1.4.0 /usr/local/bin/bun /usr/bin/bun # Install windmill CLI (node symlink needed for bun install) RUN ln -s /usr/bin/bun /usr/bin/node \ diff --git a/docker/RHEL8/Dockerfile b/docker/RHEL8/Dockerfile index 1af03cc5f4..c17c7685c2 100644 --- a/docker/RHEL8/Dockerfile +++ b/docker/RHEL8/Dockerfile @@ -86,6 +86,7 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \ CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build --release --features "$features" RUN --mount=type=cache,target=/usr/local/cargo/registry \ + --mount=type=cache,target=/usr/local/cargo/git \ cd windmill-duckdb-ffi-internal && \ CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build --release diff --git a/docker/RHEL9/Dockerfile b/docker/RHEL9/Dockerfile index 9ab5c78318..3f24e936ab 100644 --- a/docker/RHEL9/Dockerfile +++ b/docker/RHEL9/Dockerfile @@ -86,6 +86,7 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \ CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build --release --features "$features" RUN --mount=type=cache,target=/usr/local/cargo/registry \ + --mount=type=cache,target=/usr/local/cargo/git \ cd windmill-duckdb-ffi-internal && \ CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build --release diff --git a/docker/entrypoint-extra.sh b/docker/entrypoint-extra.sh index a06901e25c..b32ad2dd94 100644 --- a/docker/entrypoint-extra.sh +++ b/docker/entrypoint-extra.sh @@ -33,6 +33,58 @@ if [ ! -w "$HOME" ]; then fi export HOME +# Register CA certificates mounted into the image before anything opens a TLS connection. +# Best-effort on purpose, unlike INIT_SCRIPT below: a non-root UID cannot write /etc/ssl/certs, and +# a deployment that never needed a custom CA must still boot. Env var names and the default-off +# behavior match the server/worker binary, so one setting covers every container. What the system +# trust store does and does not reach is documented in debugger/README.md. +CA_CERT_DIR=/usr/local/share/ca-certificates + +update_ca_certificates() { + local reason="$1" + local tool="${RUN_UPDATE_CA_CERTIFICATE_PATH:-/usr/sbin/update-ca-certificates}" + local output + if [ ! -x "$tool" ]; then + echo "[entrypoint] $reason but $tool is not executable, skipping CA update" + return + fi + echo "[entrypoint] $reason, running $tool" + if output=$("$tool" 2>&1); then + echo "[entrypoint] CA certificates updated" + else + # Carry the tool's own message: the usual cause is an unwritable /etc/ssl/certs under a + # non-root UID, but guessing that in place of the real error hides everything else. + echo "[entrypoint] WARNING: $tool failed (UID $(id -u)): ${output:-no output}; continuing" >&2 + fi +} + +if [ "$(echo "${RUN_UPDATE_CA_CERTIFICATE_AT_START:-false}" | tr '[:upper:]' '[:lower:]')" = "true" ]; then + update_ca_certificates "RUN_UPDATE_CA_CERTIFICATE_AT_START=true" +elif [ -n "$(find -L "$CA_CERT_DIR" -type f -name '*.crt' -print -quit 2>/dev/null)" ]; then + # Certificates mounted there are unambiguous intent, and they do nothing until registered, so + # take the same action without making the operator also find the env var. + update_ca_certificates "Found certificates in $CA_CERT_DIR" +elif [ -n "$(ls -A "$CA_CERT_DIR" 2>/dev/null)" ]; then + # Reporting success over a mount update-ca-certificates ignores would be worse than saying + # nothing: .pem is the spelling people reach for, and only .crt is read. + echo "[entrypoint] WARNING: $CA_CERT_DIR has files but none named *.crt, the only extension" \ + "update-ca-certificates reads; they will be ignored" >&2 +fi + +# INIT_SCRIPT is the documented hook for preparing the host before anything reaches the network +# (CA certificates, proxies, mounts), matching the worker's INIT_SCRIPT. It must therefore complete +# before any service starts, and a failure has to abort: services that come up with an unprepared +# trust store fail every TLS handshake instead, which is far harder to diagnose. +if [ -n "$INIT_SCRIPT" ]; then + echo "[entrypoint] Running INIT_SCRIPT..." + bash -c "$INIT_SCRIPT" || { + code=$? + echo "[entrypoint] ERROR: INIT_SCRIPT failed with exit code $code, aborting" >&2 + exit "$code" + } + echo "[entrypoint] INIT_SCRIPT completed" +fi + # Setup NETRC if provided (for LSP) if [ -n "$NETRC" ]; then echo "$NETRC" > "$HOME/.netrc" diff --git a/docs/agent-worker-e2e.md b/docs/agent-worker-e2e.md new file mode 100644 index 0000000000..e1fd122b98 --- /dev/null +++ b/docs/agent-worker-e2e.md @@ -0,0 +1,131 @@ +# Running an agent worker locally, for e2e + +An agent worker reaches the database only through the API, so whole code paths +(`Connection::Http`) are never taken by a normal `cargo run`. Exercising them +needs a real one. Every step below has a failure mode that looks like something +else; they are listed with the error each produces. + +## 1. Build with the right features + +Four features, and the agent's own mode gate is the one that is easy to miss: + +```bash +cd backend +cargo build --features quickjs,private,enterprise,license,agent_worker_server +``` + +- `agent_worker_server` mounts `/api/agent_workers/*` on the SERVER. Without it, + `create_agent_token` returns **404** with an empty body. +- `enterprise` + `license` compile the agent MODE into the binary. Without them + the worker exits immediately with `Agent mode is only available in the EE`, + even though the server side works and mints tokens happily. + +Verify before spending time on the handshake — the panic string must be absent: + +```bash +strings target/debug/windmill | grep -c "Agent mode is only available in the EE" # want 0 +``` + +**Pin this feature set for the whole session.** `target/debug/windmill` is one +path shared by every feature combination, and cargo swaps the cached artifact in +and out as the set changes — a `cargo build --features quickjs` (or any build +with a different set) in another pane silently replaces the binary the server and +agent are about to run, and the swap back "completes" in under a second, so it +does not look like a rebuild happened. The symptom is the agent 401ing again +after it had been working, or the EE panic reappearing. Re-run the `strings` +check above whenever anything unexpected regresses, and start the server and the +agent from the SAME build. + +## 2. Run the server without a local worker + +`MODE=server` so nothing else drains the queue and the agent is provably the one +that ran the job: + +```bash +DATABASE_URL= PORT=8420 MODE=server ./target/debug/windmill +``` + +## 3. Mint a token — with an expiry, and unquoted + +```bash +TOK=$(curl -s -X POST localhost:8420/api/auth/login -H 'Content-Type: application/json' \ + -d '{"email":"admin@windmill.dev","password":"changeme"}') + +AT=$(curl -s -X POST localhost:8420/api/agent_workers/create_agent_token \ + -H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \ + -d '{"worker_group":"agentgrp","tags":["dbt"],"exp":1900000000}' | tr -d '"') +``` + +Two traps, both of which surface as a bare `401` on the agent and a decoded +reason only in the SERVER log: + +- **`exp` must be a real timestamp.** `"exp": null` mints a token the validator + rejects with `Missing required claim: exp`. +- **The response is JSON, so it arrives quoted.** Keeping the `"` gives + `Base64 error: Invalid byte 34, offset 0` — hence the `tr -d '"'`. + +Pass the token **exactly as minted**. It looks like `jwt_agent_` and the +client appends its own hostname-derived suffix to form `jwt_agent__`, +which is what the server splits on. Adding a suffix yourself yields +`Base64 error: Encoded text cannot have a 6-bit remainder`. + +## 4. Start the agent + +`WORKER_TAGS` must contain the tag the JOBS carry, not a tag you invent — a job +whose tag nothing serves sits in `v2_job_queue` forever and looks like a hang. +dbt scripts default to the `dbt` tag. + +```bash +AGENT_TOKEN="$AT" BASE_INTERNAL_URL=http://localhost:8420 MODE=agent \ + WINDMILL_DIR=/home/$USER/wmagent \ + WORKER_GROUP=agentgrp WORKER_TAGS=dbt PORT=8499 ./target/debug/windmill +``` + +`WINDMILL_DIR` off `/tmp` matters on a dev box. Jobs fail with `IoErr: Disk quota +exceeded (os error 122)` while writing the project's files, and `df` looks +healthy — free space and free inodes both. `/tmp` is a tmpfs and Linux supports +per-user quotas on it, so the limit is the user's, not the filesystem's; several +agent sessions' caches under `/tmp` are enough to reach it. Point the worker at a +real disk instead of trying to clean up under the quota. + +Confirm it registered rather than trusting a quiet log: + +```sql +SELECT worker FROM worker_ping + WHERE worker_group = 'agentgrp' AND ping_at > now() - interval '2 min'; +-- ag-agentgrp-- +``` + +## Reading failures + +The agent only ever prints `Agent worker cannot connect to server. Please check +AGENT_TOKEN and BASE_INTERNAL_URL`. The actual reason is in the server log, from +`windmill-api-agent-workers/src/ee.rs` — grep it for `JWT_AGENT auth error`. + +## Confirming the agent is what ran the job + +`worker` on the completed job starts with `ag-`: + +```bash +curl -s -H "Authorization: Bearer $TOK" \ + "localhost:8420/api/w//jobs_u/completed/get/" | jq -r .worker +``` + +## What dbt does on an agent worker + +Runs, retries, and publishes its graph — including a per-run snapshot for a +dynamic descriptor, which it POSTs to `/api/agent_workers/dbt_graph/{workspace}` +rather than writing itself. + +What it does not get is LIVE progress: the reporter tails a JSON event log and +needs a SQL connection, so per-model state is settled from `run_results.json` +when the run ends. Retry state lives only in the worker-local generation, since +there is no database row to arbitrate against — which is why `state_dir` is keyed +by principal. + +Confirming a run really exercised that path: + +```sql +SELECT job_id, count(*) FROM dbt_node WHERE script_path = '' GROUP BY job_id; +-- a row keyed to the JOB id (not the zero UUID) means the agent published a snapshot +``` diff --git a/docs/autonomous-mode.md b/docs/autonomous-mode.md deleted file mode 100644 index 95d68136a6..0000000000 --- a/docs/autonomous-mode.md +++ /dev/null @@ -1,83 +0,0 @@ -# Autonomous Mode (Bypass Permissions) - -When running in bypass/auto permission mode, follow these instructions to work end-to-end without human intervention. - -## Available Tools - -The Nix devShell provides these tools for documentation and testing: - -- **`mmdc`** (mermaid-cli): Generate diagrams from Mermaid markup. Uses Nix-provided headless Chrome via `$PUPPETEER_EXECUTABLE_PATH`. -- **`asciinema`**: Record terminal sessions as `.cast` files for demo videos. -- **`playwright`** CLI: Take screenshots of the running frontend. - -### When to Use Them - -- **Designing a feature**: Use `mmdc` to generate Mermaid diagrams (architecture, data flow, sequence diagrams) during the planning phase. Include them in the PR description. -- **Frontend changes**: Take screenshots with the Playwright CLI after manual testing. Attach them to the PR. -- **CLI / terminal changes**: Record a demo with `asciinema` showing the feature in action. Attach to the PR. - -### Quick Reference - -```bash -# Generate a diagram -echo 'graph LR; A-->B; B-->C;' | mmdc -i - -o diagram.png - -# Take a screenshot of a page -playwright screenshot --browser chromium http://localhost:3000 screenshot.png - -# Record a terminal demo -asciinema rec demo.cast -# ... do the demo ... -# ctrl-d to stop -``` - -## Always Plan First - -Even in bypass mode, **enter plan mode before starting non-trivial work**. Ask all important questions upfront: -- Clarify ambiguous requirements before writing code -- Identify which files, crates, and features are affected -- Read `docs/validation.md` to know what checks you'll need to run -- Break large features into stages — commit each stage separately - -## Manual Testing - -After code changes compile and type-check, verify the feature works: - -1. **Check backend logs** (`tmux capture-pane -t .1 -p -S -50`) — confirm no panics or errors -2. **Check frontend logs** (`tmux capture-pane -t .2 -p -S -50`) — confirm no build errors -3. **Use Playwright MCP** to test the UI flow: - - Navigate to `http://localhost:3000/user/login` - - Click "Log in without third-party" - - Login with `admin@windmill.dev` / `changeme` - - Navigate to the page affected by your change - - Verify the feature works as expected -4. **Test edge cases**: empty states, error states, permissions - -### Playwright Gotchas - -- Backend takes ~60s to compile on first change; check logs for `health check completed` -- Frontend rebuilds in ~5s -- `critical_alerts` 404s are expected on CE builds (EE-only endpoint) — ignore them -- VSCode worker 404s are dev-mode artifacts — ignore them -- The `` component hides the checkbox (`sr-only`). Click the `