Files
windmill/frontend/src/lib/components/workspaceSettings/projectBundle.ts
T
GuilhemandClaude Opus 5 582761e37c feat: reuse an existing workspace resource in the project import wizard (#10935)
* feat: let the import wizard reuse an existing workspace resource

The project import wizard always opened the create-resource drawer, so a
workspace that already had, say, an SMTP resource still ended up with a second
one. Step 4 now offers a choice: fill in a new resource as before, or pick an
existing one of the same type.

Picking an existing resource rewrites the deployed items to point at it and
then deletes the imported stub. The rewrite covers scripts, flows, apps, raw
apps and every workspace trigger kind, and holds two rules: it writes nothing
unless every referrer can be rewritten, and it only touches items under the
target folder.

Raw apps re-upload the bundle shipped in the project export instead of
rebuilding it, and the retarget refuses when the deployed sources have moved on
since the import — that bundle was built from the export's sources, so
re-uploading it over edited sources would revert them.

Adds `update` to the trigger-kind table for the eleven kinds whose service
takes a plain config body; schedule keeps its own branch because
updateSchedule takes a different shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* feat: only ask about resources the project actually points at

A project declares one resource per `resource-<type>` input schema as well as one
per `$res:` reference, so an app that pins `f/calendly/google_calendar` for a
script whose schema says `resource-gcal` ships an unreferenced `f/calendly/gcal`
alongside it. Step 4 listed both and asked you to fill in each.

Only the referenced ones have to hold a credential for the project to work. The
rest are still created — a standalone run picks from them in the argument picker
— but they no longer reach the checklist, and `resourceCount` counts the same
set so the wizard does not offer a fourth step that has nothing on it. Across
the twelve published hub projects this drops 9 of 19 rows, including three
non-credential input shapes in `typeform`.

Also fixes a miss in the retarget: a trigger holds its resource as a bare path in
its own `*_resource_path` field rather than as a `$res:` token, so a token-only
scan left it pointing at a stub that was then deleted. Detection now mirrors
`rewriteTriggerConfig` through a shared `referencesResourcePath`, which matches
the parsed structure rather than its serialization — keeping `f/proj/db` out of
`$res:f/proj/db_prod` as well.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: refuse a resource retarget the scan or the rewriters cannot cover

Uncompiled trigger features 404 on their list route; that is the instance not
having the kind, not a listing that failed, so it no longer blocks every
retarget on a stock build. The `listSearch*` endpoints cap server-side with no
ordering and no pagination, so a full page is refused rather than read as the
whole workspace. An item that names the resource path outside a `$res:` token is
refused at plan time — no rewriter relocates it — and the trigger row keeps its
own `script_path` so a runnable sharing the path is not repointed. A raw app
whose sources the export cannot yield carries no entry at all, so the refusal
its comment promises actually fires. The reused row offers text instead of a
button that leads to a deleted resource.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* refactor: let an incomplete scan keep the stub instead of refusing the retarget

The scan behind "nothing is written unless every referrer can be rewritten"
cannot be proven complete: the listings come back capped, a trigger kind can
fail to list, and a reference can sit where no rewriter reaches. Gating the
whole run on that claim made every such case a refusal.

Rewriting an item onto the chosen resource is safe on its own — the item
resolves whether or not the stub survives — so only the delete needs the claim.
`planRetarget` now answers with the referrers it can move plus the gaps it
cannot account for, `applyRetarget` always moves the first set, and a gap keeps
the stub rather than stopping the run. A referrer outside the project's folder
is one of those gaps: the listings are workspace-wide, so it is seen for free,
it stays the user's own, and its existence is why the stub stays.

The outcome carries what moved and why the stub was kept, so the row settles to
the chosen resource either way and says when the placeholder is still there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: preserve a retargeted item's deployed identity, and send back its own bundle

Every write here edits a deployed item in place, but none of them said so.
Without `preserve_on_behalf_of` the backend replaces the item's stored run
identity with whoever opened the wizard, and `updatePolicy(next, undefined)`
rebuilt an app's policy from nothing — dropping its sandbox rules and forcing
`execution_mode: publisher`, which puts a viewer app on the publisher's
identity even though the backend would otherwise have kept the deployed mode.
The policy is now recomputed from the deployed one, which is what the
triggerables rekeying actually needs.

The raw-app bundle no longer comes from the project export. The browser can
read a deployed bundle back — mint the app's public secret and fetch
`/apps/get_data/v/{secret}.{ext}`, the same route the Hub publish reads — so
the bundle sent back is the deployed one whoever last edited it. That removes
`ExportedAppFiles`, its plumbing through the setup step, `rawSourcesDiverged`,
and the two raw-app gaps: an app "edited since the import" is no longer a case
that exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* perf: carry the trigger row from the scan into its write

`rewriteTrigger` listed the whole kind again to find the row it had just read,
once per trigger — and for schedules a listing is itself a listing plus a
detail fetch per row. The scan already holds the row, so the referrer carries
it.

Pins two properties that nothing covered: the trigger update body leaves
`enabled` out, so pointing a trigger at a credential cannot also start it; and
a write that fails partway keeps the stub while reporting what had already
moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: keep unfilled resources out of the reuse chooser

The chooser offered every resource of the row's type except the ones this import
created, so a stub left behind by an earlier import of the same project showed up
as a credential to reuse. Pointing a project at another project's empty
placeholder is never the answer, and nothing downstream would have complained.

Candidates are now read back and the unfilled ones dropped, using the same test
the checklist uses to call one of the project's own resources blank. Past a cap
they are all offered rather than costing a request each: a workspace with that
many resources of the outstanding types is not the case this filters for.

Also drops the chooser's promise that the imported placeholder is removed. That
was true when the delete was unconditional; the stub is now kept whenever the
scan cannot account for everything, and the row says which happened once it has.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: move a retargeted item's bundle and identity, and see the paths it spells out

Four gaps between what the retarget claimed and what it did.

A trigger states its run identity as `permissioned_as`, not the `on_behalf_of`
the other kinds use, and the backend keeps the row's value only when
`preserve_permissioned_as` says so. Without the pair, a trigger created under a
folder's `default_permissioned_as` started running as whoever picked the
credential.

A raw app's bundle is compiled from its sources, so a `$res:` a source spells out
is baked into it. The import rewrites that copy — `retargetProjectExport` runs
while `/bundle.js` is still one of `files` — but the retarget fetched the
deployed bundle after that split and sent it back untouched, then deleted the
stub the app still read. The fetched bundle is now rewritten too, and a path it
names any other way keeps the stub instead.

A script's content is one string, so the whole-string match that finds a bare
path in a flow or an app could not see one written inside it. `getResource("f/…")`
was invisible to both the scan, which then deleted the stub under it, and the
step-4 filter, which dropped the row so nobody was asked to fill it.

Trigger listings cap at the server's DEFAULT_PER_PAGE, which this table does not
page past. A full page is now read the way a full `listSearch*` page is: as a
listing that cannot account for the rest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: see a path a flow or app spells out, and name why an item did not move

The script scan was taught to see a resource path written inside code; flows and
apps were left on the whole-string test, which cannot. A flow whose inline module
runs `getResource("f/proj/db")`, or a raw app whose source does, was neither
rewritten nor recorded as a gap, so the stub was deleted while the deployed item
still read it. Reachable from the wizard, because the step-4 filter does see such
a reference and offers the row.

Both branches now use the same test as the script branch, and gap rather than
rewrite: the stub survives either way, so a `$res:` token in the same item still
resolves, and rewriting half an item would only make the plan and the write
disagree about what moved.

Each rewriter now says why it left an item alone instead of answering yes or no,
so a raw-app bundle that spells the path out is reported as a reference nothing
could move rather than as a concurrent edit.

Also corrects the resource-listing comment — `perPage` bounds the answer, the
route does not default to 30 — and asks the askable-resource question against the
export as published rather than the retargeted copy, so the step and the stepper
that decides whether to offer it give one answer. A path spelled out in code is
not retargeted, so only the raw export has its references and its resource paths
agreeing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: a kept placeholder is still something to fill in

Reuse marked the row done and replaced its action with static text even when the
stub survived. A kept stub is empty and is still what every item the scan could
not move reads, so the step reported "You're all set" over a project running on a
placeholder, with no way back to filling it. Reachable from one hub project: a
raw app whose source spells the resource path out gaps everything, nothing is
rewritten, and the row went green anyway.

Such a row now stays outstanding, keeps its button, says which path items still
read, and re-checks on refresh so filling that placeholder in closes it.

Flows and apps also went back to being rewritten as well as gapped, matching what
the script branch already did — the reason given for skipping them was
contradicted by that branch, and a comment merely naming the path was enough to
strand an item's real `$res:` token on the stub.

Two things had to become precise for that to hold. What counts as rewritable is
now the presence of a `$res:` token rather than any reference, since a whole
string equal to the path is the unreachable case, not a movable one. And the
post-rewrite check reads tokens only: a path the item also spells out is the
plan's gap to record, and re-reading it at write time reported one item twice,
as both unmovable and changed underfoot. Writers now skip a write that would
change nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: rewrite only the tokens, and let a filled placeholder close its row

The import's flow and app rewriters also remap a runnable's own path on an exact
match. That is right for the folder-wide map the import hands them, where every
path is moving. Here the map holds one entry, a resource path — and scripts,
flows and resources share a namespace, so a project shipping both a script and a
resource named `smtp` had the step calling it repointed at the credential.
Triggers were already guarded against exactly this; flows, apps and raw apps were
not. All three now rewrite the serialized value, which moves the tokens and
leaves every path alone.

A kept placeholder that the user then fills in now closes its row: `stubKept` is
cleared by the read that finds it filled, so the row stops saying items still
need it while showing a green check beside "You're all set".

A kept-stub row's button also goes straight to filling that placeholder rather
than reopening the chooser. A second retarget from there can only be a no-op —
every rewritable referrer is already off the stub — and it would have relabelled
the row after moving nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: check staleness where it can be seen, and stop trusting a client-side licence

The post-rewrite check could no longer fail: since the rewrite became token-only
it ran over exactly what the check looked for, so it read as a guard while
guarding nothing. The staleness it named is real — the plan classifies items from
the search listings and each write re-reads its item by path — so the check now
happens on that fresh read, and looks for the spelling no rewrite reaches. A
referrer the plan already recorded as unreachable skips it: the stub survives
either way, and re-reporting the same item would say it was both unmovable and
changed underfoot.

Trigger kinds are no longer skipped by the client-side licence store. That store
is empty on an EE instance whose licence is unset or whose fetch failed, while
the rows are still in the database and the routes still answer — and a kind
skipped that way left no gap, so the stub went while an EE trigger still pointed
at it. On CE those routes are not registered and the 404 branch already says so,
from the server rather than from a store.

`askableResources` now pairs the export's resources with the retargeted ones by
position, the way `retargetProjectExport` maps them, instead of rebuilding the
path by slicing a prefix. An external path the bundle pulled in lands at
`f/<folder>/<name>` with a `_2` suffix on collision, which no slicing recovers —
and the row would have gone missing from a checklist the stepper still counted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: a scan the caller is not shown all of cannot clear the stub for deletion

The listings the scan reads run as the caller, and row-level security filters
them inside the query. For anyone but a workspace admin that means an item they
cannot read is not absent from the answer so much as invisible in it: it does not
appear, and it does not count towards the full-page test that catches a truncated
listing either. A colleague's private script referencing the stub is exactly that
shape, so the scan reported a clean sweep and the stub was deleted out from under
it, with nothing said.

That is the one input to the completeness proof the destructive step rests on
that was never checked. A caller who is not shown the whole workspace now records
a gap like any other, so the rewrite still happens in full and the placeholder
stays.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: ask whether this workspace's listings are complete, not a stale record's

`UserExt` is per-workspace and outlives a workspace change, which is why it
carries `workspace_id`. Reading `is_admin` off it without checking which
workspace it describes answers for the wrong one. Step 4 is reachable by reload —
it is built to be — and nothing on that path re-fetches the record, so it still
describes the workspace the user came from. An admin of their own workspace
importing into a shared one they are a plain member of got a clean scan over
row-level-security-filtered listings, and the stub was deleted under a referrer
they were never shown.

The question is now asked of the target workspace, through a predicate that can
be tested. An instance superadmin bypasses the policies everywhere, so that is
asked separately rather than read off the same stale record.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* style: format the wizard retarget files

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: leave a trigger's runnable references alone, and read the app kind rather than guess it

A trigger's `on_failure`, `on_recovery`, `on_success` and `url` name a runnable,
and `rewriteTriggerConfig` remaps one on an exact match — right for the
folder-wide map the import hands it, wrong for a map holding a single resource
path. A schedule whose error handler ran a script sharing that path had the
handler pointed at the credential instead. The same reason `path` and
`script_path` were already restored; only the two prefixed shapes it remaps are,
so a field holding a `$res:` token still moves.

The scan guessed raw from low-code by looking for `files` and `runnables`,
because `list_search_apps` returns only the path and the value. Both writers
re-read the app anyway, and that record carries `raw_app`, so the write now
dispatches on it. A guess wrong in either direction was a deploy the backend
refuses for changing an app's kind, which aborted the run at that referrer.

Also drops the past-tense clauses from four test comments. Each already states
the invariant it guards; the rest described iterations of this branch that no
reader will have seen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

* fix: restore a trigger's bare runnable references too

The prefixed spellings were put back after the rewrite; the bare ones were not.
`dynamic_skip`, `error_handler_path` and a websocket initial message's
`runnable_result.path` each hold a plain script path, which `rewriteTriggerConfig`
remaps on a whole-string match — so a trigger whose error handler ran a script
sharing the stub's path had that handler pointed at the credential.

All of them now come back from the row, taken from what `triggerHandlerRefs`
reads rather than enumerated by hand. A prefixed field is still restored only
when it holds the runnable spelling, so a `$res:` token in one still moves; a
bare field is a path and nothing else, so it is always restored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fuzkt6NqsqzvSYSVKpR3pj

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-03 11:28:54 +02:00

736 lines
27 KiB
TypeScript

// Pure logic for the "project = folder" Hub bundle. A project is one folder
// `f/<slug>/...`. Bundling: collect the transitive closure, relocate external
// refs (`u/<user>/<name>`, `f/<other>/<name>` -> `f/<slug>/<name>`, `_2`/`_3`…
// on collision) and rewrite them. Hub refs stay external; runtime string-concat
// paths are out of scope. No API/Svelte deps so it's unit-testable.
import { getAllModules } from '$lib/components/flows/flowExplorer'
import { isRunnableByPath } from '$lib/components/apps/inputType'
export type RefKind = 'resource' | 'script' | 'flow'
export interface Ref {
kind: RefKind
/** Bare path, without the `$res:` / `res://` prefix for resources. */
path: string
}
export type PathClass = 'internal' | 'hub' | 'external'
/** A single `$res:PATH` / `res://PATH` token (path captured in group 1). */
const RES_TOKEN_RE = /(?:\$res:|res:\/\/)([\w\-./]+)/g
// A whole-string `$var:PATH` / `$jsonvar:PATH` value. The worker substitutes these
// only when an argument value *is* the reference (walking nested JSON), never a
// token embedded in inline code, so the whole value must match. `_KIND` captures
// the prefix (group 1) and path (group 2) so a rewrite can preserve `var`/`jsonvar`.
const VAR_VALUE_RE = /^\$(?:json)?var:([\w\-./]+)$/
const VAR_VALUE_RE_KIND = /^\$(var|jsonvar):([\w\-./]+)$/
// Variable paths a value will resolve at runtime (flow static inputs, flow_env,
// app runnable inputs, trigger config fields). Walk the parsed structure and match
// whole string values so inline code carrying a literal `$var:` string is ignored.
export function extractVarRefsFromValue(value: any): string[] {
const out = new Set<string>()
const walk = (v: any) => {
if (typeof v === 'string') {
const m = VAR_VALUE_RE.exec(v)
if (m) out.add(m[1])
} else if (Array.isArray(v)) {
for (const x of v) walk(x)
} else if (v && typeof v === 'object') {
for (const k of Object.keys(v)) walk(v[k])
}
}
walk(value)
return [...out]
}
export function classifyPath(path: string, slug: string): PathClass {
if (path.startsWith(`f/${slug}/`) || path === `f/${slug}`) return 'internal'
if (path.startsWith('hub/')) return 'hub'
return 'external'
}
export function extractScriptRefs(content: string): Ref[] {
const out: Ref[] = []
const seen = new Set<string>()
let m: RegExpExecArray | null
RES_TOKEN_RE.lastIndex = 0
while ((m = RES_TOKEN_RE.exec(content)) !== null) {
if (!seen.has(m[1])) {
seen.add(m[1])
out.push({ kind: 'resource', path: m[1] })
}
}
return out
}
/**
* References inside a flow value:
* - inline rawscript code with `$res:` (resource)
* - static step inputs whose value is a `$res:` literal (resource)
* - `type: script` steps that reference a script by path (script)
* - `type: flow` steps that reference a sub-flow by path (flow)
*/
export function extractFlowRefs(value: any): Ref[] {
const out: Ref[] = []
const seen = new Set<string>()
const add = (kind: RefKind, path: string) => {
const key = `${kind}:${path}`
if (!seen.has(key)) {
seen.add(key)
out.push({ kind, path })
}
}
// getAllModules flattens the whole tree (loops, branches, aiagent tools,
// failure module) so each module only needs local inspection; the
// preprocessor module sits outside `modules` and is walked the same way.
for (const mod of allFlowModules(value)) {
const v: any = (mod as any)?.value
if (!v || typeof v !== 'object') continue
if (v.type === 'script' && typeof v.path === 'string') add('script', v.path)
if (v.type === 'flow' && typeof v.path === 'string') add('flow', v.path)
if (typeof v.content === 'string') {
for (const r of extractScriptRefs(v.content)) add('resource', r.path)
}
const it = v.input_transforms
if (it && typeof it === 'object') {
for (const key of Object.keys(it)) {
const t = it[key]
// Static values can be a bare `$res:` string or arbitrary JSON with
// refs nested anywhere — the worker resolves both, so scan the full
// serialization.
if (t?.type === 'static' && t.value !== undefined) {
const text = typeof t.value === 'string' ? t.value : JSON.stringify(t.value)
for (const r of extractScriptRefs(text)) add('resource', r.path)
}
}
}
}
// flow_env values support `$res:path` references — as whole string values or
// nested inside JSON values (the worker resolves both), so scan the full
// serialization.
if (value?.flow_env && typeof value.flow_env === 'object') {
for (const r of extractScriptRefs(JSON.stringify(value.flow_env))) add('resource', r.path)
}
return out
}
// Every module of a flow value: the tree under `modules`, the failure module,
// and the preprocessor module (which lives outside `modules`). Any walk over a
// flow's modules must go through this — a walk that misses a module class
// silently drops its dependencies from bundles or migrations. All three go in
// the root list (not getAllModules' failure_module parameter, which appends
// the module without expanding its descendants) so nested children of a
// failure or preprocessor module are walked too.
export function allFlowModules(value: any) {
return getAllModules([
...(value?.modules ?? []),
...(value?.preprocessor_module ? [value.preprocessor_module] : []),
...(value?.failure_module ? [value.failure_module] : [])
])
}
// Visit every object node in an app value tree (JSON-safe, no cycles).
function walkAppNodes(value: any, visit: (node: Record<string, any>) => void): void {
if (value == null || typeof value !== 'object') return
if (Array.isArray(value)) {
for (const v of value) walkAppNodes(v, visit)
return
}
visit(value)
for (const k of Object.keys(value)) walkAppNodes(value[k], visit)
}
// `runnableByPath`/`path` nodes reference a workspace runnable by path.
function runnableRef(node: Record<string, any>): Ref | undefined {
if (!isRunnableByPath(node as any) || typeof node.path !== 'string') return undefined
if (node.runType === 'flow') return { kind: 'flow', path: node.path }
if (node.runType === 'script') return { kind: 'script', path: node.path }
return undefined // hubscript -> external hub, ignored
}
// App refs: `$res:` resources anywhere in the value, plus script/flow runnables
// referenced by path in components.
export function extractAppRefs(value: any): Ref[] {
const out: Ref[] = []
const seen = new Set<string>()
const add = (kind: RefKind, path: string) => {
const key = `${kind}:${path}`
if (!seen.has(key)) {
seen.add(key)
out.push({ kind, path })
}
}
walkAppNodes(value, (node) => {
const r = runnableRef(node)
if (r) add(r.kind, r.path)
})
for (const r of extractScriptRefs(JSON.stringify(value ?? {}))) add('resource', r.path)
return out
}
/**
* Build the relocation map. Internal paths (`f/<slug>/...`) map to themselves
* and are reserved first; external paths relocate to `f/<slug>/<name>` (`_2`/`_3`…
* on collision). Input is sorted so suffix assignment is deterministic.
*/
export function buildPathMap(paths: Iterable<string>, slug: string): Map<string, string> {
const map = new Map<string, string>()
const used = new Set<string>()
const sorted = [...new Set(paths)].sort()
for (const p of sorted) {
if (classifyPath(p, slug) === 'internal') {
map.set(p, p)
used.add(p)
}
}
for (const old of sorted) {
if (map.has(old)) continue
const name = old.split('/').filter(Boolean).pop() ?? old
let candidate = `f/${slug}/${name}`
let n = 2
while (used.has(candidate)) candidate = `f/${slug}/${name}_${n++}`
used.add(candidate)
map.set(old, candidate)
}
return map
}
// Both ref forms normalize to `$res:` on rewrite.
export function rewriteContent(content: string, map: Map<string, string>): string {
return content.replace(RES_TOKEN_RE, (whole, path) => {
const next = map.get(path)
return next ? `$res:${next}` : whole
})
}
// Structurally relocate whole-string `$var:`/`$jsonvar:` values — the only form the
// worker resolves. Walks the parsed value so an inert token embedded in inline code
// or arbitrary text is left untouched, unlike token replacement over serialized
// strings. Only paths present in the map move (the retarget map carries variables).
export function rewriteVarRefsInValue(value: any, map: Map<string, string>): any {
if (typeof value === 'string') {
const m = VAR_VALUE_RE_KIND.exec(value)
if (m) {
const next = map.get(m[2])
if (next) return `$${m[1]}:${next}`
}
return value
}
if (Array.isArray(value)) return value.map((v) => rewriteVarRefsInValue(v, map))
if (value && typeof value === 'object') {
const out: Record<string, any> = {}
for (const k of Object.keys(value)) out[k] = rewriteVarRefsInValue(value[k], map)
return out
}
return value
}
/**
* `$res:`/`res://` tokens anywhere in a trigger config — schedule args,
* on_*_extra_args, error_handler_args, … (e.g. the built-in Slack handler
* stores its channel resource this way). These must enter the bundle path map
* so `rewriteTriggerConfig` relocates them and a stub is exported.
*/
export function extractTriggerConfigResourceRefs(config: any): string[] {
return extractScriptRefs(JSON.stringify(config ?? {})).map((r) => r.path)
}
/**
* Whether a value reaches a resource, in either spelling a rewrite has to handle: a
* `$res:`/`res://` token embedded in content, or the bare path standing alone as a string
* the way trigger configs hold it (`kafka_resource_path: "f/slug/db"`).
*
* Matching the parsed structure rather than its serialization is what keeps
* `f/slug/db` out of `$res:f/slug/db_prod`.
*/
export function referencesResourcePath(value: unknown, path: string): boolean {
const walk = (v: any): boolean => {
if (typeof v === 'string') {
if (v === path) return true
RES_TOKEN_RE.lastIndex = 0
let m: RegExpExecArray | null
while ((m = RES_TOKEN_RE.exec(v)) !== null) if (m[1] === path) return true
return false
}
if (Array.isArray(v)) return v.some(walk)
if (v && typeof v === 'object') return Object.values(v).some(walk)
return false
}
return walk(value)
}
/**
* Whether a `$res:`/`res://` token for this path survives anywhere in the value — the one
* spelling the rewriters relocate, and so the only one whose survival means a rewrite did
* not take. A bare path is deliberately not matched: nothing here moves one, so its presence
* says nothing about whether the rewrite worked.
*/
export function holdsResourceToken(value: unknown, path: string): boolean {
const walk = (v: any): boolean => {
if (typeof v === 'string') {
RES_TOKEN_RE.lastIndex = 0
let m: RegExpExecArray | null
while ((m = RES_TOKEN_RE.exec(v)) !== null) if (m[1] === path) return true
return false
}
if (Array.isArray(v)) return v.some(walk)
if (v && typeof v === 'object') return Object.values(v).some(walk)
return false
}
return walk(value)
}
/**
* Whether the text names the resource somewhere no rewriter reaches — a path written on its
* own rather than inside a `$res:` token, the way `getResource("f/proj/db")` does. The
* tokens are stripped first so the ones a rewrite would move do not count, and the match is
* bounded so `f/proj/db` is not found inside `f/proj/db_prod`.
*/
export function textHoldsBarePath(text: string, path: string): boolean {
const withoutTokens = text.replace(RES_TOKEN_RE, '')
const boundary = /[\w\-./]/
for (let i = withoutTokens.indexOf(path); i !== -1; i = withoutTokens.indexOf(path, i + 1)) {
const before = withoutTokens[i - 1] ?? ''
const after = withoutTokens[i + path.length] ?? ''
if (!boundary.test(before) && !boundary.test(after)) return true
}
return false
}
/**
* Whether anything the project ships points at one of its own resources.
*
* A project declares two kinds of resource. One is referenced — an app pins `$res:` for a
* script argument, a trigger names it — and the project does not work until it holds a
* credential. The other is minted from a `resource-<type>` input schema: it names a type
* a script accepts, nothing points at it, and a standalone run picks a resource in the
* argument picker instead. Only the first kind is worth asking anyone to fill in.
*
* The `resources` list is excluded from the walk because a stub's own declaration carries
* its path, which would make every stub look referenced.
*/
export function projectReferencesResource(bundle: ProjectExport, path: string): boolean {
const { resources: _resources, ...rest } = bundle as any
if (referencesResourcePath(rest, path)) return true
// A script that reads the resource by name rather than through a `$res:` token still needs
// it filled in. Asked about is the safe side of this answer: the cost of a wrong yes is a
// row nobody had to act on, and of a wrong no a credential nobody was told to set up.
return textHoldsBarePath(JSON.stringify(rest ?? {}), path)
}
/**
* Trigger configs reference resources as plain path strings (e.g.
* `kafka_resource_path: "f/slug/db"`), not `$res:` tokens, so token rewriting
* misses them. Deep-walk the config and remap any string that exact-matches a
* map key (map keys are full bundle paths, so an exact match is a reference),
* or a `script/<path>`/`flow/<path>` handler reference (schedules' on_failure
* et al.), falling back to `$res:` token rewriting for embedded refs.
*/
// Top-level config fields whose string values are prefixed runnable refs.
// Prefixed forms are remapped ONLY in these known positions: deciding meaning
// from string shape alone rewrote literal payloads that merely looked like
// refs. Bare-path exact matches and $res: tokens stay position-independent.
const HANDLER_REF_FIELDS = new Set(['on_failure', 'on_recovery', 'on_success'])
export function rewriteTriggerConfig(config: any, map: Map<string, string>, depth = 0): any {
if (typeof config === 'string') {
const direct = map.get(config)
if (direct) return direct
return rewriteContent(config, map)
}
if (Array.isArray(config)) return config.map((v) => rewriteTriggerConfig(v, map, depth + 1))
if (config && typeof config === 'object') {
return Object.fromEntries(
Object.entries(config).map(([k, v]) => {
if (depth === 0 && typeof v === 'string') {
// Websocket url: $script:<path> / $flow:<path>.
if (k === 'url') {
const m = /^\$(script|flow):(.+)$/.exec(v)
if (m && map.has(m[2])) return [k, `$${m[1]}:${map.get(m[2])}`]
}
// Schedule handlers: script/<path> / flow/<path>.
if (HANDLER_REF_FIELDS.has(k)) {
const m = /^(script|flow)\/(.+)$/.exec(v)
if (m && map.has(m[2])) return [k, `${m[1]}/${map.get(m[2])}`]
}
}
return [k, rewriteTriggerConfig(v, map, depth + 1)]
})
)
}
return config
}
export function rewriteFlowValue(value: any, map: Map<string, string>): any {
const cloned = JSON.parse(JSON.stringify(value ?? {}))
for (const mod of allFlowModules(cloned)) {
const v: any = (mod as any)?.value
if (!v || typeof v !== 'object') continue
if (
(v.type === 'script' || v.type === 'flow') &&
typeof v.path === 'string' &&
map.has(v.path)
) {
v.path = map.get(v.path)
}
if (typeof v.content === 'string') v.content = rewriteContent(v.content, map)
const it = v.input_transforms
if (it && typeof it === 'object') {
for (const key of Object.keys(it)) {
const t = it[key]
// Mirror extraction: rewrite refs wherever they sit, preserving the
// value's type (a string stays a string, JSON round-trips).
if (t?.type === 'static' && t.value !== undefined) {
if (typeof t.value === 'string') {
t.value = rewriteContent(t.value, map)
} else {
t.value = JSON.parse(rewriteContent(JSON.stringify(t.value), map))
}
}
}
}
}
if (cloned?.flow_env && typeof cloned.flow_env === 'object') {
// Tokens can sit inside nested JSON values, not just string values; the
// serialize→rewrite→parse round-trip reaches all of them (paths contain
// no characters that would break JSON string literals).
cloned.flow_env = JSON.parse(rewriteContent(JSON.stringify(cloned.flow_env), map))
}
return cloned
}
// Relocate `$res:` tokens (one round-trip, also produces a fresh clone) then
// runnable-by-path refs structurally. Incidental `f/<slug>/` strings stay intact.
export function rewriteAppValue(value: any, map: Map<string, string>): any {
if (value == null) return value
const cloned = JSON.parse(rewriteContent(JSON.stringify(value), map))
walkAppNodes(cloned, (node) => {
if (runnableRef(node) && map.has(node.path)) node.path = map.get(node.path)
})
return cloned
}
// Raw/compiled apps store their structure as a JSON string (`{ runnables, files }`).
// Parse it so runnable-by-path refs in the runnables map are seen, reusing the
// same walk; fall back to plain `$res:` scanning if it isn't valid JSON.
export function extractRawAppRefs(content: string): Ref[] {
let parsed: any
try {
parsed = JSON.parse(content)
} catch {
return extractScriptRefs(content)
}
return extractAppRefs(parsed)
}
export function rewriteRawAppContent(content: string, map: Map<string, string>): string {
let parsed: any
try {
parsed = JSON.parse(content)
} catch {
return rewriteContent(content, map)
}
return JSON.stringify(rewriteAppValue(parsed, map))
}
// ---------------------------------------------------------------------------
// Hub project export format (what /projects/{slug}/export returns) and its
// retargeting into a destination folder. Kept here, next to the rewriters,
// so the bundle format is defined in one module for both publish and install.
// ---------------------------------------------------------------------------
export type ExportItem = Record<string, any>
export interface ProjectMigration {
datatable_name: string
sql: string
sql_down?: string
enabled: boolean
}
export interface ProjectExport {
project: { slug: string; name: string; summary: string; readme: string | null }
scripts: ExportItem[]
flows: ExportItem[]
apps: ExportItem[]
resources: ExportItem[]
triggers: ExportItem[]
migrations?: ProjectMigration[]
}
// Map bundled paths `f/<fromSlug>/...` -> `f/<folder>/...`. Only enumerated
// paths go in, so rewriters touch real refs, never incidental text.
export function buildRetargetMap(
bundle: ProjectExport,
fromSlug: string,
folder: string
): Map<string, string> {
const map = new Map<string, string>()
const prefix = `f/${fromSlug}/`
const add = (p: unknown) => {
if (typeof p === 'string' && p.startsWith(prefix)) {
map.set(p, `f/${folder}/${p.slice(prefix.length)}`)
}
}
for (const s of bundle.scripts) add(s.path)
for (const f of bundle.flows) add(f.path)
for (const a of bundle.apps) add(a.path)
for (const r of bundle.resources) add(r.path)
for (const t of bundle.triggers) {
add(t.path)
add(t.runnable_path)
}
// Variables aren't enumerated in the export; their `$var:`/`$jsonvar:` refs live
// inside item values. Relocate the internal ones so a renamed-folder import
// rewrites them into the target folder instead of retaining the old prefix.
for (const p of collectExportVarPaths(bundle)) add(p)
return map
}
// Internal-or-external variable paths referenced by the export's flows, apps and
// triggers. Scripts carry no variable args. Raw apps hold their structure in the
// `value.raw` JSON string.
export function collectExportVarPaths(bundle: ProjectExport): string[] {
const out = new Set<string>()
const collect = (value: any) => {
for (const p of extractVarRefsFromValue(value)) out.add(p)
}
for (const f of bundle.flows) collect(f.value)
for (const a of bundle.apps) collect(a.app_type === 'raw' ? safeParseRaw(a.value?.raw) : a.value)
for (const t of bundle.triggers) collect(t.config)
return [...out]
}
function safeParseRaw(raw: unknown): any {
if (typeof raw !== 'string') return undefined
try {
return JSON.parse(raw)
} catch {
return undefined
}
}
// Structural retarget: rewrite each item's path and its internal refs,
// leaving Hub refs and arbitrary content untouched.
export function retargetProjectExport(
bundle: ProjectExport,
fromSlug: string,
folder: string
): ProjectExport {
if (folder === fromSlug) return bundle
const map = buildRetargetMap(bundle, fromSlug, folder)
const remap = (p: unknown) => (typeof p === 'string' ? (map.get(p) ?? p) : p)
return {
...bundle,
scripts: bundle.scripts.map((s) => ({
...s,
path: remap(s.path),
content: rewriteContent(s.content ?? '', map)
})),
flows: bundle.flows.map((f) => ({
...f,
path: remap(f.path),
value: rewriteVarRefsInValue(rewriteFlowValue(f.value, map), map)
})),
apps: bundle.apps.map((a) => ({
...a,
path: remap(a.path),
// Raw apps keep their structure in the `value.raw` JSON string.
value:
a.app_type === 'raw'
? {
...a.value,
raw: rewriteRawVarRefs(rewriteRawAppContent(a.value?.raw ?? '', map), map)
}
: rewriteVarRefsInValue(rewriteAppValue(a.value, map), map)
})),
resources: bundle.resources.map((r) => ({ ...r, path: remap(r.path) })),
triggers: bundle.triggers.map((t) => ({
...t,
path: remap(t.path),
runnable_path: remap(t.runnable_path),
// Configs hold `$res:` tokens, plain resource paths (kafka_resource_path
// etc.) and whole-string `$var:` values — rewrite all three.
config: t.config ? rewriteVarRefsInValue(rewriteTriggerConfig(t.config, map), map) : t.config
}))
}
}
// Var relocation for a raw app's `value.raw` JSON string: parse, structurally
// rewrite whole-string var values, re-serialize; leave invalid JSON untouched.
function rewriteRawVarRefs(raw: string, map: Map<string, string>): string {
const parsed = safeParseRaw(raw)
if (parsed === undefined) return raw
return JSON.stringify(rewriteVarRefsInValue(parsed, map))
}
export type ItemKind = 'script' | 'flow' | 'app' | 'raw_app'
export interface ItemRef {
kind: ItemKind
path: string
}
export interface FetchedItem {
kind: ItemKind
path: string
summary?: string
description?: string
/** scripts + raw_apps */
content?: string
/** flows + apps */
value?: any
/** scripts */
language?: string
schema?: any
lock?: string
scriptKind?: string
}
export interface BundleDeps {
/** Fetch a workspace item by ref, or undefined if it doesn't exist. */
fetchItem: (ref: ItemRef) => Promise<FetchedItem | undefined>
/** Resolve a resource path to its type, or undefined if missing. */
resolveResourceType: (path: string) => Promise<string | undefined>
}
export interface BundledItem extends FetchedItem {
/** Path the item takes inside the project folder. */
newPath: string
}
export interface ResourceStub {
originalPath: string
newPath: string
resource_type: string
}
export interface ProjectBundle {
items: BundledItem[]
resourceStubs: ResourceStub[]
/** Original -> relocated path for every item and resource (incl. unresolved). */
pathMap: Map<string, string>
/** External paths we couldn't fetch/resolve (missing items or untyped resources). */
unresolved: string[]
}
function refsForFetched(item: FetchedItem): Ref[] {
if (item.kind === 'script') return extractScriptRefs(item.content ?? '')
if (item.kind === 'flow') return extractFlowRefs(item.value)
if (item.kind === 'app') return extractAppRefs(item.value)
if (item.kind === 'raw_app') return extractRawAppRefs(item.content ?? '')
return []
}
// Whole-string `$var:`/`$jsonvar:` paths an item resolves at runtime. Scripts carry
// no variable args; raw apps hold their structure in the `content` JSON string.
function varRefsForFetched(item: FetchedItem): string[] {
if (item.kind === 'flow' || item.kind === 'app') return extractVarRefsFromValue(item.value)
if (item.kind === 'raw_app') return extractVarRefsFromValue(safeParseRaw(item.content))
return []
}
// Walks the transitive closure: scripts referenced by path are pulled in
// recursively, resources become empty stubs, hub refs stay external.
export async function buildProjectBundle(
seed: ItemRef[],
slug: string,
deps: BundleDeps,
extraResourcePaths: string[] = [],
extraVarPaths: string[] = []
): Promise<ProjectBundle> {
const fetched = new Map<string, FetchedItem>()
const queued = new Set<string>()
const resourcePaths = new Set<string>()
const varPaths = new Set<string>()
const unresolved: string[] = []
// Resources and variables referenced by triggers (by config value, not `$res:`
// in code) — relocated through the same map so the export stays slug-relative.
for (const p of extraResourcePaths) {
if (classifyPath(p, slug) !== 'hub') resourcePaths.add(p)
}
for (const p of extraVarPaths) varPaths.add(p)
// Key by `${kind}:${path}`, not bare path: a script and flow can share a path,
// and keying by path alone would silently drop one.
const refKey = (kind: string, path: string) => `${kind}:${path}`
// Refs at the same BFS depth are independent: fetch each level concurrently.
let level: ItemRef[] = []
for (const s of seed) {
const key = refKey(s.kind, s.path)
if (!queued.has(key)) {
queued.add(key)
level.push(s)
}
}
while (level.length > 0) {
const results = await Promise.all(
level.map(async (ref) => ({ ref, item: await deps.fetchItem(ref) }))
)
const next: ItemRef[] = []
for (const { ref, item } of results) {
if (!item) {
unresolved.push(ref.path)
continue
}
fetched.set(refKey(ref.kind, ref.path), item)
for (const r of refsForFetched(item)) {
if (classifyPath(r.path, slug) === 'hub') continue
if (r.kind === 'resource') {
resourcePaths.add(r.path)
} else if (r.kind === 'script' || r.kind === 'flow') {
const key = refKey(r.kind, r.path)
if (!queued.has(key)) {
queued.add(key)
next.push({ kind: r.kind, path: r.path })
}
}
}
// Relocate the item's runtime variable refs into the project folder too, so
// the export is slug-relative regardless of the source folder (import then
// materializes them as placeholders). Variables are never hub-hosted.
for (const p of varRefsForFetched(item)) varPaths.add(p)
}
level = next
}
const fetchedItems = [...fetched.values()]
const itemPaths = fetchedItems.map((it) => it.path)
const map = buildPathMap([...itemPaths, ...resourcePaths, ...varPaths], slug)
const items: BundledItem[] = fetchedItems.map((it) => {
const rewritten: BundledItem = { ...it, newPath: map.get(it.path) ?? it.path }
if (it.kind === 'script') {
rewritten.content = rewriteContent(it.content ?? '', map)
} else if (it.kind === 'raw_app') {
rewritten.content = rewriteRawVarRefs(rewriteRawAppContent(it.content ?? '', map), map)
} else if (it.kind === 'flow') {
rewritten.value = rewriteVarRefsInValue(rewriteFlowValue(it.value, map), map)
} else if (it.kind === 'app') {
rewritten.value = rewriteVarRefsInValue(rewriteAppValue(it.value, map), map)
}
return rewritten
})
const resourceStubs: ResourceStub[] = []
const resolved = await Promise.all(
[...resourcePaths].map(async (path) => ({ path, type: await deps.resolveResourceType(path) }))
)
for (const { path, type } of resolved) {
if (!type) {
unresolved.push(path)
continue
}
resourceStubs.push({ originalPath: path, newPath: map.get(path) ?? path, resource_type: type })
}
// `unresolved` keys missing items by kind:path but stores the bare path, so a
// missing script and flow (or a runnable and resource) sharing a path can push
// the same string twice. Dedupe: callers use it as a display/blocker list where
// duplicate keys would break keyed rendering.
return { items, resourceStubs, pathMap: map, unresolved: [...new Set(unresolved)] }
}