feat: let raw app backend runnables import each other (#11777)

* fix: iterate workspace picker nav items without spreading a NodeList

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: let raw app backend runnables import each other

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: preview only imported sibling runnables and reserve the main id

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: resolve a module's workspace imports from the entrypoint's folder

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: scope run-mode app scripts to their app and key bundles on module imports

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: clone app_version_lite into forks with remapped app_script ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: update the run-mode raw_code comment for app-scoped ids

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: run module-bearing app scripts only from the deployed version

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: run deployed inline code only through the deployed version's rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: in-workspace app viewers run through the deployed rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: let the app embed token read the app's lite value

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: pass sibling modules to the agent-worker Python preview resolver

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: list the lite route in the app embed scope comment

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: update ee-repo-ref to 455f98540fb042d4f624dcec681dab88b2c8df22

This commit updates the EE repository reference after PR #883 was merged in windmill-ee-private.

Previous ee-repo-ref: 927d3592a6b2127ffe16fb14eb47f5bd66afc026

New ee-repo-ref: 455f98540fb042d4f624dcec681dab88b2c8df22

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
Ruben Fiszel
2026-10-15 13:36:34 +02:00
committed by GitHub
co-authored by Claude Opus 5.5 windmill-internal-app[bot]
parent 813c080d46
commit 526da60da9
38 changed files with 810 additions and 138 deletions
@@ -0,0 +1,30 @@
{
"db_name": "PostgreSQL",
"query": "SELECT l.id IS NOT NULL as \"has_lite!\",\n coalesce(jsonb_path_exists(l.value, '$.**.inlineScript.id ? (@ == $id)',\n jsonb_build_object('id', $3::bigint)), false) as \"is_deployed_row!\"\n FROM app a\n LEFT JOIN app_version_lite l ON l.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.path = $1 AND a.workspace_id = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "has_lite!",
"type_info": "Bool"
},
{
"ordinal": 1,
"name": "is_deployed_row!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Int8"
]
},
"nullable": [
null,
null
]
},
"hash": "0d0156a21dd4bf64ad74ed6a44cbdd22c18a5f748497824d41b1b6eae975b091"
}
@@ -1,22 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT format('rawscript/%s', code_sha256) as \"path!: String\"\n FROM app_script WHERE id = $1 LIMIT 1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path!: String",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Int8"
]
},
"nullable": [
null
]
},
"hash": "1bae415f9440cc1334f24ce3009242cf3a6287e7b4548c7f01ad888230c27013"
}
@@ -0,0 +1,27 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO app_script (app, hash, lock, code, code_sha256, modules)\n VALUES ($1, $2, $3, $4, $5, $6)\n ON CONFLICT (hash) DO UPDATE SET app = EXCLUDED.app -- trivial update to return the id\n RETURNING id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Int8"
}
],
"parameters": {
"Left": [
"Int8",
"Bpchar",
"Text",
"Text",
"Bpchar",
"Jsonb"
]
},
"nullable": [
false
]
},
"hash": "4448d2cc6bc73635bb8c252ab9f4efaa33600c3ecd266e8339b9bfb32ae18499"
}
@@ -1,18 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO app_script (app, hash, lock, code, code_sha256)\n VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int8",
"Bpchar",
"Text",
"Text",
"Bpchar"
]
},
"nullable": []
},
"hash": "5c72e6e4102039c4426e8eb47d86ec152d4ee52b58f208f2fa1639eaa448c6ad"
}
@@ -0,0 +1,28 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, value AS \"value!: serde_json::Value\" FROM app_version_lite WHERE id = ANY($1)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "value!: serde_json::Value",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Int8Array"
]
},
"nullable": [
false,
true
]
},
"hash": "73a22a814ae983ae6b0188ee2b9f110dadb3db328f0589f7de27bcb418e70371"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT lock, code FROM app_script WHERE id = $1 LIMIT 1",
"query": "SELECT lock, code, modules AS \"modules: serde_json::Value\"\n FROM app_script WHERE id = $1 LIMIT 1",
"describe": {
"columns": [
{
@@ -12,6 +12,11 @@
"ordinal": 1,
"name": "code",
"type_info": "Text"
},
{
"ordinal": 2,
"name": "modules: serde_json::Value",
"type_info": "Jsonb"
}
],
"parameters": {
@@ -21,8 +26,9 @@
},
"nullable": [
true,
false
false,
true
]
},
"hash": "ffa86babfcab107caffb8dda31a66b142fa628b8983b966357deb3d5e0a1df3c"
"hash": "7b1d7a1404c3a17e1852e1fe7d32d04d8c95e1e87e75b9a07b01c03f3e3d4373"
}
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO app_version_lite (id, value)\n SELECT * FROM UNNEST($1::bigint[], $2::jsonb[])\n ON CONFLICT (id) DO NOTHING",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Int8Array",
"JsonbArray"
]
},
"nullable": []
},
"hash": "9f6bb810ebe383c848d8256f594fe701ee064dd9c880ad4df89ec47279aff7de"
}
@@ -1,32 +1,42 @@
{
"db_name": "PostgreSQL",
"query": "SELECT app, hash, lock, code, code_sha256\n FROM app_script\n WHERE app = ANY(SELECT id FROM app WHERE workspace_id = $1)",
"query": "SELECT id, app, hash, lock, code, code_sha256, modules\n FROM app_script\n WHERE app = ANY(SELECT id FROM app WHERE workspace_id = $1)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "app",
"name": "id",
"type_info": "Int8"
},
{
"ordinal": 1,
"name": "app",
"type_info": "Int8"
},
{
"ordinal": 2,
"name": "hash",
"type_info": "Bpchar"
},
{
"ordinal": 2,
"ordinal": 3,
"name": "lock",
"type_info": "Text"
},
{
"ordinal": 3,
"ordinal": 4,
"name": "code",
"type_info": "Text"
},
{
"ordinal": 4,
"ordinal": 5,
"name": "code_sha256",
"type_info": "Bpchar"
},
{
"ordinal": 6,
"name": "modules",
"type_info": "Jsonb"
}
],
"parameters": {
@@ -35,12 +45,14 @@
]
},
"nullable": [
false,
false,
false,
true,
false,
false
false,
true
]
},
"hash": "34979901e417b6eef8bda80f6213db46e3e68ee3d756c2540d924831879ce9e9"
"hash": "a3582b2b9089d85d8fa273aed5348394cb91064948c1c3fbb21f9461b9869af0"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n INSERT INTO app_script (app, hash, lock, code, code_sha256)\n VALUES ($1, $2, $3, $4, $5)\n ON CONFLICT (hash) DO UPDATE SET app = EXCLUDED.app -- trivial update to return the id\n RETURNING id\n ",
"query": "\n INSERT INTO app_script (app, hash, lock, code, code_sha256, modules)\n VALUES ($1, $2, $3, $4, $5, $6)\n ON CONFLICT (hash) DO UPDATE SET app = EXCLUDED.app -- trivial update to return the id\n RETURNING id\n ",
"describe": {
"columns": [
{
@@ -15,12 +15,13 @@
"Bpchar",
"Text",
"Text",
"Bpchar"
"Bpchar",
"Jsonb"
]
},
"nullable": [
false
]
},
"hash": "0c6c80746733be8f561ab0b631854799f5e8122adaf35465cb16c3dc795bdc3b"
"hash": "c798432108de19ed28dcf1810c68c838beaa5dfa5e3bcfcfeddee252ffe1334e"
}
@@ -0,0 +1,24 @@
{
"db_name": "PostgreSQL",
"query": "SELECT format('rawscript/%s', s.code_sha256) as \"path!: String\"\n FROM app_script s JOIN app a ON a.id = s.app\n WHERE s.id = $1 AND a.path = $2 AND a.workspace_id = $3",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "path!: String",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Int8",
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "f434d29f5617e1a97f9cc556ffaf47751aa5bff0741e98c00b3294d29dc36718"
}
+1 -1
View File
@@ -1 +1 @@
346bd5417537b76591465e63b9d15bd081e9228c
455f98540fb042d4f624dcec681dab88b2c8df22
@@ -0,0 +1 @@
ALTER TABLE app_script DROP COLUMN IF EXISTS modules;
@@ -0,0 +1 @@
ALTER TABLE app_script ADD COLUMN IF NOT EXISTS modules JSONB;
@@ -268,6 +268,33 @@ pub fn parse_code_for_imports(code: &str, path: &str) -> anyhow::Result<Vec<NImp
return Ok(nimports);
}
/// A runnable's Python modules as [`parse_python_imports`]' `module_sources`: content keyed
/// by the path a relative import of each resolves to from `script_path` (`shared.py` next
/// to `u/a/app/main` is `u/a/app/shared`).
#[cfg(not(target_arch = "wasm32"))]
pub fn module_sources(
script_path: &str,
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
) -> Option<HashMap<String, String>> {
let dir = script_path
.rsplit_once('/')
.map(|(dir, _)| dir)
.unwrap_or("");
let sources: HashMap<String, String> = modules?
.iter()
.filter_map(|(key, module)| {
let stem = key.strip_suffix(".py")?;
let path = if dir.is_empty() {
stem.to_string()
} else {
format!("{dir}/{stem}")
};
Some((path, module.content.clone()))
})
.collect();
(!sources.is_empty()).then_some(sources)
}
#[cfg(not(target_arch = "wasm32"))]
pub async fn parse_python_imports(
code: &str,
@@ -278,6 +305,10 @@ pub async fn parse_python_imports(
locked_v: &mut Option<pep440_rs::Version>,
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
temp_script_refs: &Option<HashMap<String, String>>,
// Content of relative imports that ship with the importer rather than being deployed
// (its modules), keyed by the path a relative import resolves to. Wins over both
// `temp_script_refs` and the deployed script, as the module file does at run time.
module_sources: Option<&HashMap<String, String>>,
) -> error::Result<(Vec<String>, Option<String>)> {
let mut compile_error_hint: Option<String> = None;
let mut imports = parse_python_imports_inner(
@@ -291,6 +322,7 @@ pub async fn parse_python_imports(
locked_v,
raw_workspace_dependencies_o,
temp_script_refs,
module_sources,
)
.await?
.into_values()
@@ -349,6 +381,7 @@ async fn parse_python_imports_inner(
locked_v: &mut Option<pep440_rs::Version>,
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
temp_script_refs: &Option<HashMap<String, String>>,
module_sources: Option<&HashMap<String, String>>,
) -> error::Result<HashMap<String, NImportResolved>> {
tracing::debug!("Parsing python imports for path: {}", path);
let PythonAnnotations { py310, py311, py312, py313, py314, py315, .. } =
@@ -517,10 +550,11 @@ async fn parse_python_imports_inner(
for n in nimports.into_iter() {
let mut nested = match n {
NImport::Relative(rpath) => {
// First try to get content from temp_script_refs cache if available
let code_from_cache = if let Some(hash) =
temp_script_refs.as_ref().and_then(|dt| dt.get(&rpath))
let code_from_cache = if let Some(content) =
module_sources.and_then(|m| m.get(&rpath))
{
Some(content.clone())
} else if let Some(hash) = temp_script_refs.as_ref().and_then(|dt| dt.get(&rpath)) {
tracing::debug!(
"Found relative import '{}' in temp_script_refs with hash '{}'",
rpath,
@@ -570,6 +604,7 @@ async fn parse_python_imports_inner(
locked_v,
raw_workspace_dependencies_o,
temp_script_refs,
module_sources,
)
.await?
.into_values()
@@ -29,6 +29,7 @@ def main():
&mut None,
&None,
&None,
None,
)
.await?;
// println!("{}", serde_json::to_string(&r)?);
@@ -71,6 +72,7 @@ def main():
&mut None,
&None,
&None,
None,
)
.await?;
println!("{}", serde_json::to_string(&r)?);
@@ -103,6 +105,7 @@ def main():
&mut None,
&None,
&None,
None,
)
.await?;
println!("{}", serde_json::to_string(&r)?);
+1 -1
View File
@@ -48,7 +48,7 @@ alerts: id(int), alert_type(char), message(text), created_at(ts), acknowledged(b
app: id(bigint), workspace_id(char), path(char), summary(char), policy(jsonb), versions(bigint[]), extra_perms(jsonb), draft_only(bool), custom_path(text), labels(text[])
FK: (workspace_id) -> workspace(id)
app_bundles: app_version_id(bigint), w_id(char), file_type(char), data(bytes)
app_script: id(bigint), app(bigint), hash(char(64)), lock(text), code(text), code_sha256(char(64))
app_script: id(bigint), app(bigint), hash(char(64)), lock(text), code(text), code_sha256(char(64)), modules(jsonb)
FK: (app) -> app(id)
app_version: id(bigint), app_id(bigint), value(json), created_by(char), created_at(ts), raw_app(bool)
FK: (app_id) -> app(id)
+6 -3
View File
@@ -462,8 +462,9 @@ fn app_embed_route_denied(domain: ScopeDomain, suffix: &str) -> bool {
}
}
/// App routes a running app uses: its own definition (`apps/get/p/<path>`, further
/// path-scoped by `apps:read:<path>`) and the public app-serving endpoints
/// App routes a running app uses: its own definition (`apps/get/p/<path>`, or
/// `apps/get/lite/<path>` that the viewer runs from, both further path-scoped by
/// `apps:read:<path>`) and the public app-serving endpoints
/// (`apps_u/*`: public_app, public_resource, get_data, and the path-taking
/// `execute_component` / `download_s3_file`, which re-check `apps:run|read:<path>`
/// in their handlers so they stay confined to this app). Everything else in the
@@ -477,7 +478,9 @@ fn app_embed_apps_route_allowed(suffix: &str) -> bool {
if suffix.starts_with("apps_u/embed_token") {
return false;
}
suffix.starts_with("apps/get/p/") || suffix.starts_with("apps_u/")
suffix.starts_with("apps/get/p/")
|| suffix.starts_with("apps/get/lite/")
|| suffix.starts_with("apps_u/")
}
/// Routes a guest session is denied: the app-embed allowlist, plus the embed-token
@@ -8547,7 +8547,7 @@ async fn clone_apps(
// Clone app scripts with recomputed hashes
let app_scripts = sqlx::query!(
"SELECT app, hash, lock, code, code_sha256
"SELECT id, app, hash, lock, code, code_sha256, modules
FROM app_script
WHERE app = ANY(SELECT id FROM app WHERE workspace_id = $1)",
source_workspace_id
@@ -8555,34 +8555,104 @@ async fn clone_apps(
.fetch_all(&mut **tx)
.await?;
let mut app_script_id_mapping: HashMap<i64, i64> = HashMap::new();
for app_script in app_scripts {
if let Some(&new_app_id) = app_id_mapping.get(&app_script.app) {
// Recompute hash using app_id, code_sha256, and lock
// Recompute hash using app_id, code_sha256, lock and modules
let mut hasher = Sha256::new();
hasher.update(new_app_id.to_be_bytes());
hasher.update(hex::decode(&app_script.code_sha256)?);
if let Some(lock) = &app_script.lock {
hasher.update(lock.as_bytes());
}
if let Some(modules) = &app_script.modules {
hasher.update(modules.to_string());
}
let new_hash = hex::encode(hasher.finalize());
sqlx::query!(
"INSERT INTO app_script (app, hash, lock, code, code_sha256)
VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING",
let new_id = sqlx::query_scalar!(
"INSERT INTO app_script (app, hash, lock, code, code_sha256, modules)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (hash) DO UPDATE SET app = EXCLUDED.app -- trivial update to return the id
RETURNING id",
new_app_id,
new_hash,
app_script.lock,
app_script.code,
app_script.code_sha256,
app_script.modules,
)
.execute(&mut **tx)
.fetch_one(&mut **tx)
.await?;
app_script_id_mapping.insert(app_script.id, new_id);
}
}
// A version's lite value is what points each inline script at its `app_script` row (lock and
// modules). Without it the fork serves the full value, whose inline scripts carry no id and
// so run as bare code: no lock, and no modules for a raw app's sibling imports. Every
// version, not just `versions.last()`: the fork's `versions` is rebuilt from all rows above,
// so its current one can be a row the parent's array never listed.
let source_versions: Vec<i64> = version_id_mapping.keys().copied().collect();
let lite_versions = sqlx::query!(
"SELECT id, value AS \"value!: serde_json::Value\" FROM app_version_lite WHERE id = ANY($1)",
&source_versions,
)
.fetch_all(&mut **tx)
.await?;
let (mut lite_ids, mut lite_values) = (vec![], vec![]);
for lite in lite_versions {
let Some(&new_version_id) = version_id_mapping.get(&lite.id) else {
continue;
};
let mut value = lite.value;
// Lite inline scripts have no code of their own, so one left without a row cannot run.
if !remap_app_script_ids(&mut value, &app_script_id_mapping) {
continue;
}
if let Some(object) = value.as_object_mut() {
object.insert("version".to_string(), serde_json::json!(new_version_id));
}
lite_ids.push(new_version_id);
lite_values.push(value);
}
sqlx::query!(
"INSERT INTO app_version_lite (id, value)
SELECT * FROM UNNEST($1::bigint[], $2::jsonb[])
ON CONFLICT (id) DO NOTHING",
&lite_ids,
&lite_values,
)
.execute(&mut **tx)
.await?;
Ok(app_id_mapping)
}
/// Points every `inlineScript.id` of a cloned lite app value at the fork's copy of that row.
/// False when some id has no copy.
fn remap_app_script_ids(value: &mut serde_json::Value, mapping: &HashMap<i64, i64>) -> bool {
match value {
serde_json::Value::Object(object) => {
if let Some(serde_json::Value::Object(script)) = object.get_mut("inlineScript") {
if let Some(old_id) = script.get("id").and_then(serde_json::Value::as_i64) {
let Some(new_id) = mapping.get(&old_id) else {
return false;
};
script.insert("id".to_string(), serde_json::json!(new_id));
}
}
object
.values_mut()
.all(|child| remap_app_script_ids(child, mapping))
}
serde_json::Value::Array(array) => array
.iter_mut()
.all(|child| remap_app_script_ids(child, mapping)),
_ => true,
}
}
async fn clone_raw_apps(
tx: &mut Transaction<'_, Postgres>,
source_workspace_id: &str,
+6
View File
@@ -15600,6 +15600,12 @@ paths:
type: integer
tag:
type: string
modules:
type: object
nullable: true
description: "Additional modules keyed by relative file path. Honored in preview only: a deployed app runs the modules stored with its runnable."
additionalProperties:
$ref: "#/components/schemas/ScriptModule"
required:
- content
- language
+66 -18
View File
@@ -61,7 +61,7 @@ use windmill_common::{
APP_WORKSPACED_ROUTE,
},
auth::{APP_EMBED_TOKEN_LABEL_PREFIX, RAW_APP_SDK_TOKEN_LABEL_PREFIX, TOKEN_PREFIX_LEN},
cache::{self, future::FutureCachedExt},
cache,
db::{DbWithOptAuthed, UserDB},
error::{to_anyhow, Error, JsonResult, Result},
jobs::{
@@ -1502,8 +1502,9 @@ async fn get_public_app_by_secret(
/// - `users:read` → `users/whoami` only.
/// - `folders:read` → `folders/listnames` only.
/// Plus two path-scoped scopes minted per app (see `mint_app_embed_token`):
/// - `apps:read:<path>` → the app's own definition (`apps/get/p/<path>`); no
/// `apps:write`, so management routes are unreachable.
/// - `apps:read:<path>` → the app's own definition (`apps/get/p/<path>` and
/// `apps/get/lite/<path>`); no `apps:write`, so management
/// routes are unreachable.
/// - `apps:run:<path>` → run THIS app's components (`execute_component`, which
/// re-checks the path); `apps_u/*` public-serving routes.
pub const APP_EMBED_SCOPES: [&str; 5] = [
@@ -4674,34 +4675,81 @@ async fn execute_component(
// `app_script` id. Its resolved `rawscript/<sha>` key must be present
// in the policy triggerables below — it must never resolve via the
// Viewer default fallback. Without `id` the caller supplies the code
// verbatim; with `id` it selects any `app_script` row by number (no
// app/workspace scoping), so both let a caller run code the publisher
// never pinned for this app.
// verbatim, which lets a caller run code the publisher never pinned
// for this app; with `id` the row must also belong to this app.
let is_inline_raw_code = payload.raw_code.is_some();
// The policy key covers only an inline script's own code, not the lock and modules
// its `app_script` row runs with, so it cannot tell the deployed row from an older
// one (or from id-less code, which runs with neither). Once the deployed version has
// its lite value, inline code runs only through the rows that value points at.
// Checked on every run: a deploy moves it.
if is_inline_raw_code {
let deployed = sqlx::query!(
"SELECT l.id IS NOT NULL as \"has_lite!\",
coalesce(jsonb_path_exists(l.value, '$.**.inlineScript.id ? (@ == $id)',
jsonb_build_object('id', $3::bigint)), false) as \"is_deployed_row!\"
FROM app a
LEFT JOIN app_version_lite l ON l.id = a.versions[array_upper(a.versions, 1)]
WHERE a.path = $1 AND a.workspace_id = $2",
path,
&w_id,
payload.id,
)
.fetch_optional(&db)
.await?;
if deployed.is_some_and(|d| d.has_lite && !d.is_deployed_row) {
return Err(Error::NotAuthorized(format!(
"This runnable is not part of the deployed version of app {path}, reload the app"
)));
}
}
// Compute the path for the triggerables map:
// - flow: `flow/<payload.path>`
// - script: `script/<payload.path>`
// - inline script: `rawscript/<sha256(raw_code.content)>`
let app_script_key: Arc<String>;
let path = match &payload {
// flow or script: just use the `payload.path`.
ExecuteApp { path: Some(path), .. } => path,
// inline script: without entry in the `app_script` table.
ExecuteApp { raw_code: Some(raw_code), id: None, .. } => &digest(&raw_code.content),
// inline script: with an entry in the `app_script` table.
// Scoped to this app: the policy key covers only the entrypoint's code, while the
// row also carries the modules (and lock) the job runs, so another app's row with
// the same entrypoint would run its own modules under this app's identity.
ExecuteApp { raw_code: Some(_), id: Some(id), .. } => {
let cache = cache::anon!({ u64 => Arc<String> } in "appscriptpath" <= 10000);
// `id` is unique, cache the result.
&*sqlx::query_scalar!(
"SELECT format('rawscript/%s', code_sha256) as \"path!: String\"
FROM app_script WHERE id = $1 LIMIT 1",
id
)
.fetch_one(&db)
.map_err(Into::<Error>::into)
.map_ok(Arc::new)
.cached(cache, *id as u64)
.await?
lazy_static! {
static ref APP_SCRIPT_KEY_CACHE: cache::Cache<(String, String, i64), Arc<String>> =
cache::Cache::new(10000);
}
let cache_key = (w_id.to_string(), path.to_string(), *id);
app_script_key = match APP_SCRIPT_KEY_CACHE.get(&cache_key) {
Some(key) => key,
None => {
let key = sqlx::query_scalar!(
"SELECT format('rawscript/%s', s.code_sha256) as \"path!: String\"
FROM app_script s JOIN app a ON a.id = s.app
WHERE s.id = $1 AND a.path = $2 AND a.workspace_id = $3",
id,
path,
&w_id,
)
.fetch_optional(&db)
.await?
.map(Arc::new)
.ok_or_else(|| {
Error::NotAuthorized(format!(
"App script {id} does not belong to app {path}"
))
})?;
// A row never changes app, so a hit stays valid.
APP_SCRIPT_KEY_CACHE.insert(cache_key, key.clone());
key
}
};
&**app_script_key
}
_ => unreachable!(),
};
+10 -2
View File
@@ -2223,14 +2223,22 @@ pub async fn fetch_raw_script_from_app_query(
loc: &'static Location<'_>,
) -> error::Result<RawScript> {
sqlx::query!(
"SELECT lock, code FROM app_script WHERE id = $1 LIMIT 1",
"SELECT lock, code, modules AS \"modules: serde_json::Value\"
FROM app_script WHERE id = $1 LIMIT 1",
id,
)
.fetch_optional(db)
.await
.map_err(Into::into)
.and_then(unwrap_or_error(&loc, "Application script", id))
.map(|r| RawScript { content: r.code, lock: r.lock, meta: None, modules: None })
.and_then(|r| {
Ok(RawScript {
content: r.code,
lock: r.lock,
meta: None,
modules: r.modules.map(serde_json::from_value).transpose()?,
})
})
}
/// Returns the number of jobs the row already accounted for: non-zero when a worker of the
+32 -4
View File
@@ -41,7 +41,10 @@ const p = {
imp.path.startsWith("/f/")) &&
!imp.path.endsWith(".ts")
) {
code = code.replaceAll(imp.path, imp.path + ".ts");
// Whole quoted specifiers only: `./util` is also a substring of `../util_ws`.
for (const q of ['"', "'"]) {
code = code.replaceAll(q + imp.path + q, q + imp.path + ".ts" + q);
}
}
}
}
@@ -55,6 +58,17 @@ const p = {
return replaceRelativeImports(code);
});
// Module files (written by write_module_files next to main.ts) need the same `.ts`
// rewrite: an extensionless relative import misses the resolve filter below, and the
// lockfile builder then treats it as an external package named ".".
const filterLoadModule = new RegExp(
`^${cdir}\/(?!node_modules\/)(?!_wm_)(?!main\\.ts$).*\\.ts$`
);
build.onLoad({ filter: filterLoadModule }, async (args) => {
const code = readFileSync(args.path, "utf8");
return replaceRelativeImports(code);
});
// A stalled fetch would otherwise hold the whole build for bun's own 5-minute
// default, with nothing naming the script it was waiting on.
const RELATIVE_IMPORT_FETCH_TIMEOUT_MS = 120000;
@@ -109,18 +123,32 @@ const p = {
// Check if the import resolves to a local module file (written by write_module_files).
// Only check relative paths — absolute/bare specifiers should fall through to the
// remote resolver, matching the Windows loader pattern.
if (args.path.startsWith(".")) {
const localPath = resolve(cdir, args.path);
// Only for local importers (main.ts or a module, both `.ts` in the job dir): a fetched
// workspace script (`.url`) resolves within its own workspace folder, where a module
// file of the same name would shadow its real neighbour.
const importer = args.importer ? resolve(args.importer) : "";
if (
args.path.startsWith(".") &&
importer.endsWith(".ts") &&
importer.startsWith(cdir + "/")
) {
// From the importing module's own folder: a module may sit in a subfolder (`lib/a.ts`).
const localPath = resolve(dirname(importer), args.path);
try {
readFileSync(localPath);
return { path: localPath };
} catch {}
}
const importerRel = args.importer.replace(cdir + "/", "");
// A module file stands in the job dir for a script in the entrypoint's folder, so its
// own relative imports resolve from that folder's workspace path, not from the job dir.
const file_path =
args.importer == "./main.ts" || args.importer == resolve("./main.ts")
? current_path
: args.importer.replace(cdir + "/", "");
: importerRel.endsWith(".ts")
? join(dirname(current_path), importerRel.replace(/\.ts$/, ""))
: importerRel;
const isRelative = !args.path.startsWith("/");
const endExt = args.path.endsWith(".ts") ? "" : ".ts";
@@ -1542,6 +1542,26 @@ pub async fn compute_bundle_local_and_remote_path(
input_src.push_str(&path);
input_src.push_str(&hash.to_string());
}
// Modules are inlined too, and so is every deployed script they import: those are
// reached through the module files, never through the entrypoint's own imports.
let dir = script_path
.rsplit_once('/')
.map(|(dir, _)| dir)
.unwrap_or("");
let mut module_entries = modules.into_iter().flatten().collect_vec();
module_entries.sort_by(|a, b| a.0.cmp(b.0));
for (key, module) in module_entries {
let Some(stem) = key.strip_suffix(".ts") else {
continue;
};
let module_path = format!("{dir}/{stem}");
for (path, hash) in
collect_transitive_import_versions(db, w_id, &module_path, &module.content).await
{
input_src.push_str(&path);
input_src.push_str(&hash.to_string());
}
}
};
// Keep temp-script-ref (preview) bundles in a distinct cache slot: their
+44 -1
View File
@@ -791,7 +791,8 @@ pub async fn handle_python_job(
None
};
let (py_version, mut additional_python_paths) = handle_python_deps(
let module_sources = windmill_parser_py_imports::module_sources(&script_path, modules.as_ref());
let (py_version, mut additional_python_paths) = handle_python_deps_with_modules(
job_dir,
requirements_o,
inner_content,
@@ -807,6 +808,7 @@ pub async fn handle_python_job(
precomputed_agent_info,
annotations.clone(),
&temp_script_refs,
module_sources.as_ref(),
)
.await?;
@@ -1939,6 +1941,7 @@ async fn replace_pip_secret(
}
}
#[cfg(feature = "private")]
pub(crate) async fn handle_python_deps(
job_dir: &str,
requirements_o: Option<&String>,
@@ -1955,6 +1958,45 @@ pub(crate) async fn handle_python_deps(
precomputed_agent_info: Option<PrecomputedAgentInfo>,
annotations: PythonAnnotations,
temp_script_refs: &Option<HashMap<String, String>>,
) -> error::Result<(PyV, Vec<String>)> {
handle_python_deps_with_modules(
job_dir,
requirements_o,
inner_content,
w_id,
script_path,
job_id,
conn,
worker_name,
worker_dir,
mem_peak,
canceled_by,
occupancy_metrics,
precomputed_agent_info,
annotations,
temp_script_refs,
None,
)
.await
}
async fn handle_python_deps_with_modules(
job_dir: &str,
requirements_o: Option<&String>,
inner_content: &str,
w_id: &str,
script_path: &str,
job_id: &Uuid,
conn: &Connection,
worker_name: &str,
worker_dir: &str,
mem_peak: &mut i32,
canceled_by: &mut Option<CanceledBy>,
occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
precomputed_agent_info: Option<PrecomputedAgentInfo>,
annotations: PythonAnnotations,
temp_script_refs: &Option<HashMap<String, String>>,
module_sources: Option<&HashMap<String, String>>,
) -> error::Result<(PyV, Vec<String>)> {
create_dependencies_dir(job_dir).await;
@@ -1984,6 +2026,7 @@ pub(crate) async fn handle_python_deps(
&mut locked_v,
&None,
temp_script_refs,
module_sources,
))
.await?;
+175 -25
View File
@@ -1,5 +1,5 @@
use std::borrow::Cow;
use std::collections::HashMap;
use std::collections::{BTreeMap, HashMap, HashSet};
use std::fs::{create_dir_all, remove_dir_all};
#[cfg(feature = "python")]
@@ -2093,8 +2093,10 @@ async fn insert_app_script(
code: String,
lock: Option<String>,
language: Option<ScriptLang>,
modules: Option<BTreeMap<String, ScriptModule>>,
) -> Result<AppScriptId> {
let code_sha256 = format!("{:x}", sha2::Sha256::digest(&code));
let modules = modules.map(|m| serde_json::to_value(m)).transpose()?;
let hash = {
let mut hasher = sha2::Sha256::new();
hasher.update(app.to_le_bytes());
@@ -2106,14 +2108,18 @@ async fn insert_app_script(
relative_imports_bytes(db, Some(&code), &format!("{path}/app"), language).await?,
);
}
// Only when present, so a runnable without modules keeps the id it already had.
if let Some(modules) = modules.as_ref() {
hasher.update(modules.to_string());
}
format!("{:x}", hasher.finalize())
};
// Insert the app script if it doesn't exist.
sqlx::query_scalar!(
r#"
INSERT INTO app_script (app, hash, lock, code, code_sha256)
VALUES ($1, $2, $3, $4, $5)
INSERT INTO app_script (app, hash, lock, code, code_sha256, modules)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (hash) DO UPDATE SET app = EXCLUDED.app -- trivial update to return the id
RETURNING id
"#,
@@ -2121,7 +2127,8 @@ async fn insert_app_script(
hash,
lock,
code,
code_sha256
code_sha256,
modules
)
.fetch_one(db)
.await
@@ -2294,11 +2301,103 @@ async fn reduce_flow<'c>(
Ok(tx)
}
/// A raw app's inline runnables, keyed by the path a relative import of one resolves to:
/// `<app path>/<runnable id>`, the path the runnable itself runs under. A runnable ships
/// the ones it imports as its modules, so they resolve from the job dir at run time.
#[derive(Default)]
struct RawAppRunnables(HashMap<String, (String, ScriptLang, String)>);
impl RawAppRunnables {
fn from_value(app_path: &str, value: &Value) -> Self {
let mut runnables = HashMap::new();
if let Some(Value::Object(by_id)) = value.get("runnables") {
for (id, runnable) in by_id {
let Some(script) = runnable.get("inlineScript") else {
continue;
};
let content = script.get("content").and_then(Value::as_str);
let language = script
.get("language")
.and_then(|l| from_value::<ScriptLang>(l.clone()).ok());
if let (Some(content), Some(language)) = (content, language) {
// `main.ts` is the Bun entrypoint's own file in the job dir, so a
// runnable with that id cannot ship as a module next to it.
let ext = module_extension(language);
if ext.is_some() && !(ext == Some("ts") && id == "main") {
runnables.insert(
format!("{app_path}/{id}"),
(id.clone(), language, content.to_string()),
);
}
}
}
}
Self(runnables)
}
/// Whether an import of `path` by `language` code resolves to a runnable shipped as a
/// module, rather than to the workspace script at that path.
fn resolves(&self, path: &str, language: ScriptLang) -> bool {
self.0.get(path).is_some_and(|(_, sibling_language, _)| {
module_extension(*sibling_language) == module_extension(language)
})
}
/// The runnables `content` imports, directly or through one another.
fn modules_of(
&self,
app_path: &str,
content: &str,
language: ScriptLang,
) -> Option<BTreeMap<String, ScriptModule>> {
let ext = module_extension(language)?;
let mut queue =
extract_relative_imports(content, &format!("{app_path}/app"), &Some(language))
.unwrap_or_default();
let mut visited = HashSet::new();
let mut modules = BTreeMap::new();
while let Some(path) = queue.pop() {
if !visited.insert(path.clone()) {
continue;
}
let Some((id, sibling_language, sibling_content)) = self.0.get(&path) else {
continue;
};
if module_extension(*sibling_language) != Some(ext) {
continue;
}
queue.extend(
extract_relative_imports(sibling_content, &path, &Some(*sibling_language))
.unwrap_or_default(),
);
modules.insert(
format!("{id}.{ext}"),
ScriptModule {
content: sibling_content.clone(),
language: *sibling_language,
lock: None,
},
);
}
(!modules.is_empty()).then_some(modules)
}
}
/// The languages whose executors resolve a relative import against a module file.
fn module_extension(language: ScriptLang) -> Option<&'static str> {
match language {
ScriptLang::Bun | ScriptLang::Bunnative => Some("ts"),
ScriptLang::Python3 => Some("py"),
_ => None,
}
}
async fn reduce_app(
db: &sqlx::Pool<sqlx::Postgres>,
path: &str,
value: &mut Value,
app: i64,
raw_app_runnables: &RawAppRunnables,
) -> Result<()> {
match value {
Value::Object(object) => {
@@ -2319,26 +2418,20 @@ async fn reduce_app(
Value::String(s) => Some(s),
_ => None,
});
let id = insert_app_script(
db,
path,
app,
code,
lock,
language.map(|v| from_value(v).ok()).flatten(),
)
.await?;
let language: Option<ScriptLang> = language.and_then(|v| from_value(v).ok());
let modules = language.and_then(|l| raw_app_runnables.modules_of(path, &code, l));
let id = insert_app_script(db, path, app, code, lock, language, modules).await?;
// insert the `id` into the `script` object:
script.insert("id".to_string(), json!(id.0));
} else {
for (_, value) in object {
Box::pin(reduce_app(db, path, value, app)).await?;
Box::pin(reduce_app(db, path, value, app, raw_app_runnables)).await?;
}
}
}
Value::Array(array) => {
for value in array {
Box::pin(reduce_app(db, path, value, app)).await?;
Box::pin(reduce_app(db, path, value, app, raw_app_runnables)).await?;
}
}
_ => {}
@@ -2383,6 +2476,7 @@ async fn lock_modules_app(
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
triggered_by_relative_import: bool,
temp_script_refs: &Option<HashMap<String, String>>,
raw_app_runnables: &RawAppRunnables,
) -> Result<Value> {
match value {
Value::Object(mut m) => {
@@ -2417,15 +2511,29 @@ async fn lock_modules_app(
.to_string();
let mut logs = "".to_string();
// An import satisfied by another runnable of the app is not a
// dependency on the workspace script at that path.
let referenced_paths = extract_referenced_paths(
&content,
&format!("{job_path}/app"),
Some(language),
);
)
.map(|paths| {
paths
.into_iter()
.filter(|p| !raw_app_runnables.resolves(p, language))
.collect_vec()
});
// The lock covers what the imported runnables import, and nothing
// records whether one of them changed, so it is never reused.
let modules =
raw_app_runnables.modules_of(job_path, &content, language);
let modules = modules.map(|m| m.into_iter().collect::<HashMap<_, _>>());
if let Some((l, id)) = locks_to_reload
.as_ref()
.zip(container_id.as_ref())
.filter(|_| modules.is_none())
// TODO: Remove fallback
.and_then(|e| {
if *WMDEBUG_NO_COMPONENTS_TO_RELOCK {
@@ -2447,9 +2555,9 @@ async fn lock_modules_app(
.await?;
return Ok(Value::Object(m.clone()));
}
} else if v
.get("lock")
.is_some_and(|x| !x.as_str().unwrap().trim().is_empty())
} else if modules.is_none()
&& v.get("lock")
.is_some_and(|x| !x.as_str().unwrap().trim().is_empty())
{
if skip_creating_new_lock(&language, &content)
&& (MIN_VERSION_SUPPORTS_DEBOUNCING_V2.met().await
@@ -2486,7 +2594,7 @@ async fn lock_modules_app(
base_internal_url,
token,
&format!("{}/app", job.runnable_path()),
None,
modules.as_ref(),
occupancy_metrics,
&None,
existing_lock,
@@ -2573,6 +2681,7 @@ async fn lock_modules_app(
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
raw_app_runnables,
)
.await?,
);
@@ -2602,6 +2711,7 @@ async fn lock_modules_app(
raw_workspace_dependencies_o,
triggered_by_relative_import,
temp_script_refs,
raw_app_runnables,
)
.await?,
);
@@ -2716,6 +2826,11 @@ pub async fn handle_app_dependency_job(
// TODO: Use transaction for entire segment?
if let Some((app_id, value, is_raw_app)) = record {
let raw_app_runnables = if is_raw_app {
RawAppRunnables::from_value(&job_path, &value)
} else {
RawAppRunnables::default()
};
let value = lock_modules_app(
value,
&job,
@@ -2735,6 +2850,7 @@ pub async fn handle_app_dependency_job(
&raw_workspace_dependencies_o,
triggered_by_relative_import,
&temp_script_refs,
&raw_app_runnables,
)
.await?;
@@ -2747,7 +2863,7 @@ pub async fn handle_app_dependency_job(
// Compute a lite version of the app value (w/ `inlineScript.{lock,code}`).
let mut value_lite = value.clone();
reduce_app(db, &job_path, &mut value_lite, app_id).await?;
reduce_app(db, &job_path, &mut value_lite, app_id, &raw_app_runnables).await?;
if let Value::Object(object) = &mut value_lite {
object.insert("version".to_string(), json!(id));
object.remove("files");
@@ -3311,6 +3427,7 @@ async fn capture_dependency_job(
&mut locked_v,
raw_workspace_dependencies_o,
temp_script_refs,
windmill_parser_py_imports::module_sources(script_path, modules).as_ref(),
)
.await?
.0
@@ -3436,6 +3553,11 @@ async fn capture_dependency_job(
// TODO: move inside gen_bun_lockfile
if !wd_exist {
write_file(job_dir, "main.ts", job_raw_code)?;
// The lock has to cover what the modules import, and the loader resolves a
// relative import against these files before asking for a deployed script.
if let Some(modules) = modules {
crate::worker::write_module_files(job_dir, modules, None).await?;
}
}
if let Some(lock) = gen_bun_lockfile(
mem_peak,
@@ -3457,10 +3579,9 @@ async fn capture_dependency_job(
)
.await?
{
// Nothing here writes the module files, so a bundle built now resolves a
// multi-file script's relative imports remotely rather than from its
// modules; caching that under a key naming them would serve the wrong
// code to every run. Leave it to the first run instead.
// A runnable with modules bundles on its first run instead: a multi-file
// script's per-module lock pass also lands here, once per module, and would
// prebundle each module as if it were an entrypoint no run ever asks for.
if !wd_exist && modules.map_or(true, |m| m.is_empty()) {
crate::bun_executor::prebundle_bun_script(
job_raw_code,
@@ -3621,3 +3742,32 @@ async fn add_lock_header(
Ok(())
}
#[cfg(test)]
mod raw_app_runnables_tests {
use super::*;
#[test]
fn modules_follow_imports_between_runnables_of_one_language() {
let value = json!({ "runnables": {
"main": { "inlineScript": { "language": "bun", "content": "import { a } from './a'" } },
"a": { "inlineScript": { "language": "bun", "content": "import { b } from './b.ts'" } },
"b": { "inlineScript": { "language": "bun", "content": "export const b = 1" } },
"unused": { "inlineScript": { "language": "bun", "content": "export const u = 1" } },
"py": { "inlineScript": { "language": "python3", "content": "x = 1" } },
}});
let runnables = RawAppRunnables::from_value("u/admin/app", &value);
let modules = runnables
.modules_of(
"u/admin/app",
"import { a } from './a'\nimport { p } from './py'",
ScriptLang::Bun,
)
.unwrap();
assert_eq!(modules.keys().collect_vec(), vec!["a.ts", "b.ts"]);
assert!(runnables
.modules_of("u/admin/app", "export const x = 1", ScriptLang::Bun)
.is_none());
}
}
+59 -1
View File
@@ -1181,6 +1181,7 @@ async function dev(opts: DevOptions, appFolder?: string) {
runnableId,
args,
appTempRefs,
previewModules(runnables, runnableId),
);
log.info(colors.gray(`[backend] Job started: ${uuid}`));
@@ -1228,6 +1229,7 @@ async function dev(opts: DevOptions, appFolder?: string) {
runnable_id,
v,
appTempRefs,
previewModules(runnables, runnable_id),
);
log.info(colors.gray(`[backendAsync] Job started: ${uuid}`));
@@ -1813,6 +1815,55 @@ async function loadRunnables(): Promise<Record<string, Runnable>> {
}
}
function moduleExtension(language: string | undefined): string | undefined {
if (language === "bun" || language === "bunnative") return "ts";
if (language === "python3") return "py";
return undefined;
}
/** Ids of the sibling runnables `content` imports as `./<id>` (TS) or `.<id>` (Python). */
function siblingImports(content: string, ext: string): string[] {
const pattern = ext === "ts"
? /(?:from|import)\s*['"]\.\/([^'"/]+?)(?:\.ts)?['"]/g
: /^\s*from\s+\.([A-Za-z_]\w*)\s+import\b/gm;
return Array.from(content.matchAll(pattern), (m) => m[1]);
}
/**
* The app's other inline runnables that `runnableId` imports, directly or through one
* another, as the modules its preview ships with so those imports resolve to the local
* files. Deploy stores the same set on the runnable.
*/
function previewModules(
runnables: Record<string, Runnable>,
runnableId: string,
): Record<string, { content: string; language: string }> | undefined {
const inlineScriptOf = (r: Runnable | undefined) =>
r && "inlineScript" in r ? r.inlineScript : undefined;
const own = inlineScriptOf(runnables[runnableId]);
const ext = moduleExtension(own?.language);
if (!own || !ext) return undefined;
const modules: Record<string, { content: string; language: string }> = {};
const queue = siblingImports(own.content, ext);
while (queue.length > 0) {
const id = queue.pop()!;
const key = `${id}.${ext}`;
const inlineScript = inlineScriptOf(runnables[id]);
// `main.ts` is the Bun entrypoint's own file, so no module may take that name. The
// runnable itself is kept when a cycle reaches it: the entrypoint is not under its id.
if (key in modules || key === "main.ts") continue;
if (!inlineScript || moduleExtension(inlineScript.language) !== ext) {
continue;
}
modules[key] = {
content: inlineScript.content,
language: inlineScript.language,
};
queue.push(...siblingImports(inlineScript.content, ext));
}
return Object.keys(modules).length > 0 ? modules : undefined;
}
async function executeRunnable(
runnable: Runnable,
workspace: string,
@@ -1820,6 +1871,7 @@ async function executeRunnable(
runnableId: string,
args: any,
tempScriptRefs?: Record<string, string>,
modules?: Record<string, { content: string; language: string }>,
): Promise<string> {
const requestBody: any = {
component: runnableId,
@@ -1852,12 +1904,18 @@ async function executeRunnable(
if (inlineScript.id !== undefined) {
requestBody.id = inlineScript.id;
}
// A lock is not regenerated when only an imported runnable is edited, so with
// modules the preview resolves dependencies afresh.
const previewMods = inlineScript.id === undefined ? modules : undefined;
requestBody.raw_code = {
content: inlineScript.id === undefined ? inlineScript.content : "",
language: inlineScript.language ?? "",
path: `${appPath}/${runnableId}`,
lock: inlineScript.id === undefined ? inlineScript.lock : undefined,
lock: inlineScript.id === undefined && !previewMods
? inlineScript.lock
: undefined,
cache_ttl: inlineScript.cache_ttl,
...(previewMods ? { modules: previewMods } : {}),
};
if (inlineScript.id === undefined && tempScriptRefs) {
requestBody.temp_script_refs = tempScriptRefs;
+4
View File
@@ -6875,6 +6875,10 @@ def main(user_id: str):
return user
\`\`\`
#### Sharing code between runnables
An inline runnable can import another inline runnable of the same app by its id, as a relative import: \`import { getDb } from './shared'\` (TypeScript, Bun) or \`from .shared import get_db\` (Python). This works across TypeScript (Bun) runnables and across Python runnables, not between the two. The imported runnable's own dependencies are picked up, and it can import further runnables in turn. A runnable that only holds helpers needs no \`main\`; the frontend just never calls it. A TypeScript runnable whose id is \`main\` cannot be imported, because that name is taken by the entrypoint file of every TypeScript job: give shared code any other id.
#### The \`wmill\` client is already authenticated
An inline runnable runs as an ordinary Windmill job. \`import * as wmill from 'windmill-client'\` (TypeScript) and \`import wmill\` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
@@ -1,4 +1,4 @@
import { AppService, type ExecuteComponentData } from '$lib/gen'
import { AppService, type ExecuteComponentData, type ScriptModule } from '$lib/gen'
import { defaultIfEmptyString } from '$lib/utils'
import { isRunnableByName, isRunnableByPath, type Runnable } from '../../inputType'
import type { InlineScript } from '../../sharedTypes'
@@ -12,7 +12,8 @@ export async function executeRunnable(
id: string,
requestBody: ExecuteComponentData['requestBody'],
inlineScriptOverride?: InlineScript,
queryParams?: Record<string, any>
queryParams?: Record<string, any>,
modules?: Record<string, ScriptModule>
) {
let appPath = defaultIfEmptyString(path, `u/${username ?? 'unknown'}/newapp`)
if (isRunnableByName(runnable)) {
@@ -24,13 +25,17 @@ export async function executeRunnable(
if (inlineScript.id !== undefined) {
requestBody['id'] = inlineScript.id
}
// A lock is not cleared when only an imported runnable is edited, so with
// modules the preview resolves dependencies afresh.
const previewModules = inlineScript.id === undefined ? modules : undefined
requestBody['raw_code'] = {
content: inlineScript.id === undefined ? inlineScript.content : '',
language: inlineScript.language ?? '',
path: appPath + '/' + id,
lock: inlineScript.id === undefined ? inlineScript.lock : undefined,
lock: inlineScript.id === undefined && !previewModules ? inlineScript.lock : undefined,
cache_ttl: inlineScript.cache_ttl,
...(inlineScript.tag ? { tag: inlineScript.tag } : {})
...(inlineScript.tag ? { tag: inlineScript.tag } : {}),
...(previewModules ? { modules: previewModules } : {})
}
}
} else if (isRunnableByPath(runnable)) {
@@ -123,8 +123,8 @@
// Viewer side — used for the inline renderings (raw, and unsandboxed low-code);
// the sandboxed low-code case loads inside the opaque /app_embed iframe instead.
// getAppByPath returns bundle_secret + runnables for raw apps, which
// PublicApp -> RawAppPreview needs.
// getAppLiteByPath returns the runnables (with their ids) that
// PublicApp -> RawAppPreview needs for raw apps.
async function loadApp() {
// Kept local and handed to PublicApp rather than written to `userStore`: this is the
// membership in `workspace`, and a session preview tab shows a workspace the rest of
@@ -134,8 +134,10 @@
// populated `userStore` for the workspace the page *is* on.
appUser = await getUserExt(workspace)
try {
const loaded: any = await AppService.getAppByPath({ workspace, path })
// Raw apps need the bundle secret to load their bundle. getAppByPath
// The lite value: its inline scripts carry the ids of their deployed rows, the only
// code a deployed app runs (with its lock, and a raw app's imported runnables).
const loaded: any = await AppService.getAppLiteByPath({ workspace, path })
// Raw apps need the bundle secret to load their bundle. getAppLiteByPath
// doesn't compute it (unlike the public handlers), so fetch it here — the
// same call the previous raw viewer used — and hand it to PublicApp ->
// RawAppPreview via bundle_secret.
@@ -5,7 +5,7 @@
import type { JobById } from '../apps/types'
import { JobService } from '$lib/gen'
import type { Runnable } from './rawAppPolicy'
import { RAW_APP_PREVIEW_RELAY } from './utils'
import { RAW_APP_PREVIEW_RELAY, rawAppPreviewModules } from './utils'
import { undefinedIfEmpty } from '$lib/utils'
interface Props {
@@ -162,7 +162,9 @@
on_behalf_of: editor ? previewOnBehalfOf : undefined,
viewer_data_access: editor ? runnable?.viewer_data_access || undefined : undefined
},
undefined
undefined,
undefined,
editor ? rawAppPreviewModules(runnables, runnable_id) : undefined
)
launchedJobs.add(uuid)
let job: JobById = { component: runnable_id, created_at: Date.now(), job: uuid }
@@ -31,6 +31,7 @@
import { userStore } from '$lib/stores'
import { isHubFlowPath } from '$lib/utils'
import { sendUserToast } from '$lib/toast'
import { rawAppPreviewModules } from './utils'
import {
useOperatingUser,
useOperatingWorkspace
@@ -51,6 +52,8 @@
runnable: Runnable
id: string
appPath: string
/** All of the app's runnables, which a test run ships as modules for `./<id>` imports */
runnables?: Record<string, Runnable>
/** Identity the test runs as instead of the user (see `appPreviewOnBehalfOf`). */
previewOnBehalfOf?: string
lastDeployedCode?: string | undefined
@@ -70,6 +73,7 @@
runnable = $bindable(),
id,
appPath,
runnables,
previewOnBehalfOf,
onSelectionChange
}: Props = $props()
@@ -193,7 +197,7 @@
undefined,
undefined,
undefined,
undefined,
runnables ? rawAppPreviewModules(runnables, id) : undefined,
undefined,
undefined,
{ onBehalfOf: previewOnBehalfOf, viewerDataAccess: runnable.viewer_data_access }
@@ -217,13 +221,7 @@
// The deployed app runs a path runnable as the app's identity, which a run
// by path cannot ask for: preview the deployed definition instead.
const flow = await FlowService.getFlowByPath({ workspace: opWs!, path: runnable.path })
await jobLoader.runFlowPreview(
args,
flow,
undefined,
runnable.path,
previewOnBehalfOf
)
await jobLoader.runFlowPreview(args, flow, undefined, runnable.path, previewOnBehalfOf)
} else {
await jobLoader.runFlowByPath(runnable.path, args)
}
@@ -40,6 +40,7 @@
{#key selectedRunnable}
<RawAppInlineScripRunnable
{appPath}
{runnables}
{previewOnBehalfOf}
on:createScriptFromInlineScript={(e) => {
createScriptFromInlineScript(selectedRunnable ?? '', e.detail, opWs ?? '', appPath)
@@ -7,10 +7,37 @@ import {
formatRuntimeLogsForChat,
genWmillTs,
normalizeRawAppRuntimeLogs,
rawAppPreviewModules,
stripRawAppDiffNoise,
type Runnable
} from './utils'
describe('rawAppPreviewModules', () => {
const inline = (language: 'bun' | 'python3', content: string) =>
({ type: 'inline', inlineScript: { language, content } }) as unknown as Runnable
it('ships only the same-language runnables reached through imports', () => {
const runnables = {
get: inline('bun', "import { a } from './a'\nimport { p } from './py'"),
a: inline('bun', 'import { b } from "./b.ts"'),
b: inline('bun', "import { g } from './get'\nexport const b = 1"),
unused: inline('bun', 'export const u = 1'),
main: inline('bun', 'export const m = 1'),
py: inline('python3', 'from .helpers import x')
}
// `get` itself ships too once the cycle through `b` reaches it, as deploy does.
expect(Object.keys(rawAppPreviewModules(runnables, 'get') ?? {}).sort()).toEqual([
'a.ts',
'b.ts',
'get.ts'
])
expect(rawAppPreviewModules(runnables, 'unused')).toBeUndefined()
expect(
rawAppPreviewModules({ ...runnables, helpers: inline('python3', 'x = 1') }, 'py')
).toEqual({ 'helpers.py': { language: 'python3', content: 'x = 1' } })
})
})
const flowSchema = {
$schema: 'https://json-schema.org/draft/2020-12/schema',
type: 'object',
+44 -1
View File
@@ -1,4 +1,4 @@
import type { ScriptLang } from '../../gen/types.gen'
import type { ScriptLang, ScriptModule } from '../../gen/types.gen'
import type { Schema } from '../../common'
import { schemaToTsType } from '../../schema'
import { isRunnableByName, isRunnableByPath, type RunnableWithFields } from '../apps/inputType'
@@ -18,6 +18,49 @@ export type RawApp = {
files: string[]
}
function moduleExtension(language: ScriptLang | undefined): string | undefined {
if (language === 'bun' || language === 'bunnative') return 'ts'
if (language === 'python3') return 'py'
return undefined
}
/** Ids of the sibling runnables `content` imports as `./<id>` (TS) or `.<id>` (Python). */
function siblingImports(content: string, ext: string): string[] {
const pattern =
ext === 'ts'
? /(?:from|import)\s*['"]\.\/([^'"/]+?)(?:\.ts)?['"]/g
: /^\s*from\s+\.([A-Za-z_]\w*)\s+import\b/gm
return Array.from(content.matchAll(pattern), (m) => m[1])
}
/**
* The app's other inline runnables that runnable `id` imports, directly or through one
* another, as the modules its preview ships with so those imports resolve to the
* editor's code. Deploy stores the same set on the runnable.
*/
export function rawAppPreviewModules(
runnables: Record<string, Runnable>,
id: string
): Record<string, ScriptModule> | undefined {
const own = runnables[id]?.inlineScript
const ext = moduleExtension(own?.language)
if (!own || !ext) return undefined
const modules: Record<string, ScriptModule> = {}
const queue = siblingImports(own.content, ext)
while (queue.length > 0) {
const siblingId = queue.pop()!
const key = `${siblingId}.${ext}`
const inlineScript = runnables[siblingId]?.inlineScript
// `main.ts` is the Bun entrypoint's own file, so no module may take that name. The
// runnable itself is kept when a cycle reaches it: the entrypoint is not under its id.
if (key in modules || key === 'main.ts') continue
if (!inlineScript || moduleExtension(inlineScript.language) !== ext) continue
modules[key] = { content: inlineScript.content, language: inlineScript.language }
queue.push(...siblingImports(inlineScript.content, ext))
}
return Object.keys(modules).length > 0 ? modules : undefined
}
// Server-managed columns the deployed app row (getAppByPath) carries but the
// editor's current value never does — leaving them in renders as spurious diff.
const RAW_APP_DEPLOYED_METADATA_KEYS = [
@@ -58,7 +58,8 @@
console.warn('Anonymous user')
}
try {
app = await AppService.getAppByPath({ workspace, path })
// The lite value, whose inline scripts carry their deployed rows' ids: see InWorkspaceAppViewer.
app = await AppService.getAppLiteByPath({ workspace, path })
noPermission = false
notExists = false
} catch (e: any) {
+4
View File
@@ -1099,6 +1099,10 @@ def main(user_id: str):
return user
\`\`\`
#### Sharing code between runnables
An inline runnable can import another inline runnable of the same app by its id, as a relative import: \`import { getDb } from './shared'\` (TypeScript, Bun) or \`from .shared import get_db\` (Python). This works across TypeScript (Bun) runnables and across Python runnables, not between the two. The imported runnable's own dependencies are picked up, and it can import further runnables in turn. A runnable that only holds helpers needs no \`main\`; the frontend just never calls it. A TypeScript runnable whose id is \`main\` cannot be imported, because that name is taken by the entrypoint file of every TypeScript job: give shared code any other id.
#### The \`wmill\` client is already authenticated
An inline runnable runs as an ordinary Windmill job. \`import * as wmill from 'windmill-client'\` (TypeScript) and \`import wmill\` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
@@ -432,6 +432,10 @@ def main(user_id: str):
return user
```
#### Sharing code between runnables
An inline runnable can import another inline runnable of the same app by its id, as a relative import: `import { getDb } from './shared'` (TypeScript, Bun) or `from .shared import get_db` (Python). This works across TypeScript (Bun) runnables and across Python runnables, not between the two. The imported runnable's own dependencies are picked up, and it can import further runnables in turn. A runnable that only holds helpers needs no `main`; the frontend just never calls it. A TypeScript runnable whose id is `main` cannot be imported, because that name is taken by the entrypoint file of every TypeScript job: give shared code any other id.
#### The `wmill` client is already authenticated
An inline runnable runs as an ordinary Windmill job. `import * as wmill from 'windmill-client'` (TypeScript) and `import wmill` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
+4
View File
@@ -193,6 +193,10 @@ def main(user_id: str):
return user
```
#### Sharing code between runnables
An inline runnable can import another inline runnable of the same app by its id, as a relative import: `import { getDb } from './shared'` (TypeScript, Bun) or `from .shared import get_db` (Python). This works across TypeScript (Bun) runnables and across Python runnables, not between the two. The imported runnable's own dependencies are picked up, and it can import further runnables in turn. A runnable that only holds helpers needs no `main`; the frontend just never calls it. A TypeScript runnable whose id is `main` cannot be imported, because that name is taken by the entrypoint file of every TypeScript job: give shared code any other id.
#### The `wmill` client is already authenticated
An inline runnable runs as an ordinary Windmill job. `import * as wmill from 'windmill-client'` (TypeScript) and `import wmill` (Python) are already pointed at this instance and this workspace — there is nothing to configure.