mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-17 00:02:59 +00:00
feat: let raw app backend runnables import each other (#11777)
* fix: iterate workspace picker nav items without spreading a NodeList Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat: let raw app backend runnables import each other Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: preview only imported sibling runnables and reserve the main id Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: resolve a module's workspace imports from the entrypoint's folder Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: scope run-mode app scripts to their app and key bundles on module imports Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: clone app_version_lite into forks with remapped app_script ids Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: update the run-mode raw_code comment for app-scoped ids Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: run module-bearing app scripts only from the deployed version Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: run deployed inline code only through the deployed version's rows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: in-workspace app viewers run through the deployed rows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: let the app embed token read the app's lite value Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: pass sibling modules to the agent-worker Python preview resolver Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: list the lite route in the app embed scope comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore: update ee-repo-ref to 455f98540fb042d4f624dcec681dab88b2c8df22 This commit updates the EE repository reference after PR #883 was merged in windmill-ee-private. Previous ee-repo-ref: 927d3592a6b2127ffe16fb14eb47f5bd66afc026 New ee-repo-ref: 455f98540fb042d4f624dcec681dab88b2c8df22 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
windmill-internal-app[bot]
parent
813c080d46
commit
526da60da9
+30
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT l.id IS NOT NULL as \"has_lite!\",\n coalesce(jsonb_path_exists(l.value, '$.**.inlineScript.id ? (@ == $id)',\n jsonb_build_object('id', $3::bigint)), false) as \"is_deployed_row!\"\n FROM app a\n LEFT JOIN app_version_lite l ON l.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.path = $1 AND a.workspace_id = $2",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "has_lite!",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "is_deployed_row!",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"Text",
|
||||
"Int8"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "0d0156a21dd4bf64ad74ed6a44cbdd22c18a5f748497824d41b1b6eae975b091"
|
||||
}
|
||||
-22
@@ -1,22 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT format('rawscript/%s', code_sha256) as \"path!: String\"\n FROM app_script WHERE id = $1 LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "path!: String",
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "1bae415f9440cc1334f24ce3009242cf3a6287e7b4548c7f01ad888230c27013"
|
||||
}
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO app_script (app, hash, lock, code, code_sha256, modules)\n VALUES ($1, $2, $3, $4, $5, $6)\n ON CONFLICT (hash) DO UPDATE SET app = EXCLUDED.app -- trivial update to return the id\n RETURNING id",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Bpchar",
|
||||
"Text",
|
||||
"Text",
|
||||
"Bpchar",
|
||||
"Jsonb"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "4448d2cc6bc73635bb8c252ab9f4efaa33600c3ecd266e8339b9bfb32ae18499"
|
||||
}
|
||||
-18
@@ -1,18 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO app_script (app, hash, lock, code, code_sha256)\n VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Bpchar",
|
||||
"Text",
|
||||
"Text",
|
||||
"Bpchar"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "5c72e6e4102039c4426e8eb47d86ec152d4ee52b58f208f2fa1639eaa448c6ad"
|
||||
}
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT id, value AS \"value!: serde_json::Value\" FROM app_version_lite WHERE id = ANY($1)",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "value!: serde_json::Value",
|
||||
"type_info": "Jsonb"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8Array"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "73a22a814ae983ae6b0188ee2b9f110dadb3db328f0589f7de27bcb418e70371"
|
||||
}
|
||||
+9
-3
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT lock, code FROM app_script WHERE id = $1 LIMIT 1",
|
||||
"query": "SELECT lock, code, modules AS \"modules: serde_json::Value\"\n FROM app_script WHERE id = $1 LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
@@ -12,6 +12,11 @@
|
||||
"ordinal": 1,
|
||||
"name": "code",
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "modules: serde_json::Value",
|
||||
"type_info": "Jsonb"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
@@ -21,8 +26,9 @@
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
false
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "ffa86babfcab107caffb8dda31a66b142fa628b8983b966357deb3d5e0a1df3c"
|
||||
"hash": "7b1d7a1404c3a17e1852e1fe7d32d04d8c95e1e87e75b9a07b01c03f3e3d4373"
|
||||
}
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO app_version_lite (id, value)\n SELECT * FROM UNNEST($1::bigint[], $2::jsonb[])\n ON CONFLICT (id) DO NOTHING",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8Array",
|
||||
"JsonbArray"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "9f6bb810ebe383c848d8256f594fe701ee064dd9c880ad4df89ec47279aff7de"
|
||||
}
|
||||
+19
-7
@@ -1,32 +1,42 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT app, hash, lock, code, code_sha256\n FROM app_script\n WHERE app = ANY(SELECT id FROM app WHERE workspace_id = $1)",
|
||||
"query": "SELECT id, app, hash, lock, code, code_sha256, modules\n FROM app_script\n WHERE app = ANY(SELECT id FROM app WHERE workspace_id = $1)",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "app",
|
||||
"name": "id",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "app",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "hash",
|
||||
"type_info": "Bpchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"ordinal": 3,
|
||||
"name": "lock",
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"ordinal": 3,
|
||||
"ordinal": 4,
|
||||
"name": "code",
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"ordinal": 4,
|
||||
"ordinal": 5,
|
||||
"name": "code_sha256",
|
||||
"type_info": "Bpchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 6,
|
||||
"name": "modules",
|
||||
"type_info": "Jsonb"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
@@ -35,12 +45,14 @@
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
false,
|
||||
false
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "34979901e417b6eef8bda80f6213db46e3e68ee3d756c2540d924831879ce9e9"
|
||||
"hash": "a3582b2b9089d85d8fa273aed5348394cb91064948c1c3fbb21f9461b9869af0"
|
||||
}
|
||||
+4
-3
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n INSERT INTO app_script (app, hash, lock, code, code_sha256)\n VALUES ($1, $2, $3, $4, $5)\n ON CONFLICT (hash) DO UPDATE SET app = EXCLUDED.app -- trivial update to return the id\n RETURNING id\n ",
|
||||
"query": "\n INSERT INTO app_script (app, hash, lock, code, code_sha256, modules)\n VALUES ($1, $2, $3, $4, $5, $6)\n ON CONFLICT (hash) DO UPDATE SET app = EXCLUDED.app -- trivial update to return the id\n RETURNING id\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
@@ -15,12 +15,13 @@
|
||||
"Bpchar",
|
||||
"Text",
|
||||
"Text",
|
||||
"Bpchar"
|
||||
"Bpchar",
|
||||
"Jsonb"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "0c6c80746733be8f561ab0b631854799f5e8122adaf35465cb16c3dc795bdc3b"
|
||||
"hash": "c798432108de19ed28dcf1810c68c838beaa5dfa5e3bcfcfeddee252ffe1334e"
|
||||
}
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT format('rawscript/%s', s.code_sha256) as \"path!: String\"\n FROM app_script s JOIN app a ON a.id = s.app\n WHERE s.id = $1 AND a.path = $2 AND a.workspace_id = $3",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "path!: String",
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Text",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "f434d29f5617e1a97f9cc556ffaf47751aa5bff0741e98c00b3294d29dc36718"
|
||||
}
|
||||
@@ -1 +1 @@
|
||||
346bd5417537b76591465e63b9d15bd081e9228c
|
||||
455f98540fb042d4f624dcec681dab88b2c8df22
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE app_script DROP COLUMN IF EXISTS modules;
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE app_script ADD COLUMN IF NOT EXISTS modules JSONB;
|
||||
@@ -268,6 +268,33 @@ pub fn parse_code_for_imports(code: &str, path: &str) -> anyhow::Result<Vec<NImp
|
||||
return Ok(nimports);
|
||||
}
|
||||
|
||||
/// A runnable's Python modules as [`parse_python_imports`]' `module_sources`: content keyed
|
||||
/// by the path a relative import of each resolves to from `script_path` (`shared.py` next
|
||||
/// to `u/a/app/main` is `u/a/app/shared`).
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
pub fn module_sources(
|
||||
script_path: &str,
|
||||
modules: Option<&HashMap<String, windmill_common::scripts::ScriptModule>>,
|
||||
) -> Option<HashMap<String, String>> {
|
||||
let dir = script_path
|
||||
.rsplit_once('/')
|
||||
.map(|(dir, _)| dir)
|
||||
.unwrap_or("");
|
||||
let sources: HashMap<String, String> = modules?
|
||||
.iter()
|
||||
.filter_map(|(key, module)| {
|
||||
let stem = key.strip_suffix(".py")?;
|
||||
let path = if dir.is_empty() {
|
||||
stem.to_string()
|
||||
} else {
|
||||
format!("{dir}/{stem}")
|
||||
};
|
||||
Some((path, module.content.clone()))
|
||||
})
|
||||
.collect();
|
||||
(!sources.is_empty()).then_some(sources)
|
||||
}
|
||||
|
||||
#[cfg(not(target_arch = "wasm32"))]
|
||||
pub async fn parse_python_imports(
|
||||
code: &str,
|
||||
@@ -278,6 +305,10 @@ pub async fn parse_python_imports(
|
||||
locked_v: &mut Option<pep440_rs::Version>,
|
||||
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
// Content of relative imports that ship with the importer rather than being deployed
|
||||
// (its modules), keyed by the path a relative import resolves to. Wins over both
|
||||
// `temp_script_refs` and the deployed script, as the module file does at run time.
|
||||
module_sources: Option<&HashMap<String, String>>,
|
||||
) -> error::Result<(Vec<String>, Option<String>)> {
|
||||
let mut compile_error_hint: Option<String> = None;
|
||||
let mut imports = parse_python_imports_inner(
|
||||
@@ -291,6 +322,7 @@ pub async fn parse_python_imports(
|
||||
locked_v,
|
||||
raw_workspace_dependencies_o,
|
||||
temp_script_refs,
|
||||
module_sources,
|
||||
)
|
||||
.await?
|
||||
.into_values()
|
||||
@@ -349,6 +381,7 @@ async fn parse_python_imports_inner(
|
||||
locked_v: &mut Option<pep440_rs::Version>,
|
||||
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
module_sources: Option<&HashMap<String, String>>,
|
||||
) -> error::Result<HashMap<String, NImportResolved>> {
|
||||
tracing::debug!("Parsing python imports for path: {}", path);
|
||||
let PythonAnnotations { py310, py311, py312, py313, py314, py315, .. } =
|
||||
@@ -517,10 +550,11 @@ async fn parse_python_imports_inner(
|
||||
for n in nimports.into_iter() {
|
||||
let mut nested = match n {
|
||||
NImport::Relative(rpath) => {
|
||||
// First try to get content from temp_script_refs cache if available
|
||||
let code_from_cache = if let Some(hash) =
|
||||
temp_script_refs.as_ref().and_then(|dt| dt.get(&rpath))
|
||||
let code_from_cache = if let Some(content) =
|
||||
module_sources.and_then(|m| m.get(&rpath))
|
||||
{
|
||||
Some(content.clone())
|
||||
} else if let Some(hash) = temp_script_refs.as_ref().and_then(|dt| dt.get(&rpath)) {
|
||||
tracing::debug!(
|
||||
"Found relative import '{}' in temp_script_refs with hash '{}'",
|
||||
rpath,
|
||||
@@ -570,6 +604,7 @@ async fn parse_python_imports_inner(
|
||||
locked_v,
|
||||
raw_workspace_dependencies_o,
|
||||
temp_script_refs,
|
||||
module_sources,
|
||||
)
|
||||
.await?
|
||||
.into_values()
|
||||
|
||||
@@ -29,6 +29,7 @@ def main():
|
||||
&mut None,
|
||||
&None,
|
||||
&None,
|
||||
None,
|
||||
)
|
||||
.await?;
|
||||
// println!("{}", serde_json::to_string(&r)?);
|
||||
@@ -71,6 +72,7 @@ def main():
|
||||
&mut None,
|
||||
&None,
|
||||
&None,
|
||||
None,
|
||||
)
|
||||
.await?;
|
||||
println!("{}", serde_json::to_string(&r)?);
|
||||
@@ -103,6 +105,7 @@ def main():
|
||||
&mut None,
|
||||
&None,
|
||||
&None,
|
||||
None,
|
||||
)
|
||||
.await?;
|
||||
println!("{}", serde_json::to_string(&r)?);
|
||||
|
||||
@@ -48,7 +48,7 @@ alerts: id(int), alert_type(char), message(text), created_at(ts), acknowledged(b
|
||||
app: id(bigint), workspace_id(char), path(char), summary(char), policy(jsonb), versions(bigint[]), extra_perms(jsonb), draft_only(bool), custom_path(text), labels(text[])
|
||||
FK: (workspace_id) -> workspace(id)
|
||||
app_bundles: app_version_id(bigint), w_id(char), file_type(char), data(bytes)
|
||||
app_script: id(bigint), app(bigint), hash(char(64)), lock(text), code(text), code_sha256(char(64))
|
||||
app_script: id(bigint), app(bigint), hash(char(64)), lock(text), code(text), code_sha256(char(64)), modules(jsonb)
|
||||
FK: (app) -> app(id)
|
||||
app_version: id(bigint), app_id(bigint), value(json), created_by(char), created_at(ts), raw_app(bool)
|
||||
FK: (app_id) -> app(id)
|
||||
|
||||
@@ -462,8 +462,9 @@ fn app_embed_route_denied(domain: ScopeDomain, suffix: &str) -> bool {
|
||||
}
|
||||
}
|
||||
|
||||
/// App routes a running app uses: its own definition (`apps/get/p/<path>`, further
|
||||
/// path-scoped by `apps:read:<path>`) and the public app-serving endpoints
|
||||
/// App routes a running app uses: its own definition (`apps/get/p/<path>`, or
|
||||
/// `apps/get/lite/<path>` that the viewer runs from, both further path-scoped by
|
||||
/// `apps:read:<path>`) and the public app-serving endpoints
|
||||
/// (`apps_u/*`: public_app, public_resource, get_data, and the path-taking
|
||||
/// `execute_component` / `download_s3_file`, which re-check `apps:run|read:<path>`
|
||||
/// in their handlers so they stay confined to this app). Everything else in the
|
||||
@@ -477,7 +478,9 @@ fn app_embed_apps_route_allowed(suffix: &str) -> bool {
|
||||
if suffix.starts_with("apps_u/embed_token") {
|
||||
return false;
|
||||
}
|
||||
suffix.starts_with("apps/get/p/") || suffix.starts_with("apps_u/")
|
||||
suffix.starts_with("apps/get/p/")
|
||||
|| suffix.starts_with("apps/get/lite/")
|
||||
|| suffix.starts_with("apps_u/")
|
||||
}
|
||||
|
||||
/// Routes a guest session is denied: the app-embed allowlist, plus the embed-token
|
||||
|
||||
@@ -8547,7 +8547,7 @@ async fn clone_apps(
|
||||
|
||||
// Clone app scripts with recomputed hashes
|
||||
let app_scripts = sqlx::query!(
|
||||
"SELECT app, hash, lock, code, code_sha256
|
||||
"SELECT id, app, hash, lock, code, code_sha256, modules
|
||||
FROM app_script
|
||||
WHERE app = ANY(SELECT id FROM app WHERE workspace_id = $1)",
|
||||
source_workspace_id
|
||||
@@ -8555,34 +8555,104 @@ async fn clone_apps(
|
||||
.fetch_all(&mut **tx)
|
||||
.await?;
|
||||
|
||||
let mut app_script_id_mapping: HashMap<i64, i64> = HashMap::new();
|
||||
for app_script in app_scripts {
|
||||
if let Some(&new_app_id) = app_id_mapping.get(&app_script.app) {
|
||||
// Recompute hash using app_id, code_sha256, and lock
|
||||
// Recompute hash using app_id, code_sha256, lock and modules
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(new_app_id.to_be_bytes());
|
||||
hasher.update(hex::decode(&app_script.code_sha256)?);
|
||||
if let Some(lock) = &app_script.lock {
|
||||
hasher.update(lock.as_bytes());
|
||||
}
|
||||
if let Some(modules) = &app_script.modules {
|
||||
hasher.update(modules.to_string());
|
||||
}
|
||||
let new_hash = hex::encode(hasher.finalize());
|
||||
|
||||
sqlx::query!(
|
||||
"INSERT INTO app_script (app, hash, lock, code, code_sha256)
|
||||
VALUES ($1, $2, $3, $4, $5) ON CONFLICT DO NOTHING",
|
||||
let new_id = sqlx::query_scalar!(
|
||||
"INSERT INTO app_script (app, hash, lock, code, code_sha256, modules)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
ON CONFLICT (hash) DO UPDATE SET app = EXCLUDED.app -- trivial update to return the id
|
||||
RETURNING id",
|
||||
new_app_id,
|
||||
new_hash,
|
||||
app_script.lock,
|
||||
app_script.code,
|
||||
app_script.code_sha256,
|
||||
app_script.modules,
|
||||
)
|
||||
.execute(&mut **tx)
|
||||
.fetch_one(&mut **tx)
|
||||
.await?;
|
||||
app_script_id_mapping.insert(app_script.id, new_id);
|
||||
}
|
||||
}
|
||||
|
||||
// A version's lite value is what points each inline script at its `app_script` row (lock and
|
||||
// modules). Without it the fork serves the full value, whose inline scripts carry no id and
|
||||
// so run as bare code: no lock, and no modules for a raw app's sibling imports. Every
|
||||
// version, not just `versions.last()`: the fork's `versions` is rebuilt from all rows above,
|
||||
// so its current one can be a row the parent's array never listed.
|
||||
let source_versions: Vec<i64> = version_id_mapping.keys().copied().collect();
|
||||
let lite_versions = sqlx::query!(
|
||||
"SELECT id, value AS \"value!: serde_json::Value\" FROM app_version_lite WHERE id = ANY($1)",
|
||||
&source_versions,
|
||||
)
|
||||
.fetch_all(&mut **tx)
|
||||
.await?;
|
||||
let (mut lite_ids, mut lite_values) = (vec![], vec![]);
|
||||
for lite in lite_versions {
|
||||
let Some(&new_version_id) = version_id_mapping.get(&lite.id) else {
|
||||
continue;
|
||||
};
|
||||
let mut value = lite.value;
|
||||
// Lite inline scripts have no code of their own, so one left without a row cannot run.
|
||||
if !remap_app_script_ids(&mut value, &app_script_id_mapping) {
|
||||
continue;
|
||||
}
|
||||
if let Some(object) = value.as_object_mut() {
|
||||
object.insert("version".to_string(), serde_json::json!(new_version_id));
|
||||
}
|
||||
lite_ids.push(new_version_id);
|
||||
lite_values.push(value);
|
||||
}
|
||||
sqlx::query!(
|
||||
"INSERT INTO app_version_lite (id, value)
|
||||
SELECT * FROM UNNEST($1::bigint[], $2::jsonb[])
|
||||
ON CONFLICT (id) DO NOTHING",
|
||||
&lite_ids,
|
||||
&lite_values,
|
||||
)
|
||||
.execute(&mut **tx)
|
||||
.await?;
|
||||
|
||||
Ok(app_id_mapping)
|
||||
}
|
||||
|
||||
/// Points every `inlineScript.id` of a cloned lite app value at the fork's copy of that row.
|
||||
/// False when some id has no copy.
|
||||
fn remap_app_script_ids(value: &mut serde_json::Value, mapping: &HashMap<i64, i64>) -> bool {
|
||||
match value {
|
||||
serde_json::Value::Object(object) => {
|
||||
if let Some(serde_json::Value::Object(script)) = object.get_mut("inlineScript") {
|
||||
if let Some(old_id) = script.get("id").and_then(serde_json::Value::as_i64) {
|
||||
let Some(new_id) = mapping.get(&old_id) else {
|
||||
return false;
|
||||
};
|
||||
script.insert("id".to_string(), serde_json::json!(new_id));
|
||||
}
|
||||
}
|
||||
object
|
||||
.values_mut()
|
||||
.all(|child| remap_app_script_ids(child, mapping))
|
||||
}
|
||||
serde_json::Value::Array(array) => array
|
||||
.iter_mut()
|
||||
.all(|child| remap_app_script_ids(child, mapping)),
|
||||
_ => true,
|
||||
}
|
||||
}
|
||||
|
||||
async fn clone_raw_apps(
|
||||
tx: &mut Transaction<'_, Postgres>,
|
||||
source_workspace_id: &str,
|
||||
|
||||
@@ -15600,6 +15600,12 @@ paths:
|
||||
type: integer
|
||||
tag:
|
||||
type: string
|
||||
modules:
|
||||
type: object
|
||||
nullable: true
|
||||
description: "Additional modules keyed by relative file path. Honored in preview only: a deployed app runs the modules stored with its runnable."
|
||||
additionalProperties:
|
||||
$ref: "#/components/schemas/ScriptModule"
|
||||
required:
|
||||
- content
|
||||
- language
|
||||
|
||||
@@ -61,7 +61,7 @@ use windmill_common::{
|
||||
APP_WORKSPACED_ROUTE,
|
||||
},
|
||||
auth::{APP_EMBED_TOKEN_LABEL_PREFIX, RAW_APP_SDK_TOKEN_LABEL_PREFIX, TOKEN_PREFIX_LEN},
|
||||
cache::{self, future::FutureCachedExt},
|
||||
cache,
|
||||
db::{DbWithOptAuthed, UserDB},
|
||||
error::{to_anyhow, Error, JsonResult, Result},
|
||||
jobs::{
|
||||
@@ -1502,8 +1502,9 @@ async fn get_public_app_by_secret(
|
||||
/// - `users:read` → `users/whoami` only.
|
||||
/// - `folders:read` → `folders/listnames` only.
|
||||
/// Plus two path-scoped scopes minted per app (see `mint_app_embed_token`):
|
||||
/// - `apps:read:<path>` → the app's own definition (`apps/get/p/<path>`); no
|
||||
/// `apps:write`, so management routes are unreachable.
|
||||
/// - `apps:read:<path>` → the app's own definition (`apps/get/p/<path>` and
|
||||
/// `apps/get/lite/<path>`); no `apps:write`, so management
|
||||
/// routes are unreachable.
|
||||
/// - `apps:run:<path>` → run THIS app's components (`execute_component`, which
|
||||
/// re-checks the path); `apps_u/*` public-serving routes.
|
||||
pub const APP_EMBED_SCOPES: [&str; 5] = [
|
||||
@@ -4674,34 +4675,81 @@ async fn execute_component(
|
||||
// `app_script` id. Its resolved `rawscript/<sha>` key must be present
|
||||
// in the policy triggerables below — it must never resolve via the
|
||||
// Viewer default fallback. Without `id` the caller supplies the code
|
||||
// verbatim; with `id` it selects any `app_script` row by number (no
|
||||
// app/workspace scoping), so both let a caller run code the publisher
|
||||
// never pinned for this app.
|
||||
// verbatim, which lets a caller run code the publisher never pinned
|
||||
// for this app; with `id` the row must also belong to this app.
|
||||
let is_inline_raw_code = payload.raw_code.is_some();
|
||||
|
||||
// The policy key covers only an inline script's own code, not the lock and modules
|
||||
// its `app_script` row runs with, so it cannot tell the deployed row from an older
|
||||
// one (or from id-less code, which runs with neither). Once the deployed version has
|
||||
// its lite value, inline code runs only through the rows that value points at.
|
||||
// Checked on every run: a deploy moves it.
|
||||
if is_inline_raw_code {
|
||||
let deployed = sqlx::query!(
|
||||
"SELECT l.id IS NOT NULL as \"has_lite!\",
|
||||
coalesce(jsonb_path_exists(l.value, '$.**.inlineScript.id ? (@ == $id)',
|
||||
jsonb_build_object('id', $3::bigint)), false) as \"is_deployed_row!\"
|
||||
FROM app a
|
||||
LEFT JOIN app_version_lite l ON l.id = a.versions[array_upper(a.versions, 1)]
|
||||
WHERE a.path = $1 AND a.workspace_id = $2",
|
||||
path,
|
||||
&w_id,
|
||||
payload.id,
|
||||
)
|
||||
.fetch_optional(&db)
|
||||
.await?;
|
||||
if deployed.is_some_and(|d| d.has_lite && !d.is_deployed_row) {
|
||||
return Err(Error::NotAuthorized(format!(
|
||||
"This runnable is not part of the deployed version of app {path}, reload the app"
|
||||
)));
|
||||
}
|
||||
}
|
||||
|
||||
// Compute the path for the triggerables map:
|
||||
// - flow: `flow/<payload.path>`
|
||||
// - script: `script/<payload.path>`
|
||||
// - inline script: `rawscript/<sha256(raw_code.content)>`
|
||||
let app_script_key: Arc<String>;
|
||||
let path = match &payload {
|
||||
// flow or script: just use the `payload.path`.
|
||||
ExecuteApp { path: Some(path), .. } => path,
|
||||
// inline script: without entry in the `app_script` table.
|
||||
ExecuteApp { raw_code: Some(raw_code), id: None, .. } => &digest(&raw_code.content),
|
||||
// inline script: with an entry in the `app_script` table.
|
||||
// Scoped to this app: the policy key covers only the entrypoint's code, while the
|
||||
// row also carries the modules (and lock) the job runs, so another app's row with
|
||||
// the same entrypoint would run its own modules under this app's identity.
|
||||
ExecuteApp { raw_code: Some(_), id: Some(id), .. } => {
|
||||
let cache = cache::anon!({ u64 => Arc<String> } in "appscriptpath" <= 10000);
|
||||
// `id` is unique, cache the result.
|
||||
&*sqlx::query_scalar!(
|
||||
"SELECT format('rawscript/%s', code_sha256) as \"path!: String\"
|
||||
FROM app_script WHERE id = $1 LIMIT 1",
|
||||
id
|
||||
)
|
||||
.fetch_one(&db)
|
||||
.map_err(Into::<Error>::into)
|
||||
.map_ok(Arc::new)
|
||||
.cached(cache, *id as u64)
|
||||
.await?
|
||||
lazy_static! {
|
||||
static ref APP_SCRIPT_KEY_CACHE: cache::Cache<(String, String, i64), Arc<String>> =
|
||||
cache::Cache::new(10000);
|
||||
}
|
||||
let cache_key = (w_id.to_string(), path.to_string(), *id);
|
||||
app_script_key = match APP_SCRIPT_KEY_CACHE.get(&cache_key) {
|
||||
Some(key) => key,
|
||||
None => {
|
||||
let key = sqlx::query_scalar!(
|
||||
"SELECT format('rawscript/%s', s.code_sha256) as \"path!: String\"
|
||||
FROM app_script s JOIN app a ON a.id = s.app
|
||||
WHERE s.id = $1 AND a.path = $2 AND a.workspace_id = $3",
|
||||
id,
|
||||
path,
|
||||
&w_id,
|
||||
)
|
||||
.fetch_optional(&db)
|
||||
.await?
|
||||
.map(Arc::new)
|
||||
.ok_or_else(|| {
|
||||
Error::NotAuthorized(format!(
|
||||
"App script {id} does not belong to app {path}"
|
||||
))
|
||||
})?;
|
||||
// A row never changes app, so a hit stays valid.
|
||||
APP_SCRIPT_KEY_CACHE.insert(cache_key, key.clone());
|
||||
key
|
||||
}
|
||||
};
|
||||
&**app_script_key
|
||||
}
|
||||
_ => unreachable!(),
|
||||
};
|
||||
|
||||
@@ -2223,14 +2223,22 @@ pub async fn fetch_raw_script_from_app_query(
|
||||
loc: &'static Location<'_>,
|
||||
) -> error::Result<RawScript> {
|
||||
sqlx::query!(
|
||||
"SELECT lock, code FROM app_script WHERE id = $1 LIMIT 1",
|
||||
"SELECT lock, code, modules AS \"modules: serde_json::Value\"
|
||||
FROM app_script WHERE id = $1 LIMIT 1",
|
||||
id,
|
||||
)
|
||||
.fetch_optional(db)
|
||||
.await
|
||||
.map_err(Into::into)
|
||||
.and_then(unwrap_or_error(&loc, "Application script", id))
|
||||
.map(|r| RawScript { content: r.code, lock: r.lock, meta: None, modules: None })
|
||||
.and_then(|r| {
|
||||
Ok(RawScript {
|
||||
content: r.code,
|
||||
lock: r.lock,
|
||||
meta: None,
|
||||
modules: r.modules.map(serde_json::from_value).transpose()?,
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
/// Returns the number of jobs the row already accounted for: non-zero when a worker of the
|
||||
|
||||
@@ -41,7 +41,10 @@ const p = {
|
||||
imp.path.startsWith("/f/")) &&
|
||||
!imp.path.endsWith(".ts")
|
||||
) {
|
||||
code = code.replaceAll(imp.path, imp.path + ".ts");
|
||||
// Whole quoted specifiers only: `./util` is also a substring of `../util_ws`.
|
||||
for (const q of ['"', "'"]) {
|
||||
code = code.replaceAll(q + imp.path + q, q + imp.path + ".ts" + q);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -55,6 +58,17 @@ const p = {
|
||||
return replaceRelativeImports(code);
|
||||
});
|
||||
|
||||
// Module files (written by write_module_files next to main.ts) need the same `.ts`
|
||||
// rewrite: an extensionless relative import misses the resolve filter below, and the
|
||||
// lockfile builder then treats it as an external package named ".".
|
||||
const filterLoadModule = new RegExp(
|
||||
`^${cdir}\/(?!node_modules\/)(?!_wm_)(?!main\\.ts$).*\\.ts$`
|
||||
);
|
||||
build.onLoad({ filter: filterLoadModule }, async (args) => {
|
||||
const code = readFileSync(args.path, "utf8");
|
||||
return replaceRelativeImports(code);
|
||||
});
|
||||
|
||||
// A stalled fetch would otherwise hold the whole build for bun's own 5-minute
|
||||
// default, with nothing naming the script it was waiting on.
|
||||
const RELATIVE_IMPORT_FETCH_TIMEOUT_MS = 120000;
|
||||
@@ -109,18 +123,32 @@ const p = {
|
||||
// Check if the import resolves to a local module file (written by write_module_files).
|
||||
// Only check relative paths — absolute/bare specifiers should fall through to the
|
||||
// remote resolver, matching the Windows loader pattern.
|
||||
if (args.path.startsWith(".")) {
|
||||
const localPath = resolve(cdir, args.path);
|
||||
// Only for local importers (main.ts or a module, both `.ts` in the job dir): a fetched
|
||||
// workspace script (`.url`) resolves within its own workspace folder, where a module
|
||||
// file of the same name would shadow its real neighbour.
|
||||
const importer = args.importer ? resolve(args.importer) : "";
|
||||
if (
|
||||
args.path.startsWith(".") &&
|
||||
importer.endsWith(".ts") &&
|
||||
importer.startsWith(cdir + "/")
|
||||
) {
|
||||
// From the importing module's own folder: a module may sit in a subfolder (`lib/a.ts`).
|
||||
const localPath = resolve(dirname(importer), args.path);
|
||||
try {
|
||||
readFileSync(localPath);
|
||||
return { path: localPath };
|
||||
} catch {}
|
||||
}
|
||||
|
||||
const importerRel = args.importer.replace(cdir + "/", "");
|
||||
// A module file stands in the job dir for a script in the entrypoint's folder, so its
|
||||
// own relative imports resolve from that folder's workspace path, not from the job dir.
|
||||
const file_path =
|
||||
args.importer == "./main.ts" || args.importer == resolve("./main.ts")
|
||||
? current_path
|
||||
: args.importer.replace(cdir + "/", "");
|
||||
: importerRel.endsWith(".ts")
|
||||
? join(dirname(current_path), importerRel.replace(/\.ts$/, ""))
|
||||
: importerRel;
|
||||
|
||||
const isRelative = !args.path.startsWith("/");
|
||||
const endExt = args.path.endsWith(".ts") ? "" : ".ts";
|
||||
|
||||
@@ -1542,6 +1542,26 @@ pub async fn compute_bundle_local_and_remote_path(
|
||||
input_src.push_str(&path);
|
||||
input_src.push_str(&hash.to_string());
|
||||
}
|
||||
// Modules are inlined too, and so is every deployed script they import: those are
|
||||
// reached through the module files, never through the entrypoint's own imports.
|
||||
let dir = script_path
|
||||
.rsplit_once('/')
|
||||
.map(|(dir, _)| dir)
|
||||
.unwrap_or("");
|
||||
let mut module_entries = modules.into_iter().flatten().collect_vec();
|
||||
module_entries.sort_by(|a, b| a.0.cmp(b.0));
|
||||
for (key, module) in module_entries {
|
||||
let Some(stem) = key.strip_suffix(".ts") else {
|
||||
continue;
|
||||
};
|
||||
let module_path = format!("{dir}/{stem}");
|
||||
for (path, hash) in
|
||||
collect_transitive_import_versions(db, w_id, &module_path, &module.content).await
|
||||
{
|
||||
input_src.push_str(&path);
|
||||
input_src.push_str(&hash.to_string());
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// Keep temp-script-ref (preview) bundles in a distinct cache slot: their
|
||||
|
||||
@@ -791,7 +791,8 @@ pub async fn handle_python_job(
|
||||
None
|
||||
};
|
||||
|
||||
let (py_version, mut additional_python_paths) = handle_python_deps(
|
||||
let module_sources = windmill_parser_py_imports::module_sources(&script_path, modules.as_ref());
|
||||
let (py_version, mut additional_python_paths) = handle_python_deps_with_modules(
|
||||
job_dir,
|
||||
requirements_o,
|
||||
inner_content,
|
||||
@@ -807,6 +808,7 @@ pub async fn handle_python_job(
|
||||
precomputed_agent_info,
|
||||
annotations.clone(),
|
||||
&temp_script_refs,
|
||||
module_sources.as_ref(),
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -1939,6 +1941,7 @@ async fn replace_pip_secret(
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "private")]
|
||||
pub(crate) async fn handle_python_deps(
|
||||
job_dir: &str,
|
||||
requirements_o: Option<&String>,
|
||||
@@ -1955,6 +1958,45 @@ pub(crate) async fn handle_python_deps(
|
||||
precomputed_agent_info: Option<PrecomputedAgentInfo>,
|
||||
annotations: PythonAnnotations,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
) -> error::Result<(PyV, Vec<String>)> {
|
||||
handle_python_deps_with_modules(
|
||||
job_dir,
|
||||
requirements_o,
|
||||
inner_content,
|
||||
w_id,
|
||||
script_path,
|
||||
job_id,
|
||||
conn,
|
||||
worker_name,
|
||||
worker_dir,
|
||||
mem_peak,
|
||||
canceled_by,
|
||||
occupancy_metrics,
|
||||
precomputed_agent_info,
|
||||
annotations,
|
||||
temp_script_refs,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn handle_python_deps_with_modules(
|
||||
job_dir: &str,
|
||||
requirements_o: Option<&String>,
|
||||
inner_content: &str,
|
||||
w_id: &str,
|
||||
script_path: &str,
|
||||
job_id: &Uuid,
|
||||
conn: &Connection,
|
||||
worker_name: &str,
|
||||
worker_dir: &str,
|
||||
mem_peak: &mut i32,
|
||||
canceled_by: &mut Option<CanceledBy>,
|
||||
occupancy_metrics: &mut Option<&mut OccupancyMetrics>,
|
||||
precomputed_agent_info: Option<PrecomputedAgentInfo>,
|
||||
annotations: PythonAnnotations,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
module_sources: Option<&HashMap<String, String>>,
|
||||
) -> error::Result<(PyV, Vec<String>)> {
|
||||
create_dependencies_dir(job_dir).await;
|
||||
|
||||
@@ -1984,6 +2026,7 @@ pub(crate) async fn handle_python_deps(
|
||||
&mut locked_v,
|
||||
&None,
|
||||
temp_script_refs,
|
||||
module_sources,
|
||||
))
|
||||
.await?;
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
use std::borrow::Cow;
|
||||
use std::collections::HashMap;
|
||||
use std::collections::{BTreeMap, HashMap, HashSet};
|
||||
use std::fs::{create_dir_all, remove_dir_all};
|
||||
|
||||
#[cfg(feature = "python")]
|
||||
@@ -2093,8 +2093,10 @@ async fn insert_app_script(
|
||||
code: String,
|
||||
lock: Option<String>,
|
||||
language: Option<ScriptLang>,
|
||||
modules: Option<BTreeMap<String, ScriptModule>>,
|
||||
) -> Result<AppScriptId> {
|
||||
let code_sha256 = format!("{:x}", sha2::Sha256::digest(&code));
|
||||
let modules = modules.map(|m| serde_json::to_value(m)).transpose()?;
|
||||
let hash = {
|
||||
let mut hasher = sha2::Sha256::new();
|
||||
hasher.update(app.to_le_bytes());
|
||||
@@ -2106,14 +2108,18 @@ async fn insert_app_script(
|
||||
relative_imports_bytes(db, Some(&code), &format!("{path}/app"), language).await?,
|
||||
);
|
||||
}
|
||||
// Only when present, so a runnable without modules keeps the id it already had.
|
||||
if let Some(modules) = modules.as_ref() {
|
||||
hasher.update(modules.to_string());
|
||||
}
|
||||
format!("{:x}", hasher.finalize())
|
||||
};
|
||||
|
||||
// Insert the app script if it doesn't exist.
|
||||
sqlx::query_scalar!(
|
||||
r#"
|
||||
INSERT INTO app_script (app, hash, lock, code, code_sha256)
|
||||
VALUES ($1, $2, $3, $4, $5)
|
||||
INSERT INTO app_script (app, hash, lock, code, code_sha256, modules)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
ON CONFLICT (hash) DO UPDATE SET app = EXCLUDED.app -- trivial update to return the id
|
||||
RETURNING id
|
||||
"#,
|
||||
@@ -2121,7 +2127,8 @@ async fn insert_app_script(
|
||||
hash,
|
||||
lock,
|
||||
code,
|
||||
code_sha256
|
||||
code_sha256,
|
||||
modules
|
||||
)
|
||||
.fetch_one(db)
|
||||
.await
|
||||
@@ -2294,11 +2301,103 @@ async fn reduce_flow<'c>(
|
||||
Ok(tx)
|
||||
}
|
||||
|
||||
/// A raw app's inline runnables, keyed by the path a relative import of one resolves to:
|
||||
/// `<app path>/<runnable id>`, the path the runnable itself runs under. A runnable ships
|
||||
/// the ones it imports as its modules, so they resolve from the job dir at run time.
|
||||
#[derive(Default)]
|
||||
struct RawAppRunnables(HashMap<String, (String, ScriptLang, String)>);
|
||||
|
||||
impl RawAppRunnables {
|
||||
fn from_value(app_path: &str, value: &Value) -> Self {
|
||||
let mut runnables = HashMap::new();
|
||||
if let Some(Value::Object(by_id)) = value.get("runnables") {
|
||||
for (id, runnable) in by_id {
|
||||
let Some(script) = runnable.get("inlineScript") else {
|
||||
continue;
|
||||
};
|
||||
let content = script.get("content").and_then(Value::as_str);
|
||||
let language = script
|
||||
.get("language")
|
||||
.and_then(|l| from_value::<ScriptLang>(l.clone()).ok());
|
||||
if let (Some(content), Some(language)) = (content, language) {
|
||||
// `main.ts` is the Bun entrypoint's own file in the job dir, so a
|
||||
// runnable with that id cannot ship as a module next to it.
|
||||
let ext = module_extension(language);
|
||||
if ext.is_some() && !(ext == Some("ts") && id == "main") {
|
||||
runnables.insert(
|
||||
format!("{app_path}/{id}"),
|
||||
(id.clone(), language, content.to_string()),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Self(runnables)
|
||||
}
|
||||
|
||||
/// Whether an import of `path` by `language` code resolves to a runnable shipped as a
|
||||
/// module, rather than to the workspace script at that path.
|
||||
fn resolves(&self, path: &str, language: ScriptLang) -> bool {
|
||||
self.0.get(path).is_some_and(|(_, sibling_language, _)| {
|
||||
module_extension(*sibling_language) == module_extension(language)
|
||||
})
|
||||
}
|
||||
|
||||
/// The runnables `content` imports, directly or through one another.
|
||||
fn modules_of(
|
||||
&self,
|
||||
app_path: &str,
|
||||
content: &str,
|
||||
language: ScriptLang,
|
||||
) -> Option<BTreeMap<String, ScriptModule>> {
|
||||
let ext = module_extension(language)?;
|
||||
let mut queue =
|
||||
extract_relative_imports(content, &format!("{app_path}/app"), &Some(language))
|
||||
.unwrap_or_default();
|
||||
let mut visited = HashSet::new();
|
||||
let mut modules = BTreeMap::new();
|
||||
while let Some(path) = queue.pop() {
|
||||
if !visited.insert(path.clone()) {
|
||||
continue;
|
||||
}
|
||||
let Some((id, sibling_language, sibling_content)) = self.0.get(&path) else {
|
||||
continue;
|
||||
};
|
||||
if module_extension(*sibling_language) != Some(ext) {
|
||||
continue;
|
||||
}
|
||||
queue.extend(
|
||||
extract_relative_imports(sibling_content, &path, &Some(*sibling_language))
|
||||
.unwrap_or_default(),
|
||||
);
|
||||
modules.insert(
|
||||
format!("{id}.{ext}"),
|
||||
ScriptModule {
|
||||
content: sibling_content.clone(),
|
||||
language: *sibling_language,
|
||||
lock: None,
|
||||
},
|
||||
);
|
||||
}
|
||||
(!modules.is_empty()).then_some(modules)
|
||||
}
|
||||
}
|
||||
|
||||
/// The languages whose executors resolve a relative import against a module file.
|
||||
fn module_extension(language: ScriptLang) -> Option<&'static str> {
|
||||
match language {
|
||||
ScriptLang::Bun | ScriptLang::Bunnative => Some("ts"),
|
||||
ScriptLang::Python3 => Some("py"),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
async fn reduce_app(
|
||||
db: &sqlx::Pool<sqlx::Postgres>,
|
||||
path: &str,
|
||||
value: &mut Value,
|
||||
app: i64,
|
||||
raw_app_runnables: &RawAppRunnables,
|
||||
) -> Result<()> {
|
||||
match value {
|
||||
Value::Object(object) => {
|
||||
@@ -2319,26 +2418,20 @@ async fn reduce_app(
|
||||
Value::String(s) => Some(s),
|
||||
_ => None,
|
||||
});
|
||||
let id = insert_app_script(
|
||||
db,
|
||||
path,
|
||||
app,
|
||||
code,
|
||||
lock,
|
||||
language.map(|v| from_value(v).ok()).flatten(),
|
||||
)
|
||||
.await?;
|
||||
let language: Option<ScriptLang> = language.and_then(|v| from_value(v).ok());
|
||||
let modules = language.and_then(|l| raw_app_runnables.modules_of(path, &code, l));
|
||||
let id = insert_app_script(db, path, app, code, lock, language, modules).await?;
|
||||
// insert the `id` into the `script` object:
|
||||
script.insert("id".to_string(), json!(id.0));
|
||||
} else {
|
||||
for (_, value) in object {
|
||||
Box::pin(reduce_app(db, path, value, app)).await?;
|
||||
Box::pin(reduce_app(db, path, value, app, raw_app_runnables)).await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
Value::Array(array) => {
|
||||
for value in array {
|
||||
Box::pin(reduce_app(db, path, value, app)).await?;
|
||||
Box::pin(reduce_app(db, path, value, app, raw_app_runnables)).await?;
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
@@ -2383,6 +2476,7 @@ async fn lock_modules_app(
|
||||
raw_workspace_dependencies_o: &Option<RawWorkspaceDependencies>,
|
||||
triggered_by_relative_import: bool,
|
||||
temp_script_refs: &Option<HashMap<String, String>>,
|
||||
raw_app_runnables: &RawAppRunnables,
|
||||
) -> Result<Value> {
|
||||
match value {
|
||||
Value::Object(mut m) => {
|
||||
@@ -2417,15 +2511,29 @@ async fn lock_modules_app(
|
||||
.to_string();
|
||||
let mut logs = "".to_string();
|
||||
|
||||
// An import satisfied by another runnable of the app is not a
|
||||
// dependency on the workspace script at that path.
|
||||
let referenced_paths = extract_referenced_paths(
|
||||
&content,
|
||||
&format!("{job_path}/app"),
|
||||
Some(language),
|
||||
);
|
||||
)
|
||||
.map(|paths| {
|
||||
paths
|
||||
.into_iter()
|
||||
.filter(|p| !raw_app_runnables.resolves(p, language))
|
||||
.collect_vec()
|
||||
});
|
||||
// The lock covers what the imported runnables import, and nothing
|
||||
// records whether one of them changed, so it is never reused.
|
||||
let modules =
|
||||
raw_app_runnables.modules_of(job_path, &content, language);
|
||||
let modules = modules.map(|m| m.into_iter().collect::<HashMap<_, _>>());
|
||||
|
||||
if let Some((l, id)) = locks_to_reload
|
||||
.as_ref()
|
||||
.zip(container_id.as_ref())
|
||||
.filter(|_| modules.is_none())
|
||||
// TODO: Remove fallback
|
||||
.and_then(|e| {
|
||||
if *WMDEBUG_NO_COMPONENTS_TO_RELOCK {
|
||||
@@ -2447,9 +2555,9 @@ async fn lock_modules_app(
|
||||
.await?;
|
||||
return Ok(Value::Object(m.clone()));
|
||||
}
|
||||
} else if v
|
||||
.get("lock")
|
||||
.is_some_and(|x| !x.as_str().unwrap().trim().is_empty())
|
||||
} else if modules.is_none()
|
||||
&& v.get("lock")
|
||||
.is_some_and(|x| !x.as_str().unwrap().trim().is_empty())
|
||||
{
|
||||
if skip_creating_new_lock(&language, &content)
|
||||
&& (MIN_VERSION_SUPPORTS_DEBOUNCING_V2.met().await
|
||||
@@ -2486,7 +2594,7 @@ async fn lock_modules_app(
|
||||
base_internal_url,
|
||||
token,
|
||||
&format!("{}/app", job.runnable_path()),
|
||||
None,
|
||||
modules.as_ref(),
|
||||
occupancy_metrics,
|
||||
&None,
|
||||
existing_lock,
|
||||
@@ -2573,6 +2681,7 @@ async fn lock_modules_app(
|
||||
raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
raw_app_runnables,
|
||||
)
|
||||
.await?,
|
||||
);
|
||||
@@ -2602,6 +2711,7 @@ async fn lock_modules_app(
|
||||
raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
temp_script_refs,
|
||||
raw_app_runnables,
|
||||
)
|
||||
.await?,
|
||||
);
|
||||
@@ -2716,6 +2826,11 @@ pub async fn handle_app_dependency_job(
|
||||
|
||||
// TODO: Use transaction for entire segment?
|
||||
if let Some((app_id, value, is_raw_app)) = record {
|
||||
let raw_app_runnables = if is_raw_app {
|
||||
RawAppRunnables::from_value(&job_path, &value)
|
||||
} else {
|
||||
RawAppRunnables::default()
|
||||
};
|
||||
let value = lock_modules_app(
|
||||
value,
|
||||
&job,
|
||||
@@ -2735,6 +2850,7 @@ pub async fn handle_app_dependency_job(
|
||||
&raw_workspace_dependencies_o,
|
||||
triggered_by_relative_import,
|
||||
&temp_script_refs,
|
||||
&raw_app_runnables,
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -2747,7 +2863,7 @@ pub async fn handle_app_dependency_job(
|
||||
|
||||
// Compute a lite version of the app value (w/ `inlineScript.{lock,code}`).
|
||||
let mut value_lite = value.clone();
|
||||
reduce_app(db, &job_path, &mut value_lite, app_id).await?;
|
||||
reduce_app(db, &job_path, &mut value_lite, app_id, &raw_app_runnables).await?;
|
||||
if let Value::Object(object) = &mut value_lite {
|
||||
object.insert("version".to_string(), json!(id));
|
||||
object.remove("files");
|
||||
@@ -3311,6 +3427,7 @@ async fn capture_dependency_job(
|
||||
&mut locked_v,
|
||||
raw_workspace_dependencies_o,
|
||||
temp_script_refs,
|
||||
windmill_parser_py_imports::module_sources(script_path, modules).as_ref(),
|
||||
)
|
||||
.await?
|
||||
.0
|
||||
@@ -3436,6 +3553,11 @@ async fn capture_dependency_job(
|
||||
// TODO: move inside gen_bun_lockfile
|
||||
if !wd_exist {
|
||||
write_file(job_dir, "main.ts", job_raw_code)?;
|
||||
// The lock has to cover what the modules import, and the loader resolves a
|
||||
// relative import against these files before asking for a deployed script.
|
||||
if let Some(modules) = modules {
|
||||
crate::worker::write_module_files(job_dir, modules, None).await?;
|
||||
}
|
||||
}
|
||||
if let Some(lock) = gen_bun_lockfile(
|
||||
mem_peak,
|
||||
@@ -3457,10 +3579,9 @@ async fn capture_dependency_job(
|
||||
)
|
||||
.await?
|
||||
{
|
||||
// Nothing here writes the module files, so a bundle built now resolves a
|
||||
// multi-file script's relative imports remotely rather than from its
|
||||
// modules; caching that under a key naming them would serve the wrong
|
||||
// code to every run. Leave it to the first run instead.
|
||||
// A runnable with modules bundles on its first run instead: a multi-file
|
||||
// script's per-module lock pass also lands here, once per module, and would
|
||||
// prebundle each module as if it were an entrypoint no run ever asks for.
|
||||
if !wd_exist && modules.map_or(true, |m| m.is_empty()) {
|
||||
crate::bun_executor::prebundle_bun_script(
|
||||
job_raw_code,
|
||||
@@ -3621,3 +3742,32 @@ async fn add_lock_header(
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod raw_app_runnables_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn modules_follow_imports_between_runnables_of_one_language() {
|
||||
let value = json!({ "runnables": {
|
||||
"main": { "inlineScript": { "language": "bun", "content": "import { a } from './a'" } },
|
||||
"a": { "inlineScript": { "language": "bun", "content": "import { b } from './b.ts'" } },
|
||||
"b": { "inlineScript": { "language": "bun", "content": "export const b = 1" } },
|
||||
"unused": { "inlineScript": { "language": "bun", "content": "export const u = 1" } },
|
||||
"py": { "inlineScript": { "language": "python3", "content": "x = 1" } },
|
||||
}});
|
||||
let runnables = RawAppRunnables::from_value("u/admin/app", &value);
|
||||
|
||||
let modules = runnables
|
||||
.modules_of(
|
||||
"u/admin/app",
|
||||
"import { a } from './a'\nimport { p } from './py'",
|
||||
ScriptLang::Bun,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(modules.keys().collect_vec(), vec!["a.ts", "b.ts"]);
|
||||
assert!(runnables
|
||||
.modules_of("u/admin/app", "export const x = 1", ScriptLang::Bun)
|
||||
.is_none());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1181,6 +1181,7 @@ async function dev(opts: DevOptions, appFolder?: string) {
|
||||
runnableId,
|
||||
args,
|
||||
appTempRefs,
|
||||
previewModules(runnables, runnableId),
|
||||
);
|
||||
log.info(colors.gray(`[backend] Job started: ${uuid}`));
|
||||
|
||||
@@ -1228,6 +1229,7 @@ async function dev(opts: DevOptions, appFolder?: string) {
|
||||
runnable_id,
|
||||
v,
|
||||
appTempRefs,
|
||||
previewModules(runnables, runnable_id),
|
||||
);
|
||||
log.info(colors.gray(`[backendAsync] Job started: ${uuid}`));
|
||||
|
||||
@@ -1813,6 +1815,55 @@ async function loadRunnables(): Promise<Record<string, Runnable>> {
|
||||
}
|
||||
}
|
||||
|
||||
function moduleExtension(language: string | undefined): string | undefined {
|
||||
if (language === "bun" || language === "bunnative") return "ts";
|
||||
if (language === "python3") return "py";
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/** Ids of the sibling runnables `content` imports as `./<id>` (TS) or `.<id>` (Python). */
|
||||
function siblingImports(content: string, ext: string): string[] {
|
||||
const pattern = ext === "ts"
|
||||
? /(?:from|import)\s*['"]\.\/([^'"/]+?)(?:\.ts)?['"]/g
|
||||
: /^\s*from\s+\.([A-Za-z_]\w*)\s+import\b/gm;
|
||||
return Array.from(content.matchAll(pattern), (m) => m[1]);
|
||||
}
|
||||
|
||||
/**
|
||||
* The app's other inline runnables that `runnableId` imports, directly or through one
|
||||
* another, as the modules its preview ships with so those imports resolve to the local
|
||||
* files. Deploy stores the same set on the runnable.
|
||||
*/
|
||||
function previewModules(
|
||||
runnables: Record<string, Runnable>,
|
||||
runnableId: string,
|
||||
): Record<string, { content: string; language: string }> | undefined {
|
||||
const inlineScriptOf = (r: Runnable | undefined) =>
|
||||
r && "inlineScript" in r ? r.inlineScript : undefined;
|
||||
const own = inlineScriptOf(runnables[runnableId]);
|
||||
const ext = moduleExtension(own?.language);
|
||||
if (!own || !ext) return undefined;
|
||||
const modules: Record<string, { content: string; language: string }> = {};
|
||||
const queue = siblingImports(own.content, ext);
|
||||
while (queue.length > 0) {
|
||||
const id = queue.pop()!;
|
||||
const key = `${id}.${ext}`;
|
||||
const inlineScript = inlineScriptOf(runnables[id]);
|
||||
// `main.ts` is the Bun entrypoint's own file, so no module may take that name. The
|
||||
// runnable itself is kept when a cycle reaches it: the entrypoint is not under its id.
|
||||
if (key in modules || key === "main.ts") continue;
|
||||
if (!inlineScript || moduleExtension(inlineScript.language) !== ext) {
|
||||
continue;
|
||||
}
|
||||
modules[key] = {
|
||||
content: inlineScript.content,
|
||||
language: inlineScript.language,
|
||||
};
|
||||
queue.push(...siblingImports(inlineScript.content, ext));
|
||||
}
|
||||
return Object.keys(modules).length > 0 ? modules : undefined;
|
||||
}
|
||||
|
||||
async function executeRunnable(
|
||||
runnable: Runnable,
|
||||
workspace: string,
|
||||
@@ -1820,6 +1871,7 @@ async function executeRunnable(
|
||||
runnableId: string,
|
||||
args: any,
|
||||
tempScriptRefs?: Record<string, string>,
|
||||
modules?: Record<string, { content: string; language: string }>,
|
||||
): Promise<string> {
|
||||
const requestBody: any = {
|
||||
component: runnableId,
|
||||
@@ -1852,12 +1904,18 @@ async function executeRunnable(
|
||||
if (inlineScript.id !== undefined) {
|
||||
requestBody.id = inlineScript.id;
|
||||
}
|
||||
// A lock is not regenerated when only an imported runnable is edited, so with
|
||||
// modules the preview resolves dependencies afresh.
|
||||
const previewMods = inlineScript.id === undefined ? modules : undefined;
|
||||
requestBody.raw_code = {
|
||||
content: inlineScript.id === undefined ? inlineScript.content : "",
|
||||
language: inlineScript.language ?? "",
|
||||
path: `${appPath}/${runnableId}`,
|
||||
lock: inlineScript.id === undefined ? inlineScript.lock : undefined,
|
||||
lock: inlineScript.id === undefined && !previewMods
|
||||
? inlineScript.lock
|
||||
: undefined,
|
||||
cache_ttl: inlineScript.cache_ttl,
|
||||
...(previewMods ? { modules: previewMods } : {}),
|
||||
};
|
||||
if (inlineScript.id === undefined && tempScriptRefs) {
|
||||
requestBody.temp_script_refs = tempScriptRefs;
|
||||
|
||||
Generated
+4
@@ -6875,6 +6875,10 @@ def main(user_id: str):
|
||||
return user
|
||||
\`\`\`
|
||||
|
||||
#### Sharing code between runnables
|
||||
|
||||
An inline runnable can import another inline runnable of the same app by its id, as a relative import: \`import { getDb } from './shared'\` (TypeScript, Bun) or \`from .shared import get_db\` (Python). This works across TypeScript (Bun) runnables and across Python runnables, not between the two. The imported runnable's own dependencies are picked up, and it can import further runnables in turn. A runnable that only holds helpers needs no \`main\`; the frontend just never calls it. A TypeScript runnable whose id is \`main\` cannot be imported, because that name is taken by the entrypoint file of every TypeScript job: give shared code any other id.
|
||||
|
||||
#### The \`wmill\` client is already authenticated
|
||||
|
||||
An inline runnable runs as an ordinary Windmill job. \`import * as wmill from 'windmill-client'\` (TypeScript) and \`import wmill\` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { AppService, type ExecuteComponentData } from '$lib/gen'
|
||||
import { AppService, type ExecuteComponentData, type ScriptModule } from '$lib/gen'
|
||||
import { defaultIfEmptyString } from '$lib/utils'
|
||||
import { isRunnableByName, isRunnableByPath, type Runnable } from '../../inputType'
|
||||
import type { InlineScript } from '../../sharedTypes'
|
||||
@@ -12,7 +12,8 @@ export async function executeRunnable(
|
||||
id: string,
|
||||
requestBody: ExecuteComponentData['requestBody'],
|
||||
inlineScriptOverride?: InlineScript,
|
||||
queryParams?: Record<string, any>
|
||||
queryParams?: Record<string, any>,
|
||||
modules?: Record<string, ScriptModule>
|
||||
) {
|
||||
let appPath = defaultIfEmptyString(path, `u/${username ?? 'unknown'}/newapp`)
|
||||
if (isRunnableByName(runnable)) {
|
||||
@@ -24,13 +25,17 @@ export async function executeRunnable(
|
||||
if (inlineScript.id !== undefined) {
|
||||
requestBody['id'] = inlineScript.id
|
||||
}
|
||||
// A lock is not cleared when only an imported runnable is edited, so with
|
||||
// modules the preview resolves dependencies afresh.
|
||||
const previewModules = inlineScript.id === undefined ? modules : undefined
|
||||
requestBody['raw_code'] = {
|
||||
content: inlineScript.id === undefined ? inlineScript.content : '',
|
||||
language: inlineScript.language ?? '',
|
||||
path: appPath + '/' + id,
|
||||
lock: inlineScript.id === undefined ? inlineScript.lock : undefined,
|
||||
lock: inlineScript.id === undefined && !previewModules ? inlineScript.lock : undefined,
|
||||
cache_ttl: inlineScript.cache_ttl,
|
||||
...(inlineScript.tag ? { tag: inlineScript.tag } : {})
|
||||
...(inlineScript.tag ? { tag: inlineScript.tag } : {}),
|
||||
...(previewModules ? { modules: previewModules } : {})
|
||||
}
|
||||
}
|
||||
} else if (isRunnableByPath(runnable)) {
|
||||
|
||||
@@ -123,8 +123,8 @@
|
||||
|
||||
// Viewer side — used for the inline renderings (raw, and unsandboxed low-code);
|
||||
// the sandboxed low-code case loads inside the opaque /app_embed iframe instead.
|
||||
// getAppByPath returns bundle_secret + runnables for raw apps, which
|
||||
// PublicApp -> RawAppPreview needs.
|
||||
// getAppLiteByPath returns the runnables (with their ids) that
|
||||
// PublicApp -> RawAppPreview needs for raw apps.
|
||||
async function loadApp() {
|
||||
// Kept local and handed to PublicApp rather than written to `userStore`: this is the
|
||||
// membership in `workspace`, and a session preview tab shows a workspace the rest of
|
||||
@@ -134,8 +134,10 @@
|
||||
// populated `userStore` for the workspace the page *is* on.
|
||||
appUser = await getUserExt(workspace)
|
||||
try {
|
||||
const loaded: any = await AppService.getAppByPath({ workspace, path })
|
||||
// Raw apps need the bundle secret to load their bundle. getAppByPath
|
||||
// The lite value: its inline scripts carry the ids of their deployed rows, the only
|
||||
// code a deployed app runs (with its lock, and a raw app's imported runnables).
|
||||
const loaded: any = await AppService.getAppLiteByPath({ workspace, path })
|
||||
// Raw apps need the bundle secret to load their bundle. getAppLiteByPath
|
||||
// doesn't compute it (unlike the public handlers), so fetch it here — the
|
||||
// same call the previous raw viewer used — and hand it to PublicApp ->
|
||||
// RawAppPreview via bundle_secret.
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
import type { JobById } from '../apps/types'
|
||||
import { JobService } from '$lib/gen'
|
||||
import type { Runnable } from './rawAppPolicy'
|
||||
import { RAW_APP_PREVIEW_RELAY } from './utils'
|
||||
import { RAW_APP_PREVIEW_RELAY, rawAppPreviewModules } from './utils'
|
||||
import { undefinedIfEmpty } from '$lib/utils'
|
||||
|
||||
interface Props {
|
||||
@@ -162,7 +162,9 @@
|
||||
on_behalf_of: editor ? previewOnBehalfOf : undefined,
|
||||
viewer_data_access: editor ? runnable?.viewer_data_access || undefined : undefined
|
||||
},
|
||||
undefined
|
||||
undefined,
|
||||
undefined,
|
||||
editor ? rawAppPreviewModules(runnables, runnable_id) : undefined
|
||||
)
|
||||
launchedJobs.add(uuid)
|
||||
let job: JobById = { component: runnable_id, created_at: Date.now(), job: uuid }
|
||||
|
||||
@@ -31,6 +31,7 @@
|
||||
import { userStore } from '$lib/stores'
|
||||
import { isHubFlowPath } from '$lib/utils'
|
||||
import { sendUserToast } from '$lib/toast'
|
||||
import { rawAppPreviewModules } from './utils'
|
||||
import {
|
||||
useOperatingUser,
|
||||
useOperatingWorkspace
|
||||
@@ -51,6 +52,8 @@
|
||||
runnable: Runnable
|
||||
id: string
|
||||
appPath: string
|
||||
/** All of the app's runnables, which a test run ships as modules for `./<id>` imports */
|
||||
runnables?: Record<string, Runnable>
|
||||
/** Identity the test runs as instead of the user (see `appPreviewOnBehalfOf`). */
|
||||
previewOnBehalfOf?: string
|
||||
lastDeployedCode?: string | undefined
|
||||
@@ -70,6 +73,7 @@
|
||||
runnable = $bindable(),
|
||||
id,
|
||||
appPath,
|
||||
runnables,
|
||||
previewOnBehalfOf,
|
||||
onSelectionChange
|
||||
}: Props = $props()
|
||||
@@ -193,7 +197,7 @@
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
runnables ? rawAppPreviewModules(runnables, id) : undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
{ onBehalfOf: previewOnBehalfOf, viewerDataAccess: runnable.viewer_data_access }
|
||||
@@ -217,13 +221,7 @@
|
||||
// The deployed app runs a path runnable as the app's identity, which a run
|
||||
// by path cannot ask for: preview the deployed definition instead.
|
||||
const flow = await FlowService.getFlowByPath({ workspace: opWs!, path: runnable.path })
|
||||
await jobLoader.runFlowPreview(
|
||||
args,
|
||||
flow,
|
||||
undefined,
|
||||
runnable.path,
|
||||
previewOnBehalfOf
|
||||
)
|
||||
await jobLoader.runFlowPreview(args, flow, undefined, runnable.path, previewOnBehalfOf)
|
||||
} else {
|
||||
await jobLoader.runFlowByPath(runnable.path, args)
|
||||
}
|
||||
|
||||
@@ -40,6 +40,7 @@
|
||||
{#key selectedRunnable}
|
||||
<RawAppInlineScripRunnable
|
||||
{appPath}
|
||||
{runnables}
|
||||
{previewOnBehalfOf}
|
||||
on:createScriptFromInlineScript={(e) => {
|
||||
createScriptFromInlineScript(selectedRunnable ?? '', e.detail, opWs ?? '', appPath)
|
||||
|
||||
@@ -7,10 +7,37 @@ import {
|
||||
formatRuntimeLogsForChat,
|
||||
genWmillTs,
|
||||
normalizeRawAppRuntimeLogs,
|
||||
rawAppPreviewModules,
|
||||
stripRawAppDiffNoise,
|
||||
type Runnable
|
||||
} from './utils'
|
||||
|
||||
describe('rawAppPreviewModules', () => {
|
||||
const inline = (language: 'bun' | 'python3', content: string) =>
|
||||
({ type: 'inline', inlineScript: { language, content } }) as unknown as Runnable
|
||||
|
||||
it('ships only the same-language runnables reached through imports', () => {
|
||||
const runnables = {
|
||||
get: inline('bun', "import { a } from './a'\nimport { p } from './py'"),
|
||||
a: inline('bun', 'import { b } from "./b.ts"'),
|
||||
b: inline('bun', "import { g } from './get'\nexport const b = 1"),
|
||||
unused: inline('bun', 'export const u = 1'),
|
||||
main: inline('bun', 'export const m = 1'),
|
||||
py: inline('python3', 'from .helpers import x')
|
||||
}
|
||||
// `get` itself ships too once the cycle through `b` reaches it, as deploy does.
|
||||
expect(Object.keys(rawAppPreviewModules(runnables, 'get') ?? {}).sort()).toEqual([
|
||||
'a.ts',
|
||||
'b.ts',
|
||||
'get.ts'
|
||||
])
|
||||
expect(rawAppPreviewModules(runnables, 'unused')).toBeUndefined()
|
||||
expect(
|
||||
rawAppPreviewModules({ ...runnables, helpers: inline('python3', 'x = 1') }, 'py')
|
||||
).toEqual({ 'helpers.py': { language: 'python3', content: 'x = 1' } })
|
||||
})
|
||||
})
|
||||
|
||||
const flowSchema = {
|
||||
$schema: 'https://json-schema.org/draft/2020-12/schema',
|
||||
type: 'object',
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { ScriptLang } from '../../gen/types.gen'
|
||||
import type { ScriptLang, ScriptModule } from '../../gen/types.gen'
|
||||
import type { Schema } from '../../common'
|
||||
import { schemaToTsType } from '../../schema'
|
||||
import { isRunnableByName, isRunnableByPath, type RunnableWithFields } from '../apps/inputType'
|
||||
@@ -18,6 +18,49 @@ export type RawApp = {
|
||||
files: string[]
|
||||
}
|
||||
|
||||
function moduleExtension(language: ScriptLang | undefined): string | undefined {
|
||||
if (language === 'bun' || language === 'bunnative') return 'ts'
|
||||
if (language === 'python3') return 'py'
|
||||
return undefined
|
||||
}
|
||||
|
||||
/** Ids of the sibling runnables `content` imports as `./<id>` (TS) or `.<id>` (Python). */
|
||||
function siblingImports(content: string, ext: string): string[] {
|
||||
const pattern =
|
||||
ext === 'ts'
|
||||
? /(?:from|import)\s*['"]\.\/([^'"/]+?)(?:\.ts)?['"]/g
|
||||
: /^\s*from\s+\.([A-Za-z_]\w*)\s+import\b/gm
|
||||
return Array.from(content.matchAll(pattern), (m) => m[1])
|
||||
}
|
||||
|
||||
/**
|
||||
* The app's other inline runnables that runnable `id` imports, directly or through one
|
||||
* another, as the modules its preview ships with so those imports resolve to the
|
||||
* editor's code. Deploy stores the same set on the runnable.
|
||||
*/
|
||||
export function rawAppPreviewModules(
|
||||
runnables: Record<string, Runnable>,
|
||||
id: string
|
||||
): Record<string, ScriptModule> | undefined {
|
||||
const own = runnables[id]?.inlineScript
|
||||
const ext = moduleExtension(own?.language)
|
||||
if (!own || !ext) return undefined
|
||||
const modules: Record<string, ScriptModule> = {}
|
||||
const queue = siblingImports(own.content, ext)
|
||||
while (queue.length > 0) {
|
||||
const siblingId = queue.pop()!
|
||||
const key = `${siblingId}.${ext}`
|
||||
const inlineScript = runnables[siblingId]?.inlineScript
|
||||
// `main.ts` is the Bun entrypoint's own file, so no module may take that name. The
|
||||
// runnable itself is kept when a cycle reaches it: the entrypoint is not under its id.
|
||||
if (key in modules || key === 'main.ts') continue
|
||||
if (!inlineScript || moduleExtension(inlineScript.language) !== ext) continue
|
||||
modules[key] = { content: inlineScript.content, language: inlineScript.language }
|
||||
queue.push(...siblingImports(inlineScript.content, ext))
|
||||
}
|
||||
return Object.keys(modules).length > 0 ? modules : undefined
|
||||
}
|
||||
|
||||
// Server-managed columns the deployed app row (getAppByPath) carries but the
|
||||
// editor's current value never does — leaving them in renders as spurious diff.
|
||||
const RAW_APP_DEPLOYED_METADATA_KEYS = [
|
||||
|
||||
@@ -58,7 +58,8 @@
|
||||
console.warn('Anonymous user')
|
||||
}
|
||||
try {
|
||||
app = await AppService.getAppByPath({ workspace, path })
|
||||
// The lite value, whose inline scripts carry their deployed rows' ids: see InWorkspaceAppViewer.
|
||||
app = await AppService.getAppLiteByPath({ workspace, path })
|
||||
noPermission = false
|
||||
notExists = false
|
||||
} catch (e: any) {
|
||||
|
||||
@@ -1099,6 +1099,10 @@ def main(user_id: str):
|
||||
return user
|
||||
\`\`\`
|
||||
|
||||
#### Sharing code between runnables
|
||||
|
||||
An inline runnable can import another inline runnable of the same app by its id, as a relative import: \`import { getDb } from './shared'\` (TypeScript, Bun) or \`from .shared import get_db\` (Python). This works across TypeScript (Bun) runnables and across Python runnables, not between the two. The imported runnable's own dependencies are picked up, and it can import further runnables in turn. A runnable that only holds helpers needs no \`main\`; the frontend just never calls it. A TypeScript runnable whose id is \`main\` cannot be imported, because that name is taken by the entrypoint file of every TypeScript job: give shared code any other id.
|
||||
|
||||
#### The \`wmill\` client is already authenticated
|
||||
|
||||
An inline runnable runs as an ordinary Windmill job. \`import * as wmill from 'windmill-client'\` (TypeScript) and \`import wmill\` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
|
||||
|
||||
@@ -432,6 +432,10 @@ def main(user_id: str):
|
||||
return user
|
||||
```
|
||||
|
||||
#### Sharing code between runnables
|
||||
|
||||
An inline runnable can import another inline runnable of the same app by its id, as a relative import: `import { getDb } from './shared'` (TypeScript, Bun) or `from .shared import get_db` (Python). This works across TypeScript (Bun) runnables and across Python runnables, not between the two. The imported runnable's own dependencies are picked up, and it can import further runnables in turn. A runnable that only holds helpers needs no `main`; the frontend just never calls it. A TypeScript runnable whose id is `main` cannot be imported, because that name is taken by the entrypoint file of every TypeScript job: give shared code any other id.
|
||||
|
||||
#### The `wmill` client is already authenticated
|
||||
|
||||
An inline runnable runs as an ordinary Windmill job. `import * as wmill from 'windmill-client'` (TypeScript) and `import wmill` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
|
||||
|
||||
@@ -193,6 +193,10 @@ def main(user_id: str):
|
||||
return user
|
||||
```
|
||||
|
||||
#### Sharing code between runnables
|
||||
|
||||
An inline runnable can import another inline runnable of the same app by its id, as a relative import: `import { getDb } from './shared'` (TypeScript, Bun) or `from .shared import get_db` (Python). This works across TypeScript (Bun) runnables and across Python runnables, not between the two. The imported runnable's own dependencies are picked up, and it can import further runnables in turn. A runnable that only holds helpers needs no `main`; the frontend just never calls it. A TypeScript runnable whose id is `main` cannot be imported, because that name is taken by the entrypoint file of every TypeScript job: give shared code any other id.
|
||||
|
||||
#### The `wmill` client is already authenticated
|
||||
|
||||
An inline runnable runs as an ordinary Windmill job. `import * as wmill from 'windmill-client'` (TypeScript) and `import wmill` (Python) are already pointed at this instance and this workspace — there is nothing to configure.
|
||||
|
||||
Reference in New Issue
Block a user